\documentclass[10 pt]{article}
\usepackage{amsmath, amsthm, amssymb, graphicx, color, subfigure, algorithm, algorithmic, cite, url, footnote}
\usepackage[utf8]{inputenc}
\addtolength{\textwidth}{3 cm}
\addtolength{\oddsidemargin}{-1.5 cm}
\newtheorem{lem}{Lemma}
\setlength{\parindent}{0pt}
\newtheorem{thm}{Theorem}
\newcommand{\argmax}{\operatornamewithlimits{argmax}}
\newcommand{\argmin}{\operatornamewithlimits{argmin}}
\newcommand{\var}{\operatornamewithlimits{var}}
\begin{document}

\begin{center} 
\huge \textbf{Can a single image denoising neural network handle all levels of Gaussian noise?}
\end{center}

\vspace{5mm}
\begin{center}
\large Yi-Qing Wang, and Jean-Michel Morel\footnote{Email: yqwang9@gmail.com; morel@cmla.ens-cachan.fr}
\end{center}
\begin{center}
CMLA, Ecole Normale Supérieure de Cachan, Cachan, 94230, France 
\end{center}
\vspace{3mm}
\begin{center}
EDICS: SAS-MALN, IMD-ANAL
\end{center}
\vspace{5mm}
\begin{center}
\textbf{Abstract}
\end{center}
A recently introduced set of deep neural networks designed for the image denoising task achieves state-of-the-art performance. However, they are specialized networks in that each of them can handle just one noise level fixed in their respective training process. In this note, by investigating the distribution invariance of the natural image patches with respect to certain linear transforms, we show how to make a single existing deep neural network work well across all levels of Gaussian noise, thereby allowing to significantly reduce the training time of a general purpose neural network powered denoising algorithm.% Furthermore, our experiments seem to suggest that it may be possible to reduce these networks' domain of definition, too. 

\section{Introduction}

Recently, a set of deep neural networks, or multi-layer perceptrons (MLP) designed for the image denoising task \cite{burger2012image} has been shown to outperform BM3D \cite{dabov2008image}, widely accepted as the state-of-the-art. Although these enormous deep networks only work at the noise levels which they were trained for, this turn of events clearly demonstrates the potential of a pure learning strategy. A philosophical difference sets the two patch-based methods apart: BM3D, a major spin-off of the original non-local means \cite{buades2005review}, seeks information exclusively inside the noisy image while the neural network derives all its power by looking at noisy and clean patch pairs gathered from other images. Other algorithms exist \cite{elad2006image, mairal2009non, zoran2011learning, wang2013sure, burger2013learning} which fall between these two ends of the spectrum.\\ 


It is well known \cite{hornik1989multilayer, barron1994approximation} that neural networks form a class of universal approximators. Recent studies \cite{bengio2009learning} further suggest that multi-layer neural networks tend to be more efficient in signal representation than their traditional one-hidden-layer counterpart. Whatever their architectures, neural networks in a regression framework seek to approximate the conditional expectation under some input distribution. In our setting, let $x$, $\tilde{x}$, and $y$ denote the clean, noisy and denoised patch. Note that $y$ does not necessarily have the same dimension as $\tilde{x}$. All the neural networks \cite{burger2012image} under this study, for instance, produce a $y \in \mathbb{R}^{17\times 17}$ based on a noisy observation $\tilde{x} \in \mathbb{R}^{39\times 39}$ which includes not only $y$'s corresponding noisy pixels but also its surrounding ones. Also note that with the \textit{true} patch distribution beyond reach, a huge number of patches are drawn from a large natural image dataset to provide the conditional expectation's underlying distribution
\begin{align}
\theta^* &=\argmin_{\theta}\mathbb{E}\|f(\tilde{x}, \theta) - x\|_2^2 \label{ann} \\
	&= \argmin_{\theta}\mathbb{E}\|f(\tilde{x}, \theta) - \mathbb{E}[x | \tilde{x}]\|_2^2 \notag 
\end{align}
where $f(\cdot, \theta): \tilde{x} \mapsto y$ is a neural network parametrized by its connection weights $\theta$. Due to this problem's non-convex nature in general, instead of the potentially intractable $\theta^*$, a good $\theta$, judged on the basis of the resulting network's generalization error, is usually accepted as a solution.\\ 

Despite their impressive performance, the proposed deep networks are impractical. As stated in the paper itself \cite{burger2012image}: \textit{our most competitive MLP is tailored to a single level of noise and does not generalize well to other noise levels. This is a serious limitation which we already tried to overcome with an MLP trained on several noise levels. However, the latter does not yet achieve the same performance for $\sigma = 25$ as the specialized MLP}. Since a standard general purpose algorithm for Gaussian noise removal ought to be able to handle all levels of noise, this limitation seems to require a series of such networks, one for each noise level, which is impractical and even unrealistic especially in view of their prohibitive training time \cite{burgerthesis2013}.\\ 

In this note, by analyzing the interplay between the neural networks and the underlying patch distribution they seek to learn, we show how to construct a linear transform that moves natural image patches within the support of their distribution, which is then used to make a single existing deep neural network work well across all levels of Gaussian noise. In the concluding section, building on the natural patch distribution invariance, we hint at the possibility that further patch normalization might help scale down these networks by reducing their domain of definition without compromising their power.


\section{A single neural network for all noise levels}

To make a single network work for all noise levels, first we need to investigate the statistical regularity of the \textit{natural patch space}, or the support of the natural patch distribution, with respect to some linear transforms: we drew $10^6$ $39$-by-$39$ random patches from the \textit{Berkeley Segmentation Dataset} (BSD500) rendered to grayscale with \textit{Matlab}'s \texttt{rgb2gray} function. These patches were then normalized using the formula
\begin{align}
\bar{p} = (p / 255 - 0.5) \cdot 5 \label{lscale}
\end{align} 
provided in \cite{burger2012image} so as to conform them to the neural networks' training patch distribution. For each normalized patch, we computed the mean and standard deviation of its $1521$ pixels and then plotted their population distribution along these two dimensions. For comparison, we did the same with the \textit{PASCAL VOC 2012} dataset.\\

\begin{figure*}[htb]
\centering
\subfigure[][]{\label{bsd}\includegraphics[width = 2.8in, height = 2in]{Figures/heatmap}}
\subfigure[][]{\label{kodak}\includegraphics[width = 2.8in, height = 2in]{Figures/heatmapVOC}}
\caption{2D histogram of random patches from \subref{bsd} the grayscale BSD500 and \subref{kodak} the grayscale \textit{PASCAL VOC 2012}. The horizontal (resp. vertical) axis represents the normalized patch's mean (resp. standard variation). In both cases, patches concentrate around $(-0.5, 0.6)$. Also note that at the two ends of the horizontal axis, there are two important flat patch clusters because of saturation. } \label{heatmap}
\vspace{-3mm} 
\end{figure*}

The results (Fig.\ref{heatmap}) show that most normalized natural patches are distributed around $(-0.5, 0.6)$. Even without access to the supervised pairs used in \cite{burger2012image}, we can therefore expect their neural networks trained according to the criterion (\ref{ann}) to do well with patches from this neighborhood because of the sheer number of examples available. Moreover, the patch-wide variance peaking at the patch-wide means around $-0.5$ strongly indicates a higher tolerance for linear transforms there, that is, it is more probable for a natural patch $p$ transformed by 
\begin{align*}
q = a \cdot ( p - s ) \;\; \text{with $s$ some constant patch and $a \geq 0$} 
\end{align*}
to remain \textit{natural} if $q$'s mean is close to $-0.5$.\\ 

To verify this conjecture that a linear transform exists which only moves patches within the support of the natural patch distribution, we used Algorithm \ref{setMean} which shifts the means of the individual patches to the same value before denoising them. It is important to note that thanks to a high ratio between the patch size and $\sigma$, estimating the clean patch mean from its noisy version (Line 7 in Algorithm \ref{setMean}) is very reliable.\\ 

Running the algorithm on $10^5$ $39$-by-$39$ random patches drawn from the grayscale BSD500, we computed the resultant root mean square error (RMSE). Fig.\ref{meanShift} shows that regardless of the applied noise strength $\sigma$, the best empirical performance is attained with the patch mean shift set to $-0.5$, thereby fully confirming our supposition. In addition, observe that the optimal patch mean shift incurs surprisingly little RMSE loss, which is less than $0.1$ for a noise level as high as $75$.\\ 
 
\begin{algorithm}
\caption{Patch Mean Normalization Test}
\fontsize{3.0mm}{4.0mm}\selectfont
\begin{algorithmic}[1]
\STATE \textbf{Input:} $n$ clean patches $p_j$ of dimension $39 \times 39$ 
\STATE \textbf{Output:} $n$ denoised (resp. clean) $17\times 17$ patches $\widehat{\mathfrak{p}}_j$ (resp. $\mathfrak{p}_j$)
\STATE \textbf{Parameter:} noise variance $\sigma^2$ and desired patch mean value $\mathfrak{m}$ 
\FOR{$j=1$ to $n$}
\STATE retrieve the $17\times 17$ clean patch $\mathfrak{p}_j$ in the center of $p_j$
\STATE generate the normalized noisy patch  
\begin{align*}
\tilde{x}_j \gets ((p_j + n_j)/255 - 0.5)\cdot 5
\end{align*} 
with $n_j$ having $1521$ i.i.d. Gaussian random variables $\mathcal{N}(0, \sigma^2)$ and $n_j$ independent of $n_{j'}$ for $j \neq j'$
\STATE compute the patch mean shift $s_j = \frac{1}{1521} \sum_{k=1}^{1521}\tilde{x}_{jk} - \mathfrak{m}$ where $\tilde{x}_{jk}$ is the $k$-th pixel in $\tilde{x}_j$
\STATE shift the network's input $\forall k,\; \tilde{x}_{jk} \gets \tilde{x}_{jk} - s_j$ 
\STATE denoise $y_j = f_\sigma(\tilde{x}_{j}, \theta)$ where $f_\sigma(\cdot, \theta)$ is the deep neural network \cite{burger2012image} trained with Gaussian noise standard deviation $\sigma$
\STATE shift the network's output by the same amount $\forall k,\; y_{jk} \gets y_{jk}+ s_j$ in the opposite direction
\STATE reverse the normalization $\widehat{\mathfrak{p}}_{j} \gets (y_j /5 + 0.5)\cdot 255$ 
\ENDFOR
\end{algorithmic}\label{setMean}
\end{algorithm}

\begin{figure}[htb]
\centering
\subfigure[][]{\label{s25}\includegraphics[width = 2.8in, height = 1.8in]{Figures/rmses25}}
%\hspace{1cm}
\subfigure[][]{\label{s75}\includegraphics[width = 2.9in, height = 1.8in]{Figures/rmses75}}
\caption{\subref{s25} (resp. \subref{s75}) plots the RMSEs resulting from Algorithm \ref{setMean} with $\sigma = 25$ (resp. $75$). The blue curve records the dedicated network's performance with the patch mean values ranging from $-1.5$ to $0.9$. And the red line is the RMSE achieved on the same test data without patch mean shift. Other available neural networks have also been tested with very similar results. } \label{meanShift}
\vspace{-3mm} 
\end{figure}

The previous analysis paves the way for a generic network able to handle all levels of noise. Let us use the neural network trained with the noise level $\sigma^*$ for instance. To restore a noisy patch $\tilde{p}$ with noise standard deviation at $\sigma$, one multiplies $\tilde{p}$ by $\sigma^*\sigma^{-1}$, apply the patch mean shift and let the neural network operate on the normalized patch (Algorithm \ref{lt}). Henceforth $\sigma^*$ itself becomes a parameter for further optimization, too.\\ 

\begin{algorithm}
\caption{Generic Neural Network Denoising}
\fontsize{3.0mm}{4.0mm}\selectfont
\begin{algorithmic}[1]
\STATE \textbf{Input:} noisy patch $\tilde{p}$ of dimension $39\times 39$ and its noise level $\sigma$ 
\STATE \textbf{Output:} denoised $17\times 17$ patch $\widehat{\mathfrak{p}}$ 
\STATE \textbf{Parameter:} noise level $\sigma^*$ of the trained neural network $f_{\sigma^*}(\cdot, \theta)$ and optimal shift $\mathfrak{m} = -0.5$ 
\STATE Scale the noise $\tilde{p} \gets \sigma^* \sigma^{-1}\tilde{p}$  
\STATE Normalize the patch $\tilde{x} \gets (\tilde{p}/255 - 0.5)\cdot 5$
\STATE Compute the patch mean shift $s = \frac{1}{1521} \sum_{k=1}^{1521}\tilde{x}_{k} - \mathfrak{m}$ where $\tilde{x}_{k}$ is the $k$-th pixel in $\tilde{x}$
\STATE Shift the network's input mean $\forall k,\; \tilde{x}_{k} \gets \tilde{x}_{k} - s$ 
\STATE Run the generic neural network denoising $y = f_{\sigma^*}(\tilde{x}, \theta)$
\STATE Shift the network's output by the same amount $\forall k,\; y_{k} \gets y_{k}+ s$ in the opposite direction
\STATE Reverse the normalization $\widehat{\mathfrak{p}} \gets (y /5 + 0.5)\cdot 255 \cdot (\sigma^*\sigma^{-1})^{-1}$ 
\end{algorithmic}\label{lt}
\end{algorithm}

\begin{figure}[htb]
\centering
\subfigure[][]{\label{all}\includegraphics[width = 2.8in, height = 1.8in]{Figures/singleNN}}
%\hspace{1cm}
\subfigure[][]{\label{sole}\includegraphics[width = 2.8in, height = 1.8in]{Figures/singleNNsole}}
\caption{(\ref{all}) Performance discrepancy between dedicated neural networks and $\sigma^*$-indexed generic neural networks (using Algorithm \ref{lt}) at handling different noise levels. The horizontal axis marks various test Gaussian noise levels and the vertical axis the achieved RMSEs on $10^5$ random patches from BSD500. (\ref{sole}) singles out the best generic network with $\sigma^* = 25$.} \label{allsole}
\end{figure}

A comparison among various $\sigma^*$-indexed generic networks constructed this way (see Fig.\ref{allsole}) shows that the one built with $\sigma^* = 25$ is the best, which is hardly surprising because a neural network can learn the most about the underlying patch space when noise is weak. Moreover, except for the extremely noisy case, one sees no tangible difference in Fig.\ref{sole} between a dedicated network and the best generic network. Also observe that the higher the $\sigma^*$, the worse the resulting RMSEs at low noise levels, an expected phenomenon since a high $\sigma^*\sigma^{-1}$ exaggerates the patch-wide variation so much that resultant patches no longer remain in the natural patch space.\\ 

The implication of this analysis for future research is straightforward: training a network with the same architecture but at an even lower noise level may be rewarding. But it should also be said that too low a $\sigma^*$ is not likely to work well in a strong noise environment, as already observed in Fig.\ref{allsole}. Because the factor $\sigma^*\sigma^{-1}$ will then flatten all the meaningful patch-wide variations, leaving the network unable to tell one patch from another.\\

Put differently, what we have shown is that the trained deep network is not very different from its most informative section. Hence, one may instead train a network on mean normalized patches to improve performance, thanks to a denser data distribution. However, in so doing, one implicitly trains on a marginal distribution and thus risks losing information. For lack of a complete probabilistic description of the natural patch distribution, the exact behavior of the laws conditioned on the patch means is elusive. Yet it can still be argued that they are continuous with respect to the patch means. The similar second moments for the two laws indexed by the patch means not far away from each other, revealed in Fig.\ref{superflous}, indicate just that. Hence one might quantify \cite{linder2001learning} the patch means before training, again for benefiting from a denser data distribution. Note that here the test shown in Fig.\ref{superflous} is not intended as a modeling attempt since the first and second moments are the sufficient statistics of the Gaussian law \cite{wainwright2008graphical}, known to be rather inadequate for describing the natural patches as a whole \cite{zoran2011learning, wang2013sure}.\\

\begin{figure}[!h]\vspace{-1mm}
\centering
\label{all}\includegraphics[width = 2.8in, height = 1.8in]{Figures/caseEigens}
\caption{The first $200$ eigenvalues resulting from the principal component analysis on $10^5$ normalized random patches with the common patch-wide standard deviation in $[0.5, 0.6]$ and mean in $[-0.6, -0.5]$ and $[-1.1, -1]$ respectively. Both patch groups were drawn from \textit{PASCAL VOC 2012}. These eigenvalues, as well as their associated eigenvectors, not displayed here for lack of space, are very similar.}\vspace{-1mm} \label{superflous} 
\end{figure}

To conclude, we tested Algorithm \ref{lt} with $\sigma^* = 25$ on some real images. The test set (Fig.\ref{suite}) comprises six noiseless images proposed for benchmarking various denoising algorithms \cite{bm3dIPOL} and the results are compiled in Tab.\ref{fullres}. Recall that even for $\sigma = 25$, our generic network is different from the dedicated one, in that ours involves one additional step of patch mean shift. The obtained results are thus consistent with Fig.\ref{s25}.\\ 
 
\begin{table}
\centering 
\fontsize{3.3mm}{5.5mm}\selectfont
\caption{Comparison between dedicated neural networks and our algorithm (generic $\sigma^* = 25$)}
\begin{tabular}{cc}
\begin{tabular}{l*{2}{c}r}
\\[3pt]\hline
\textbf{$\sigma=25$} & \textbf{ dedicated} & \textbf{ generic}\\
\hline
\textbf{computer} & \textbf{7.99} & 8.10\\
\textbf{dice} & 2.77 & 2.77\\
\textbf{flower} & \textbf{4.59} & 4.63\\
\textbf{girl} & \textbf{3.38} & 3.41\\
\textbf{traffic} & \textbf{9.16} & 9.22\\
\textbf{valldemossa} & \textbf{11.85} & 11.99\\
\textsl{avg.} & \textbf{6.62} & 6.68\\
\hline
\end{tabular}
&
\begin{tabular}{l*{2}{c}r}
\\[3pt]\hline
\textbf{$\sigma=35$} & \textbf{ dedicated} & \textbf{ generic}\\
\hline
\textbf{computer} & \textbf{9.63} & 9.82\\
\textbf{dice} & \textbf{3.29} & 3.45\\
\textbf{flower} & \textbf{5.53} & 5.64\\
\textbf{girl} & \textbf{3.88} & 4.07\\
\textbf{traffic} & \textbf{10.81} & 10.94\\
\textbf{valldemossa} & \textbf{14.21} & 14.28\\
\textsl{avg.} & \textbf{7.89} & 8.03\\
\hline
\end{tabular}
\\
\begin{tabular}{l*{2}{c}r}
\\[3pt]\hline
\textbf{$\sigma=50$} & \textbf{ dedicated} & \textbf{ generic}\\
\hline
\textbf{computer} & \textbf{11.59} & 11.92\\
\textbf{dice} & \textbf{4.17} & 4.50\\
\textbf{flower} & \textbf{6.85} & 7.06\\
\textbf{girl} & \textbf{4.60} & 4.92\\
\textbf{traffic} & \textbf{12.83} & 13.07\\
\textbf{valldemossa} & \textbf{16.98} & 17.06\\
\textsl{avg.} & \textbf{9.50} & 9.75\\
\hline
\end{tabular}
&
\begin{tabular}{l*{2}{c}r}
\\[3pt]\hline
\textbf{$\sigma=65$} & \textbf{ dedicated} & \textbf{ generic}\\
\hline
\textbf{computer} & \textbf{13.16} & 13.82\\
\textbf{dice} & \textbf{4.83} & 5.39\\
\textbf{flower} & \textbf{7.89} & 8.20\\
\textbf{girl} & \textbf{5.28} & 5.74\\
\textbf{traffic} & \textbf{14.32} & 14.78\\
\textbf{valldemossa} & \textbf{18.67} & 18.99\\
\textsl{avg.} & \textbf{10.69} & 11.15\\
\hline
\end{tabular}
\\
\begin{tabular}{l*{2}{c}r}
\\[3pt]\hline
\textbf{$\sigma=75$} & \textbf{ dedicated} & \textbf{ generic}\\
\hline
\textbf{computer} & \textbf{14.10} & 14.78\\
\textbf{dice} & \textbf{5.48} & 6.14\\
\textbf{flower} & \textbf{8.57} & 8.96\\
\textbf{girl} & \textbf{5.66} & 6.20\\
\textbf{traffic} & \textbf{15.08} & 15.56\\
\textbf{valldemossa} & \textbf{19.97} & 20.34\\
\textsl{avg.} & \textbf{11.47} & 11.99\\
\hline
\end{tabular}
&
\begin{tabular}{l*{2}{c}r}
\\[3pt]\hline
\textbf{$\sigma=170$} & \textbf{ dedicated} & \textbf{ generic}\\
\hline
\textbf{computer} & \textbf{20.51} & 21.81\\
\textbf{dice} & \textbf{9.23} & 10.92\\
\textbf{flower} & \textbf{12.84} & 13.64\\
\textbf{girl} & \textbf{8.79} & 10.54\\
\textbf{traffic} & \textbf{20.71} & 21.72\\
\textbf{valldemossa} & \textbf{26.42} & 27.26\\
\textsl{avg.} & \textbf{16.41} & 17.64\\
\hline
\end{tabular}
\end{tabular}\label{fullres}
\end{table}

\begin{figure}[H]
\centering
\subfigure[][]{\label{computer}\includegraphics[width = 0.8in, height = 0.54in]{Figures/computer}}
\subfigure[][]{\label{dice}\includegraphics[width = 0.8in, height = 0.54in]{Figures/dice}}
\subfigure[][]{\label{flower}\includegraphics[width = 0.8in, height = 0.54in]{Figures/flower}}
\subfigure[][]{\label{girl}\includegraphics[width = 0.8in, height = 0.54in]{Figures/girl}}
\subfigure[][]{\label{traffic}\includegraphics[width = 0.8in, height = 0.54in]{Figures/traffic}}
\subfigure[][]{\label{valldemossa}\includegraphics[width = 1.36in, height = 0.54in]{Figures/valldemossa}}
\caption[]{Test images \subref{computer} computer \subref{dice} dice \subref{flower} flower \subref{girl} girl \subref{traffic} traffic \subref{valldemossa} valldemossa. All images are of dimension $704\times469$ except for valldemossa ($769\times 338$). } \label{suite} 
\end{figure}

\section{A better linear transform}

So far, through a statistical investigation of the natural patch distribution, we have shown that a linear transform exists which can be used to make a single neural network work well across all levels of noise. A question then arises naturally as to whether this linear transform is optimal in a certain sense. To address it, we define the following objective for a random pair comprised of a clean patch $x$ and its noisy version $\tilde{x}$ affected by some Gaussian noise of standard deviation $\sigma$
\begin{align}
\min_{V, W, U, b, c} \mathbb{E}\|V \sigma f(W \sigma^{-1} \tilde{x}+b, \theta) + U\tilde{x} + \sigma c - x\|_2^2. \label{obj}
\end{align}
Due to the previous analysis, here we may assume that the parameters involved in the two linear transforms $(V, W, U, b, c)$ do not depend on $\sigma$,  and assign a distribution on $\sigma$ to convey a prior belief on the noise environment under which the resultant generic neural network 
\begin{align*}
f^*: \tilde{x} \in \mathbb{R}^d \mapsto V \sigma f(W \sigma^{-1} \tilde{x}+b, \theta) + U\tilde{x} + \sigma c  \in \mathbb{R}^d
\end{align*} 
would operate. In other words, the probability under which the expectation in \eqref{obj} is computed is induced by three independent random variables $(x, \sigma, \mathfrak{n})$ governed respectively by the natural patch distribution, a uniform probability over some positive interval and $\mathcal{N}(0,1)$. With the natural patch distribution beyond reach, it is common practice to replace \eqref{obj} by its penalized surrogate  
\begin{align}
\min_{V, W, U, b, c}\frac{1}{n}\sum_{i=1}^n\|V\sigma_i f(W\sigma_i^{-1} \tilde{x}_i+b, \theta) + U\tilde{x}_i + \sigma_ic - x_i\|_2^2 + \beta (\|W\|_2^2 + \|V\|_2^2 + \|U\|_2^2). \label{surog}
\end{align}
Since the dedicated neural network $f(\cdot, \theta)$ is composed of $\tanh(\cdot)$, though non-convex, the surrogate \eqref{surog} is differentiable with respect to all of its parameters, and thus can be heuristically minimized using gradient descent with our previously shown transforms serving as a starting point
\begin{align*}
&W_0 = \frac{dI - \mathfrak{1}\mathfrak{1}^t}{51d} \sigma^*\\
&b_0 = - 0.5\mathfrak{1}\\
&V_0 = \frac{51}{\sigma^*}I\\
&U_0 = \frac{\mathfrak{1}\mathfrak{1}^t}{d}\\
&c_0 = \frac{25.5}{\sigma^*}\mathfrak{1}.\\ 
\end{align*}
The partial derivatives 
\begin{align*}
\frac{\partial}{\partial W}f(W\sigma_i^{-1} \tilde{x}_i+b, \theta) 
\end{align*} 

\section{Conclusion}


In this note, we have shown how to make a single existing neural network work well across all levels of Gaussian noise, thereby allowing to reduce significantly the training time for a general purpose neural network powered denoising algorithm.\\ 

To make deep neural network based algorithm more practical, the other major challenge is to reduce their sizes. This might be achieved through further patch normalization so as to reduce the neural network's input complexity. Rotation and scale invariances of natural image statistics might be used for that purpose.\\

% Viewed as an ordinary function, one major source of a neural network's redundancy comes from its domain of definition \cite{lisani2013explore}. If we can both restrict its domain and keep all the training examples at the same time, a neural network shall be able to learn better for two reasons. First, the image patches' intrinsic geometry makes it essential for a good but rudimentary neural network to acquire a certain symmetry, although this property does not seem to follow naturally from the way a network is parametrized, thereby leading to a larger than necessary architecture and training pool. It is thus desirable to remove this degree of freedom from the network's inputs. Second, if we can somehow convert and bring training examples to a smaller area that would otherwise be scattered around in the domain, in a spirit akin to importance sampling in Monte Carlo, the network should learn better with more data (see the previous section).\\ 



% This is feasible, again due to the natural patch distribution being invariant to many linear operations. For instance, we can normalize a patch geometrically by comparing the variances of its row-wise sum and column-wise sum and rotating it if necessary to have a generally horizontal pattern. Next, such a patch can be flipped upside down whenever its upper portion is brighter than its lower one. The resultant normalizing operator is reversible, non patch size specific, and thus can be applied to a possibly smaller denoised patch as in our case. In addition, invariant to linear transforms by design, it is also insensitive to noise because only summing is involved and the operated patch is large enough. Even if error occurs due to noise, it is unlikely to change much the result in RMSE because it affects only those patches with insignificant intrinsic orientation. For an empirical validation of this hypothesis, we took $10^5$ random noisy patches ($\sigma = 25$), geometrically normalized them, put them through the dedicated neural network, and applied the reversed operators on the outputs. The resulting RMSE ($9.96$), compared with that ($9.95$) from a direct application of the neural network using the same noisy patches without geometrical normalization, fully confirmed our approach.  

% \begin{figure*}[htb]
% \centering
% \subfigure[][]{\label{before}\includegraphics[width = 1.90in, height = 2in]{Figures/beforea2}}
% \subfigure[][]{\label{after}\includegraphics[width = 1.90in, height = 2in]{Figures/aftera2}}
% \subfigure[][]{\label{aftern}\includegraphics[width = 1.90in, height = 2in]{Figures/aftera2n}}
% \caption{(\ref{before}) Some random patches of dimension $39\times 39$ from the grayscale BSD500 (\ref{after}) their geometrically normalized versions (\ref{aftern}) the geometrically normalized noisy versions of (\ref{before}) with $\sigma = 25$ (using Algorithm \ref{pvr}). There is one noticeable noise induced error (the fourth from left in the second row). }  \label{flipRot}
% \end{figure*}


% \begin{algorithm}
% \caption{Patch Geometrical Normalization}
% \fontsize{3.0mm}{4.0mm}\selectfont
% \begin{algorithmic}[1]
% \STATE \textbf{Input:} a possibly noisy patch $p$ of dimension $39 \times 39$ 
% \STATE \textbf{Output:} its geometrically normalized version $p_g$ and the associated geometrically normalizing operator $T_p: p \mapsto p_g$ 
% \STATE Let $\mathfrak{1}$ be a column vector of $39$ ones and $\var(\cdot)$ be the variance of the argument vector's entries.  
% \IF{ $\var (p \mathfrak{1}) > \var (p'\mathfrak{1})$}
% \STATE $p_g \gets p'$ and $T_p \gets \mathfrak{t}$ with $\mathfrak{t}$ the standard matrix transposition
% \ELSE
% \STATE $p_g \gets p$ and $T_p \gets \mathfrak{I}$ with $\mathfrak{I}$ the identity
% \ENDIF
% \IF{the sum of the top half of $p_g$ is bigger than that of its bottom half}
% \STATE Flip $p_g$ upside down and $T_p \gets \mathfrak{f} \circ T_p$ with $\mathfrak{f}$ the flipping operation 
% \ELSE
% \STATE $T_p \gets \mathfrak{I} \circ T_p$ 
% \ENDIF 
% \end{algorithmic}\label{pvr}
% \end{algorithm}


\textbf{Acknowledgment}\\

This work was supported in part by DxO Labs, ERC(AG Twelve Labours) and ONR N00014-97-1-0839.

\clearpage
\bibliographystyle{plain}
\bibliography{refs}
\end{document}


