
\graphicspath{{./figures/}}


\chapter{Movie Scale-spaces. \label{chapitremovie} \label{chapter21}}
\thispagestyle{empty}

\def\conv{\ast}
\def\fgra{|\nabla u|}

This chapter is concerned with the axiomatic characterization of the
multiscale analyses $\{T_t\}_{t\geq 0}$ of movies. We shall formalize a movie as a
bounded function $u_0(x, y, \theta)$ defined on $\real^3$, where $x$
and $y$ are the spatial variables and $\theta$ the time variable. We
note $\fxx = (x, y, \theta)$.


As in the preceding chapters, we assume that $T_t$ is {\bf causal}  (Definition \ref{defcausalite}), {\bf Translation invariant} (Definition \ref{scalespaceinvtranslation}) and {\bf invariant by grey level translation} (Definition \ref{GLSI}).
Therefore,
as shown in Chapter \ref{Scalespace},
 there exists $T_{t,s}$ such that
$T_t= T_{t,s} T_s$, for all $t \geq s \geq 0$. And,
$$
((T_{t+h,t} u -u)/h)(\fxx) \rightarrow F(D^2u(\fxx) ,Du(\fxx),t)
$$ as $h$ tends to $0^+$ for all $u$ and $\fxx$ where $u$ is $C^2$.
 The properties of $F$ are the same as in chapter
\ref{Scalespace}, that is, $F(A, \fxp, t)$ is nondecreasing with respect to its
 first argument,
$F(A, \fxp, t)$ is continuous at all points where $\fxp \neq 0$. But, now $F$ has ten scalar arguments.

Finally, we assume that the equation
$$ \frac{\partial u}{\partial t} = F(D^2 u , Du, t)$$
a unique viscosity solution $u(x, y, \theta, t)$,
(this will of course be checked a posteriori for the models we derive).


\section{Geometrical axioms for the movie scale-space. \index{Scale-space!Movie!Axioms}}

Let us first define the geometrical axioms for the
multiscale analysis of movies.
All axioms considered in chapter \ref{Scalespace} make sense, but we need to specify them in order to take into account the
special role of time ($\theta$).  (For example, we shall not consider invariance by
spatio-temporal rotations as an essential property...) This will change a
little the assumptions on geometrical invariance.
As usual we will denote for any  affine operator  $C$ of $\real^3$, by $Cu$ the function
$Cu(\fxx ) = u(C\fxx  )$.

\vff
The first property states that the  analysis
be invariant under all linear transforms of the spatial plane
$\real^2*\{0\}$. That is, when we apply the same affine transform on each image of the
movie.
\begin{Defi}
\label{mvaffdef} We shall say that a movie scale-space $T_t$ is {\bf affine invariant} if,
for any linear map B of the form
$$
\left(
\begin{array}{ccc}
a & b & 0 \\
c & d & 0 \\
0 & 0 & 1 \end{array} \right)
$$
there exists $t'(t, B)$ such that $  B ( T_{t'(t,B)} u) = T_t (B u)$, and $B ( T_{t'(t,B),t'(s,B)} u) = T_{t,s} (B u)$.
\end{Defi}
We also state a weaker property than the affine invariance, by restricting the invariance to
the rotations of the two first coordinates, and the homotheties.
\begin{Defi} \label{mveucdef}
We shall say that a movie scale-space $T_t$ is euclidean invariant if
for any linear map
$$ A=
\left(
\begin{array}{ccc}
a\cos(b) & - a\sin(b) & 0 \\
a\sin(b) & a\cos(b) & 0 \\
0 & 0 & 1 \end{array} \right)
$$
there exists a scale $t' ( t,A)$ such that
$A ( T_{t'(t,A)} u) = T_t (A u)$ and $A ( T_{t'(t,A),t'(s,A)} u) = T_{t,s} (A u)$
\end{Defi}
Note that the $t'$ is the same for the two definitions \ref{mvaffdef} and
\ref{mveucdef}. It establishs the link between the space dimension and the
scale. Since in the following either the affine or the Euclidean invariance
will be considered, we shall always have this link.
We now establish the link between time and scale, by considering the homotheties with respect to time $\theta$.
( We accelerate or decelerate uniformly
the movie.)

\begin{Defi} \label{mvtimdef} \index{Time scale invariance}
For any $e$ in $\real^{+}$ we define by $S_e$ the linear map
$ S_e( x, y, \theta) = (x, y, e \theta)$
We shall say that a movie scale-space $T_t$ is {\bf time scale invariant} if
there exists $t'' ( t,e)$ such that
$$  S_e ( T_{t''(t,e)} u) = T_t (S_e u) \qquad \hbox{ and } S_e ( T_{t''(t,e),t''(s,e)} u) = T_{t,s} (S_e u)$$
Of course, the function $t''$ can be different from the function $t'$
 of definitions \ref{mvaffdef} and \ref{mveucdef}.
\end{Defi}

\vfp

\label{AFTI}Now, we want to state the scale invariance, as done in chapter \ref{Invariance}. We
begin by noticing that the combination of the affine (or Euclidean) invariance and the time scale invariance
implies invariance with respect to homotheties of
$\real^3$. That is, setting $H_\lambda = \lambda Id$, we have for some function $\tau(t,\lambda)$~:
$$H_\lambda ( T_{\tau(t,\lambda)} u) = T_t (H_\lambda u)$$
So, for scale invariance we could impose that the function $\tau$ is differentiable with respect
to $\lambda$ and that $\partial \tau / \partial \lambda (t,1)$ is continuous and positive.
Now, we prefer to obtain the scale-invariance assumption by using the affine and time scale invariances.

Lemma \ref{Nsc} implies that $t'$ is a function only of $t$ and of the determinant
of $B$. Then, setting $\lambda = det(B)$, we assume that $t'(t,\lambda)$ is differentiable
with respect to $\lambda$ at $\lambda=1$, and that the function
$g(t) = \frac{\partial t'}{\partial \lambda}(t,1)$ is continuous for $t>0$. We assume the same
thing for the time:
We assume that $t''(t,e)$ is differentiable with respect to $e$ at $e=1$, and that
$h(t) = \frac{\partial t''}{\partial e}(t,1)$ is continuous.
For the scale normalization we must impose in addition that at least one of $g(t)$ or $h(t)$ is positive for
$t>0$. If we assume $g(t)>0$, then the scale normalization is established with respect to spatial
variables.
 And, by an easy adaptation of Lemma \ref{Nsc}, we deduce that we can
normalize the relation between $t,B$ and $t'$ so that  \fb& \label{scMV} t' =
(det(B))^\frac{1}{2} t  \fe&
Thus the affine invariance is reduced to the property~:
\fb& \label{AFformu} F(BA^tB, Bp, t) = \vert det(B) \vert^\frac{1}{2} F(A,
p, t\vert det(B) \vert^\frac{1}{2}) \fe&
If now we assume $h(t)>0$, then the scale normalization is established with respect to time. And
then time scale invariance is reduced to
\fb& F(S_e A S_e, S_e p, t) = e F(A, p, et) \fe&
Of course, since these assumptions imply a re-normalisation, we can not assume both.
In the following, we shall assume that at least one of the two conditions is achieved.
We then state the regular scale invariance axiom~:
\begin{Defi} \label{mvrtidef}
We shall say that a scale-space $T_t$ satisfying the Affine or Euclidean invariance
and the time-scale invariance is {\bf scale-invariant} if

(i) $t'(t,\lambda)$ is differentiable
with respect to $\lambda$ at $\lambda=1$, and
$g(t) = \frac{\partial t'}{\partial \lambda}(t,1)$ is continuous for $t>0$

(ii) $t''(t,e)$ is differentiable with respect to $e$ at $e=1$, and
$h(t) = \frac{\partial t''}{\partial e}(t,1)$ is continuous for $t>0$.

(iii) One of the function $g$ or $h$ is positive, and the other one is continuous at $t=0$.

(iv) $t \rightarrow T_t$ is injective.

(where $t'$ and $t''$ are these defined in \ref{mvaffdef} or \ref{mveucdef}
 and \ref{mvtimdef}).
\end{Defi}

For the last ``geometrical axiom" we assume that the analysis is invariant under ``travelling"~: a
motion of a whole single picture with constant velocity $v$ does not alter the
analysis.  We denote by $B_v$ the galilean translation operator,
$$ B_{v=(v_x, v_y)} u(x,y,\theta) = u( x - v_x  \theta,y - v_y \theta, \theta) $$
In fact $B_v$ is an affine operator,
$$B_{v = (v_x, v_y)} = \left(
\begin{array}{ccc}
1 & 0 & - v_x  \\
0 & 1 & - v_y  \\
0 & 0 & 1 \end{array} \right)$$

\begin{Defi} \label{mvgaldef} \index{Scale-space!Galilean Invariance}\index{Invariance!Galilean} \index{Galilean invariance}
We shall say that a movie scale-space is {\bf Galilean invariant} if for any $v$
and $t$, there exists $t^*(t, B_v)$ such that $$  B_v ( T_{t^*} u) =
T_t (B_v u), \hbox{ and }  B_v ( T_{t^*(t,v),t^*(s,v)} u) =
T_{t,s} (B_v u) $$
 $t^*(t, B_{-v}) = t^*(t,B_{v})$, and $t^*$ is nondecreasing with respect to $t$.
\end{Defi}


The second part means that reversing time should not alter the analysis.
 Let us simplify the definition. By using Lemma \ref{Nsc}(i), we have
 $$ t^*(t^*(t, B_v), B_v) = t^*(t^*(t, B_v), B_{-v}) = t^*(t, B_vB_{-v}) = t^*(t, Id) = t.$$
Repeating the argument of the step (ii) of the proof of the Lemma \ref{Nsc}, we deduce from
this relation that $t^*(t, B(v)) = t.$  Thus the Galilean invariance reduces to the simpler relation
(to which we give the same name)
\fb&  B_v ( T_t u) = T_t (B_v u) \Leftrightarrow F(^tB_v A B_v, ^tB_v \fxp, t) = F(A, \fxp, t) \label{mvgaldef2} \qquad \forall A \hbox{ in } S^3, \fxp \in \real^3 \fe&

Finally, we state the morphological property, (as in definition \ref{contrastinvariance}):
\begin{Defi} \label{mvmordef}
We shall say that a movie scale-space is {\bf contrast invariant} if
for any monotone and continuous function h from $\real$ into $\real$,  $ T_{t} h(u) = h(T_{t} u) $
\end{Defi}
We have seen in lemma \ref{lemmeinvariancemorphologique} that this implies
\fb& F(\mu A+\lambda \fxp \otimes \fxp, \mu \fxp, t)=\mu F(A, \fxp, t),
\label{13MV}
\fe&
for every real
values $\lambda$, $\mu$, every symmetric matrix $A$ and every three-dimensional
vector $\fxp$.




\section{Optical flow and properties for a movie scale-space.}
The aim of this section is not to do a exhaustive list of the techniques
for optical flow estimation, but from general considerations
we will remark that lot of methods
involve a step of smoothing, which could be modelized by a scale-space.
In parallel, we will notice that the contrast and the Galilean invariances
are not only compatible but somehow justified by the aim of estimating
an optical flow. This will make more clear what motivated the choice of the
 properties stated in the preceding section.

The notion of optical flow has been introduced in the studies of
human preattentive perception of motion. \index{Optical flow!Generalities}
The optical flow associates
with each point of the movie, a vector representing the optical velocity
of this point. We shall denote by $v$ the optical flow vector ( $v= (v_x, v_y)$ is
in $\real^2$ ), and by $\fxv$ the vector $(v_x, v_y, 1)$. So that if $\Delta \theta$ is the time interval between two frames, $\fxx +\fxv(\fxx)\Delta \theta$ denotes the
point $\fxx$ shifted by $v(\fxx)$ in the next frame.

The classical definition involves a conservation assumption, which generally is
 that the points move with a constant gray level
($u$~: the gray level value). \index{Optical flow!Conservation law}
 From a discrete point of view, we are looking for $\fxv(\fxx)$ such that (\cite{Fennema:1979:VDS,Gibson:1950:PVW,Hi,Horn:1981:DOF,Verri:1987:AQO},...)
\fb& u(\fxx + \fxv(\fxx) \fde&) = u(\fxx) \label{opf} + o(\fde&) \fe& \fb& \Leftrightarrow  Du . \fxv  = 0
\label{encore2} \fe& This leads us to compare the gray level value from one frame to the next and to associate the
points which have the same intensity. Considering that the single value $u(\fxx)$ is not a reliable information because
of the many perturbation in capturing the image, the images are often smoothed before doing this matching. Of course,
it would be possible to use an image scale-space, that is to smooth each frame independently. But, we might probably do
better by smoothing the whole movie, with interactions between the different frames. Following the idea of Marr,
Hildreth, Koenderink, and Witkin many authors proposed to use the
 convolution by the 3D Gaussian function $G_t$ (the 3D heat
equation). And, then they check~:
\fb& \label{nimportequoi} ( G_t \conv u ) ( \fxx + \fxv(\fxx) \fde&) = (G_t \conv u) (\fxx)\fe&
where $\conv$ denotes the convolution operator.
The main problem of this formulation is that it is not equivalent for
two movies $u$ and $\tilde u$ representing the same object with different constant
velocity.  For example, consider that the movie $\tilde u$ is an accelerated version of $u$, $\tilde u(x, y,
\theta) = u(x, y, 2\theta) = u(A \fxx)$.  Set $\fxv_1$ (resp. $\fxv_2$) the velocity at the point $\fxx$
in the movie $u$ (resp. at the point $A\fxx$ in the movie $\tilde u$). We have $\fxv_2 =
2 \fxv_1$. Now, after the smoothing, using the formula
(\ref{nimportequoi}), $\fxv_2$ must satisfy \fb& (G_t \conv u(A .))( \fxx
+ \fxv_2 \fde&) = (G_t \conv u(A .)) (\fxx)\fe& And, we easily see that
since in general $(G_t \conv u(A .) ) \neq (G_t \conv u)( A .)$, after a such
smoothing we
shall not always obtain with formula (\ref{nimportequoi}), $\fxv_2 = 2
\fxv_1$. Indeed, in the two cases, the smoothing is not done in the same
way~: because this linear smoothing \underbar{is not Galilean invariant}.
 Therefore a such smoothing implies
some perturbation into the estimation of the velocities.




Adelson and  Bergen \cite{Adelson:1985:SEM}, and Heeger \cite{Heeger:1987:OFS}
propose in order to avoid such problem, to design ``oriented
smoothing". Such an approach yields more Galilean invariance, even
if, of course, we cannot exactly recover all the directions. (It would
involve an infinite number of filters !)

\vspace{2mm}


Let us note also that the equation (\ref{opf}) is contrast invariant. Indeed
one can apply a change of contrast for the entire movie~: change $u$ into  $\tilde u = g (u)$, where
$g$ is strictly monotonous function from $\real$ into $\real$, then the equation (\ref{opf}) with
$\tilde u$ is strictly equivalent to the equation with $u$~:
$$  u(\fxx + \fxv(\fxx)\fde&) = u(\fxx) \Leftrightarrow (g (u))(\fxx + \fxv(\fxx)\fde&) = (g (u))(\fxx) $$
for any strictly monotonous change of contrast $g$.

It is important that this property be conserved after a smoothing of the movie $u$.
Once more if we apply the linear smoothing defined by the convolution by the 3D Gaussian kernel, we
lost this property. Indeed
$$ ( G_t \conv u ) ( \fxx + \fxv(\fxx) \fde&) = (G_t \conv u) (\fxx) \qquad \hbox{is not equivalent to} $$
$$( G_t \conv (g (u)) ) ( \fxx + \fxv(\fxx) \fde&) = (G_t \conv (g (u))) (\fxx)$$
except for some specific change of contrast, or kind of motion.
In order to keep the equivalence after smoothing it is necessary that the scale-space
be contrast invariant as it has been defined in the preceding section.


As well known, the conservation law (\ref{encore2}) only gives the component of the optical flow in the direction of the spatial gradient. The other component remains indeterminated.
The usual approach to determine the optical flow then involves balance between the conservation law and some smoothing constraint on the flow. Since it is not our
subject here, we refer to the papers of Barron and al \cite{Barron:1994:POF}, Snyder \cite{Snyder:1991:MFS}, Nagel
\cite{Nagel:1988:IST}, Nagel and Enkelmann \cite{Nagel:1988:ISC}...

First, we can remark that most of the approaches involve derivatives of the intensity of the movie, that by itself can justify the fact to smooth the movie before.

Secondly, the question occurs to know whether of not it is possible to smooth the movie
so that resulting trajectories (this needs to be defined, but at least say the level surfaces, since due to conservation law trajectories are embedded within them) will be smoothed as well.

In conclusion, optical flow approaches often lead back to the problem of the
definition of a smoothing. And we do not know a priori how
much we have to smooth~: the degree of smoothing is a free scale
parameter. This indicates that a multi-scale analysis must be
applied. In addition we have seen that the conservation law justifies
the contrast and the Galilean invariances for the scale-space.




\section{The axioms lead to an equation.}

%
%N   NOTATIONS
%


We are now going to introduce some useful notation.

\begin{enumerate}

\item We denote by $\nabla u =
(\frac{\partial u}{\partial x},\frac{\partial u}{\partial y}, 0)$ the spatial gradient of the movie $u(x,y,\theta)$. When $\nabla u \neq 0$, we  associate with $Du = (\frac{\partial u}{\partial x},\frac{\partial u}{\partial y},\frac{\partial u}{\partial \theta})$ the two normal vectors
$\fff&$ and $\ffe& $ defined by
$$ \fff&  = \frac{1}{|\nabla u|}(-\frac{\partial u}{\partial y},\frac{\partial u}{\partial x},0) \qquad \ffe&   = \frac{1}{|\nabla u||Du|}(\frac{\partial u}{\partial x}\frac{\partial u}{\partial \theta},\frac{\partial u}{\partial y}\frac{\partial u}{\partial \theta},-
((\frac{\partial u}{\partial x})^2+(\frac{\partial u}{\partial x})^2))$$
When $\nabla u$ is not equal to zero, $\{Du, \fff&, \ffe& \}$ is an
orthonormal basis of $\real^3$. To be noted that $\fff&$ is spatial, that is it does not have
a temporal component.

\item Again when $\nabla u \neq 0$, we then define
$$\Gamma_1 = (D^2u) (\fff& ,\fff& ) , \qquad \Gamma_2 = (D^2u) (\fff& ,\ffe& ), \qquad \Gamma_3 = (D^2u) (\ffe& ,\ffe& ).$$
Then $\Gamma_1$ is the second derivative of $u$ in the direction $Du^\perp$,
$\Gamma_3$ in
 the direction of $Du^\pm$, and $\Gamma_2$ the cross derivative in  both
directions.

\item Then, the spatial curvature $curv(u)$ is given by
$$ curv(u) = \frac{\Gamma_1}{\vert
\nabla u \vert}.$$

\item The gaussian curvature $\Gaussiancurv (u)$ is given by
$$\Gaussiancurv (u)=
\frac{\Gamma_1 \Gamma_3 - \Gamma_2^2}{\vert Du \vert^2}$$
At last, we introduce the ``apparent acceleration", as a normalized ratius between the gaussian curvature and the spatial curvature~:
given by \index{Differential operators!Apparent acceleration!Definition} \index{Apparent acceleration} \index{accel}
$$ accel(u) = \frac{\Gaussiancurv (u)}{curv(u)} \frac{\vert Du \vert^4}{\vert \nabla u \vert^4}
 =(\frac{\vert Du \vert}{\vert \nabla u \vert})^2 (\Gamma_3
-\frac{\Gamma_2^2}{\Gamma_1})/\vert \nabla u \vert $$

\end{enumerate}
\vfp





\begin{th1} \label{thmo} \index{Scale-space!Movie!Equations}
Let a multiscale analysis $T_t$ be causal (as defined in theorem \ref{A-P}),
translation, Euclidean, Galilean, and constrast invariant.
Then, there exists a function $F$ such that $T_t$ is governed by the equation
\fb&
\frac{\partial u}{\partial t} = \vert \nabla u \vert \; F(curv(u),accel(u),t)
\fe&
(for the exact meaning of ``governed by", we refer to the theorem \ref{eqgen}.)

If in addition, $T_t$ is affine, time-scale and time invariant then the only
possible scale-space equations are
\fb&
\label{fund} \index{Affine, Morphological, and Galilean Invariant Scale-space for movie} \index{AMG|see{Affine, Morphological, and Galilean Invariant Scale-space for movie}}
\hbox{(AMG)} \qquad \frac{\partial u}{\partial t} = \vert \nabla u \vert \;
curv(u)^{\frac{1-q}{3}}
\; ( \hbox{sgn}(curv(u)) accel(u)^q )^{+}
\fe&
for some $q \in ]0,1[$, or
\fb& (q=0) \qquad \ut& = \vert \nabla u \vert curv(u)^\frac{1}{3}  \label{1}\fe&
\fb& (q=0) \qquad \ut& = \vert \nabla u \vert curv(u)^\frac{1}{3} (\hbox{sgn}(accel(u)curv(u))^+ \label{2}\fe&
\fb& (q=1) \qquad \ut& = \vert \nabla u \vert \hbox{sgn}(curv(u)) ( \hbox{sgn}(curv(u)) accel(u)
)^{+} \label{temps}\fe&
\end{th1}
In the above formulae, we use the convention that the power preserves the
sign, that is $a^q = \vert a\vert ^q \hbox{sgn}(a)$. And we set $x^+= sup(0,x)$.


\paragraph{Remark.\label{contamg} }

Before begining with the proof of the theorem, let us notice that the terms appearing in
equation (\ref{fund}) are not defined everywhere. Indeed, we can write $ curv(u)$ only when
$|\nabla u| \neq 0$, and $accel(u)$ only when $\nabla u \neq 0$ and $\Gamma_1 \neq 0$ (then $curv(u)
\neq 0$).
So, we must specify what happens when one of these conditions does not hold.
Equation (\ref{fund}) is equivalent to
$$ \ut& = |\nabla u|^{\frac{2-8q}{3}}  \Gamma_1^{\frac{1-4q}{3}} (\Gamma_1 \Gamma_3 -
\Gamma_2^2)^{q+} | Du|^{2q}$$
By continuity, when $\Gamma_1$ tends to zero, we set $\ut& = 0$.

The case $\nabla u = 0$ is more problematic. We distinguish three cases~:

\begin{itemize}
\item If $q< 1/4$, the right hand side ot the equation is continuous and we obtain, when $\nabla u$
tends to zero, $\ut& = 0$.

\item In the case $q=1/4$, which is a limit case, $\nabla u$ does not appear in the equation.
Now, the definitions of $\Gamma_1$, $\Gamma_2$,... depend on the direction of $\nabla u$.
We have in this case $$\ut& = |Du|^\frac{1}{2} (\Gamma_1 \Gamma_3 -
\Gamma_2^2)^{\frac{1}{4}+}$$
where, $(\Gamma_1 \Gamma_3 -
\Gamma_2^2)$ is the determinant of $D^2u$ restricted to the orthogonal plan to $Du$. If $|Du|
\neq 0$, this determinant is defined independently of the $\Gamma_i$, and the formulation makes
sense.
Now, if $|Du|$ tends to 0, by continuity we have $\ut& = 0$.

\item At last, if $q > 1/4$, Equation (\ref{fund}) has singularities since the right hand side of
this equation may tend to infinity when $\nabla u$ tends to zero.
\end{itemize}




\vff
\noindent Let us now set the obtained relation between space, time and scale.
\begin{cor}
Let $A$ be an affine transform of the coordinates
$$
\left(
\begin{array}{ccc}
a & b & 0 \\
c & d & 0 \\
0 & 0 & e \end{array} \right)
 \hbox{ for any } a,\; b,\; c,\; d, \; e \in \real
$$
and let $p = \sqrt{ad-bc}$. Then, the multiscale analysis defined by equation
(\ref{fund}) satifies
$  A ( T_{\tau } u) = T_t (A u) $ with
\fb& \label{scal}
\tau( A, t)  =  (p^{4(\frac{1-q}{3})}e^{2q}) t
\fe&
\end{cor}

We see in relation (\ref{scal}), that q is a parameter which represents the
respective weights between space variables and time variables in the equation.
For example, by taking $q = 0$, we remove the time dependance in the
equation and we obtain the purely spatial affine and constrast invariant scale-space (or
a slight variant).
On the other side by taking $q=1$, we remove the space dependance of the
scale~: we obtain the equation (\ref{temps}).
At last, by taking $q=\frac{1}{4}$, we impose an homogeneous dependance in
time and space.
$\tau = pe^\frac{1}{2}\;t = (det(A)^\frac{1}{2})\;t$ \index{Gaussian Curvature}
In that case, by formulating the equation with $\Gaussiancurv (u)$ the gaussian curvature of $u$, we obtain
\fb&
\frac{\partial u}{\partial t} = \vert Du \vert (\Gaussiancurv (u)^+)^\frac{1}{4}
\fe&
which is the unique contrast and 3D affine invariant scale-space as described
in chapter \ref{AMSS 3-D}.

Let us before begining the proof of the theorem give a hint on the
 kind of smoothing the equation (\ref{fund}) should do on a movie.
Let us decompose this equation into two parts
$$
 \frac{\partial u}{\partial t} =\qquad  \vert \nabla u \vert \; curv(u)^{power...}
\; \qquad ( \hbox{sgn}(curv(u)) accel(u)^{power...} )^{+}
$$
The first term $curv(u)^{power...}$ is roughly a term of spatial diffusion, and then
tends to remove objects when $t \rightarrow \infty$.
It's quite close from the diffusion term of affine and contrast invariant scale-space of
static images.

The second term $accel(u)$... can be seen as the speed of this spatial diffusion.
The bigger is $accel$, faster the spatial diffusion is executed.
As we shall see in the following the differential operator $accel$
can be interpreted as some kind of acceleration of objects in the movie.
So, we can conclude that the equation will smooth (and then remove ) faster the
object with big acceleration, than object with low acceleration.
Therefore we can expect that this will produce a discrimination between trajectories (smooth and
unsmooth).

\vfp

{\bf Proof of Theorem \ref{thmo}}
The proof is essentially based on algebraic calculations. Its main ingredient is
that the terms $\vert \nabla u \vert^3  curv(u)$ and $\vert Du \vert^4 G(u) = \vert \nabla u \vert^4 curv(u)
accel(u)$ are affine covariant of  degree 2,2,0  and 2,2,2, with respect to the coordinates
$(x,y,\theta)$.

Since the proof is quite long and technical, we refer to \cite{Alvarez:1993:AFE}.
\endproof.




\section{Optical flow and apparent acceleration. \index{Differential operators!Apparent Acceleration!Interpretation}}

In this section, we shall give to $accel(u)$ a cinematic interpretation as an ``apparent
acceleration".
As pointed before, the conservation law related to the optical flow
fixes only the component of the flow in the direction of the spatial
gradient.

First,  we shall see that the model
(\ref{fund}) and the definition of $accel(u)$  can be associated with
a special choice for the other component the apparent velocity. This choice
 corresponds to
the a priori assumption that only objects in translation are  observed.
In other terms,  $accel(u)$  gives the correct
estimate of
 the acceleration of objects when they are in
translation motion. Secondly, we will establish a formula that provides an estimation of $accel$
without any calculating of the apparent velocity.

In all this section, we work only at points where $\nabla u \neq 0$.

\paragraph{What are the possible velocities ?}

We define the optical flow $\vec
v(x,y,\theta)$ as a function from $\real^3$ into $\real^2$
representing the velocity of the point $(x,y)$ at time $\theta$. As before, we  add a third component to
the flow, which  will always be equal to 1~:  $\fxv(x,y,\theta) = (\vec
v(x,y,\theta),1)$. We denote by $\fW$ the set of ``possible" velocity
vectors \fb& \fW = \{ \; \fxv = (\vec v,1) \hbox{ for all } \vec v \hbox{ in
} \real^2 \; \} \fe&

Assuming the conservation law, the optical flow is a vector of $\fW$ which is orthogonal to $Du$, therefore  when $Du \neq 0$, it belongs to the
set $\fV$~:
\fb&  {\fV}  = \{ \; \fxv_\mu = \frac{|Du|}{\vert \nabla u
\vert} (\mu \fff& - \ffe& ), \hbox{ for all } \mu \in \real \; \} \fe&
All $\fxv_\mu$ have their component in the direction of $\nabla u$ fixed to
$ -\frac{u_\theta}{\vert \nabla u \vert}$.
We have one free parameter $\mu$ left. It corresponds to
  the component of the velocity vector in the spatial direction orthogonal to
$\nabla u$, that is
  by definition~: $\fff& $.
In the next paragraph, we define $\mu$ so that $accel(u)$ is an apparent
acceleration.

\begin{Defi}\label{Defeflow} Definition of the ``velocity vector''.
\label{flow} \index{Velocity vector}
When $\nabla u$ and $curv(u) \neq 0$, we define the ``velocity vector'': \fxV  by
\fb& \label{eflow} \fxV = \frac{|Du|}{\vert \nabla u \vert} (
\frac{\Gamma_2}{\Gamma_1}\fff&  - \ffe& )  \fe&
\end{Defi}


\vspace{5mm}
Then, if we set $v_1 =  (\fxV .\nabla u) /\vert \nabla u \vert$
(resp.  $v_2 = (\fxV . \fff&) / \vert
\fff& \vert$), the  component of $\fxV$ in the direction (resp.
orthogonal direction) of the spatial gradient $\nabla u$,
we have: \fb&
v_1 = -\frac{u_\theta}{\vert \nabla u \vert} \qquad v_2 = \frac{|Du|}{|\nabla u|}
\frac{\Gamma_2}{\Gamma_1}
\fe&

\begin{pro}
Let $\vec i, \vec j$ be an orthonormal basis of the image plane.
Consider a picture in translation motion with velocity $\vec v = (v^x, v^y)$~: $u(x,y,\theta)=w(x -
\int^\theta_0
 v^x(\theta) d\theta , y - \int^\theta_{0} v^y(\theta)  d\theta)$. Then, at every points such that $\nabla u \neq 0$ and $curv(u) \neq 0$,  $\vec v$ satisfies the explicit formula
$$
(\vec v,1) = \fxV
$$
\end{pro}

In other terms, the definition (\ref{Defeflow}) of the flow $\fxV$ is exact for any
translation motion.


The definition of the optical flow that fixes one component of the flow corresponds to say that points
move on their space-time level surface (gray-level does not change). Fixing the other
component as we do with the definition \ref{flow} is to make
the choice of a travelling
direction on the space-time level surface.
With the definition \ref{flow}, we choose the direction which does not change
the orientation of the spatial gradient.

\begin{figure}[htb]
\begin{center}
\begin{tabular}{cc}
\epsfxsize=4.5cm \epsfbox{flow1.ps} &
\epsfxsize=4.5cm \epsfbox{flow2.ps} \\
\end{tabular}
\end{center}
\caption[Fig : Optical flow definition]{ According to the optical flow definition, all above drawn velocity vectors are
possible, since they allow the moving point to remain on the same level surface. One possibility to get rid of this
ambiguity is choose as velocity the  direction which does not change the orientation of the spatial gradient.}
\end{figure}

Of course, in general, the velocity vector $\fxV$ is not equal to the real velocity
for others motions than the translations, but we shall consider it, for any type of
movement. In others words we make for a point a choice of trajectory along the the iso-surface
it belongs.



We shall now look for simpler expressions and interpretation of $accel(u)$.
The next proposition shows that
first, $accel$ can be seen as an apparent acceleration and second as a curvature
in space-time of our choice of trajectories along iso-surface.


\begin{pro} 1. $accel$ as an apparent acceleration.  \label{mvpro02} For all points such that $\nabla u \neq 0$ and $curv(u) \neq 0$, let $\fxV= (v_x, v_y, 1)$ be the velocity vector defined as above (\ref{flow}),
and $v_1$ its component
 in the direction of the spatial gradient.
$$
accel(u)=  - \deriv& =  -( v^x \frac{\partial v_1}{\partial x} + v^y
\frac{\partial v_1}{\partial y} + \frac{\partial v_1}{\partial \theta} ) $$
\fb& \label{cououc}
\qquad = -  ((Dv_1) .\fxV)  = - ( D(\fxV .\nabla u) . \fxV )
\fe&

This formula\footnote{ We denote by $\frac{\hbox{D}f}{\hbox{D}\theta}$ the
variation of $f$ along the trajectory of the considered point ( = $
((Df). \fxV) $ where $\fxV$ is the velocity of the point). This is generally different
from $\frac{\partial f}{\partial \theta}$ which is the partial variation of $f$
with respect to $\theta$. }
 shows that $accel(u)$ is the acceleration in the direction of $-\nabla u$. As $v_1$ the component of the velocity in the spatial gradient direction is called the ``apparent'' velocity, $accel(u)$ can be called the ``apparent acceleration".

2. Let $\fxV$ be the ``velocity vector'' defined in Definition \ref{eflow}, then
\fb&
accel(u) = \frac{ (D^2u) (\fxV,\fxV)}{\vert \nabla u \vert}
\fe&
\end{pro}


\paragraph{Proof of proposition \ref{mvpro02}} The proof is just some simple calculations. \endproof


\paragraph{ Discretization of the apparent acceleration.}

We shall prove some equalities allowing a robust
computation of the term $accel(u)$. As we have seen before,
the ``possible velocity" vectors are in $\fW$.
They also must be orthogonal to the gradient of the movie $Du$, and therefore
lie in $\fV$
We will first obtain a formula for $accel(u)$ that involves a minization over the vectors of $\fV$, and secondly we will extend this minimization over the vectors of $\fW$.

\begin{lem}

Whenever the spatial gradient $\nabla u$ and the spatial curvature $curv(u)$
are not equal to zero,
\fb& \label{minim}
\vert \nabla u \vert (sgn(curv(u)) \; accel(u) )^+ = min_{\fxv \in \fV} \vert  (D^2u) (\fxv , \fxv)  \vert
\fe&

\end{lem}

\paragraph{Proof}
Let us recall that the set $\fV$ is the set of the vectors
$$\fxv_\mu = \frac{|Du|}{\nabla u}(\mu \fff& - \ffe& )$$
We have
$$ ( (D^2u) \fxv_\mu . \fxv_\mu)   = \frac{|Du|^2}{\vert \nabla u \vert^2}
(\Gamma_1 \mu^2 - 2 \Gamma_2 \mu + \Gamma_3) = P(\mu),$$
where $P(\mu)$ is a polynomial of degree 2 in $\mu$.
 When $\vert \nabla u \vert$ and $curv(u)$, (and therefore $\Gamma_1$)
are not equal to zero the extremum of $P(\mu)$ is reached
when
 $\mu = \Gamma_2 / \Gamma_1$, that is when $\fxv_\mu = \fxV$.
 Thus the extremum value of $P(\mu)$ is $\fgra accel(u)$,  by proposition
\ref{mvpro02}. We obtain
 $$ ext_{\fxv \in \fV} (D^2u) (\fxv , \fxv)  = \fgra accel(u),$$
 where by $ext_{\fxv \in \fV}$ we denote the finite extremal value in $\fV$.

Assume first that $curv(u)$ and $accel(u)$ have the same sign. This implies that
the second order coefficient and the extremum of the polynomial have
 the same sign. Thus the expression $(D^2u) (\fxv,\fxv)$ has the same
sign for all $\fxv \in \fV$.
 This yields $\vert \nabla u \vert (sgn(curv(u)) \; accel(u) ) = min_{\fxv \in \fV}
\vert (D^2u) (\fxv , \fxv)  \vert$.

 If now, $curv(u)$ and $accel(u)$ have opposite signs then $\vert \nabla u
\vert (sgn(curv(u)) \; accel(u) )^+ = 0$. And $P(\mu)$ is equal to zero for at
least one vector $\fxv$ of $\fV$. Thus, for this vector,
 $\vert (D^2u) (\fxv,\fxv) \vert = 0$, and $min_{\fxv \in \fV} \vert
(D^2u) (\fxv
  , \fxv)  \vert = 0$.
 So (\ref{daccel}) is still satisfied. \endproof

From a numerical viewpoint, the minimization on the set of vectors $\fV$ is
not easy.
Indeed, first, the direction of the gradient of the movie is quite unstable
because
$\Delta \theta$, the time interval between two images, can be large.

We will restrict $\fW$ to the vectors that stand in a ball $B(0,R)$ for an
arbitrary $R$ that can be chosen large enough. In others words, we will only
consider bounded possible velocities, which is not a real restriction in pratice.

\begin{lem} \label{acinw}
Let $\nabla u$ and $curv(u)$ be not equal to zero, and $u$ be $C^2$, then the
expression
\fb&
min_{\fxv \in \fW} (\frac{1}{\Delta \theta^2} ( \vert u(\fxx  - \fxv \fde&) - u(\fxx )
\vert + \vert u(\fxx  + \fxv \fde&) - u(\fxx ) \vert ))
\fe&
converges towards $\vert \nabla u \vert (sgn(curv(u)) \; accel(u) )^+$ when
$\fde&$ tends to zero.
\end{lem}

\vfp

\paragraph{Proof} Due to the fact that $\fxv \in \fW$ are assumed to be bounded, we have
that $\fxv \Delta \theta$ tends to 0 as $\Delta \theta$ tends to 0. As consequence,
we can restrict the proof to the case where $u$ is a quadratic form without loss
of generality.

So, let $u$ be a quadratic form~: $u(\fxx ) = \frac{1}{2} A(\fxx,
\fxx)  +  \fxp.\fxx   + c$, and define
$$F(\fxv,h) = (\vert u(\fxx -\fxv h) - u(\fxx ) \vert + \vert u(\fxx +\fxv h) - u(\fxx
) \vert) / h^2$$
We have
\fb& \label{mv037} F(\fxv,h) = \vert - \frac{\fxp.\fxv}{h} + \frac{1}{2}  A(\fxv ,\fxv)
\vert + \vert \frac{ \fxp.\fxv  }{h} + \frac{1}{2} A(\fxv ,\fxv) \vert
\fe&
\noindent Let $\fxw \in {\cal V}$ be a vector which minimizes the $min$ in (\ref{minim}),
$\fxw \in {\cal V}$ then
$ \fxw . \fxp  = 0$), thus (\ref{mv037}) becomes
$$F(\fxw,h) = \vert A(\fxw , \fxw) \vert $$
Therefore
\fb& \label{dem1}
\lim_{h \rightarrow 0} (min_{\fxv  \in \fW} F(\fxv ,h)) \leq F(\fxw,h) =\vert \nabla u \vert
(sgn(curv(u)) \; accel(u) )^+
\fe&
Moreover $min_{\fxv  \in \fW} F(\fxv ,h)$ exists for every $h$ and is bounded.
We denote by $\fxv_h$ a vector of $\fW$ such that $F(\fxv_h,h) = min_{\fxv  \in \fW}
F(\fxv ,h)$.
Since $F(\fxv_h,h)$ is bounded and $ F(\fxv_h,h) \geq 2 \vert   (\fxp.\fxv_h)/h \vert$,
we necessarly have
\fb& \label{dem3}   \vert (\fxp. \fxv_h )  \vert =
O(h) \fe&
Let decompose $\fxv_h$ into two vectors~: $ \fxv_h = \fxv_h^\perp + h \fxv_h^\pm$
such that
 $ \fxv_h^\perp $ is orthogonal to $\fxp$, and (\ref{dem3}) leads that $\vert \fxv_h^\pm \vert$ is bounded when h tends to
zero.
As before, we have
$$F(\fxv_h,h)\geq \vert A (\fxv_h, \fxv_h)  \vert \geq \vert  A( (\fxv_h^\perp + \fxv_h^\pm) , (\fxv_h^\perp +
\fxv_h^\pm)) \vert \geq $$
$$ \vert A(\fxv_h^\perp, \fxv_h^\perp) + 2h
A(\fxv_h^\perp,\fxv_h^\pm) + h^2 A(\fxv_h^\pm,a\fxv_h^\pm)  \vert $$
Since $\vert \fxv_h^\pm \vert$ is bounded, we get
$ \lim_{h \rightarrow 0} F(\fxv_h,h) \geq \vert  A(\fxv_h^\perp,
\fxv_h^\perp)  \vert$
Now, $\fxv_h^\perp$ is in $\fV$ then $\vert A(\fxv_h^\perp, \fxv_h^\perp)
\vert \geq min_{\fxv  \in \fV} \vert  A(\fxv ,\fxv)  \vert$, so
$$
\lim_{h \rightarrow 0} (min_{\fxv  \in \fW} F(\fxv ,h)) =  \lim_{h \rightarrow 0}
F(\fxv_h,h) $$
 \fb& \label{dem2} \geq
 min_{\fxv  \in \fV} \vert A(\fxv ,\fxv) \vert
 = \vert \nabla u \vert (sgn(curv(u)) \; accel(u) )^+
\fe&
\noindent (\ref{dem1}) and (\ref{dem2}) conclude the proof of the proposition.  \endproof

In addition to a quantization problem, if we wish to recover an ``acceleration"
interpretation of the term ``accel" we need somehow to make
appearing in the formulation of accel the velocities before and after the
considered point.

\begin{lem} Let $u$ be $C^2$, $\nabla u$ and $curv(u)$ not zero, then
\fb&
min_{\fxv\in \fW} ( \vert u(\fxx  - \fxv \fde& ) - u(\fxx ) \vert + \vert u(\fxx  + \fxv \fde& ) -
u(\fxx  ) \vert ) =
\fe&
$$
min_{\fxv_b,\fxv_a \in \fW} (\vert u(\fxx  - \fxv_b \fde& ) - u(\fxx ) \vert + \vert u(\fxx  +
\fxv_a \fde& ) - u(\fxx  ) \vert + \fde& \vert  \nabla u .(\fxv_b - \fxv_a) \vert ) +
o(\fde&^2)
$$
\end{lem}

\paragraph{Proof}
First, we remark by taking $\fxv_b = \fxv_a$ that the first part is larger than the
second part of the expression.
$$(\vert u(\fxx  - \fxv_b h ) - u(\fxx ) \vert + \vert u(\fxx  + \fxv_a h ) - u(\fxx  )
\vert + h \vert  \nabla u. (\fxv_b - \fxv_a) \vert )$$
$$ = \qquad \vert -h  (Du.\fxv_b)  + \frac{h^2}{2} (D^2u) (\fxv_b,
\fxv_b)  \vert + \vert h  (Du.\fxv_a)  + \frac{h^2}{2} (D^2u)
(\fxv_a, \fxv_a) \vert $$ $$ + h \vert Du. (\fxv_b - \fxv_a) \vert ) + o(h^2)
$$
$$ \geq \qquad \frac{h^2}{2} (\vert (D^2u) (\fxv_b, \fxv_b)  \vert +
\vert (D^2u)(\fxv_a, \fxv_a) \vert ) + o(h^2)$$
$$ \geq \qquad min_{\fxv \in \fW} (\vert   (D^2 u) (\fxv,\fxv)  \vert) +o (h^2)$$
$$ = \qquad min_{\fxv\in \fW} ( \vert u(\fxx  - \fxv h ) - u(\fxx ) \vert + \vert
u(\fxx  + \fxv h ) - u(\fxx  ) \vert ) + o(h^2)$$
by Proposition \ref{acinw}. \endproof

\paragraph{ Interpretation.}

We deduce from all  of these propositions an explicit formula for the apparent
acceleration \index{Apparent acceleration}
\fb& \label{HEUI}
\vert \nabla u \vert (sgn(curv(u)) \; accel(u) )^+ = \fe&
$$
min_{\fxv_b,\fxv_a \in \fW}  \frac{1}{\fde&^2} (\vert u(\fxx  - \fxv_b \fde& ) - u(\fxx
) \vert + \vert u(\fxx  + \fxv_a \fde& ) - u(\fxx  ) \vert + \fde& \vert
\nabla u .(\fxv_b - \fxv_a) \vert ) + o(1)
$$

\vfp
Of course for numerical experiments, we shall not compute the minimum for
all
vectors in $\fW$, but only for the vectors on the grid.
We have two differents parts in the second term~:
The first part is the variations of the grey level value of the point $\fxx$,
for candidate velocity vectors~: $\fxv_b$ between $\theta - \fde&$ and $\theta$
(velocity $b$efore $\theta$), and
$\fxv_a$ between $\theta$ and $\theta + \fde&$ (velocity $a$fter $\theta$).
These variations must be as small as possible, because a point is not
supposed to change its grey level value during its motion.
The second part is nothing but the ``acceleration",  or the difference between
$\fxv_b$ and $\fxv_a$ in the direction of the spatial gradient $\vert \nabla u
\vert$.

\section{Destruction of the non-smooth trajectories.}

Since trajectories are included into the spatio-temporal gray-level surfaces (level
surfaces), it
 is interesting to look at the evolution of such surfaces.  According to the
equation, the surfaces move (in scale) at each point with a speed in the direction of $\nabla u$
given by
$ curv(u)^{\frac{1-q}{3}} \; ( \hbox{sgn}(curv(u)) accel(u)^q )^{+}$.
(We do not
consider the case where $q=0$ that corresponds to a pure spatial smoothing).

Therefore any level surfaces that corresponds to an uniform motion  does not move
in scale (it is a steady state for the equation AMG). Such surfaces are straight in one
direction of the space-time.

We see also that parts of the surfaces where the curvature and the operator $accel$
have opposite signs do not move as well. Then if we take example of a uniform circle
under acceleration, the level surface corresponding to the circle moves only in one
of its side.

 \begin{figure}
\centerline{\hbox{\epsfxsize=8cm \epsfbox{mvfigure3.eps} }} \caption[Fig : The AMG model erodes a circle in
acceleration only only on one side.]{ The AMG model erodes a circle in acceleration only on one side. Indeed, when the
curvature and the acceleration have opposite signs, the evolution in scale is zero. (see the AMG equation).}
\end{figure}

 More geometrically the smoothing can only occur at points where the
level surface is strictly convex or strictly concave.
We can give an intuitive hint of why the smoothing is stopped on saddle points.
This property of the
model AMG, comes directly from the contrast invariance and the causality.
 They
imply a independent and continuous motion of level surfaces that makes that two level surfaces
can not cross them-selves. Now as shown in the picture \ref{saddle}, we can bound
non-convex and non-concave part of  surfaces by straight surfaces that have no
evolution, and then easily see why such parts does not move.

\begin{figure}
\centerline{\hbox{\epsfxsize=8cm \epsfbox{mvfigure4.eps} }} \label{saddle} \caption[Fig : Saddle points of level
surfaces are not moved by the AMG model.]{ Saddle points of level surfaces remain steady by the AMG model. Indeed, our
scale-space can be seen as a motion in scale of gray level-surfaces (isophotes). The level-surfaces that are straight
in time  correspond to a uniform translation and are  not changed by the smoothing. Therefore, the two thin cylindric
level-surfaces drawn left and right in the  figures above do not move in scale. Now, by the inclusion principle, two
level surfaces can never cross during the evolution in scale. Since, as displayed in the picture, it is possible to
squeeze any surface saddle point  between two such steady cylinders,  it follows that saddle points do not move in
scale as well. This property is readable in the scale space equation : at saddle points, the positive part of the
product of the curvature and of the acceleration is zero.}
\end{figure}

As a consequence, we can not expect from a such modelization to obtain a
smoothing of the trajectories. Non-smooth trajectories are not really smoothed
by the model but are simply destroyed.
Let us take an example. In figure \ref{traject}, we display a
oscillatory trajectory (in gray). The limit of  a smoothing of
this trajectory should be a straight trajectory.
Now using the same argument as in the preceding paragraph the gray surface can
not cross the white surface which has no evolution. Therefore the gray surface
can not become straight, because it should have to cross the white one.
A such trajectory is shrunk by the AMG model and disappears at a finite scale of smoothing (see figure \ref{syntheticsequencedenoised}).

We conclude that the assumptions we made for our model are incompatible
with the notion of smoothing trajectories.
Indeed non-straight trajectories are not more and more smoothed, but are
more and more removed. And by consequence a small perturbation in a straight trajectory
might imply a destruction of this trajectory although it would have been
kept without the perturbation.

\begin{figure}
\centerline{\hbox{\epsfxsize=4.5cm \epsfbox{mvfigure5.eps} }} \label{traject} \caption[Fig : The AMG model removes the
rotating trajectories]{ The level-surface in gray cannot become straight~: it would have to cross the white
level-surface which is invariant by the scale space. Now, during the smoothing process, the level-surface in gray will
be eroded on  its convex part, and will eventually disappear at a fixed scale : it cannot converge to any steady
surface since all of them  are straight in time. Thus, trajectories that are contained in the grey level surface end
being removed from the movie.   }
\end{figure}

%\input{EXPMV.TEX}
%\input{insImMv.tex}

\section{Conclusion.}

We have seen that there exists an unique affine,
contrast and Galilean  invariant scale-space
for movies, the AMG.  This model does a spatial smoothing
with a speed depending on the spatial curvature and
an apparent acceleration. The larger is the acceleration the
larger is the speed of smoothing. Therefore, as shown on the experiments it has a strong denoising property
since the noise does not generally generate
 regular trajectories.

Now we have seen that the properties asked to the scale-space
are compatible with the definition of the optical flow. In the sense that
the definition of the optical flow satisfies as well the contrast,
the affine, and the Galilean invariance.
But, the contrast invariance added to the causality (that defines the
scale-space) is incompatible with the notion of smoothing trajectories.
In others terms, non-smooth level-surfaces (on which are contained the trajectories
by definition of the optical flow) are more shrunk than smoothed.
In fact the AMG model as to be seen as a riddle that progressively
remove
non-smooth trajectories.

\begin{figure}[!ht]
\centerline{\hbox{ \epsfxsize=12cm \epsfbox{mam2.ps} }} \caption[AMG  model used for image sequence ``denoising'',
synthetic images]{The affine, morphological, galilean (AMG) model used for image sequence restoration (extraction of
coherent trajectories). Above : three successive images extracted from a synthetic sequence, made of salt and pepper
noise, plus some squares placed at random locations. In addition, a little black square in uniform motion has been
added in the whole sequence. Bottom : resulting images at calibrated scale 500p (scale at which a spatio-temporal
sphere of 500 pixels  disapears by AMG). Only the little black square remains, as it has a coherent motion. }
\label{syntheticsequencedenoised}
\end{figure}


\begin{figure}[!ht]
\centerline{\hbox{ \epsfxsize=12cm \epsfbox{mam1.ps} }} \caption[AMG model (Affine, Morphological, Galilean) used for
image sequence denoising] { AMG model (Affine, Morphological, Galilean) used for image sequence ``denoising''. Above :
three successive images extracted from a sequence. Second row : resulting images at calibrated scale 100 pixels (scale
at which a spatiotemporal sphere of 100 pixels disapears). Third row : Some noise has been added to the original
sequence (25\% of the pixels are corrupted). Bottom : resulting images at scale 100.}
 \end{figure}


\zbibnote

\noindent
{\bf The Optical Flow:}
The problem of estimating dense velocities field from image sequence is a entire research topic
by itself. Since it is not the main point of this book we refer to some articles dealing with
 that subject:
\cite{Fennema:1979:VDS,Horn:1981:DOF,Hildreth:1984:MVM,Adelson:1985:SEM,
Heeger:1987:OFS,Verri:1987:AQO,Nagel:1988:IST,Anandan:1989:CFA,Barron:1994:POF,Tekalp:1995:DVP}...
The aperture problem of the optical flow - that is its non uniqueness- has appeared very
early and has been often adressed by e.g. some smoothness constraint on the flow it self,
see e.g. \cite{Horn:1981:DOF,Nagel:1986:ISC,Snyder:1991:MFS,Barron:1994:POF} or in some cases by
an implicit smoothing of image sequences see e.g. \cite{Adelson:1985:SEM}...

\noindent
{\bf Smoothing images sequences:}
Explicit smoothing of image sequences, for the purpose of estimating the optical flow or for other
purposes has first appeared as a direct extension of the 2D smoothing to the 3D. That is no specific
rule was given to the time. In that sense most all 2D filters can be adapted to N-dimensional data,
and in particular the images sequences.

In \cite{Adelson:1985:SEM}, it is implicitely proposed to tune the sequence filtering to few different
orientations in space-time. All designed filters give different answers, answers that were used as basis
of the optical flow estimates. Even if it was impossible to use a filter for all
spatio-temporal directions, the idea to orient the filtering in the direction of the
(unknown) motion was there.

In \cite{Alvarez:1993:AFE} the basic principles explained in this chapter were proposed.
In particular the "Galilean Invariance". Surprisingly, these formal principles yield an
 anisotropic diffusion oriented, for each point, in the direction of the (unknown) optical flow
 \cite{Alvarez:1993:AFE,Guichard:1998:MAG}...
Several other works have introduced other smoothings depending on its aim and where
the time plays a specific rule. In \cite{Moisan:1995:MAM}  the author formalizes a smoothing compatible
with the aim of estimating depth from an image sequence. In \cite{Niessen:1997:MSA} and
one could find adaptations of the 2D linear smoothing theory to an anisotropic diffusion in the direction
of an estimated optical flow.

\ifodd\count0
    \clearpage
    \leavevmode
    \thispagestyle{empty}
    \clearpage
\else\relax \fi
