“JMMBookOct04” 30/10/2009 page 1

[figure] [figure]

Contrast invariant image analysis and PDE’s

Frédéric Guichard  Jean-Michel Morel  Robert Ryan

“JMMBookOct04” 30/10/2009 page 2

[figure] [figure]

2 “JMMBookOct04” 30/10/2009 page i

[figure] [figure]

Contents

Introduction 1 I.1 Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 I.2 Image processing . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 I.3 PDEs and image processing . . . . . . . . . . . . . . . . . . . . . 11

1 Notation and background material 33 I.0.1 A framework for sets and images . . . . . . . . . . . . . . 36

I Linear Image Analysis 39

2 The Heat Equation 41 2.1 Linear smoothing and the Laplacian . . . . . . . . . . . . . . . . 41 2.2 Existence and uniqueness of solutions of the heat equation …. 43 2.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 2.4 Comments and references . . . . . . . . . . . . . . . . . . . . . . 49

3 Iterated Linear Filters and the Heat Equation 51 3.1 Smoothing and the Laplacian . . . . . . . . . . . . . . . . . . . . 51 3.2 The convergence theorem . . . . . . . . . . . . . . . . . . . . . . 53 3.3 Directional averages and directional heat equations . . . . . . . . 56 3.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.5 Comments and references . . . . . . . . . . . . . . . . . . . . . . 58 3.6 Preliminaries on sub-pixel interpolation . . . . . . . . . . . . . . 62

4 Le cas discret 65

4.1 Transforme de Fourier Discrte, applications . . . . . . . . . . . . 65 4.1.1 La dimension 1 . . . . . . . . . . . . . . . . . . . . . . . . 65 4.1.2 La dimension 2 . . . . . . . . . . . . . . . . . . . . . . . . 67 4.1.3 Le phnomne du repliement de spectre ou aliasage . . . . . 68 4.1.4 Manipulations propres et simples et leur visualisation . . 71 4.1.5 L’quation de la chaleur . . . . . . . . . . . . . . . . . . . 72 4.1.6 L’utilisation de la transforme de Fourier discrte pour dfinir

zoom, translations et rotations des images . . . . . . . . . 73

5 From continuous to digital images, and back 77 5.0.7 The practical Shannon interpolation: zero-padding …. 78 5.0.8 Shannon or zero-padding? . . . . . . . . . . . . . . . . . . 79

5.1 The Gaussian semigroup . . . . . . . . . . . . . . . . . . . . . . . 80

i “JMMBookOct04” 30/10/2009 page ii

[figure] [figure]

ii CONTENTS

5.2 The right gaussian blur for well-sampling . . . . . . . . . . . . . 81 5.2.1 Discrete sampling . . . . . . . . . . . . . . . . . . . . . . 83

6 The SIFT Method 87 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 6.2 A Short Guide to SIFT Encoding . . . . . . . . . . . . . . . . . . 88

6.2.1 Scale-Space Extrema . . . . . . . . . . . . . . . . . . . . . 89 6.2.2 Accurate Key Point Detection . . . . . . . . . . . . . . . 90 6.2.3 Construction of the SIFT descriptor . . . . . . . . . . . . 91 6.2.4 Final matching . . . . . . . . . . . . . . . . . . . . . . . . 92

6.3 Image acquisition model underlying SIFT . . . . . . . . . . . . . 93 6.3.1 The camera model . . . . . . . . . . . . . . . . . . . . . . 93 6.3.2 Condensed description of the SIFT method . . . . . . . . 94

6.4 Scale and SIFT: consistency of the method . . . . . . . . . . . . 94 6.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.6 Comments and references . . . . . . . . . . . . . . . . . . . . . . 97

7 Linear Scale Space and Edge Detection 99

7.1 The edge detection doctrine . . . . . . . . . . . . . . . . . . . . . 99

7.1.1 Discussion and critique . . . . . . . . . . . . . . . . . . . 100 7.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 7.3 Comments and references . . . . . . . . . . . . . . . . . . . . . . 106

8 Four Algorithms to Smooth a Shape 107 8.1 Dynamic shape . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 8.2 Curve evolution using the heat equation . . . . . . . . . . . . . . 108 8.3 Restoring locality and causality . . . . . . . . . . . . . . . . . . . 109

8.3.1 Localizing the dynamic shape method . . . . . . . . . . . 110 8.3.2 Renormalized heat equation for curves . . . . . . . . . . . 110

8.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 8.5 Comments and references . . . . . . . . . . . . . . . . . . . . . . 113

9 Affine Invariant Image Comparison 115 9.1 The affine camera model . . . . . . . . . . . . . . . . . . . . . . . 116 9.2 A-SIFT : combining simulation and normalization . . . . . . . . 120 9.3 High transition tilts . . . . . . . . . . . . . . . . . . . . . . . . . 121 9.4 Parameter sampling and complexity . . . . . . . . . . . . . . . . 125

9.4.1 Sampling ranges . . . . . . . . . . . . . . . . . . . . . . . 125 9.4.2 Sampling steps . . . . . . . . . . . . . . . . . . . . . . . . 126 9.4.3 Acceleration with multi-resolution . . . . . . . . . . . . . 128 9.4.4 A-SIFT Complexity . . . . . . . . . . . . . . . . . . . . . 129

10 The mathematical justification 133 10.0.5 The image formation model . . . . . . . . . . . . . . . . . 133 10.0.6 Inverting tilts . . . . . . . . . . . . . . . . . . . . . . . . . 134 10.0.7 Proof that A-SIFT works . . . . . . . . . . . . . . . . . . 136 10.0.8 Conclusion on the algorithms . . . . . . . . . . . . . . . . 139 “JMMBookOct04” 30/10/2009 page iii

[figure] [figure]

CONTENTS iii

11 Experiments on affine invariant methods 141 11.1 Affine normalization methods: are they fully affine invariant? . . 141 11.2 Global Normalization and Encoding . . . . . . . . . . . . . . . . 142

11.2.1 Global Affine Normalization . . . . . . . . . . . . . . . . . 142 11.2.2 Maximally Stable Extremal Regions (MSER) . . . . . . . 144

11.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 11.3.1 Exploring tilts and zooms . . . . . . . . . . . . . . . . . . 145 11.3.2 Absolute Tilt Tests . . . . . . . . . . . . . . . . . . . . . . 147 11.3.3 Transition Tilt Tests . . . . . . . . . . . . . . . . . . . . . 151 11.3.4 Comparative experiments . . . . . . . . . . . . . . . . . . 152 11.3.5 Symmetry detection in perspective . . . . . . . . . . . . . 160

11.4 Comments and references . . . . . . . . . . . . . . . . . . . . . . 160

II Contrast-Invariant Image Analysis 163

12 Contrast-Invariance and Level Sets 165 12.1 From an image to its level sets and back . . . . . . . . . . . . . . 166 12.2 Contrast changes and level sets . . . . . . . . . . . . . . . . . . . 168 12.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 12.4 Comments and references . . . . . . . . . . . . . . . . . . . . . . 173

13 Specifying the contrast of images 175 13.1 Midway equalization . . . . . . . . . . . . . . . . . . . . . . . . . 177 13.2 Midway equalization on image pairs . . . . . . . . . . . . . . . . 179

13.2.1 Movie equalization . . . . . . . . . . . . . . . . . . . . . . 182 13.3 Comments and references . . . . . . . . . . . . . . . . . . . . . . 185

14 Contrast-Invariant Monotone Operators 187

14.1 Contrast-invariance . . . . . . . . . . . . . . . . . . . . . . . . . . 188 14.1.1 Set monotone operators . . . . . . . . . . . . . . . . . . . 188 14.1.2 Monotone function operators . . . . . . . . . . . . . . . . 189

14.2 Stack filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 14.3 The level set extension . . . . . . . . . . . . . . . . . . . . . . . 193 14.4 A first application: the extrema killer . . . . . . . . . . . . . . . 195 14.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 14.6 Comments and references . . . . . . . . . . . . . . . . . . . . . . 201

15 Sup-Inf Operators 203 15.1 Translation-invariant monotone set operators . . . . . . . . . . . 203 15.2 The Sup-Inf form . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 15.3 Locality and isotropy . . . . . . . . . . . . . . . . . . . . . . . . . 207 15.4 The operators who’s who . . . . . . . . . . . . . . . . . . . . . . 209

15.4.1 Commutation with thresholds almost everywhere . . . . . 210 15.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 15.6 Comments and references . . . . . . . . . . . . . . . . . . . . . . 212 “JMMBookOct04” 30/10/2009 page iv

[figure] [figure]

iv CONTENTS

16 Erosions and Dilations 213 16.1 Set and function erosions and dilations . . . . . . . . . . . . . . 213 16.2 Multiscale aspects . . . . . . . . . . . . . . . . . . . . . . . . . . 214 16.3 The PDEs associated with erosions and dilations . . . . . . . . . 216 16.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 16.5 Comments and references . . . . . . . . . . . . . . . . . . . . . . 221

17 Median Filters and Mathematical Morphology 223 17.1 Set and function medians . . . . . . . . . . . . . . . . . . . . . . 223 17.2 Self-dual median filters . . . . . . . . . . . . . . . . . . . . . . . . 225

17.2.1 Chessboard dilemma and fattening effect . . . . . . . . . 227 17.3 Discrete median filters and the ”usual” median value . . . . . . . 227 17.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 17.5 Comments and references . . . . . . . . . . . . . . . . . . . . . . 231

III Local Asymptotic Analysis of Operators 233

18 Curves and Curvatures 235 18.1 Tangent, normal, and curvature . . . . . . . . . . . . . . . . . . . 235 18.2 The structure of the set of level lines . . . . . . . . . . . . . . . . 238 18.3 Curvature of the level lines . . . . . . . . . . . . . . . . . . . . . 240 18.4 The principal curvatures of a level surface . . . . . . . . . . . . . 242 18.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 18.6 Comments and references . . . . . . . . . . . . . . . . . . . . . . 248

19 The Main Curvature Equations 249 19.1 The definition of a shape and how it is recognized . . . . . . . . 249 19.2 Multiscale features and scale space . . . . . . . . . . . . . . . . . 250 19.3 From image motion to curve motion . . . . . . . . . . . . . . . . 251

19.3.1 A link between image and curve evolution . . . . . . . . . 251 19.3.2 Introduction to the affine curve and function equations . . 253 19.3.3 The affine scale space as an intrinsic heat equation …. 253

19.4 Curvature motion in N dimensions . . . . . . . . . . . . . . . . . 255 19.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 19.6 Comments and references . . . . . . . . . . . . . . . . . . . . . . 256

20 Finite Difference Schemes 259 20.1 Case of Mean curvature motion. . . . . . . . . . . . . . . . . . . 259 20.2 FDS for AMSS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 20.3 IL MANQUE UNE EXPERIENCE AMSS SUR L’INVARIANCE

AFFINE! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 20.4 Numerical normalization of scale. . . . . . . . . . . . . . . . . . . 266 20.5 Contrast invariance and the level set extension . . . . . . . . . . 267 20.6 Problems at extrema . . . . . . . . . . . . . . . . . . . . . . . . 268 20.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 20.8 Comments and references . . . . . . . . . . . . . . . . . . . . . . 270 “JMMBookOct04” 30/10/2009 page 1

[figure] [figure]

Introduction

This book addresses the problem of low-level image analysis and, as such, is a contribution to image processing and imaging science. While the material touches on several aspects of image analysis—and peripherally on other parts of image processing—the main subject is image smoothing using partial differential equations (PDEs). The rational for a book devoted to smoothing is the assumption that a digital image must be smoothed before reliable features can be extracted.

The purpose of this introduction is to establish some of the language, conventions, and assumptions that are used throughout the book, to review part of the history of PDEs in image processing, and to introduce notation and background material.

I.1 Images

Since the objects of our study are ultimately digital images, we begin by defining what we mean by “digital image” and by describing some of the ways these images are obtained and some current assumptions about the “original images” from which the digital images are derived.

Most of the images dealt with will be natural images, that is, images from nature (people, landscapes, cityscapes, etc.). We include medical images and astronomical images, and we do not exclude drawings, paintings, and other manmade images. All of the images we consider will be grayscale images. Thus, mathematically, an image is a real-valued function u defined on some subset Ω of the plane ℝ^{2}. The value u(x), x = (x, y) ∈ Ω, represents the gray level of the image at the point x. If u is a digital image, then its domain of definition is a finite grid with evenly spaced points. It is often square with 2^{n} ×2^{n} points. The gray levels u(x) are typically coded with the integers 0–255, where 0 represents black and 255 represents white. If h is the distance between grid lines, then the squares with sides of length h centered at the points u(x) are called pixels, where “pix” is slang for “picture” and “el” stands for “element.”

The mathematical development in this book proceeds along two parallel lines. The first is theoretical and deals with images u that belong to function spaces, generally spaces of continuous functions that are defined on domains of ℝ^{2}. The second line concerns numerical algorithms, and for this the images are digital images. To understand the relations between the digital and continuous images, it is useful to consider some examples of how images are obtained and some of the assumptions we make about the processes and the images. Perhaps the simplest example is that of taking a picture of a natural scene with a dig-

1 “JMMBookOct04” 30/10/2009 page 2

[figure] [figure]

2 Introduction

ital camera. The scene—call it S—is focused at the focal plane of the camera forming a representation of S that we denote by u_{f}. When we take the picture, the image u_{f} is sampled, or captured, by an array of charged coupled devices (CCDs) producing the digital image u_{d}. This image, u_{d}, is the only representation of S that is directly available to us; the image u_{f} is not directly available to us. Even more elusive is the completely hypothetical image that we call u_{S}. This is the representation of S that would be formed at the focal plane of an ideal camera having perfect optics. A variation on this example is to capture u_{f} on film as the image u_{p}. Then u_{p} can be sampled (scanned) to produce a digital image u_{d}. For example, before the advent of CCDs, astronomical images were captured on Schmidt plates. Many of these plates have been scanned recently, and the digital images have been made available to astronomers via the Internet.

Aspects of the photographic example could be recast for medical imaging. Although photography plays an important role in medicine, images for diagnostic use are often obtained using other kinds of radiation. X-rays are perhaps closest to our photographic example. In this case, there is an image corresponding to u_{p} that can be scanned to produce a digital image u_{d}. Other medical imaging processes, such as scintigraphy and nuclear magnetic resonance, are more complicated, but these processes yield digital images. The images examined by the experts are often “negatives” produced from an original digital images. Irrespective of the process, digital images captured by some technology all have one characteristic in common: They are all noisy.

One way to relate the different representations of S, is to write

u_{d} = T u_{S} + n,

where T is a hypothetical operator representing some technology and n is noise. In the case of photography, we might write this in two steps,

\begin{cases}u_{f} = P ∗ u_{S} + n_{1}, \\ u_{d} = Ru_{f} + n_{2}, \end{cases}

where P represents the optics and R represents the sampling. This is a useful model in optical astronomy, since astronomers have considerable knowledge about the operators P and R and about the noises n_{1} and n_{2}. Similarly, experts in other technologies know a great deal about the processes and noise sources. Noise and pixels are illustrated in Figure I.1

In the photographic example, the image u_{f} is a smoothed version of u_{S}. Furthermore, Ru_{f}(x) is not exactly u_{f}(x) but rather an average of values of u_{f} in a small neighborhood of x, which is to say that the operator R does some smoothing. Thus, in this example, u_{d} is sampled from a smoothed version of S. We are going to assume that this is the case for the digital images considered in the book, except for digital images that are artificially generated. This is realistic, since all of the processes T that we can imagine for capturing images, smooth the original photon flux. In fact, this is more of an observation about technology than it is an assumption. We are also going to assume that, for any technology considered, the sampling rate used to produce u_{d} is high enough so that u_{d} is a “good” representation of the smoothed version of S, call it u_{f}, from which it was derived. Here, “good” means that the parallel development “JMMBookOct04” 30/10/2009 page 3

[figure] [figure]

I.2. IMAGE PROCESSING 3

in the book mentioned above make sense; it means that, from a practical point of view, the theoretical development that uses smooth functions to model the images u_{f} is indeed related to the algorithmic development that uses the digital images u_{d}. We will say more about smoothing and sampling in section I.2.

Figure I.1: A noisy image magnified to show the pixels.

It is widely assumed that the underlying “real image” u_{S} is either a measure or, for more optimistic authors, a function that has strong discontinuities. Rudin in 1987 [214] and De Giorgi and Ambrosio in 1988 [92] proposed independently the space BV (ℝ^{2}) of functions with bounded variation as the correct function space for modeling the images u_{S}. A function f is in BV (ℝ^{2}) if its partial derivatives ∂f/∂x and ∂f/∂y, taken as distributions, are Radon measures with finite total mass. BV (ℝ^{2}) looked at first well adapted to modeling digital images because it contains functions having step discontinuities. In fact, the characteristic functions of smooth domains in ℝ^{2} belong to BV (ℝ^{2}). However, in 1999, Alvarez, Gousseau, and Morel used a statistical device on digital images u_{d} to estimate how the corresponding images u_{S} oscillate [6]. They deduced by geometric-measure arguments, that the u_{S} have, in fact, unbounded variation. We may therefore accept the idea that these high-resolution images contain very strong oscillations. Although the images u_{f} are smoothed versions of the u_{S}, and hence the oscillations have been averaged, common sense tells us that they also have large derivatives at transitions between different observed objects, that is, on the apparent contours of physical objects. Furthermore, we expect that these large derivatives (along with noise) are passed to the digital images u_{d}.

I.2 Image processing

For the convenience of exposition, we divide image processing into separate disciplines. These are distinguished not so much by their techniques, which often overlap, as they are by their goals. We will briefly describe two of these areas: compression and restoration. The third area, image analysis, is the main subject of the book and will be discussed in more detail.

“JMMBookOct04” 30/10/2009 page 4

[figure] [figure]

4 Introduction

Image compression

Compression is based on the discrete nature of digital images, and it is motivated by economic necessity: Each form of storage and transmission has an associated cost, and hence one wishes to represent an image with the least number of bits that is compatible with end usage. There are two kinds of compression: lossless compression and lossy compression. Lossless compression algorithms are used to compress digital files where the decompressed file must agree bit-by-bit with the original file. Perhaps the best known example of lossless compression is the zip format. Lossless algorithms can be used on any digital file, including digital images. These algorithms take advantage of the structure of the file itself and have nothing to do with what the file represents. On the other hand, lossy compression algorithms take advantage of redundancies in natural images and subtleties of the human visual system. Done correctly, one can throw away information contained in an image without impairing its usefulness. The goal is to develop algorithms that provide high compression factors without objectionable visible alterations. Naturally, what is visually objectionable depends on how the decompressed image is used. This is nicely illustrated with our photographic example. Suppose that we capture the image u_{f} at our camera’s highest resolution. If we are going to send u_{d} over the Internet to a publisher to be printed in a high-quality publication, then we want no loss of information and will probably send the entire file in the zip format. If, however, we just want the publisher to have a quick look at the image, then we would probably send u_{d} compressed as a .jpg file, using the Joint Photographic Expert Group (JPEG) standard for still image compression. This kind of compression is illustrated in Figure I.2.

Figure I.2: Compression. Left to right: the original image and its increasingly compressed versions. The compression factors are roughly 7, 10, and 25. Up too a 10 factor, alterations are hardly visible.

Image restoration

A second area is restoration or denoising. Restoring digital images is much like restoring dirty or damaged paintings or photographs. Beginning with a digital image that contains blurs or other perturbations (all of which may be considered as noise), one wishes to produce a better version of the image; one wishes to enhance aspects of the image that have been attenuated or degraded. Image restoration plays an important role in law enforcement and legal proceedings. For example, surveillance cameras generally produce rather poor images that must often be denoised and enhanced as needed. Image restoration is also “JMMBookOct04” 30/10/2009 page 5

[figure] [figure]

I.2. IMAGE PROCESSING 5

important in science. When the Hubble Space Telescope was first launched in 1990, and until it was repaired in 1993, the images it returned were all blurred due to a spherical aberration in the telescope’s primary mirror. Elaborate (and costly) algorithms were developed to restore these poor images, and indeed useful images were obtained during this period. Restoration is illustrated in Figure I.3 with an artificial example. The image on the left has been ostensibly destroyed by introducing random-valued pixels amounting to 75% of the total pixel count. Nevertheless, the image can be significantly restored, and a restored version is shown on the right, by using a Vincent and Serra operator which we will study in Chapter ??, the “area opening”.

Figure I.3: Denoising. Left: an image with up to 75% of its pixels contaminated by simulated noise. Right: a denoised version by the Vincent-Serra algorithm (area opening).

Image analysis

A third area of image processing is low-level image analysis, and since this is the main topic of the book, it is important to explain what we mean by “low-level” and “analysis.” “Analysis” is widely used in mathematics, with various shades of meaning. Our use of “analyze,” and thus of “analysis,” is very close to its common meaning, which is to decompose a whole into its constituent parts, to study the parts, and to study their relation to the whole. For our purposes, the constituent parts are, for the most part, the “edges” and “shapes” in an image. These objects, which are often called features, are things that we could, for a given image, point to and outline, although for a complex natural image this would be a tedious process. The goal of image analysis is to create algorithms that do this automatically.

The term “low-level” comes from the study of human vision and means extracting reliable, local geometric information from an image. At the same time, we would like the information to be minimal but rich enough to characterize the image. The goal here is not compression, although some of the techniques may provide a compressed representation of the image. Our goal is rather to “JMMBookOct04” 30/10/2009 page 6

[figure] [figure]

6 Introduction

answer questions like, Does a feature extracted from image A exist in image B? We are also interested in comparing features extracted from an image with features stored in a database. As an example, consider the level set at the left of Figure I.4. It consists of major features (roughly, the seven appendages) and noise. The noise, which is highly variable, prevents us from comparing the image directly with other images having similar shapes. Thus we ask for a sketchy version, where, however, all essential features are kept. The images on the right are such a sketchy versions, where most of the spurious details (or noise) have disappeared, but the main structures are maintained. These sketchy versions may lead to concise invariant encoding of the shape. Notice how the number of inflexion points of the shape has decreased in the simplification process. This is an example of what we mean by image analysis. The aim is not denoising or compression. The aim is to construct an invariant code that puts in evidence the “main parts” of an image (in this case, the appendages) and that facilitates fast recognition in a large database of shapes.

Figure I.4: Analysis of a shape. The original scanned shape is on the left. Simplified versions are to the right.

Edge detection and scale space

Since the earliest work in the 1960s, one of the goals of image analysis has been to locate the strong discontinuities in an image. This search is called edge detection, and it derives from early research that involved working with images of cubes. This seemingly simple goal turned out to be exceedingly difficult. Here is what David Marr wrote about the problem in the early 1980s ([163], p. 16):

The first great revelation was that the problems are difficult. Of course, these days this fact is a commonplace. But in the 1960s almost no one realized that machine vision was difficult. The field had to go through the same experience as the machine translation field did in its fiascoes of the 1950s before it was at last realized that here were some problems that had to be taken seriously. The reason for this misconception is that we humans are ourselves so good at vision. The notion of a feature detector was well established by Barlow and by Hubel and Wiesel, and the idea that extracting edges and lines from images might be at all difficult simply did not occur to those who had not tried to do it. It turned out to be an elusive problem: Edges that are of critical importance from a threedimensional point of view often cannot be found at all by looking at the intensity changes in an image. Any kind of textured image gives a multitude of noisy edge segments; variations in reflectance “JMMBookOct04” 30/10/2009 page 7

[figure] [figure]

I.2. IMAGE PROCESSING 7

and illumination cause no end of trouble; and even if an edge has a clear existence at one point, it is as likely as not to fade out quite soon, appearing only in patches along its length in the image. The common and almost despairing feeling of the early investigators like B.K.P. Horn and T.O. Binford was that practically anything could happen in an image and furthermore that practically everything did.

The point we wish to emphasize is that textures and noise (which are often lumped together in image analysis) produce unwanted edges. The challenge was to separate the “true edges” from the noise. For example, one did not want to extract all of the small edges in a textured wall paper; one wanted the outline of the wall. The response was to blur out the textures and noise in a way that left the “true edges” intact, and then to extract these features. More formally, image analysis was reformulated as two processes: smoothing followed by edge detection. At the same time, a new doctrine, the scale space, was proposed. Scale space means that instead of speaking of features of an image at a given location, we speak of them at a given location and at a given scale, where the scale quantifies the amount of smoothing performed on the image before computing the features. We will see in experiments that “edges at scale 4” and “edges at scale 7” are different outputs of an edge detector.

Three requirements for image smoothing operators

We have advertised that this book is about image analysis, which we have just defined to be smoothing followed by edge detection, or feature extraction. In fact, the text focuses on smoothing and particularly on discussing and answering the question, What kind of smoothing should be used? To approach this problem, we need to introduce three concepts associated with image analysis operators. These concepts will be used to narrow the field of smoothing operators. We introduce them informally at first; more precise meanings will follow.

Localization

The first notion is localization. Roughly speaking, to say that an operator T is localized means it essentially uses information from a small neighborhood of x to compute the output T u(x). Recall that the sampling operator R in the photographic example was well localized. As another example, consider the classic Gaussian smoothing operators G_{t} defined by

G_{t}u(x) = G_{t} ∗ u(x) = ∫_{ℝ^{2}} G_{t}(y)u(x − y) dy,

where G_{t}(x) = (1/4πt)e^{−|x|^{2}/4t}. If t > 0 is small, then the Gaussian G_{t} is well localized around zero and G_{t}u(x) is essentially an average of the values of u(x) in a small neighborhood of x. The importance of localization is related to the occlusion problem: Most optical images consist of a superposition of different objects that partially obscure one another. It is clear that we must avoid confusing them in the analysis, as would, for example, G_{t} if t is large. It is for reasons like this that we want the analysis to be as local as possible.

“JMMBookOct04” 30/10/2009 page 8

[figure] [figure]

8 Introduction

We will prove in Chapter 1 under rather general conditions that u(t, x) = G_{t} ∗ u_{0}(x) is the unique solution of the heat equation

\frac{∂u}{∂t} = Δu

with initial value u_{0}. Thus, we can say that smoothing u_{0} with the Gaussian G_{t} is equivalent to applying the heat equation to u_{0}. We will see that the heat equation is possibly the worst candidate in our search for the ideal smoothing operator, since, except for small t, it is poorly localized and produces a very blurred image.

Iteration

One might conjecture that a way around this problem with the heat equation would be to replace G_{t} with a more suitable positive kernel. This is not the case, but it does serve to introduce the second concept, which is iteration. We will show in Chapter 2 that under reasonable assumptions and appropriate rescalings, iterating a convolution with a positive kernel leads to the Gaussian, and thus directly back to the heat equation. There is, however, a different point of view that leads to useful smoothing operators: Instead of looking for a different kernel, look for other PDEs that provide smoothing. This program leads to a class of nonlinear PDEs, where the Laplacian in the heat equation is replaced by various nonlinear operators. We will see that for these operators it is generally better, from the localization point of view, to iterate a well localized operator than to apply it directly at a large scale. This, of course, is just not true for the heat equation; if you iterate n times the convolution G_{t} ∗ u you get exactly G_{nt} ∗ u. This is a good place to point out that if we are dealing with smoothing, localization, and iteration, then we are talking about parabolic PDEs. This announcement is heuristic, and the object of the book is to formalize and to make precise the necessity and the role of several PDEs in image analysis.

Invariance

Our last concept is invariance. Invariance requirements play a central role in image analysis because the objects to be recognized must be recognized under varying conditions of illumination (contrast invariance) and from different points of view (projective invariance). Contrast invariance is one of the central requirements of the theory of image analysis called mathematical morphology (see, for example, Matheron [169] or Serra [228]). This theory involves a number of contrast-invariant image analysis operators, including dilations, erosions, median filters, openings, and closings. We are going to use this theory by attempting to localize as much as possible these morphomath operators to exploit their behavior at small scales. We will then iterate these operators. This will lead to the proof that several geometric PDEs, namely, the curvature motions, are asymptotically related to certain morphomath operators in much the same way that linear smoothing is related to the heat equation. Thus, through these PDEs, one is able to combine the scale space doctrine and mathematical morphology. In particular, affine-invariant morphomath operators, which seemed at first to be computationally impractical, turn out to yield in their local iterated “JMMBookOct04” 30/10/2009 page 9

[figure] [figure]

I.2. IMAGE PROCESSING 9

Figure I.5: Shannon theory and sampling. Left to right: original image; smoothed image; sampled version of the original image; sampled version of the smoothed image. This illustrates the famous Shannon-Nyquist law that an image must be smoothed before sampling in order to avoid the aliasing artifacts.

version a very affordable PDE, the so called affine morphological scale space (AMSS) equation.

Shannon’s sampling theory

We mentioned in section I.1 that most of the digital images u_{d} that come to us in practice have been sampled from a smoothed version, call it u_{f}, of the “real image” u_{S}. This was basically a comment about the technology. Another comment (or assumption) was that the sampling rate was high enough to capture all of the information in u_{f} that is needed in practice. What we mean by this is that the representations of u_{f} that we reconstruct from u_{d} show no signs that u_{f} was undersampled. This is an empirical statement; we will comment on the theory in a moment, but first we wish to illustrate in Figure I.5 what can happen if an image is undersampled.

We call the original image on the left Victor. Notice that Victor’s sweater contains a striped pattern, which has a spatial frequency that is high relative to other aspects of the picture. If we attempt to reduce the size of Victor by simply sampling, for example, by taking one pixel in sixteen in a square pattern, we obtain a new image (the third panel) in which the sampling has created new and unstable patterns. Notice how new stripes have been created with a frequency and direction that has nothing to do with the original. This is called aliasing, and it is caused by high spatial frequencies being projected onto lower frequencies, which creates new patterns. If this had been a video instead of being a still photo, these newly created patterns would move and flicker in a totally uncontrolled way. This kind of moving pattern often appears in recent commercial DVDs. They have simply not been sampled at a high enough rate. The second panel in Figure I.5 is a version of Victor that has been smoothed enough so that we no longer see the stripes in the sweater. This image is sampled the same way—every fourth pixel horizontally and vertically—and appears in panel four. It is not a good image, but there are no longer the kinds of artifacts that appear in the third image. To compare the images we have magnified the sampled versions by a factor of four. This example also shows that simply subsampling an image is a poor way to compress it.

This pragmatic discussion and the experiment have their theoretical counterpart, namely, Shannon’s theory of sampling. Briefly, Shannon’s theorem, in the “JMMBookOct04” 30/10/2009 page 10

[figure] [figure]

10 Introduction

two-dimensional case, states that for an image to be accurately reconstructed from samples, the image must be bandlimited, which means that it contains no spatial frequencies greater than some bound λ, and the sampling rate must be higher than a factor of λ. Some implications of these statements are that the image u must be infinitely differentiable, that its domain of definition is all of ℝ^{2}, and that there must be an infinite number of samples to accurately reconstruct u. Furthermore, in Shannon’s theory, the image u is reconstructed as an infinite series of trigonometric functions. Note that this is very different from what was done in Figure I.5. So what does this have to do with the problems addressed in this book? What does this have to do with, say, a hypothesized u_{S} in BV (ℝ^{2}) that is definitely not bandlimited? Our answer, which may smack of smoke and mirrors, is that we always are working in two parallel worlds, the theoretical one and the practical one based on numerical computations, and that these two worlds live together in harmony at a certain scale. Here is an example of what we mean: Suppose that u is not a bandlimited image. To sample it properly we would first have to smooth it with a bandlimited kernel. Suppose that instead we smooth it with the Gaussian G_{t}, which is not bandlimited. Theoretically this is wrong, but practically, the spectrum of G_{t}, which is G_{t} itself, decays exponentially. If |x|^{2}/4t is sufficiently large, then G_{t}(x) appears as zero in computations, and thus it is “essentially” bandlimited. Arguments like this could be made for other situations, but the important point for the reader to keep in mind is that the parallel developments, theory and practice, make sense in the limit.

In the next section, we present a survey of most of the PDEs that have been proposed for image analysis. This provides an informal account of the mathematics that will be developed in detail in the following chapters.

We wish to end this section with a mild disclaimer, and for this we take a page from Theory of Games and Economic Behavior by John von Neumann and Oskar Morgenstern where they comment on their theory of a zero-sum two-person game [254] p. 147:

We are trying to find a satisfactory theory,—at this stage for the zero-sum two-person game. Consequently we are not arguing deductively from the firm basis of an existing theory—which has already stood all reasonable tests—but we are searching for such a theory.. . . This consists in imagining that we have a satisfactory theory of a certain desired type, trying to picture the consequences of this imaginary intellectual situation, and then drawing conclusions from this as to what the hypothetical theory must be like in detail. If this process is applied successfully, it may narrow the possibilities for the hypothetical theory of the type in question to such an extent that only one possibility is left,—i.e. that the theory is determined, discovered by this device. Of course, it can happen that the application is even more “successful,” and that it narrows the possibilities down to nothing—i.e. that it demonstrates that a consistent theory of the kind desired is inconceivable.

We take much the same philosophical position, and here is our variation on the von Neumann–Morgenstern statement: We do not suggest that what will be developed here is a necessary future for image analysis. However, if image analysis requires a smoothing theory, then here is how it should be done, and “JMMBookOct04” 30/10/2009 page 11

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 11

here is the proof that there is no other way to do it. This statement does not exclude the possibility of other theories, based on different principles, or even the impossibility of making any theory.

I.3 PDEs and image processing

We have argued that smoothing—suppressing high spatial frequencies—is a necessary part of image processing in at least two situations: An image needs to be smoothed before features can be extracted, and images must be smoothed before they are sampled. We have also mentioned that, while smoothing with the Gaussian is not a good candidate for the first situation (we will see that it is not contrast invariant, and it is not well localized except for small t), it is not unreasonable to use it numerically in the second situation, since it does a good job of suppressing high frequencies. These smoothing requirements and the fact that the Gaussian is the fundamental solution of the heat equation mean that the heat equation appears completely naturally in image processing, and indeed it is the first PDE to enter the picture in Chapters 1 and 2. Smoothing with the heat equation is illustrated in Figure I.6.

Figure I.6: Heat equation and smoothing. The original image is on the left; the heat equation has been applied at some scale, and the resulting blurred image is on the right.

There is another path hinted at in section I.1 that leads to the Gaussian and thus to the heat equation. Suppose that k is any positive kernel such that k(x) = k(|x|) and such that k is localized in the sense that k(x) → 0 sufficiently rapidly as |x| → ∞. If k is normalized properly and if we write k_{h}(x) = (1/h)k(x/h^{1/2}), then

\frac{k_{h} ∗ u_{0}(x) − u_{0}(x)}{h} → Δu_{0}(x)

as h → 0 whenever the image u_{0} is sufficiently smooth. We write this as

k_{h} ∗ u_{0}(x) − u_{0}(x) = hΔu_{0}(x) + o(h).  (I.1)

Now let u(t, x) denote the solution of the heat equation

\frac{∂u}{∂t} = Δu, u(0, x) = u_{0}(x).

If u_{0} is sufficiently smooth, then we can write

u(t, x) − u(0, x) = tΔu_{0}(x) + o(t).  (I.2)

“JMMBookOct04” 30/10/2009 page 12

[figure] [figure]

12 Introduction

The reverse heat equation

Equations (I.1) and (I.2) suggest that blurring u_{0} with a kernel k_{h} for small h is equivalent to applying the heat equation to u_{0} at some small scale t. This is true, and it will be made precise in Chapter 2. These equations also lead to another idea: We read in the paper [147] by Lindenbaum, Fischer, and Bruckstein that Kovasznay and Joseph [136] introduced in 1955 the notion that a slightly blurred image could be deblurred by subtracting a small amount of its Laplacian. Numerically, this amounts to subtracting a fraction λ of the Laplacian of the observed image from itself:

u_{restored} = u_{observed} − λΔu_{observed}.

Dennis Gabor, who received the Nobel prize in 1971 for his invention of optical holography, studied this process and determined that the best value of λ was the one that doubled the steepest slope in the image [147]. Empirically, one can start with a small value of λ and repeat the process until a good image is obtained; with further repetitions the process blows up. Indeed, this process is just applying the reverse heat equation to the observed image, and the reverse heat equation is notoriously ill-posed. On the other hand, the Kovasznay– Joseph–Gabor method is efficient for sufficiently small λ and can be successfully applied to most images obtained from optical devices. This process is illustrated in Figure I.7. A few iterations can enhance the image (second panel), but the inverse heat equation finally blows up (third panel).

Figure I.7: Kovasznay–Joseph–Gabor deblurring. Left to right: original image; three iterations of the algorithm; ten iterations of the algorithm.

Figure I.8 shows that same experiment applied to an image of Victor that has been numerically blurred. Again, the process blows up, but it yields a significant improvement at some scales.

We have now seen the heat equation used in two senses, each with a different objective. In both cases, we have noted drawbacks. In the first instance, the heat equation (or Gaussian) was used to smooth an image, but as we have mentioned, this operator is not contrast invariant, and thus is not appropriate for any theory of image analysis that requires contrast-invariant operators. This does not mean that the Gaussian should be dismissed; it only means that it is not appropriate for our version of image analysis. To meet our objectives, we will replace the Laplacian, which is a linear isotropic operator, with nonlinear, “JMMBookOct04” 30/10/2009 page 13

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 13

Figure I.8: Kovasznay–Joseph–Gabor deblurring. This is the same deblurring experiment as in Figure I.7, but it is applied to a much more blurred image.

nonisotropic smoothing operators. This will bring us to the central theme of the book: appropriate smoothing for a possible theory of image analysis.

In the second instance, the heat equation is run backward (the inverse heat equation) with the objective of restoring a blurred image. As we have seen, this is successful to some extent, but the drawback is that it is an unstable process. The practical problem is more complex than the fact that the inverse heat equation is not well posed. In the absence of noise, the best way to deblurr a slightly blurred image is to use the inverse heat equation. However, in the presence of noise, this isotropic operator acts equally in all direction, and while it enhances the definition of edges, the edges become jagged due to the noise. This observation led Gabor to try to improve matters by using more directional operators in place of the Laplacian. Gabor was concerned with image restoration, but his ideas will appear later in our story in connection with smoothing. (For an account of Gabor’s work see [147].)

Shock filters

The objective for running the heat equation backward is image restoration, and although restoration is not the main subject of the book, we are going to pause here to describe two ways to improve the stability of the inverse heat equation. Image restoration is an extremely important area of image processing, and the techniques we describe illustrate another use of PDEs in image processing. There are indeed stable ways to “reverse” the heat equation. More precisely, there are “inverse diffusions” that deblurr an image and reach a steady state. The first example, due to Rudin in 1987 [214] and Osher and Rudin in 1990 [198] is a pseudoinverse for the heat equation, where the propagation term |Du| = |(u_{x}, u_{y})| is controlled by the sign of the Laplacian:

\frac{∂u}{∂t} = −sign(Δu)|Du|.  (I.3)

This equation is called a shock filter. We will see later that this operator propagates the level lines of an image with a constant speed and in the same direction as the reverse heat equation would propagate these lines; hence it acts as a pseudoinverse for the heat equation. This motion enhances the definition of “JMMBookOct04” 30/10/2009 page 14

[figure] [figure]

14 Introduction

the contours and thus sharpens the image. Equation (I.3) is similar to a classic nonlinear filter introduced by Kramer in the seventies [137]. Kramer’s filter can be interpreted in terms of a PDE using the same kinds of heuristic arguments that have been used to derive the heat equation. This equation is

\frac{∂u}{∂t} = −sign(D^{2}u(Du, Du))|Du|,  (I.4)

where the Laplacian has been replaced by

D^{2}u(Du, Du) = u_{xx}(u_{x})^{2} + 2u_{xy}u_{x}u_{y} + u_{yy}(u_{y})^{2}.  (I.5)

We will see in Chapter 2 that D^{2}u(Du, Du)/|Du|^{2} is the second derivative of u in the direction of its gradient Du, and we will interpret the differential operator (I.5) as Haralick’s edge detector. Kramer’s equation yields a slightly better version of a shock filter. The actions of these filters are illustrated in Figure I.9. The image on the left is a blurred image of Victor. The next image has been deblurred using the Rudin–Osher shock filter. This is a pseudoinverse of the heat equation that attains a steady state. The third image has been deblurred using Kramer’s improved shock filter, which also attains steady state. The fourth image was deblurred using the Rudin–Osher–Fatemi restoration scheme, which is described below [215].

Figure I.9: Deblurring with shock filters and a variational method. Left to right: blurred image; Rudin–Osher shock filter; Kramer’s improved shock filter; Rudin–Osher–Fatemi restoration method.

The deblurring algorithms (I.3) and (I.4) work to the extent that, experimentally, they attain steady states and do not blow up. However, a third deblurring method, the Rudin–Osher–Fatemi algorithm, is definitely better. It poses the deblurring problem as an inverse problem. It is very efficient when the observed image u_{0} is of the form k ∗ u + n, where k is known and where the statistics of the noise n are also known. Given the observed image u_{0}, one tries to find a restored version u such that k ∗ u is as close as possible to u_{0} and such that the oscillation of u is nonetheless bounded. This is done by finding u that minimizes the functional

∫ (|Du(x)| + λ(k ∗ u(x) − u_{0}(x))^{2}) dx.  (I.6)

The parameter λ controls the oscillation in the restored version u. If λ is large, the restored version will closely satisfy the equation k ∗ u = u_{0}, but it may be very oscillatory. If instead λ is small, the solution is smooth but inaccurate. This parameter can be computed in principle as a Lagrange multiplier. The “JMMBookOct04” 30/10/2009 page 15

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 15

obtained restoration can be remarkable. The best result we can obtain with the blurred Victor is shown in the fourth panel of Figure I.9. This scheme was selected by the French Space Agency (CNES) after a benchmark for satellite image deblurring, and it is currently being used by the CNES for satellite image restoration. This total variation restoration method also has fast wavelet packets versions.

From the heat equation to wavelets

The observation by Kovasznay, Joseph, and Gabor (and undoubtedly others) that the difference between a smoothed image and the original image is related to the Laplacian of the original image is also the departure of one of the paths that lead to wavelet theory. Here, very briefly, is the idea: If we convolve an image with an appropriate smoothing kernel and then take the difference, we obtain a new image related to the Laplacian of the original image (see equation (I.1)). This new “Laplacian image” turns out to be faded with respect to the original, and if one retains only the values greater than some threshold, the image is often sparse. This is illustrated in Figure I.10. The last panel on the right shows in black the values of this Laplacian image of Victor that differ significantly from zero. Here, and in most natural images, this representation is sparse and thus useful for compression. This experiment simulates the first step of a well-known algorithm due to Burt and Adelson.

In 1983, Burt and Adelson developed a compression algorithm called the Laplacian pyramid based on this idea [36]. Their algorithm consists of iterating two operations: a convolution followed by subsampling. After each convolution, one keeps only the difference k_{n} ∗ u_{n} − u_{n}, where n is used here to indicate that each step takes place at a different scale due to the subsampling. The image is then coded by the (finite) sequence of these differences. These differences resemble the Laplacian of u_{n}, hence the name “Laplacian pyramid.” An important aspect of this algorithm is that the discrete kernels k_{n}, which are low-pass filters, are all the same kernel k; the index n merely indicates that k is adjusted for the scale of the space where the subsampled image u_{n} lives. Ironically, the smoothing function cannot be the Gaussian, since the requirements for reconstructing the image from its coded version rule out the Gaussian. Burt and Adelson’s algorithm turned out to be one of the key steps that led to multiresolution analyses and wavelets. Burt and Adelson were interested in compression, and, indeed, the differences k_{n} ∗ u_{n} − u_{n} tend to be sparse for natural images. On the other hand, we are interested in image analysis, and for us, the Burt and Adelson algorithm has the drawback that it is not translation invariant or isotropic because of the multiscale subsampling.

Back to edge detection

Early research in computer vision focused on edge detection as a main tool for image representation and analysis. It was assumed that the apparent contours of objects, and also the boundaries of the facets of objects, produce step discontinuities, while inside these boundaries, the image oscillates mildly. The apparent contour points, or edges points, were to be computed as points where the gradient is in some sense largest. Two ways were proposed to do this: Marr and Hildreth proposed computing the points where Δu crosses zero, the now-famous “JMMBookOct04” 30/10/2009 page 16

[figure] [figure]

16 Introduction

Figure I.10: The Laplacian pyramid of Burt and Adelson. Left to right: the original image; the image blurred by Gaussian convolution; the difference between the original image and the blurred version, which approximates the Laplacian of the original image; the points where this Laplacian image is large.

zero-crossings [165]. A significant improvement was made by Harakick who defined the boundaries, or edges, of an image as those points where |Du| attains a local maximum along the gradient lines [101]. Two years later, Canny implemented Haralick’s detector in an algorithm that consists of Gaussian smoothing followed by computing the (edge) points where D^{2}u(Du, Du) = 0 and |Du| is above some threshold [38]. We refer to this algorithm as the Haralick–Canny edge detector. The fourth panel in Figure I.11 displays what happens when we smooth the image with the Gaussian (the heat equation) and then compute the points where D^{2}u(Du, Du) = 0 and |Du| is above some threshold. If this computation is done on the raw image (first panel), then ”edges” show up everywhere (second panel) because the raw image is a highly oscillatory function and contains a very dense set of inflexion points. After applying the heat equation and letting it evolve to some scale (third panel), we see that the Haralick–Canny edge detector is able to extract some meaningful structure.

Figure I.11: Heat equation and Haralick’s edge detector. Left to right: original image; edge points found in the original image using Haralick’s detector; blurred image; edges found in the blurred image using the Haralick–Canny detector. The image “edges” are singled out after the image has been smoothed. This smoothing eliminates tiny oscillations and maintains the big ones.

The Perona-Malik equation

Given certain natural requirements such as isotropy, localization, and scale invariance, the heat equation is the only good linear smoothing operator. There are, however, many nonlinear ways to smooth an image. The first one was proposed by Perona and Malik in 1987 [203, 204]. Roughly, the idea is to smooth “JMMBookOct04” 30/10/2009 page 17

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 17

what needs to be smoothed, namely, the irrelevant homogeneous regions, and to enhance the boundaries. With this in mind, the diffusion should look like the heat equation when |Du| is small, but it should act like the inverse heat equation when |Du| is large. Here is an example of a Perona–Malik equation in divergence form:

\frac{∂u}{∂t} = div(g(|Du|)Du),  (I.7)

where g(s) = 1/(1+λ^{2}s^{2}). It is easily checked that we have a diffusion equation when λ|Du| ≤ 1 and an inverse diffusion equation when λ|Du| > 1. To see this, consider the second derivative of u in the direction of Du,

u_{ξξ} = D^{2}u(\frac{Du}{|Du|}, \frac{Du}{|Du|}),

and the second derivative of u in the orthogonal direction,

u_{ηη} = D^{2}u(\frac{Du^{⊥}}{|Du|} , \frac{Du^{⊥}}{|Du|} ),

where Du = (u_{x}, u_{y}) and Du^{⊥} = (−u_{y}, u_{x}). The Laplacian can be rewritten in the intrinsic coordinates (ξ, η) as Δu = u_{ξξ} + u_{ηη}. The Perona–Malik equation then becomes

\frac{∂u}{∂t} = \frac{1}{1 + λ^{2}|Du|^{2}} u_{ηη} + \frac{1 − λ^{2}|Du|^{2}}{(1 + λ^{2}|Du|^{2})^{2}} u_{ξξ}.

The first term in this representation always appears as a one-dimensional diffusion in the direction orthogonal to the gradient, tuned by the size of the gradient. The nature of the second term depends on the value of the gradient; it can be either diffusion in the direction Du or inverse diffusion in the same direction. This model indeed mixes the heat equation and the reverse heat equation. Figure I.12 is used to compare the Perona–Malik equation with the classical heat equation (illustrated in Figure I.11) in terms of accuracy of the boundaries obtained by the Haralick–Canny edge detector (see Chapter 7). At a comparable scale of smoothing, we clearly gain some accuracy in the boundaries and remove more “spurious” boundaries using this Perona–Malik equation. The representation is both more sparse and more accurate.

Figure I.12: A Perona–Malik equation and edge detection. This is the same experiment as in Figure I.11, but here the Perona–Malik equation is used in place of the heat equation. Notice that the edge map looks slightly better in this case.

“JMMBookOct04” 30/10/2009 page 18

[figure] [figure]

18 Introduction

The ambitious Perona–Malik model attempts to build into a single operator the ability to perform two very different tasks, namely, restoration and analysis. This has its cost: The model contains a “contrast threshold” λ^{−1} that must be set manually, and although experimental results have been impressive, the mathematical existence and uniqueness of solutions are not guaranteed, despite some partial results by Kichenassamy [127] and Weickert and Benhamouda [257]. There are three parameters involved in the overall smoothing and edge-detecting scheme: the gradient threshold λ^{−1} in the equation (7.2), the smoothing scale(s) t (or the time that equation (7.2) evolves), and the gradient threshold in the Haralick–Canny detector. We can use the same gradient threshold in both the Haralick–Canny detector and the Perona–Malik equation, but this still leaves us with a two-parameter algorithm. Can these parameters be dealt with automatically for an image analysis scheme? This question seems to have no general answer at present. An interesting attempt based on statistical arguments had been made, however, by Black et al. [29].

A proliferation of PDE’s

If one believes that some nonlinear diffusion might be a good image analysis model, why not try them all? This is exactly what has happened during the last ten years. We can claim with some certainty that almost all possible nonlinear parabolic equations have been proposed. A few of the proposed models are even systems of PDEs. The common theme in this proliferation of models is this: Each attempt fixes one intrinsic diffusion direction and tunes the diffusion using the size of the gradient or the value of an estimate of the gradient. To keep the size of this introduction reasonable, we will focus on a few of the simplest models.

We begin with the Rudin–Osher–Fatemi model [215]. In this model the BV norm of u, ∫ |Du(x)| dx, is one of the terms in the expression (I.6) that is minimized to obtain a restored image. It is this term that provides the smoothing. The gradient descent for ∫ |Du(x)| dx translates into the equation

\frac{∂u}{∂t} = div(\frac{Du}{|Du|}) = \frac{1}{|Du|}u_{ηη}.

Written this way, the method appears as a diffusion in the direction orthogonal to the gradient, tuned by the size of the gradient. Andreu et al. proved that this equation is well posed in the space BV of functions of bounded variation [12, 13]. A variant of this model was proposed independently by Alvarez, Lions, and Morel [8]. In this case, the relevant equation is

\frac{∂u}{∂t} = \frac{1}{|k ∗ Du|} |Du| div(\frac{Du}{|Du|}) = \frac{1}{|k ∗ Du|}u_{ηη},

and again the diffusion is in the direction Du^{⊥} orthogonal to the gradient. Note that the rate of diffusion depends on the average value k ∗ Du of the gradient in a neighborhood of x, whereas the direction of diffusion, Du^{⊥}(x)/|Du(x)|, depends on the value of Du(x) at x. The kernel k is usually the Gaussian. Kimia, Tannenbaum, and Zucker, working in a more general shape-analysis framework, proposed the simplest equation of our list [131]:

\frac{∂u}{∂t} = |Du| div(\frac{Du}{|Du|}) = D^{2}u(\frac{Du^{⊥}}{|Du|} , \frac{Du^{⊥}}{|Du|} ) = u_{ηη}.  (I.8)

“JMMBookOct04” 30/10/2009 page 19

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 19

This equation had been proposed earlier in another context by Sethian as a tool for front-propagation algorithms [233]. This equation is a “pure” diffusion in the direction orthogonal to the gradient. We call this equation the curvature equation; this is to distinguish it from other equations that depend on the curvature of u in some other way. These latter will be called curvature equations. When we refer to the action of the equations, we often write curvature motions or curvature-dependent motions. (See Chapters 18 and 19.)

The Weickert equation can be viewed as a variant of the curvature equation [256]. It uses a nonlocal estimate of the direction orthogonal to the gradient for the diffusion direction. This direction is computed as the direction v of the eigenvector corresponding to the smallest eigenvalue of k ∗ (Du ⊗ Du), where (y ⊗ y)(x) = (x · y)y. Note that if the convolution kernel is removed, then this eigenvector is simply Du^{⊥}. So the equation writes

\frac{∂u}{∂t} = u_{ηη},  (I.9)

where η denotes the coordinate in the direction v. The three models just described can be interpreted as diffusions in a direction orthogonal to the gradient (or an estimate of this direction), tuned by the size of the gradient. They are illustrated in Figure I.13. (The original image is in the first panel of Figure I.14.)

Carmona and Zhong proposed a diffusion in the direction of the eigenvector w corresponding to the smallest eigenvalue of D^{2}u [43]. So the equation is again 3.18, but this time η denotes the coordinate in the direction of w. This is illustrated in panel three of Figure I.14. Sochen, Kimmel, and Malladi propose instead a nondegenerate diffusion associated with a minimal surface variational formulation [236]. Their idea was to make a gradient descent for the area, ∫ \sqrt{1 + |Du(x)|^{2}} dx, of the graph of u. This leads to the diffusion equation

\frac{∂u}{∂t} = div(\frac{Du}{\sqrt{1 + |Du|^{2}}} ).

At points where Du is large this equation behaves like \frac{∂u}{∂t} = div(\frac{Du}{|Du|}),

where we retrieve the Rudin-Osher-Fatemi model of Section I.3. At points where Du is small we have \frac{∂u}{∂t} = div(Du) which is the heat equation. This equation is illustrated in panel four of Figure I.14. Other diffusions have also been considered. For purposes of interpolation, Caselles, Morel, and Sbert proposed a diffusion that may be interpreted as the strongest possible image smoothing [49],

\frac{∂u}{∂t} = D^{2}u(Du, Du) = |Du|^{2}u_{ξξ}.

This equation is not used for preprocessing the image as the others are; rather, it is a way to interpolate between the level lines of an image with sparse level lines (Figure I.15). Among the models mentioned, only the curvature motion proposed by Kimia, Tannenbaum, and Zucker was specifically introduced as a shape analysis tool. We are going to explain this, but to do so we must say more about image analysis.

“JMMBookOct04” 30/10/2009 page 20

[figure] [figure]

20 Introduction

Figure I.13: Diffusion models I. Left to right: Osher, Sethian 1988: the curvature equation; Rudin, Osher, Fatemi 1992: minimization of the image’s total variation; Alvarez, Lions, Morel 1992: nonlocal variant of the preceding; Weickert 1994: nonlocal variant of the curvature equation. All of these models diffuse only in the direction orthogonal to the gradient, using a more or less local estimate of this direction. This explains why the results of the filters are so similar. However, the Weickert model captures better the texture direction.

Principles of image analysis

There are probably as many ways to approach image analysis as there are uses of digital images, and today the range of applications covers much of human activity. Most scientific and technical activities, including particularly medicine, and even sound analysis (visual sonograms), involve the perceptual analysis of images. Our goal is to look for fundamental principles that underlie most of these applications and to develop algorithms that are widely applicable. From a less lofty point of view, we wish to examine the collection of existing and potential image operators to determine which among them fit our vision of “JMMBookOct04” 30/10/2009 page 21

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 21

Figure I.14: Diffusion models II. Left to right: original image; Perona–Malik equation 1987, creating blurry parts separated by sharp edges; Carmona, Zhong 1998 which actually blurs the whole image: diffusion along the least eigenvector of D^{2}u; Sochen, Kimmel, Malladi 1998: minimization of the image graph area. This last equation has effects similar to the Perona-Malik model.

Figure I.15: Diffusion models III. Left to right: original image; quantized image (only 10 levels are kept - 3.32 bits/pixel); the quantized image reinterpolated using the Caselles–Sbert algorithm 1998. They apply a diffusion on the quantized image with values on the remaining level lines taken as boundary conditions.

image analysis. Instead of examining an endless list of partial and specific requirements, we rely on a mathematical shortcut, well known in mechanics, that consists of stating a short list of invariance requirements. These invariance requirements will lead to a classification of models and point out the ones that are the most suitable as image analysis tools. The first invariance requirement is the Wertheimer principle according to which visual perception (and therefore, we add, image analysis) should be independent of the image contrast [260]. We formalize this as follows:

Contrast-invariant classes. Two images u and v are said to be (perceptually) equivalent if there is a continuous increasing function g such that v = g(u). In this case, u and v are said to belong to the same contrast-invariant class. (“Increasing” always means “strictly increasing.”)

Contrast invariance requirement. An image analysis operator T must act directly on the equivalence class. As a consequence, we ask that T (g(u)) = g(T u), which means that the image analysis operator commutes with contrast changes.

The contrast invariance requirement rules out the heat equation and all of the models described above except the curvature motion (I.8). Contrast invariance “JMMBookOct04” 30/10/2009 page 22

[figure] [figure]

22 Introduction

led Matheron in 1975 to formulate image analysis as set analysis, namely, the analysis of the level sets of an image. The upper level set of an image u at level λ is the set

X_{λ}u = {x | u(x) ≥ λ}.

We define in exactly the same way the lower level sets by changing “≥” into “≤.” The main point to retain here is the global invariance of level sets under contrast changes. if g is a continuous increasing contrast change, then

X_{g(λ)}g(u) = X_{λ}u.

According to mathematical morphology, the image analysis doctrine founded by Matheron and Serra, the essential image shape information is contained in its level sets. It can be proved (Chapter 12) that an image can be reconstructed, up to a contrast change, from its set of level sets [169]. Figure I.16 shows an image and one of its level sets.

Figure I.16: An image and one of its level sets. On the right is level set 140 of the left image. This experiment illustrates Matheron’s thesis that the main shape information is contained in the level sets of an image. Level sets are contrast invariant.

The contrast invariance requirement leads to powerful and simple denoising operators like the so-called extrema killer, or area opening, (Chapter 14) defined by Vincent in 1993 [253]. This image operator simply removes all connected components of upper and lower level sets with areas smaller than some fixed value. This operator is not a PDE; actually it’s much simpler. Its effect is amazingly good for impulse noise, which includes the local destruction of the image and spots. The action of the extrema killer is illustrated in Figure I.17. The original image is in the first panel. In the third panel, the image has been degraded by adding “salt and pepper” noise to 75% of the pixels. The next panel shows its restoration using the extrema killer set to remove upper and lower level sets with areas smaller than 80 pixels. The second panel shows the result of the same operator applied to the original.

Level lines as a complete contrast invariant representation

In 1996, Caselles, Coll, and Morel further localized the contrast invariance requirement in image analysis. They proposed as the main objects of analysis the level lines of an image, that is, the boundaries of its level sets [46]. For this program—and the previous one involving level sets—to make sense, the levels sets and level lines must have certain topological and analytic properties. Level sets and isolevel sets {x | u(x) = λ}, which we would like to be the “level lines,”

“JMMBookOct04” 30/10/2009 page 23

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 23

Figure I.17: The extrema killer filter. Left to right: original image; extrema killer applied with area threshold equal 80 pixels; 75% salt and pepper noise added to the original image; the same filter applied.

can be defined for any image (or function) u, but they will not necessarily be useful for image analysis. In particular, we cannot directly define useful level sets and level lines for a digital image u_{d}. What is needed is a representation of u_{d} for which these concepts make sense. But this is not a problem. By the assumptions of section I.1, a digital representation u_{d} of a natural image S has been obtained by suitably sampling a smooth version of S, call it u_{f}, and a smooth approximation of u_{f} is available to us by interpolation. There are, of course, different interpolation methods to produce smooth representations of u_{d}. One can also obtain a useful discontinuous representation by considering the extension of u_{d} that is constant on each pixel. For an interpolation method to be useful, the level lines should have certain minimal properties: They should be composed of a finite number of rectifiable Jordan curves, and they should be nested. This means that they do not cross, and thus that they form a tree by inclusion (Section 18.2.)

A study by Kronrod in 1950 shows that if the function u is continuous, then the isolevels sets {x | u(x) = λ} are nested and thus form a tree when ordered by inclusion [139]. These isolevel sets are not necessarily curves; they are curves, however, if u has continuous first derivatives. Monasse proved Kronrod’s result for lower semicontinuous and upper semicontinuous functions in 2000 [180] (see also [21]). His result implies that the extension of u_{d} that is constant on each pixel yields a nested set of Jordan curves bounding the pixels. Thus we have at least two ways to associate a set of nested Jordan curves with a digital image u_{d}, depending on how u_{d} is interpolated. Given an interpolation method, we call this set of nested curves a topographic map of the image.^{1} By introducing the topographic map, the search for image smoothing, which had already been reduced to set smoothing, is further reduced to curve smoothing. Of course, we require that this smoothing preserves curve inclusion. Level lines of an image at a fixed level are shown in Figure I.18.

1The use of level lines is also consistent with the “BV assumption” mentioned in section I.1, according to which the correct function space for modeling images is the space BV of functions of bounded variation. In this case, the coarea formula can be used to associate a set of Jordan curves with an image (see [11]) It is, however, in general false for BV functions that the boundaries of lower and upper level sets form a nested set of curves; these curves may cross (see again [180].)

“JMMBookOct04” 30/10/2009 page 24

[figure] [figure]

24 Introduction

Figure I.18: Level lines of an image. Level lines, defined as the boundaries of level sets, can be defined to be a nested set of Jordan curves. They provide a contrast-invariant representation of the image. On the right are the level lines at level 183 of the left image.

Contrast invariant PDE’s

Chen, Giga, and Goto [53, 54] and Alvarez et al. [7] proved that if one adds contrast invariance to the usual isotropic invariance requirement for image processing, then all multiscale image analyses should have a curvature-dependent motion of the form

\frac{∂u}{∂t} = F (curv(u), t)|Du|,  (I.10)

where F is increasing with respect to its first argument (see chapters ?? and ??). This equation can be interpreted as follows: Consider a point x on a given level curve C of u at time t. Let n(x) denote the unit vector normal to C at x and let curv(x) denote its curvature. Then the preceding equation is associated with the curve motion equation

\frac{∂x}{∂t} = F (|κ|(x), t)n(x)

that describes how the point x moves in the direction of the normal. The formula defining curv(u) at a point x is (Chapter 18)

curv(u)(x) = \frac{1}{|Du|^{3}} D^{2}u(Du^{⊥}, Du^{⊥})(x) = \frac{u_{xx}u_{y}^{2} − 2u_{xy}u_{x}u_{y} + u_{yy}u_{x}^{2}}{(u_{x}^{2} + u_{y}^{2})^{3/2}} (x).

The curvature vector at a point of a C^{2} curve is the second derivative for a curve x(s) parameterized by length : κ = d^{2}x/ds^{2}. We refer to Chapter 18 for the detailed definitions and the links between the curvature vector of a level line of u and curv(u). Not much more can be said at this level of generality about F. Two specific cases play prominent roles in this subject. The first case is F (curv(u), t) = curv(u), the curvature equation (I.8). The second case is F (curv(u), t) = (curv(u))^{1/3}.

“JMMBookOct04” 30/10/2009 page 25

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 25

This particular one-third power form for the curvature dependence provides an important additional invariance, namely, affine invariance. We would like to have complete projective invariance, but a theorem proved by Alvarez et al. shows that this is impossible [7] (Chapter ??). The best we can have is invariance with respect to the so-called Chinese perspective, which preserves parallelism. Most of these equations, particularly when F is a power of the curvature, have a viscosity solution in the sense of Crandall and Lions [62]. This was shown in 1995 by Ishii and Souganidis [118]. We refer to Chapters ?? and ?? for all details.

As we have mentioned, contrast-invariant processing can be reduced to level set processing and, finally, to level curve processing. The equations mentioned above are indeed equivalent to curve evolution models if existence and regularity have been established. These results exist for the most important cases, namely, for F (curv(u), t) = curv(u), called curve shortening, and for F (curv(u), t) = (curv(u))^{1/3}, known as affine shortening. Grayson proved existence, uniqueness, and analyticity for the curve shortening equation [98],

\frac{∂x}{∂t} = curv(x)n(x),  (I.11)

NE PAS LAISSER COMMME C’EST : curv n’est pas la meme notation qu’apres et n’est pas meme defini!

and Angenent, Sapiro, and Tannenbaum proved the same results for the affine shortening equation [14],

\frac{∂x}{∂t} = (curv(x))^{\frac{1}{3}} n(x).  (I.12)

These results are very important for image analysis because they ensure that the shortening processes do indeed reduce a curve to a more and more sketchy version of itself.

Affine invariance

An experimental verification of affine invariance for affine shortening is illustrated in Figure I.19. The numerical tests were made using a very fast numerical scheme for the affine shortening designed by Lionel Moisan [178]. The principle of this algorithm is explained in Chapter ??. Unlike many numerical schemes, this one is itself affine invariant. Each of the three panels in Figure I.19 contains three shapes. The first panel shows the action of an affine transformation A: Call the first shape in the first panel X; then the second shape is A(X) and the third shape is A^{−1}A(X) = X. The second panel shows that affine shortening, S, commutes with A: The shapes are, from left to right, S(X), SA(X), and A^{−1}SA(X). Since this third shape is the same as the first, we see that A^{−1}SA(X) = S(X), or that SA(X) = AS(X). The third panel shows the same experiment with affine shortening replaced with curve shortening. Since the first and third shapes are different, this illustrates that A does not commute with curve shortening, and hence that curve shortening is not affine invariant.

Evans and Spruck [74] (also [75, 76, 77]) and Chen, Giga, and Goto [53, 54] proved in 1991 that a continuous function moves by the curvature motion “JMMBookOct04” 30/10/2009 page 26

[figure] [figure]

26 Introduction

Figure I.19: Experimental verification of the affine invariance of the affine shortening (AMSS). The first panel contains three shapes, X, A(X), and A^{−1}A(X). The second panel contains S(X), SA(X), and A^{−1}SA(X). The congruence of the first and third shapes implies that S and A commute. In the third panel, the same procedure has been applied using equation (I.11). Here the first and third shapes are not congruent, which shows that the curve shortening is not affine invariant, as expected.

(equation (I.10) with F (curv(u), t) = curv(u)) if and only if almost all of its level curves move by curve shortening (equation (I.11)). The same result is true for the affine invariant curve evolution (equation (I.10) with F (curv(u), t) = (curv(u))^{1/3}) and affine shortening (equation (I.12)).

In the case of the curvature motion, this result provides a mathematical justification for the now-classic Osher–Sethian numerical method for moving fronts [199]: They associate with some curve or surface C its signed distance function u(x) = ±d(x, C), and the curve or surface is handled indirectly as the zero isolevel set of u. Then u is evolved by, say, the curvature motion with a classic numerical difference scheme. Thus, the evolution of the curve C is dealt with efficiently and accurately as a by-product of the evolution of u. The point of view that we adopt is slightly different from that of Osher and Sethian. We view the image as a generalized distance function to each of its level sets, since we are interested in all of them.

We show in Figure I.20 how the level lines are simplified by evolving the image numerically using affine invariant curvature motion. For clarity, we display only sixteen levels of level curves. Notice that the aim here is not subsampling; we keep the same resolution. Nor is the aim restoration; the processed image is clearly worse than the original. The aim is invariant simplification leading to shape recognition.

Figures I.21 and I.22 illustrate the effect of affine curvature motion on the values of the curvature of an image. In Figure I.21 the sea bird image has been smoothed by affine curvature motion at calibrated scale 1. In Figure I.22 the smoothing is stronger at calibrated scale 4. (A calibrated scale t means that at this scale a disk with radius t disappears.) The absolute values of the curvature of the smoothed images are shown in the upper-right panels of both figures, with the convention that the darkest points have the largest curvature. For clarity, the curvature is shown only at points where the gradient of the image was larger than 6 in a scale ranging from 0 to 255. Note how the density of points having large curvature is reduced in the second figure where the smoothing is stronger. On the other hand, the regions with large curvature are more concentrated with stronger smoothing. Each degree of smoothing produces a different curvature “JMMBookOct04” 30/10/2009 page 27

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 27

Figure I.20: The affine and morphological scale space (AMSS model). Left to right: original image; level lines of this image (16 levels only); original image smoothed using the AMSS equation; level lines of the third image.

map of the original image, and thus curvature motions can be used as a nonlinear means to compute a ”multiscale” curvature of the original image. The bottom two panels of the figures show, from left to right, the positive curvature and the negative curvature.

The snake method

Before proceeding to shape recognition, we mention that a variant of the curvature equation can be used for shape detection. This is a well-known method of contour detection, initially proposed by Kass, Witkin, and Terzopoulos [125]. Their method was very unstable. A better method is a variant of curvature motion proposed by Caselles, Catté, Coll, and Dibos [44] and improved simultaneously by Caselles, Kimmel, and Sapiro [47] and Malladi, Sethian, and Vemuri [156]. Here is how it works. The user draws roughly the desired contour in “JMMBookOct04” 30/10/2009 page 28

[figure] [figure]

28 Introduction

Figure I.21: Curvature scale space I. Top, left to right: original sea bird image smoothed by affine curvature motion at calibrated scale 1; the absolute value of the curvature. Bottom, left to right: the positive part of the curvature; the negative part. Compare with Figure I.22, where the calibrated smoothing scale is 4.

Figure I.22: Curvature scale space II. Top, left to right: original sea bird image smoothed by affine curvature motion at calibrated scale 4; the absolute value of the curvature. Bottom, left to right: the positive part of the curvature; the negative part. Compare with Figure I.21, where the calibrated smoothing scale is 1.

the image, and the algorithm then finds the best possible contour in terms of some variational criterion. This method is very useful in medical imaging. The motion of the contour is a tuned curvature motion that tends to minimize an “JMMBookOct04” 30/10/2009 page 29

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 29

energy function E. Given an original image u_{0} containing some closed contour that we wish to approximate, we start with an edge map

g(x) = \frac{1}{1 + |Du_{0}(x)|^{2}} ,

that is, a function that vanishes on the edges of the image. The user then designates the contour of interest by drawing a polygon γ_{0} roughly following the desired contour. The geodesic snake algorithm then builds a distance function v_{0} to this initial contour, so that γ_{0} is the zero level set of v_{0}. The energy to be minimized is

E(γ) = ∫_{γ} g(x(s)) ds,

where g is the edge map associated with the original image u_{0} and s denotes the parameter measuring the length along γ. The motion of the “analyzing image” v is governed by

\frac{∂v}{∂t} (x, t) = g(x)|Dv(x)|curv(v)(x) − Dv(x) · Dg(x).

This algorithm is illustrated with a medical example in Figure I.23.

Figure I.23: Active contour, or “snake.” Left to right: original image; initial contour; evolved distance function; final contour.

Shape retrieval

It seems to us that the most obvious application of invariant PDEs is shape retrieval in large databases. There are thousands of different definitions of shapes and a multitude of shape recognition algorithms. The real bottleneck has always been the ability to extract the relevant shapes. The discussion above points to a brute force strategy: All contrast-invariant local elements, or the level lines of the image, are candidates to be “shape elements.” Of course, this notion of shape element suggests the contours of some object, but there is no way to give a simple geometric definition of objects. We must give up the hope of jumping from the geometry to the common sense world. We may instead simply ask the question, Given two images, can we retrieve all the level lines that are similar in both images? This would give a factual, a posteriori, definition of shapes. They would be defined as pieces of level lines common to two different images, irrespective of their relationships to real physical objects.

Of course, this brute force strategy would be impossible without the initial invariant filtering (AMSS). It is doable only if the level lines have been significantly simplified. This simplification entails the possibility of compressed “JMMBookOct04” 30/10/2009 page 30

[figure] [figure]

30 Introduction

invariant encoding. In Figure I.24, we present an experiment due to Lisani et al. [149]. Two images of a desk and the backs of chairs, viewed from different angles, are shown in the first two panels. All of the pieces of level lines in the two images that found a match in the other image are shown in the last two panels. Notice that several of these matches are doubled. Indeed, there are two similar chairs in each image. This brings to mind a Gestalt law that states that human perception tends to group similar shapes. We now see the numerical necessity of this perceptual grouping: A preliminary self-matching of each image, with grouping of similar shapes, must be performed before we can compare it with other images.

This concludes our overview of the use of PDEs in image analysis. The rest of the book is devoted to filling in the mathematical details that support most of the results mentioned in this introduction. We have tried to prove all of the mathematical statements, assuming only two or three years of mathematical training at the university level. Thus, for most of the PDEs addressed, and for all of the relevant ones, we prove the existence and uniqueness of solutions. We also develop invariant, monotone approximation schemes. This has been technically possible by combining tools from the recent, and remarkably simple, theory of viscosity solutions with the Matheron formalism for monotone set and function operators. Thus, the really necessary mathematical knowledge amounts to elementary differential calculus, linear algebra, and some results from the theory of Lebesgue integration, which are used in the chapters on the heat equation. Mathematical statements are not introduced as art for art’s sake; all of the results are directed at proving the correctness of a model, of its properties, or of the associated numerical schemes. Numerical experiments, with detailed comments, are described throughout the text. They provide an independent development that is parallel to the central theoretical development. Most image processing algorithms mentioned in the text are accessible in the public software MegaWave. MegaWave was developed jointly by several university research groups in France, Spain and America, and it is available at

http://www.cmla.ens-cachan.fr.

“JMMBookOct04” 30/10/2009 page 31

[figure] [figure]

I.3. PDES AND IMAGE PROCESSING 31

Figure I.24: A shape parser based on level lines. The two left images are of a desk and the backs of chairs viewed from different angles. In the far left panel, one level line has been selected (in white). In the second panel we show, also in white, all matching pieces of level lines. The match is ambiguous, as must be expected when the same object is repeated in the scene. In the two panels on the right, we display all the matching pairs of pieces of level lines (in white). The non matching parts of the same level lines are shown in black. Usually, recognized shape elements are pieces of level lines, seldom whole level lines. See [] “JMMBookOct04” 30/10/2009 page 32

[figure] [figure]

32 Introduction “JMMBookOct04” 30/10/2009 page 33

[figure] [figure]

Chapter 1

Notation and background material

ℝ^{N} denotes the real N-dimensional Euclidian space. If x ∈ ℝ^{N} and N > 2, we write x = (x_{1}, x_{2}, …, x_{N}); if N = 2, we usually write x = (x, y). For x, y ∈ ℝ^{N}, we denote their scalar product by x · y = x_{1}y_{1} + x_{2}y_{2} + ⋯ + x_{N}y_{N} and write

|x| = (x · x)^{1/2} = (x_{1}^{2} + x_{2}^{2} + ⋯ + x_{N}^{2})^{1/2}.

Let Ω be an open set in ℝ^{N}, and let n ∈ ℕ be a fixed integer. C^{n}(Ω) denotes the set of real-valued functions f : Ω → ℝ that have bounded continuous derivatives of all orders up to and including n. f ∈ C^{∞}(Ω) means that f has continuous derivatives of all orders and that they are all bounded; f ∈ C(Ω) = C^{0}(Ω) means that f is continuous and bounded on Ω. We will often write “f is C^{n}” as shorthand for f ∈ C^{n}(Ω), and we often omit the domain Ω if there is no chance of confusion.

We use multi-indices of the form α = (α_{1}, α_{2}, …, α_{N}) ∈ ℕ^{N} as shorthand in several cases. For x ∈ ℝ^{N}, we write x^{α} and |x|^{α} for x_{1}^{α_{1}}x_{2}^{α_{2}} ⋯ x_{N}^{α_{N}} and |x_{1}|^{α_{1}}|x_{2}|^{α_{2}} ⋯ |x_{N}|^{α_{N}}, respectively. For f ∈ C^{n}(Ω), we abbreviate the partial derivatives of f by writing

∂^{α}f = \frac{∂^{|α|}f}{∂x_{1}^{α_{1}}∂x_{2}^{α_{2}} ⋯ ∂x_{N}^{α_{N}}} ,

where |α| = α_{1} + α_{2} + ⋯ + α_{N} and |α| ≤ n.

We also write the partial derivatives of f(x) = f(x_{1}, x_{2}, …, x_{N}) as f_{i} = ∂f/∂x_{i}, f_{ij} = ∂^{2}f/∂x_{i}∂x_{j}, and so on. In the two-dimensional case f(x) = f(x, y), we usually write ∂f/∂x = f_{x}, ∂f/∂y = f_{y}, ∂^{2}f/∂x∂y = f_{xy}, and so on.

The gradient of f is denoted by Df. Thus, if f(x) = f(x_{1}, x_{2}, …, x_{N}),

Df = (f_{1}, f_{2}, …, f_{N}),

and

Df = (f_{x}, f_{y})

in case N = 2. The Laplacian of f is denoted by Δf. Thus Δf = f_{11} + f_{22} + ⋯ + f_{NN} in general, and Δf = f_{xx} + f_{yy} if N = 2.

33 “JMMBookOct04” 30/10/2009 page 34

[figure] [figure]

34 CHAPTER 1. NOTATION AND BACKGROUND MATERIAL

We will often use the symbols O, o, and ε. They are defined as follows. We assume that h is a real variable that tends to a limit h_{0} that can be finite or infinite. We assume that g is a positive function of h and that f is any other function of h. Then f = O(g) means that there is a constant C > 0 such that |f(h)| < Cg(h) for all values of h. The expression f = o(g) means that f(h)/g(h) → 0 as h → h_{0}. We occasionally will use ε to denote a function of h that tends to zero as h → 0. Thus, f(h) = o(h) can be written equivalently as f(h) = hε(h).

Taylor’s formula

An N-dimensional form of Taylor’s formula is used several times in the book. We will first state it and then explain the notation. Assume that f ∈ C^{n}(Ω) for some open set Ω ∈ ℝ^{N}, that x, y ∈ Ω, and that the segment joining x and x + y is also in Ω. Then

f(x+y) = f(x)+ \frac{1}{1!}Df(x)y^{(1)} + \frac{1}{2!}D^{2}f(x)y^{(2)} +⋯+ \frac{1}{n!}D^{n}f(x)y^{(n)} +o(|y|^{n}).

This has been written compactly to resemble the one-dimensional case, but the price to be paid is to explain the meaning of D^{p}f(x)y^{(p)}. We have already seen special cases of this expression in section I.3, for example, D^{2}u(Du, Du) in equation (I.4). The expression D^{p}f(x)y^{(p)} is

D^{p}f(x)y^{(p)} = D^{p}f(x)(\underbrace{y, y, …, y}_{p terms}) = ∑_{(i_{1},i_{2},...,i_{p})} \frac{∂^{p}f}{∂x_{i_{1}}∂x_{i_{2}} ⋯ ∂x_{i_{p}}} (x)y_{i_{1}}y_{i_{2}} ⋯ y_{i_{p}},

where the sum is taken over all N^{p} different vectors (i_{1}, i_{2}, …, i_{p}), i_{j} = 1, 2, …, N. Notice that Df(x)y^{(1)} is just ∑_{j=1}^{N} f_{j}y_{j} = Df(x) · y, which is how we usually write it.

The so-called Lagrange variant of Taylor’s formula will be equally useful,

f(x + y) = f(x) + ⋯ + \frac{1}{(n − 1)!}D^{n−1}f(x)y^{(n−1)} + \frac{1}{n!}D^{n}f(x + θy)y^{(n)}, (I.1)

where θ = θ(y) ranges in [0, 1].

The implicit function theorem

Consider a real-valued C^{1} function f defined on an open set Ω in ℝ^{N}. For ease of notation we write z = (x, y), where x = (x_{1}, …, x_{N−1}) and y = x_{N}. Assume that f(z_{0}) = 0 for a point z_{0} ∈ Ω and that f_{y}(x_{0}) ≠ 0. Then there is a neighborhood M = M(x_{0}) and a neighborhood N = N(y_{0}) such that for every x ∈ M there is exactly one y ∈ N such that f(x, y) = 0. The function y = φ(x) is C^{1} on M and y_{0} = φ(x_{0}). Furthermore, if f ∈ C^{n}(Ω), then φ ∈ C^{n}(M).

Lebesgue integration

The Lebesgue integral, which first appeared in 1901 and is thus over a hundred years old, has become the workhorse of analysis. It plays a role in chapters 1 and 2 and appears briefly in other parts of the book. One does not need a “JMMBookOct04” 30/10/2009 page 35

[figure] [figure]

35

profound understanding of abstract measure theory and integration to follow the arguments. One should, however, be familiar with a few key results and be comfortable with the basic manipulations of the integral. With this in mind, we restate some of these fundamentals.

The functions and sets in this book are always measurable. Thus we dispense in general with phrases like “let f be a measurable function.” We denote by M the set of Lebesgue measurable subsets of ℝ^{N}. Since we shall sometimes need to complete ℝ^{N} by a point at infinity, ∞, we still denote by M the measurable sets of S_{N} = ℝ^{N} ∪ {∞} and take measure({∞}) = 0. A function f defined on a subset A of ℝ^{N} is integrable, if

∫_{A} |f(x)| dx < +∞.

The Banach space of all integrable function defined on A is denoted as usual by L^{1}(A); we write ∥f∥_{L^{1}(A)} = ∫_{A} |f(x)| dx to denote the norm of f in L^{1}(A). The most important applications in the book are the two cases A = ℝ^{N} and A = [−1, 1]^{N}. Here are two results that we use in chapters 1 and 2. We state them not in the most general form, but rather in the simplest form suitable for our work.

A density theorem for L^{1}(ℝ^{N})

If f is in L^{1}(ℝ^{N}), then there exists a sequence of continuous functions {g_{n}}, each of which has compact support, such that g_{n} → f in L^{1}(ℝ^{N}), that is, ∫ |g_{n}(x) − f(x)|dx → 0 as n → +∞. This result is true for L^{1}([−1, 1]^{N}), in which case the g_{n} are continuous on [−1, 1]^{N}.

Fubini’s theorem

Suppose that f is a measurable function defined on A × B ∈ ℝ^{N} × ℝ^{N}. Fubini’s theorem states that

∫_{A×B} |f(z)| dz = ∫_{A} ∫_{B} |f(x, y)| dx dy = ∫_{B} ∫_{A} |f(x, y)| dy dx,

where we have written z = (x, y). It further states, that if any one of the integrals is finite, then

∫_{A×B} f(z) dz = ∫_{B} ∫_{A} f(x, y) dx dy = ∫_{A} ∫_{B} f(x, y) dy dx.

Lebesgue’s dominated convergence theorem

If a sequence of functions {f_{n}} is such that f_{n}(x) → f(x) for almost every x ∈ ℝ^{N} as n → +∞, and if there is an integrable function g such that |f_{n}(x)| ≤ g(x) almost everywhere, then

∫_{ℝ^{N}} f_{n}(x) dx → ∫_{ℝ^{N}} f(x) dx.

We often use the following direct consequence: if A_{n} is a decreasing sequence of measurable sets with bounded measure then measure(A_{n}) ↦ measure(A). To “JMMBookOct04” 30/10/2009 page 36

[figure] [figure]

36 CHAPTER 1. NOTATION AND BACKGROUND MATERIAL

prove this, apply Lebesgue’s theorem to the characteristic functions of A_{n} and A, 1_{A_{n}} and 1_{A}.

We also use the following result, which is a direct consequence of the dominated convergence theorem.

Interchanging differentiation and integration

Suppose that a function f defined on (t_{0}, t_{1}) × ℝ^{N}, where (t_{0}, t_{1}) is any interval of ℝ, is such that t ↦ f(t, x) is continuously differentiable (for almost every x ∈ ℝ^{N}) on some interval [a, b] ⊂ (t_{0}, t_{1}). If there exists an integrable function g such that for all t ∈ [a, b]

|\frac{∂f}{∂t} (t, x)| ≤ g(x) almost everywhere,

then the integral I(t) = ∫_{ℝ^{N}} f(t, x) dx is differentiable for t ∈ (a, b) and

\frac{dI}{dt} (t) = ∫_{ℝ^{N}} \frac{∂f}{∂t} (t, x) dx.

A brief but comprehensive discussion of the Lebesgue integral can be found in the classic textbook by Walter Rudin [216].

I.0.1 A framework for sets and images

We start by fixing a simple and handy functional framework for images and sets, which will be maintained throughout the book. Until now, we have been vague about the domain of definition of an image. On one hand, a real digital image is defined on a finite grid. On the other hand, standard interpolation methods give a continuous representation defined on a finite domain of ℝ^{N}, usually a rectangle. Now, it is convenient to have images defined on all of ℝ^{N}, but it is not convenient to extend them by making them zero outside their original domains of definition because that would make them discontinuous. So an usual way is to extend them into a continuous function tending to a constant at infinity. One way to do that is illustrated in Figure I.1. First, an extension to a wider domain is performed by reflection across the domain’s boundary and periodization. Then, it is easy to let the function fade at infinity or to make it compactly supported. This also means that we fix a value at infinity for u, which we denote by u(∞). We denote the topological completion of ℝ^{N} by this infinity point by S_{N} = ℝ^{N} ∪\{∞\}, which can also be denoted \overline{ℝ^{N}}. Let us justify the notation.

Proposition 1.1. Consider the sphere S_{N} = {z ∈ ℝ^{N+1}, ||z|| = 1}. Then the mapping T : ℝ^{N} ∪ {∞} → S_{N} defined by

T (x) = (\frac{2x}{1 + x^{2}} , \frac{x^{2} − 1}{x^{2} + 1})

is a homeomorphism (that is, a continuous bijection with continuous inverse.)

This is easily checked (Exercise 2.6).

“JMMBookOct04” 30/10/2009 page 37

[figure] [figure]

37

Figure I.1: Image extension by symmetry, followed by periodization. Then the image can be extended continuously to the rest of the plane into a function which is constant for x large. The purpose of these successive extensions of u to all of ℝ^{N} is to facilitate the definition of certain operations on u, such as convolution with smoothing kernels, and, at the same time, to preserve the continuity of u. This method of extending a function is widely used in image processing; in particular, it is used in most compression and transmission standards. For instance, the discrete cosine transform (DCT) applied to the initial data u, restricted to [0, 1]^{N}, is easily interpreted as an application of the FFT to the symmetric extension of u.

Definition 1.2. We denote by F the set of continuous functions on S_{N}, which can be identified with the set of continuous functions on ℝ^{N} tending to some constant at infinity. The natural norm of F is

∥u∥_{F} = sup_{x∈ℝ^{N}} |u(x)|.  (I.2)

We say that an image u in F is C^{1}, if the function u is C^{1} at each point x ∈ ℝ^{N}. We define in the same way the C^{2},... C^{∞} functions of F.

Definition 1.3. We say that a function u defined on ℝ^{N} is uniformly continuous if for every x, y,

|u(x + y) − u(x)| ≤ ε(|y|),

“JMMBookOct04” 30/10/2009 page 38

[figure] [figure]

38 CHAPTER 1. NOTATION AND BACKGROUND MATERIAL

for some function ε called modulus of continuity of u, satisfying lim_{s→0} ε(s) = 0.

Continuous functions on a compact set are uniformly continuous, so functions of F are uniformly continuous. We shall often consider the level sets of functions in F, which simply are compact sets of S_{N}.

Definition 1.4. We denote by L the set of all compact sets of S_{N}.

These sets are easy to characterize:

Proposition 1.5. The elements of L are of three kinds:

• compact subsets of ℝ^{N}

• F ∪ {∞}, where F is a compact set of ℝ^{N}.

• F ∪ {∞}, where F is an unbounded closed subset of ℝ^{N}

Proof. Indeed, B ∩ ℝ^{N} is a closed set of ℝ^{N} and is therefore either a bounded compact set or an unbounded closed set of ℝ^{N}. In the latter case, B must contain ∞. □

“JMMBookOct04” 30/10/2009 page 39

[figure] [figure]

Part I

Linear Image Analysis

39 “JMMBookOct04” 30/10/2009 page 40

[figure] [figure]

“JMMBookOct04” 30/10/2009 page 41

[figure] [figure]

Chapter 2

The Heat Equation

The heat equation is the prototype of all the PDEs used in image analysis. There are strong reasons for that and it is the aim of this chapter to explain some of them. Some more will be given in Chapter ??. Our first section is dedicated to a simple example of linear smoothing illustrating the relation between linear smoothing and the Laplacian. In the next section, we prove the existence and uniqueness of its solutions, which incidentally establishes the equivalence between the convolution with a Gaussian and the heat equation.

2.1 Linear smoothing and the Laplacian

Consider a continuous and bounded function u_{0} defined on ℝ^{2}. If we wish to smooth u_{0}, then the simplest way to do so without favoring a particular direction is to replace u_{0}(x) with the average of the values of u_{0} in a disk D(x, h) of radius h centered at x. This means that we replace u_{0}(x) with

M_{h}u_{0}(x) = \frac{1}{πh^{2}} ∫_{D(x,h)} u_{0}(y) dy = \frac{1}{πh^{2}} ∫_{D(0,h)} u_{0}(x + y) dy.  (2.1)

Although the operator M_{h} is quite simple, it exhibits important characteristics of a general linear isotropic smoothing operator. For example, it is localizable: As h becomes small, M_{h} becomes more localized, that is, M_{h}u_{0}(x) depends only on the values of u_{0}(x) in a small neighborhood of x. Smoothing an image by averaging over a small symmetric area is illustrated in Figure 2.1.

Our objective is to point out the relation between the action of M_{h} and the action of the Laplacian, or the heat equation. To do so, we assume enough regularity for u_{0}, namely that it is C^{2}. We shall actually prove in Theorem 3.2 that under that condition

M_{h}u_{0}(x) = u_{0}(x) + \frac{h^{2}}{8} Δu_{0}(x) + h^{2}ε(x, h),  (2.2)

where ε(x, h) tends to 0 when h → 0. As we have seen in the introduction, (2.2) provides the theoretical basis for deblurring an image by subtracting a small amount of its Laplacian. It also suggests that M_{h} acts as one step forward in the heat equation starting with initial condition u_{0},

\frac{∂u}{∂t}(t, x) = \frac{1}{8}Δu(t, x), u(0, x) = u_{0}(x).  (2.3)

41 “JMMBookOct04” 30/10/2009 page 42

[figure] [figure]

42 CHAPTER 2. THE HEAT EQUATION

Figure 2.1: Local averaging algorithm. Left to right: original image; result of replacing the grey level at each pixel by the average of the grey levels over the neighboring pixels. The shape of the neighborhood is shown by the black spot displayed in the upper right-hand corner.

Figure 2.2: The Gaussian in two dimensions.

This statement is made more precise in Exercise 2.5. Equation (2.2) actually suggests that if we let n → +∞ and at the same time require that nh^{2} → t, then

(M_{h}^{n}u_{0})(x) → u(t, x)  (2.4)

where u(t, x) is a solution of (2.3).

This heuristics justifies the need for a thorough analysis of the heat equation. The next chapter will prove that (2.4) is true under fairly general conditions. In the next section, we shall prove that the heat equation has a unique solution for a given continuous initial condition u_{0}, and that this solution at time t is equal to the convolution G_{t} ∗ u_{0}, where G_{t} is the Gaussian (Figure 2.2). The effect on level lines of smoothing with the Gaussian is shown in Figure 2.4.

“JMMBookOct04” 30/10/2009 page 43

[figure] [figure]

2.2. EXISTENCE AND UNIQUENESS OF SOLUTIONS OF THE HEAT EQUATION43

2.2 Existence and uniqueness of solutions of the heat equation

Definition 2.1. We say that a function g defined on ℝ^{N} is rapidly decreasing, or has fast decay, if for each multi-index β there is a constant C such that

|x|^{β}|g(x)| ≤ C.

We say that g belongs to the Schwartz class S if g ∈ C^{∞}(ℝ^{N}) and if ∂^{α}g has fast decay for each multi-index α.

Proposition 2.2. If g ∈ S, then g ∈ L^{1}(ℝ^{N}), that is, ∫_{ℝ^{N}} |g(x)|dx < +∞. For each pair of multi-indices α, β, the function x^{β}∂^{α}g also belongs to S, and ∂^{α}g is uniformly continuous on ℝ^{N}.

Proof. The second statement follows from the Leibnitz rule for differentiating a product. By the definition of S, there is a constant C such that |x|^{N+2}|g(x)| ≤ C. Thus there is another C such that |g(x)| ≤ C/(1 + |x|^{N+2}); since C/(1 + |x|^{N+2}) ∈ L^{1}(ℝ^{N}), g ∈ L^{1}(ℝ^{N}). Finally, note that |∂^{α}g(x)| → 0 as |x| → ∞. But any continuous function on ℝ^{N} that tends to zero at infinity is uniformly continuous. □

Proposition 2.3 (The Gaussian and the heat equation). For all t > 0, the function x ↦ G_{t}(x) = (1/(4πt)^{N/2})e^{−|x|^{2}/4t} belongs to S and satisfies the heat equation

\frac{∂G_{t}}{∂t} − ΔG_{t} = 0.

Proof. It is sufficient to prove the first statement for the function g(x) = e^{−|x|^{2}}. An induction argument shows that ∂^{α}g(x) = P_{α}(x)e^{−|x|^{2}}, where P_{α}(x) is a polynomial of degree |α| in the variables x_{1}, x_{2}, …, x_{N}. The fact that, for every k ∈ ℕ, x^{k}e^{−x^{2}} → 0 as |x| → +∞ finishes the proof. Differentiation shows that G_{t} satisfies the heat equation. □

Exercise 2.1. Check that G_{t} is solution of the heat equation.

Linear image filtering is mainly done by convolving an image u with a positive integrable kernel g. This means that the smoothed image is given by the function g ∗ u defined as

g ∗ u(x) = ∫_{ℝ^{N}} g(x − y)u(y) dy = ∫_{ℝ^{N}} g(y)u(x − y) dy.

Exercise 2.2. Prove that the convolution, when it makes sense, is translation invariant. This means that g ∗ u(x − z) = gz ∗ u(x), where gz(x) = g(x − z). Exercise 2.3. Check that G_{t} ∗ G_{s} = G_{t+s}.

Linear filtering with the Gaussian at several scales is illustrated in Figure 2.3. The next result establishes properties of the convolution that we need for our treatment of the heat equation.

“JMMBookOct04” 30/10/2009 page 44

[figure] [figure]

44 CHAPTER 2. THE HEAT EQUATION

Figure 2.3: Convolution with Gaussian kernels (heat equation). Displayed from top-left to bottom-right are the original image and the results of convolutions with Gaussians of increasing variance. A grey level representation of the convolution kernel is put on the right of each convolved image to give an idea of the size of the involved neighborhood.

Proposition 2.4. Assume that u ∈ F and that g ∈ L^{1}(ℝ^{N}). Then the function g ∗ u belongs to F and satisfies the inequality

∥g ∗ u∥_{F} ≤ ∥g∥_{L^{1}(ℝ^{N})}∥u∥_{F}.  (2.5)

Proof.

|g ∗ u(x)| ≤ ∫_{ℝ^{N}} |g(x − y)||u(y)| dy ≤ ∥u∥_{F} ∫_{ℝ^{N}} |g(x − y)| dy = ∥u∥_{F}∥g∥_{L^{1}(ℝ^{N})}.

□

Exercise 2.4. Verify that g ∗ u indeed is continuous and tends to u(∞) at infinity : this a direct application of Lebesgue Theorem.

We are now going to focus on kernels that, like the Gaussian, belong to S.

Proposition 2.5. If u ∈ F and g ∈ S, then g ∗ u ∈ C^{∞}(ℝ^{N}) ∩ F and

∂^{α}(g ∗ u) = (∂^{α}g) ∗ u  (2.6)

for every multi-index α.

Proof. Since g ∈ S, g is in L^{1}(ℝ^{N}), and so is ∂^{α}g for any multi-index α (Proposition 2.2). Thus by Proposition 2.4, (∂^{α}g) ∗ u belongs to F. To prove (2.6), it is sufficient to prove it for α = (1, 0, …, 0). Indeed, we know that ∂^{α}g is in S if g is in S, so the general case follows from the case α = (1, 0, …, 0) by induction. Letting e_{1} = (1, 0, …, 0) and using Taylor’s formula with Lagrange’s form for the remainder, we can write “JMMBookOct04” 30/10/2009 page 45

[figure] [figure]

2.2. EXISTENCE AND UNIQUENESS OF SOLUTIONS OF THE HEAT EQUATION45

g ∗ u(x + he_{1}) − g ∗ u(x) = ∫_{ℝ^{N}} (g(x + he_{1} − y) − g(x − y))u(y) dy
= ∫_{ℝ^{N}} (g(y + he_{1}) − g(y))u(x − y) dy
= h∫_{ℝ^{N}} \frac{∂g}{∂x_{1}} (y)u(x − y) dy

+ \frac{h^{2}}{2} ∫_{ℝ^{N}} \frac{∂^{2}g}{∂x_{1}^{2}} (y + θ(y)he_{1})u(x − y) dy,

(2.7)

where 0 ≤ θ(y) ≤ 1. To complete the proof, we wish to have a bound on the last integral that is independent of x ∈ C. This last integral is of the form f ∗ u, where f is defined by f(y) = (∂^{2}g/∂x_{1}^{2})(y+θ(y)he_{1}). Since g ∈ S, ∂^{2}g/∂x_{1}^{2} ∈ S, and from this it is a simple computation to show that f is bounded and decays fast at infinity. Having done this, Proposition 2.4 applies, and we deduce that g ∗ u is differentiable in x_{1} and that ∂(g ∗ u)/∂x_{1} = (∂g/∂x_{1}) ∗ u. □

Proposition 2.6. Assume that g decreases rapidly at infinity, that g(x) ≥ 0 for all x ∈ ℝ^{N}, and that ∫_{ℝ^{N}} g(x) dx = 1 and set, for t > 0, g_{t}(x) = (1/t^{N})g(x/t). Then: If u_{0} ∈ F, g_{t} ∗ u_{0} converges to u_{0} uniformly as t → 0. In addition, we have a maximum principle :

inf_{x∈ℝ^{N}} u_{0}(x) ≤ g_{t} ∗ u_{0}(x) ≤ sup_{x∈ℝ^{N}} u_{0}(x).  (2.8)

Proof. Note first that g_{t} is normalized so that

∫_{ℝ^{N}} g_{t}(y) dy = 1.  (2.9)

A change of variable x → tx and an application of Lebesgue’s theorem shows that, for any η > 0,

∫_{|y|≥η} g_{t}(y) dy → 0 as t → 0.  (2.10)

Using (2.9), we have

g_{t} ∗ u_{0}(x) − u_{0}(x) = ∫_{ℝ^{N}} g_{t}(y)(u_{0}(x − y) − u_{0}(x)) dy.  (2.11)

As already mentioned, u_{0} ∈ F is uniformly continuous. Thus, for any ε > 0, there is an η = η(ε) > 0 such that |u_{0}(x − y) − u_{0}(x)| ≤ ε when |y| ≤ η. Using this inequality, we have

|g_{t} ∗ u_{0}(x) − u_{0}(x)| ≤ ∫_{|y|<η} g_{t}(y)|u_{0}(x − y) − u_{0}(x)| dy
+ ∫_{|y|≥η} g_{t}(y)|u_{0}(x − y) − u_{0}(x)| dy
≤ε ∫_{|y|<η} g_{t}(y) dy + 2∥u∥_{L^{∞}(C)} ∫_{|y|≥η} g_{t}(y) dy.

“JMMBookOct04” 30/10/2009 page 46

[figure] [figure]

46 CHAPTER 2. THE HEAT EQUATION

By (2.9) and (2.10), we conclude that g_{t} ∗ u tends to u_{0} uniformly in x as t → 0. Relation (2.8) is an immediate consequence of the assumption that g_{t}(x) ≥ 0 and equation (2.9). □

Lemma 2.7. Let u_{0} ∈ F and u(t, x) = (G_{t} ∗ u_{0})(x). Then for every t_{0} > 0, u(t, x) → u_{0}(∞) uniformly for t ≤ t_{0} as x → ∞.

Proof. By assumption,

∀ε > 0, ∃R, |x| ≥ R ⇒ |u_{0}(x) − u_{0}(∞)| < ε.  (2.12)

Because of the fast decay of the gaussian at infinity (or using Lebesgue’s theorem, as in the former proof), we have

∀ε > 0, ∃r(ε), r ≥ r(ε) ⇒ ∫_{|y|≥r} G_{t_{0}}(y)dy < ε.  (2.13)

By using ∫ G_{t}(y)dy = 1, we deduce that

|u(t, x)−u(∞)| ≤∫_{|y|≤r} G_{t}(y)|u_{0}(x−y)−u_{0}(∞)|dy+∫_{|y|≥r} G_{t}(y)|u_{0}(x−y)−u_{0}(∞)|dy.

(2.14)

Using (2.13), the second term in (2.14) is bound from above for r ≥ r(ε) and t ≤ t_{0} by

(2 sup |u_{0}|) ∫_{|y|≥r} G_{t_{0}}(y) ≤ (2 sup |u_{0}|)ε.

Fix therefore r ≥ r(ε). Then using ∫ G_{t} = 1, the first term in (2.14) is bound by ε by (2.12) for |x| ≥ R + r.

□

Lemma 2.8. Let u_{0} ∈ F and G_{t} the gaussian. Then

(∂G_{t}/∂t) ∗ u_{0} = ∂(G_{t} ∗ u_{0})/∂t.

Proof. Proposition 2.5 does not apply directly, since it applies to the spatial partial derivatives of G_{t} but not to the derivative with respect to t. Observe, however, that a slight modification of the proof of this proposition does the job: Replace g with G_{t} and x_{1} with t. Then the crux of the matter is to notice that, given an interval 0 < t_{0} < t_{1}, there is a rapidly decreasing function f such that |(∂^{2}G_{t}/∂t^{2})(t + θ(t)h, y)| ≤ f(y) uniformly for t ∈ [t_{0}, t_{1}], where f depends on t_{0} and t_{1} but not on h. Then Proposition 2.4 applies, and the last integral in equation (2.7) is uniformly bounded. □

All of the tools are in place to state and prove the main theorem of this chapter.

Theorem 2.9 (Existence and uniqueness of solutions of the heat equation). Assume that u_{0} ∈ F and define for t > 0 and x ∈ ℝ^{N}, u(t, x) = (G_{t} ∗ u_{0})(x), u(t, ∞) = u_{0}(∞) and u(0, x) = u_{0}(x). Then

“JMMBookOct04” 30/10/2009 page 47

[figure] [figure]

2.2. EXISTENCE AND UNIQUENESS OF SOLUTIONS OF THE HEAT EQUATION47

(i) u is C^{∞} and bounded on (0, +∞) × ℝ^{N};

(ii) x → u(t, x) belongs to F for every t ≥ 0;

(iii) for any t_{0} ≥ 0, u(t, x) tends uniformly for t ≤ t_{0} to u(∞) as x → ∞;

(iv) u(t, x) tends uniformly to u_{0}(x) as t → 0;

(v) u(t, x) satisfies the heat equation with initial value u_{0};

\frac{∂u}{∂t} = Δu and u(0, x) = u_{0}(x);  (2.15)

(vi) More specifically,

sup_{x∈ℝ^{N} , t≥0} |u(t, x)| ≤ ∥u_{0}∥_{F}.  (2.16)

Conversely, given u_{0} ∈ F, u(t, x) = (G_{t} ∗ u_{0})(x) is the only C^{2} bounded solution u of (2.15) that satisfies properties (ii)-(v).

Proof. Let us prove properties (i)-(vi). For each t > 0, G_{t} ∈ S, so by Proposition 2.5 and Lemma 2.8,

\frac{∂u}{∂t} − Δu = u_{0} ∗ (\frac{∂G_{t}}{∂t} − ΔG_{t}).  (2.17)

Proposition 2.5 also tells us that u(t, ·) ∈ C^{∞}(ℝ^{N}) ∩ F for each t > 0. The right-hand side of (2.17) is zero by Proposition 2.3, and the fact that |u(t, x) − u_{0}(x)| → 0 uniformly as t → 0 follows from Proposition 2.6. The inequality (2.16) is a direct application of Proposition 2.4. Relation (iii) comes from Lemma 2.7.

Uniqueness proof. If both v and w are solutions of the heat equation with the same initial condition u_{0} ∈ F, then u = v − w is in F and satisfies (2.15) with the initial condition u_{0}(x) = 0 for all x ∈ ℝ^{N}. Also, by the assumptions of (ii), u is bounded on [0, +∞) × ℝ^{N} and is C^{2} on (0, +∞) × ℝ^{N}. We wish to show that u(t, x) = 0 for all t > 0 and all x ∈ ℝ^{N}. Assume that this is not the case. Then there is some point (t, x) where u(t, x) ≠ 0. Assume that u(t, x) > 0, by changing u to −u if necessary.

We now consider the function u^{ε} defined by u^{ε}(t, x) = e^{−εt}u(t, x). This function tends to zero uniformly in x as t → 0 and as t → +∞. It also tends uniformly to zero for each t ≤ t_{0} when x → ∞. These conditions imply that u^{ε} attains its supremum at some point (t_{0}, x_{0}) ∈ (0, +∞) × ℝ^{N}, and this means that Δu^{ε}(t_{0}, x_{0}) = e^{−εt}Δu(t_{0}, x_{0}) ≤ 0 and (∂u^{ε}/∂t)(t_{0}, x_{0}) = 0. Here is the payoff: Using the fact that u is a solution of the heat equation, we have the following relations:

0 = \frac{∂u^{ε}}{∂t} (t_{0}, x_{0}) = −εu^{ε}(t_{0}, x_{0}) + e^{−εt} \frac{∂u}{∂t}(t_{0}, x_{0})
= −εu^{ε}(t_{0}, x_{0}) + e^{−εt}Δu(t_{0}, x_{0}) ≤ −εu^{ε}(t_{0}, x_{0}) < 0.

This contradiction completes the uniqueness proof. □

“JMMBookOct04” 30/10/2009 page 48

[figure] [figure]

48 CHAPTER 2. THE HEAT EQUATION

Figure 2.4: Level lines and the heat equation. Top, left to right: original 410×270 grey level image; level lines of original image for levels at multiples of 12. Bottom, left to right: original image smoothed by the heat equation (convolution with the Gaussian). The standard deviation of the Gaussian is 4, which means that its spatial range is comparable to a disk of radius 4. The image gets blurred by the convolution, which averages grey level values and removes all sharp edges. This can be appreciated on the right, where we have displayed all level lines for levels at multiples of 12. Note how some level lines on the boundaries of the image have split into parallel level lines that have drifted away from each other. The image has become smooth, but it is losing its structure.

2.3 Exercises

Exercise 2.5. The aim of this exercise is to prove relation (2.2) and its consequence: A local average is equivalent to one step forward of the heat equation. Theorem 3.2 yields actually a more general statement.

1) Expanding u_{0} around the point x using Taylor’s formula, write

u_{0}(x + y) = u_{0}(x) + Du_{0}(x) · y + \frac{1}{2} D^{2}u_{0}(x)(y, y) + o(|y|^{2}).  (2.18)

Expand the various terms using the coordinates (x, y) of x.

2) Apply M_{h} to both sides of this expansion and deduce relation (2.2).

3) Assume u_{0} ∈ F and consider the solution u(t, x) of the heat equation (2.3) Then, for fixed t_{0} > 0 and x, apply M_{h} to the function u^{t_{0}} : x → u(t_{0}, x) and write equation (2.2) for u^{t_{0}}. Using that u(t, x) is a solution of the heat equation and its Taylor expansion between t_{0} and t_{0} + h, deduce that

M_{h}u(t_{0}, x) = u(t_{0} + h^{2}, x) + h^{2}ε(t_{0}, x, h).  (2.19)

“JMMBookOct04” 30/10/2009 page 49

[figure] [figure]

2.4. COMMENTS AND REFERENCES 49

Exercise 2.6. Consider the sphere S_{N} = {z ∈ ℝ^{N+1}, ||z|| = 1}. Prove that the mapping T : ℝ^{N} ∪ {∞} → S_{N} defined in Proposition 1.1 by

T (x) = ( \frac{2x}{1 + x^{2}} , \frac{x^{2} − 1}{x^{2} + 1} ) , T (∞) = (0, 1).

is a homeomorphism. Exercise 2.7. Let u_{0} be a continuous function defined on ℝ^{N} having the property that there exist a constant C > 0 and an integer k such that

|u_{0}(x)| ≤ C(1 + |x|^{k})

for all x ∈ ℝ^{N}. Show that the function u defined by u(t, x) = G_{t} ∗u_{0}(x) is well defined and C^{∞} on (0, ∞) × ℝ^{N} and that it is a classical solution of the heat equation. Hints: Everything follows from the fact that the Gaussian and all of its derivatives decay exponentially at infinity. Exercise 2.8. We want to prove the general principle that any linear, translation invariant and continuous operator T is a convolution, that is T u = g∗u for some kernel g. This is one of the fundamental principles of both mechanics and signal processing, and it has many generalizations that depend on the domain, range, and continuity properties of T. For instance, assume that T is translation invariant (commutes with translations) and is continuous from L^{2}(ℝ^{N}) into L^{∞}(ℝ^{N}) ∩ C^{0}(ℝ^{N}). Show that T u = g ∗ u, where the convolution kernel g is in L^{2}(ℝ^{N}). This is a direct consequence of Riesz theorem, which states that every bounded linear functional on L^{2}(ℝ^{N}) has the form f ↦ ∫_{ℝ^{N}} f(x)g(x) dx for some g ∈ L^{2}(ℝ^{N}). Show that if u ≥ 0 (u(x) ≥ 0 for all x) implies T u ≥ 0, then g ≥ 0.

2.4 Comments and references

The heat equation. One should not conclude from Theorem 2.9 that the solutions of the heat equation are always unique. The assumption in (ii) that the solution was bounded is crucial. In fact, without this assumption, there are solutions u that grow so fast that gu is not in L^{1}(ℝ^{N}) for g ∈ S (see, for example, [242, page 217]). The existence and uniqueness proof of Theorem 2.9 is classic and can be found in most textbooks on partial differential equations, such as Evans [73], Taylor [242], or Brezis [33].

Convolution. The heat equation—its solutions and their uniqueness—has been the main topic in this chapter, but to approach this, we have studied several aspects of the convolution, such as the continuity property (2.5). We also noted that the convolution commutes with translation. Conversely, as a general principle, any linear, translation invariant and continuous operator T is a convolution, that is, T u = g ∗ u for some kernel g. This is a direct consequence of a result discovered independently by F. Riesz and M. Fréchet in 1907 (see [210, page 61] and exercise 2.8). Since we want smoothing to be translation invariant and continuous in some topology, this means that linear smoothing operators—which are called filters in the context of signal and image processing—are described by their convolution kernels. The Gaussian serves as a model for linear filters because it is the only one whose shape is stable under iteration. Other positive filters change their shape when iterated. This fact will be made precise in the next chapter where we show that a large class of iterated linear filters behaves asymptotically as a convolution with the Gaussian.

“JMMBookOct04” 30/10/2009 page 50

[figure] [figure]

50 CHAPTER 2. THE HEAT EQUATION

Smoothing and the Laplacian. One of the first tools proposed in the early days of image processing in the 1960s came, not surprisingly, directly from signal processing. The idea was to restore an image by averaging the gray levels locally (see, for example, [93] and [107]). The observation that the difference between an image and its local average is proportional to the Laplacian of the image has proved to be one of the most fruitful contributions to image processing. As noted in the Introduction, this method for deblurring an image was introduced by Kovasznay and Joseph in 1955 [136], and it was studied and optimized by Gabor in 1965 [87] (information taken from [147]). (See also [114] and [115].) Burt and Adelson based their Laplacian pyramid algorithm on this idea, and this was one of the results that led to multiresolution analysis and wavelets [36].

“JMMBookOct04” 30/10/2009 page 51

[figure] [figure]

Chapter 3

Iterated Linear Filters and the Heat Equation

The title of this chapter is self-explanatory. The next section fixes fairly general conditions so that the difference of a smoothed image and the original be proportional to the Laplacian. The second section proves the main result, namely the convergence of iterated linear filters to the heat equation. So the choice of a smoothing convolution kernel is somewhat forced : Iterating the convolution with a smoothing kernel is asymptotically equivalent to the convolution with a Gauss function. This result is known in Probability as the central limit theorem, where it has a quite different interpretation. In image processing, it justifies the prominent role of Gaussian filtering. A last section is devoted to linear directional filters and their associated differential operators.

3.1 Smoothing and the Laplacian

There are minimal requirements on the smoothing kernels g which we state in the next definition.

Definition 3.1. We say that a real-valued kernel g ∈ L^{1}(ℝ^{N}) is Laplacian consistent if it satisfies the following moment conditions:

(i) ∫_{ℝ^{N}} g(x) dx = 1.

( ii) For i = 1, 2, …, N, ∫_{ℝ^{N}} x_{i}g(x) dx = 0.

( iii) For each pair i, j = 1, 2, …, N, i ≠ j, ∫_{ℝ^{N}} x_{i}x_{j}g(x) dx = 0.

( iv) For i = 1, 2, …, N, ∫_{ℝ^{N}} x_{i}^{2}g(x) dx = σ, where σ > 0.

( v) ∫_{ℝ^{N}} |x|^{3}|g(x)| dx < +∞.

Note that we do not assume that g ≥ 0; in fact, many important filters used in signal and image processing are not positive. However, condition (i) implies that g is “on average” positive. A discussion of the necessity of the requirements (i) − (v) is performed in Exercise 3.4.

51 “JMMBookOct04” 30/10/2009 page 52

[figure] [figure]

52CHAPTER 3. ITERATED LINEAR FILTERS AND THE HEAT EQUATION

Figure 3.1: The rescalings g_{t}(x) = (1/t^{2})g(x/t) of a kernel for t=4, 3, and 2.

We say that a function g is radial if g(x) = g(|x|), x ∈ ℝ^{N}. This is equivalent to saying that g is invariant under all rotations around the origin in ℝ^{N}. As pointed out in Exercise 3.3, any radial function g ∈ L^{1}(ℝ^{N}) can be rescaled to be Laplacian consistent if it decays fast enough at infinity and if ∫_{ℝ^{N}} x_{i}^{2}g(x) dx and ∫_{ℝ^{N}} g(x) dx have the same sign.

We consider rescalings of a kernel g defined by

g_{h}(x) = \frac{1}{h^{N/2}} g(\frac{x}{h^{1/2}} )  (3.1)

for h > 0 (see Figure 3.1). Notice that this rescaling differs slightly from the one used in Section 1.2. We have used the factor h^{1/2} here because it agrees with the factor t^{1/2} in the Gaussian. We denote the convolution of g with itself n times by g^{n∗}. The main result of this section concerns the behavior of g_{h}^{n∗} as n → +∞ and h → 0.

Exercise 3.1. Prove the following two statements:

(i) g_{h} is Laplacian consistent if and only if g is Laplacian consistent. (ii) If g ∈ L^{1}(ℝ^{N}), then (g_{h})^{n∗} = (g^{n∗})_{h}.

Theorem 3.2. If g is Laplacian consistent, then for every u ∈ F ∩ C^{3}(IR^{N}),

g_{h} ∗ u(x) − u(x) = h\frac{σ}{2} Δu(x) + ε(h, x)  (3.2)

where |ε(h, x)| ≤ Ch^{3/2}.

Proof. We use condition (i), the definition of g_{h}, and rescaling y = h^{1/2}z inside the integral to see that

g_{h} ∗ u(x) − u(x) = ∫_{ℝ^{N}} \frac{1}{h^{N/2}} g(\frac{y}{h^{1/2}} )(u(x − y) − u(x)) dy
= ∫_{ℝ^{N}} g(z)(u(x − h^{1/2}z) − u(x)) dz.

Using Taylor’s formula with the Lagrange remainder (I.1), we have

u(x − h^{1/2}z) − u(x) = −h^{1/2}Du(x) · z + \frac{h}{2}D^{2}u(x)(z, z)

−\frac{1}{6}h^{3/2}D^{3}u(x − h^{1/2}θz)(z, z, z),

“JMMBookOct04” 30/10/2009 page 53

[figure] [figure]

3.2. THE CONVERGENCE THEOREM 53

where θ = θ(x, z, h) ∈ [0, 1]. By condition (ii), ∫_{ℝ^{N}} g(z)Du(x) · z dz = 0; by conditions (iii) and (iv), ∫_{ℝ^{N}} g(z)D^{2}u(x)(z, z) dz = σΔu(x). Thus,

g_{h} ∗ u(x) − u(x) = h\frac{σ}{2} Δu(x) − \frac{1}{6}h^{3/2} ∫_{ℝ^{N}} g(z)D^{3}u(x − h^{1/2}θz)(z, z, z) dz.

We denote the error term by ε(h, x). Then we have the following estimate:

|ε(h, x)| ≤\frac{1}{6}h^{3/2} ∫_{ℝ^{N}} |g(z)D^{3}u(x − h^{1/2}θz)(z, z, z)| dz
≤\frac{1}{6}h^{3/2}N^{3} sup_{α,x} |∂^{α}u(x)| ∫_{ℝ^{N}} |z|^{3}|g(z)| dz,

where the supremum is taken over all vectors α = (α_{1}, α_{2}, ⋯, α_{N}), α_{j} ∈ {1, 2, 3}, such that |α| = 3 and over all x ∈ ℝ^{N}. □

The preceding theorem shows a direct relation between smoothing with a Laplacian-consistent kernel and the heat equation. It also shows why we require σ to be positive: If it is not positive, the kernel is associated with the inverse heat equation (see Exercise 3.4.)

3.2 The convergence theorem

The result of the next theorem is illustrated in Figure 3.2.

Theorem 3.3. Let g be a nonnegative Laplacian-consistent kernel with σ = 2 and define g_{h} by (3.1). Write T_{h}u_{0} = g_{h} ∗u_{0} for u_{0} ∈ F, and let u(t, ·) = G_{t} ∗u_{0} be the solution of the heat equation (2.15). Then, for each t > 0,

(T_{h}^{n}u_{0})(x) → u(t, x) uniformly in x as n → +∞ and nh → t.  (3.3)

Proof. Let us start with some preliminaries. We have (g_{h} ∗ u_{0})(∞) = u_{0}(∞) and therefore T_{h}^{n}u_{0}(∞) = u_{0}(∞). The norm in F is ||u||_{F} = sup_{x∈S_{N}} |u(x)| = sup_{x∈ℝ^{N}} |u(x)|. The first order of business is to say precisely what is meant by the asymptotic limit (3.3): Given t > 0 and given ε > 0, there exists an n_{0} = n_{0}(t, ε) and a δ = δ(t, ε) such that ∥T_{h}^{n}u_{0} − u(t, ·)∥_{F} ≤ ε if n > n_{0} and |nh − t| ≤ δ. This is what we must prove. We will first prove the result when h = t/n. We will then show that the result is true when h is suitably close to t/n.

We begin with comments about the notation. By Exercise 3.1, (T_{h})^{n} = (T^{n})_{h}, so there is no ambiguity in writing T_{h}^{n}. We will be applying T_{h}^{n} to the solution u of the heat equation, which is C^{∞} on (0, +∞)×ℝ^{N}. In this situation, t is considered to be a parameter, and we write T_{h}^{n}u(t, x) as shorthand for T_{h}^{n}u(t, ·)(x). Throughout the proof, we will be dealing with error terms that we write as O(h^{r}). These terms invariably depend on h, t, and x. However, in all cases, given a closed interval [t_{1}, t_{2}] ⊂ (0, +∞), there will be a constant C such that |O(h^{r})| ≤ Ch^{r} uniformly for t ∈ [t_{1}, t_{2}] and x ∈ ℝ^{N}. Finally, keep in mind that all functions of x tend to u_{0}(∞) as x → ∞.

We wish to fix an interval [t_{1}, t_{2}], but since this depends on the point t in (3.3) and on ε, we must first choose these numbers. Thus, choose τ > 0 and keep it fixed. This will be the “t” in (3.3). Next, choose ε > 0. Here are the conditions we wish t_{1} and t_{2} to satisfy:

“JMMBookOct04” 30/10/2009 page 54

[figure] [figure]

54CHAPTER 3. ITERATED LINEAR FILTERS AND THE HEAT EQUATION

Figure 3.2: Iterated linear smoothing converges to the heat equation. In this experiment with one-dimensional functions, it can be appreciated how fast an iterated convolution of a positive kernel converges to a Gaussian. On the left are displayed nine iterations of the convolution of the characteristic function of an interval with itself, with appropriate rescalings. On the right, the same experiment is repeated with a much more irregular kernel. The convergence is almost as fast as the first case.

(1) t_{1} is small enough so ∥u(t_{1}, ·) − u_{0}∥_{F} < ε. (This is possible by Theorem 2.9.)

(2) t_{1} is small enough so ∥u(t_{1} + τ, ·) − u(τ, ·)∥_{F} < ε. (Again, by Theorem 2.9.)

(3) t_{2} is large enough so t_{1} + τ < t_{2}.

There is no problem meeting these conditions, so we fix the interval [t_{1}, t_{2}] ⊂ (0, +∞).

Step 1, main argument : proof that

lim_{{n→+∞ \atop nh=τ}} T_{h}^{n}u(t_{1}, x) = u(t_{1} + τ, x),  (3.4)

where the convergence is uniform for x ∈ ℝ^{N}. We can use Theorem 3.2 to write

T_{h}u(t, x) − u(t, x) = hΔu(t, x) + O(h^{3/2}),  (3.5)

where t ∈ [t_{1}, t_{2}]. That the error function is bounded uniformly by Ch^{3/2} on [t_{1}, t_{2}] × ℝ^{N} follows from the fact that sup_{α,t,x} |∂^{α}u(t, x)| is finite for (t, x) ∈ [t_{1}, t_{2}] × ℝ^{N} (see the proof of Theorem 3.2). Since u is a solution of the heat equation, we also have by Taylor-Lagrange formula (I.1),

u(t+h, x)−u(t, x) = h\frac{∂u}{∂t}(t, x)+ \frac{h^{2}}{2} \frac{∂^{2}}{∂t^{2}} u(t+θh, x) = hΔu(t, x)+O(h^{2}). (3.6)

This time the error term is bounded uniformly by Ch^{2} on [t_{1}, t_{2}] × ℝ^{N} because \frac{∂^{2}u}{∂t^{2}}(t, x) is bounded on [t_{1}, t_{1}]×ℝ^{N}. By subtracting (3.6) from (3.5) we see that

T_{h}u(t, x) = u(t + h, x) + O(h^{3/2}).  (3.7)

This shows that applying T_{h} to a solution of the heat equation at time t advances the solution to time t + h, plus an error term.

“JMMBookOct04” 30/10/2009 page 55

[figure] [figure]

3.2. THE CONVERGENCE THEOREM 55

So far we have not used the assumption that g is nonnegative. Thus, (3.7) is true for any Laplacian-consistent kernel g with σ = 2. However, we now wish to apply the linear operator T_{h} to both sides of equation (3.7), and in doing so we do not want the error term to increase. Since g ≥ 0, this is not a problem:

|T_{h}O(h^{3/2})| ≤ ∫_{ℝ^{N}} |O(h^{3/2})|g_{h}(x − y) dy ≤ ∫_{ℝ^{N}} Ch^{3/2}g_{h}(x − y) dy = Ch^{3/2}.

With this in hand, we can apply T_{h} to both sides of (3.7) and obtain

T_{h}^{2}u(t, x) = T_{h}u(t + h, x) + O(h^{3/2}).  (3.8)

If we write equation (3.7) with t + h in place of t and substitute the expression for T_{h}u(t + h, x) in equation (3.8), we have

T_{h}^{2}u(t, x) = u(t + 2h, x) + 2O(h^{3/2}).  (3.9)

We can iterate this process and get

T_{h}^{n}u(t, x) = u(t + nh, x) + nO(h^{3/2})  (3.10)

with the same constant C in the estimate |O(h^{3/2})| ≤ Ch^{3/2} as long as t + nh ∈ [t_{1}, t_{2}]. To ensure that this happens, we take t = t_{1} and h = τ/n. Then

T_{h}^{n}u(t_{1}, x) = u(t_{1} + τ, x) + O((\frac{τ}{n})^{1/2})  (3.11)

and we obtain (3.4). If we could take t_{1} = 0, this would end the proof. This is not possible because all of the O terms were based on a fixed interval [t_{1}, t_{2}]. However, we have taken t_{1} small enough to finish the proof .

Step 2 : getting rid of t_{1}. Since ∫_{ℝ^{N}} g(x) dx = 1, ∥g_{h}∥_{L^{1}(ℝ^{N})} = 1, and thus

∥g_{h}^{n∗} ∗ v∥_{F} ≤ ∥v∥_{F}.

If we take v = u(t_{1}, ·) − u_{0}, then this inequality and condition (1) imply that

∥T_{h}^{n}u(t_{1}, ·) − T_{h}^{n}u_{0}∥_{F} < ε.  (3.12)

Relations (3.12) and (3.11) imply that

∥T_{h}^{n}u_{0} − u(t_{1} + τ, ·)∥_{F} < 2ε.  (3.13)

This inequality and condition (2) show that

∥T_{h}^{n}u_{0} − u(τ, ·)∥_{F} < 3ε  (3.14)

for n > n_{0} and h = τ/n. This proves the theorem in the case h = τ/n.

Conclusion. It is a simple matter to obtain the more general result. Again, by Theorem 1.10, there is a δ = δ(τ, ε) such that |nh − τ| < δ implies that ∥u(nh, ·) − u(τ, ·)∥_{F} < ε and that nh ∈ [t_{1}, t_{2}] (by condition (3)). Combining this with (3.14) shows that

∥T_{h}^{n}u_{0} − u(nh, ·)∥_{F} < 4ε

if n > n_{0} and |nh − τ| < δ, and this completes the proof. □

“JMMBookOct04” 30/10/2009 page 56

[figure] [figure]

56CHAPTER 3. ITERATED LINEAR FILTERS AND THE HEAT EQUATION

3.3 Directional averages and directional heat equations

In this section, we list easy extensions of Theorem 3.2. They analyze local averaging processes which take averages at each point in a singular neighborhood made of a segment. In that way, we will make appear several nonlinear generalizations of the Laplacian which will accompany us throughout the book. Consider a C^{2} function from ℝ^{N} into ℝ and a vector z ∈ ℝ^{N} with |z| = 1. We wish to compute the mean value of u along a segment of the line through x parallel to the vector z. To do this, we define the operator T z_{h}, h ∈ [−1, 1], by

T z_{h}u(x) = \frac{1}{2h} ∫_{−h}^{h} u(x + sz) ds.

This operator is the directional counterpart of the isotropic operator M_{h} defined by equation (1.1).

Proposition 3.4.

T z_{h}u(x) = u(x) + \frac{h^{2}}{6} D^{2}u(x)(z, z) + o(h^{2}).

Proposition 3.4 is deduced from Theorem 3.2, and it suggests that iterations of the operator T z_{h} are associated with the directional heat equation

\frac{∂u}{∂t}(t, x) = \frac{1}{6}D^{2}u(t, x)(z, z)  (3.15)

in the same way that the iterations of the operator T_{h} in Theorem 3.3 are associated with the ordinary heat equation. If z is fixed, then the operator T z_{h} and equation (3.15) act on u along each line in ℝ^{N} parallel to z separately; there is no “cross talk” between lines. Exercise 3.5 formalizes and clarifies these comments when z is fixed. However, Proposition 3.4 remains true when z is a function of x. This means that we are able to approximate the directional second derivative by taking directional averages where z varies from point to point. The main choices considered in the book are z = Du/|Du| and z = Du^{⊥}/|Du|, where Du = (u_{x}, u_{y}) and Du^{⊥} = (−u_{y}, u_{x}). Then by Proposition 3.4 we have the following limiting relations:

• Average in the direction of the gradient. By choosing z = Du/|Du|,

\frac{1}{|Du|^{2}} D^{2}u(Du, Du) = 6 lim_{h→0} \frac{T_{h}^{Du/|Du|}u − u}{h^{2}}.

We will interpret this differential operator as Haralick’s edge detector in section 7.1.

• Average in the direction orthogonal to the gradient. By choosing z = Du/|Du^{⊥}|,

\frac{1}{|Du|^{2}} D^{2}u(Du^{⊥}, Du^{⊥}) = 6 lim_{h→0} \frac{T_{h}^{Du^{⊥}/|Du|}u − u}{h^{2}}.

This differential operator appears as the second term of the curvature equation. (See Chapter 19.)

Although we have not written them as such, the limits are pointwise in both cases.

“JMMBookOct04” 30/10/2009 page 57

[figure] [figure]

3.4. EXERCISES 57

3.4 Exercises

Exercise 3.2. We will denote the characteristic function of a set A ⊂ ℝ^{N} by 1_{A}. Thus, 1_{A}(x) = 1 if x ∈ A and 1_{A}(x) = 0 otherwise. Consider the kernel g = (1/π)1_{D(0,1)}, where D(0, 1) is the disk of radius one centered at zero. In this case, g is a radial function and it is clearly Laplacian consistent. For N = 2, let A = [−1/2, 1/2] × [−1/2, 1/2]. Then g = 1_{A} is not radial. Show that it is, however, Laplacian consistent. If we take B = [−1, 1]×[−1/2, 1/2], then g = (1/2)1_{B} is no longer Laplacian consistent because it does not satisfy condition (iv). Show that this kernel does, however, satisfy a relation similar to (3.2).

Exercise 3.3. The aim of the exercise is to prove roughly that radial functions with fast decay are Laplacian consistent. Assume g ∈ L^{1}(ℝ^{N}) is radial with finite first second moments, ∫_{ℝ^{N}} |x|^{k}|g(x)| dx < +∞, k = 0, 1, 2, 3 and such that ∫_{ℝ^{N}} x_{i}^{2}g(x) dx > 0. Show that g satisfies conditions (ii) and (iii) of Definition 3.1 and that, for suitably chosen a, b ∈ ℝ, the rescaled function x ↦ ag(x/b) satisfies conditions (i) and (iv), where σ can be taken to be an arbitrary positive number.

Exercise 3.4. The aim of the exercise is to illustrate by simple examples what happens to the iterated filter g^{n∗}, n ∈ ℕ when g does not satisfy some of the requirements of the Laplacian consistency (Definition 3.1). We recall the notation (3.1), g_{h}(x) =

\frac{1}{h^{N/2}} g( \frac{x}{h^{1/2}} ).

1) Take on ℝ, g(x) = 1 on [−1, 1], g(x) = 0 otherwise. Which one of the assumptions (i) − (v) is not satisfied in Definition 3.1 ? Compute gn∗_{\frac{1}{n}} ∗ u, where u = 1 on ℝ. Conclude : the iterated filter blows up.

2) Take on ℝ, g(x) = 1 on [0, 1], g(x) = 0 otherwise. Which one of the assumptions (i) − (v) is not satisfied in Definition 3.1 Compute g^{n∗} ∗ u, where u(x) = x on ℝ. Conclude : the iterated filter “drifts”.

3) Assume that the assumptions (i) − (v) hold, except (iii). By a simple adaptation of its proof, draw a more general form of Theorem 3.2.

4) Perform the same analysis as in 3) when all assumptions hold but (iv).

5) Take the case of dimension N = 1 and assume that (i) hold but (ii) does not hold. Set g_{h}(x) = \frac{1}{h} g( \frac{x}{h} ) and give a version of Theorem 3.2 in that case (make an order 1 Taylor expansion of u).

Exercise 3.5. Let z be a fixed vector in ℝ^{N} with |z| = 1 and let u_{0} be in F. Define a one-dimensional kernel g by g(s) = \frac{1}{2} 1_{[−1,1]}(s).

(i) Show that g is Laplacian consistent. Compute the variance σ of g.

(ii) Show that

u(t, x) = ∫_{ℝ} u_{0}(x + sz)G_{t}(s) ds

is a solution of the directional heat equation

\frac{∂u}{∂t} (t, x) = D^{2}u(t, x)(z, z), u(0, x) = u_{0}(x).  (3.16)

Give an example to show that u(t, ·) is not necessarily C^{2}. This being the case, how does one interpret the right-hand side of (3.16)?

(iii) Let g_{h}(s) = (6h)^{−1/2}g(s/(6h)^{−1/2}) and T_{h}u(x) = ∫_{ℝ} u(x + sz)g_{h}(s) ds. By applying Theorem 3.3 for N = 1, show that, for each t > 0,

T_{h}^{n}u_{0} → u(t, ·) in F as n → +∞ and nh → t.  (3.17)

“JMMBookOct04” 30/10/2009 page 58

[figure] [figure]

58CHAPTER 3. ITERATED LINEAR FILTERS AND THE HEAT EQUATION

Exercise 3.6. The Weickert equation can be viewed as a variant of the curvature equation [256]. It uses a nonlocal estimate of the direction orthogonal to the gradient for the diffusion direction. This direction is computed as the direction v of the eigenvector corresponding to the smallest eigenvalue of k ∗ (Du ⊗ Du), where (y ⊗ y)(x) = (x · y)y and k is a smooth kernel, typically a gaussian. Prove that if the convolution kernel is removed, then this eigenvector is simply Du^{⊥}. So the equation writes

\frac{∂u}{∂t} = u_{ηη},  (3.18)

where η denotes the coordinate in the direction v. Exercise 3.7. Suppose that u ∈ C^{2}(ℝ). Assuming that u′(x) ≠ 0, show that

u′′(x) = lim_{h→0} \frac{1}{h^{2}} ( max_{s∈[−h,h]} u(x + s) + min_{s∈[−h,h]} u(x + s) − 2u(x)).  (3.19)

What is the value of the right-hand side of (3.19) if u′(x) = 0?

Now consider u ∈ C^{2}(ℝ^{2}). We wish to establish an algorithm similar to (3.19) to compute the second derivative of u in the direction of the gradient Du = (u_{x}, u_{y}). For this to make sense, we must assume that Du(x) ≠ 0. With these assumptions, we know from (3.19) that

u_{ξξ}(x) = \frac{∂^{2}v}{∂ξ^{2}} (x, 0) = lim_{h→0} \frac{1}{h^{2}} ( max_{s∈[−h,h]} u(x + sz) + min_{s∈[−h,h]} u(x + sz) − 2u(x)), (3.20)

where v(x, ξ) = u(x + ξz) and z = Du/|Du|. The second part of the exercise is to prove that, in fact,

u_{ξξ}(x) = lim_{h→0} \frac{1}{h^{2}} ( max_{y∈D(0,h)} u(x + y) + min_{y∈D(0,h)} u(x + y) − 2u(x)),  (3.21)

where D(0, h) is the disk of radius h centered at the origin. Intuitively, (3.21) follows from (3.20) because the gradient indicates the direction of maximal change in u(x), so in the limit as h → 0, taking max and min in the direction of the gradient is equivalent to taking max and min in the disk. The point of the exercise is to formalize this.

3.5 Comments and references

Asymptotics. Our proof that iterated and rescaled convolutions of a Laplacianconsistent kernel tend asymptotically to the Gaussian is a version of the De Moivre–Laplace formula, or the central limit theorem, adapted to image processing [32]. This result is particularly relevant to image analysis, since it implies that iterated linear smoothing leads inevitably to convolution with the Gaussian, or equivalently, to the application of the heat equation. We do not wish to imply, however, that the Gaussian is the only important kernel for image processing. The Gaussian plays a significant role in our form of image analysis, but there are other kernels that, because of their spectral and algebraic properties, have equally important roles in other aspects of signal and image processing. This is particularly true for wavelet theory which combines recursive filtering and sub-sampling.

Directional diffusion. Directional diffusion has a long history that began when Hubel and Wiesel showed the existence of direction-sensitive cells in the visual areas of the neocortex [112]. There has been an explosion of publication “JMMBookOct04” 30/10/2009 page 59

[figure] [figure]

3.5. COMMENTS AND REFERENCES 59

on directional linear filters, beginning, for example, with influential papers such as that by Daugman [65]. We note again that Gabor’s contribution to directional filtering is described in [147].

“JMMBookOct04” 30/10/2009 page 60

[figure] [figure]

60CHAPTER 3. ITERATED LINEAR FILTERS AND THE HEAT EQUATION

Sampling and interpolation

Digital images, only defined for (n_{1}, n_{2}) ∈ IZ^{2}, will be denoted by u(n_{1}, n_{2}). The sampled image u = S_{1}u is defined on IZ^{2} by

u(n_{1}, n_{2}) = (S_{1}u)(n_{1}, n_{2}) =: u(n_{1}, n_{2});  (3.22)

The practical Shannon interpolation: zero-padding

Let a > 0 and consider a function u from [0, a]^{2} to IR such that u(x+a, y +a) = u(x, y). Fix an integer N, and consider the N^{2} samples of u, u_{k,l} = (Su) (\frac{ka}{N}, \frac{la}{N}) on [0, a]^{2}.

Definition 3.5. The discrete Fourier transform (DFT) of the N^{2} samples u = (u_{k,l})_{k,l=0,1,...,N−1} is the double sequence of discrete Fourier coefficients for m, n ∈ \{−\frac{N}{2} , ..., \frac{N}{2} − 1\} defined by

DF T (u)_{m,n} = ũ_{m,n} = \frac{1}{N^{2}} ∑_{k=0}^{N−1} ∑_{l=0}^{N−1} u_{k,l}ω_{N}^{−mk}ω_{N}^{−nl},  (3.23)

where ω_{N} =: e^{\frac{2iπ}{N}} is the first N-root of 1.

Proposition 3.6. Consider the trigonometric polynomial

Iu(x, y) = ∑_{m,n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{m,n} exp(\frac{2iπmx}{a}) exp(\frac{2iπny}{a}).  (3.24)

Then its coefficients ũ_{m,n} are the only complex numbers such that for every k, l ∈ \{0, ..., N−1\}, Iu (\frac{ka}{N}, \frac{la}{N}) = u (\frac{ka}{N}, \frac{la}{N}). In consequence, the discrete inverse transform of the DFT u → ũ is nothing but the calculation of the value of the polynomial at the samples (\frac{ka}{N}, \frac{la}{N}), 0 ≤ k, l ≤ N −1. In other terms, setting ũ = DF T (u), the inverse transform DF T^{−1} is given by

u(k, l) = ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{m,n}ω_{N}^{km+ln},

for every k, l = 0, 1, … N − 1.

Exercise 3.8. Recall that ω_{N} = exp ( \frac{2iπ}{N} ), N-th root of 1. Show that ∑_{k=0}^{N−1} ω_{N}^{k} = 0, and that ∑_{k=0}^{N−1} ω_{N}^{kl} = 0 for l ≠ 0 modulo N and finally that for every k_{0}, ∑_{k=k_{0}}^{k_{0}+N−1} ω_{N}^{kl} = 0 for all l ≠ 0 modulo N. Using these relation show the above proposition, namely that DF T (DF T^{−1}) = Id.

Proposition 3.7. If u(x, y) is a a-periodic band-limited function, then it is a trigonometric polynomial. If its highest degree is \frac{N}{2} − 1, with N even, then its coefficients ũ_{m,n} are obtained by DFT from the samples u(k, l) = u(\frac{ka}{N}, \frac{la}{N}). In consequence for such functions we have ISu = u.

“JMMBookOct04” 30/10/2009 page 61

[figure] [figure]

3.5. COMMENTS AND REFERENCES 61

Zoom in by zero-padding

Let (u_{k,l}) be a digital image and define its zoomed version (v_{i,j})_{i,j=0, ..., 2N−1} as the inverse discrete Fourier transform of ṽ_{i,j} defined for i, j = −N, ..., N−1 by

ṽ_{m,n} = ũ_{m,n} if − \frac{N}{2} ≤ m, n ≤ \frac{N}{2} − 1, ṽ_{m,n} = 0 otherwise.  (3.25)

Proposition 3.8. The image v whose Discrete Fourier Transform is given by (5.5) satisfies v_{2k,2l} = u_{k,l}, for k, l = 0, ..., N −1.

Gaussians

G_{σ} denotes the convolution operator on ℝ^{2} with the gauss kernel G_{σ}(x_{1}, x_{2}) =

\frac{1}{2π(σ)^{2}} e^{− \frac{x^{2}_{1}+x^{2}_{2}}{2(σ)^{2}}} , namely G_{σ}u(x, y) =: (G_{σ} ∗ u)(x, y).

G_{σ}G_{β} = G\sqrt{σ^{2}+β^{2}}.  (3.26)
G_{σ}H_{γ}u = H_{γ}G_{σγ}u.  (3.27)

Definition 3.9. The discrete gaussian convolution applied to a digital image u is defined as a digital operator by

G_{δ}u =: S_{1}G_{δ}Iu.  (3.28)

Proposition 3.10. This definition maintains the gaussian semi-group property,

G_{δ}G_{β} = G\sqrt{δ^{2}+β^{2}}.  (3.29)

Proposition 3.11. For every σ larger than 0.8 and every continuous and bounded image u_{0}, the gaussian blurred image G_{σ}u_{0} is well sampled, namely IS_{1}G_{σ}u_{0} = G_{σ}u_{0}.

Discrete sampling

• If u is digital and therefore only defined on IZ^{2} and if δ is an integer, then one can define any sub- or over-sampling operations on u. But this requires interpolating u first. Thus we define a digital subsampling operator by

S_{δ}u =: S_{δ}Iu.  (3.30)

Notice that S_{δ} is a discrete filter which can be interpreted as a re-sampling operator.

• Zooming in by oversampling. For every γ ≤ 1,

IS_{γ}u = H_{γ}Iu.  (3.31)

• Corollary: If γ ≤ 1, then

S_{β}S_{γ} = S_{βγ}.  (3.32)

• a discrete commutation formula : Assume u is a digital image. Then for

γ < 1,

G_{β}S_{γ}u = S_{γ}G_{βγ}u.  (3.33)

“JMMBookOct04” 30/10/2009 page 62

[figure] [figure]

62CHAPTER 3. ITERATED LINEAR FILTERS AND THE HEAT EQUATION

3.6 Preliminaries on sub-pixel interpolation

This section proves a discrete correlation formula which is faithful to the continuous image interpolates. Thanks to it, an accurate subpixel matching becomes possible. Without loss of generality, all considered images u, u_{1}, etc. are defined on a square [0, a]^{2} and are supposed to be square integrable. Thus, the Fourier series decomposition applies

u(x, y)=∑_{k,l∈Z} ũ_{k,l}e^{\frac{2iπ(kx+ly)}{a}} ,  (3.34)

where the ũ_{k,l} are the Fourier series coefficients (or shortly the Fourier coefficients) of u. By the classic Fourier series isometry, for any two square integrable functions u(x) and v(x) on [0, a]^{2},

∫_{[0,a]^{2}} u(x)\overline{v}(x)dx = a^{2} ∑_{k,l∈Z} ũ_{k,l}\overline{ṽ_{k,l}}.  (3.35)

The digital images are usually given by their N^{2} samples u(m) for m in the grid

Z_{a}^{1} = [0, a]^{2} ∩ ((\frac{a}{2N} , \frac{a}{2N}) + \frac{a}{N} Z^{2}).

Similarly, the over-sampling grid with four times more samples is denoted by

Za^{\frac{1}{2}} = [0, a]^{2} ∩ ((\frac{a}{4N} , \frac{a}{4N}) + \frac{a}{2N} Z^{2}).

N is always an even integer. In all that follows we shall assume that the images obtained by a stereo vision system are band-limited. This assumption is classical and realistic, the aliasing in good quality CCD cameras being moderate. As classical in image processing, under the (forced) a-periodicity assumption a band-limited image becomes a trigonometric polynomial. This periodicity assumption is not natural, but it only entails a minor drawback, namely a small distortion near the boundary of the image domain [0, a]^{2}. The payoff for the band-limited + periodic assumption is that the image can be interpolated, and its Fourier coefficients computed from discrete samples. Indeed, given N^{2} samples u_{m} for m in Z_{a}^{1}, there is a unique trigonometric polynomial in the form

u(x, y)= ∑_{k,l=−N/2}^{N/2−1} ũ_{k,l}e^{\frac{2iπ(kx+ly)}{a}}  (3.36)

such that u(m) = u_{m}. We shall call such polynomials N-degree trigonometric polynomials. The coefficients ũ_{k,l} are the Fourier coefficients of u in the Fourier basis e^{\frac{2iπ(kx+ly)}{a}}, k, l ∈ Z. The map u_{m} → u_{k,l} is nothing but the 2D Discrete Fourier Transform (DFT), and the map (u_{m}) → N(ũ_{k,l}) is an isometry from C^{N^{2}} to itself. The function u(x, y) is therefore usually called the DFT interpolate of the samples u_{m}. In consequence, there is an isometry between the set of N-degree trigonometric polynomials endowed with the L^{2}([0, a]^{2}) norm, and C^{N^{2}} endowed with the usual Euclidean norm:

“JMMBookOct04” 30/10/2009 page 63

[figure] [figure]

3.6. PRELIMINARIES ON SUB-PIXEL INTERPOLATION 63

∫_{[0,a]^{2}} |u(x, y)|^{2} = a^{2} ∑_{k,l=−N/2}^{N/2−1} |ũ_{k,l}|^{2} = \frac{a^{2}}{N^{2}} ∑_{m∈Z_{a}^{1}} |u(y + m)|^{2},  (3.37)

where the N^{2} samples grid can have an arbitrary origin y. If u(x) and v(x) are two degree N trigonometric polynomials, we therefore also have

∫_{[0,a]^{2}} u(x)\overline{v}(x) =a^{2} ∑_{k,l=−N/2}^{N/2−1} ũ_{k,l}\overline{ṽ_{k,l}} =\frac{a^{2}}{N^{2}} ∑_{m∈Z_{a}^{1}}u(y + m)\overline{v(y + m)}. (3.38)

where \overline{v} is the complex conjugate of v. Taking four times more samples, it follows from (3.38) that

∫_{[0,a]^{2}} u(x)\overline{v}(x) =a^{2} ∑_{k,l=−N}^{N−1} ũ_{k,l}\overline{ṽ_{k,l}} = \frac{a^{2}}{4N^{2}} ∑_{m∈Z \frac{1}{2} ; a} u(m)\overline{v(m)}.  (3.39)

which is also valid if u(x) and v(x) are up to 2N-degree trigonometric polynomials in x.

This last fact has a first important consequence in block-matching. Consider two images u_{1}(x) and u_{2}(x) on [0, a]^{2} and a window function φ(x). Block matching is the search for a value of μ minimizing the continuous quadratic distance

e_{x_{0}}(μ) := ∫_{[0,a]^{2}} φ(x − x_{0})(u_{1}(x) − u_{2}(x + (μ, 0)))^{2}dx.  (3.40)

Proposition 3.12. (Equality of the discrete and the continuous quadratic distance) Let u_{1}(x) and u_{2}(x) be two N-degree trigonometric polynomials on [0, a]^{2} and let φ(x) be a window function which we assume to be a 2N-degree trigonometric polynomial. Then

e_{x_{0}}(μ) = e_{x_{0}}^{d}(μ), where  (3.41)
e_{x_{0}}^{d}(μ) := \frac{a^{2}}{4N^{2}} ∑_{m∈Z \frac{1}{2} ; a} φ(m − x_{0})(u_{1}(m) − u_{2}(m + (μ, 0)))^{2}.  (3.42)

The proof follows from (3.39). Indeed, (u_{1}(x) − u_{2}(x + (μ, 0)))^{2} and φ(x − x_{0}) are both 2N-degree trigonometric polynomials in x, so according to (3.39) the discrete scalar product defining e_{x_{0}}^{d}(μ) equals the continues scalar product defining e_{x_{0}}(μ). Thus the continuous block distance is a finite sum of discrete samples!

The block distance function μ → e_{x_{0}}(μ), whose minimization is our main objective here, is also easily sampled. By (3.42) it is a 2N-degree trigonometric polynomial with respect to μ. This proves:

Proposition 3.13 (Sub-pixel correlation requires ×2 zoom). Let u_{1}(x) and u_{2}(x) be two N-degree trigonometric polynomials. Then the quadratic distance e_{x_{0}}^{d}(μ) is well-sampled provided it has at least 2N successive samples. Thus “JMMBookOct04” 30/10/2009 page 64

[figure] [figure]

64CHAPTER 3. ITERATED LINEAR FILTERS AND THE HEAT EQUATION

the computation of e_{x_{0}}^{d}(μ) at half samples μ ∈ \frac{aZ}{2} (via zero padding) allows the exact reconstruction of e_{x_{0}}^{d}(μ) for any real μ by DFT interpolation.

Remark that the last proposition does not require any assumption on the window function φ(x). Prop. 3.13, which opens the way to rigorous block matching with sub-pixel accuracy, has been noticed (but not implemented) in [?]. It is also used in the MARC method [?] used by the French space agency (CNES). The above simple proof of Prop. 3.13 is new.

Sub-pixel block matching will require to interpolate the noisy images. Thus, following Shannon’s classical observation, the noise itself must also be interpolated as a band-limited function. In the periodic framework it therefore becomes a trigonometric polynomial. Assume that (n_{m}), m ∈ Z_{a}^{1} are N^{2} independent N (0, σ^{2}) noise samples. This amounts to say that (n_{m}) is a Gaussian vector. Since the DFT is an isometry, the noise Fourier coefficients N(ñ_{k}) also form a Gaussian vector with diagonal covariance matrix σ^{2}Id. By (3.38), the mapping (n_{m})_{m∈Z_{a}^{1}} → (n(x + m))_{m∈Z_{a}^{1}} is an isometry from C^{N^{2}} to itself. It follows that n(x) is N (0, σ^{2}) for every x.

One can also estimate Var(n_{x}(x)), where n_{x}(x) = \frac{∂n}{∂x}(x, y).

Var(n_{x}(x)) = Var ( ∑_{k,l=−N/2}^{N/2−1} ñ_{k,l} \frac{2ikπ}{a} e^{2i \frac{kπx+lπy}{a}} )

= \frac{4π^{2}σ^{2}N}{N^{2}a^{2}} ∑_{k=−N/2}^{N/2−1} k^{2} ≃ \frac{4π^{2}σ^{2}}{a^{2}N} \frac{N^{3}}{12} = \frac{π^{2}N^{2}}{3a^{2}} σ^{2}.

Since n(x) is a normal law, n(x)^{2} is a χ^{2} law of order 1. Thus its variance is 2σ^{4}. Finally we shall need to evaluate Var(n_{1}(x)n_{2}(x)), where n_{i} are two independent interpolated white noises of the above kind. Thus n_{1}(x)n_{2}(x) is the product of two normal laws. The expectation of the product is zero and the variance is therefore Var(n_{1}n_{2}) = E(n_{1}n_{2})^{2} = En_{1}^{2}En_{2}^{2} = (En^{2})^{2} = Var(n)^{2} = σ^{4}. In summary:

Lemma 3.14. Let (n_{m})_{m∈Z_{a}^{1}} be N^{2} independent white Gaussian noise samples with variance σ^{2}. Then the DFT interpolate n(x) on [0, a]^{2} is N (0, σ^{2}) for every x. If n_{1} and n_{2} are two independent noises like n, one has

Var(n^{2}(x))=2σ^{4},  (3.43)
Var((n)_{x}(x))≃\frac{π^{2}N^{2}}{3a^{2}} σ^{2},  (3.44)
Var(n_{1}(x)n_{2}(x))=σ^{4}.  (3.45)

“JMMBookOct04” 30/10/2009 page 65

[figure] [figure]

Chapter 4

Le cas discret

4.1 Transforme de Fourier Discrte, applications

4.1.1 La dimension 1

La transforme de Fourier discrte est un moyen de calculer les coefficients de Fourier d’une fonction a-priodique u partir de ses N chantillons u(\frac{ka}{N}), k = 0, ...N−1. Cela n’est possible que si la fonction prsente un nombre de frquences infrieur ou gal N. Pour coller la pratique numrique, nous supposerons toujours dans ce chapitre que N est pair. En gnral, N est une puissance de 2. Tous les rsultats noncs (sauf ceux du paragraphe ?? relatifs la transforme de Fourier rapide) s’adaptent toutefois sans difficult au cas N impair. Soit u(x) une fonction relle ou complexe de priode a, et N un entier pair. On cherche un polynôme trigonomtrique de la forme

P (x) = ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n} exp (\frac{2iπnx}{a}) ,  (4.1)

qui soit gal u aux points \frac{ka}{N} pour k = 0, ..., N −1. On dira dans la suite que P est de degr \frac{N}{2} . Le but est donc d’interpoler les chantillons u(\frac{ka}{N}) = u_{k}.

Pourquoi choisir un polynme trigonomtrique ? La raison est physique : tous les dispositifs d’acquisition de signaux (sons) ou images ont une bande passante, c’est--direun intervalle de frquences captes par le dispositif d’enregistrement ; les autres frquences sont perdues ou tellement attnues qu’on les nglige : on suppose donc ici que la ”bande passante” est contenue dans l’intervalle [− \frac{N}{2} , \frac{N}{2} − 1]. Il n’y a par contre aucune raison de supposer que le signal ou l’image soit priodique, et en plus d’une priode qui concide avec la fentre d’observation [0, a] comme nous sommes en train de le suppposer pour P. Cette hypothse est donc impose la donne par une priodisation abusive, et provoque une distorsion prs des bords de l’image que l’on peut voir et valuer : c’est le phnomne de Gibbs. Si la fonction u dont on possde les N chantillons n’avait pas une bande de frquence contenue dans [− \frac{N}{2} , \frac{N}{2} − 1], son interpolation par un polynme trigonomtrique de degr \frac{N}{2} provoque une autre distorsion, trs grave, que nous allons valuer prcisment dans ce chapitre : l’aliasage.

On va commencer par calculer les coefficients de P.

65 “JMMBookOct04” 30/10/2009 page 66

[figure] [figure]

66 CHAPTER 4. LE CAS DISCRET

On pose ω_{N} = exp (\frac{2iπ}{N} ), racine N-ime de l’unit. Montrer que ∑_{k=0}^{N−1} ω_{N}^{k} = 0, puis que ∑_{k=0}^{N−1} ω_{N}^{kl} = 0 pour l ≠ 0 modulo N et finalement que pour tout k_{0}, ∑_{k=k_{0}}^{k_{0}+N−1} ω_{N}^{kl} = 0 pour tout l ≠ 0 modulo N.

transforme de Fourier!discrte On pose u_{k} = u(\frac{ka}{N}) et, pour n = −\frac{N}{2} , ..., \frac{N}{2} −1,

ũ_{n} = \frac{1}{N} ∑_{l=0}^{N−1} u_{l}ω_{N}^{−nl}.  (4.2)

Les N coefficients ũ_{n} sont appels transforme de Fourier discrte (TFD) des N chantillons u_{k}. On appelle transforme de Fourier discrte inverse l’application de C^{N} dans lui mme dfinie par

u_{k} = ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n}ω_{N}^{kn}, k = 0, ..., N −1.  (4.3)

Proposition 4.1. transforme de Fourier!discrte Les coefficients (ũ_{n}) dfinis par (4.2) sont les uniques coefficients tels que le polynôme trigonomtrique (4.1) vrifie P ( \frac{ka}{N} ) = u_{k}, pour tout k = 0, ..., N −1. En d’autres termes, la transforme de Fourier discrte compose avec son inverse donne bien l’identit.

Dmonstration Pour k = 0, ..., N −1,

P (\frac{ka}{N}) = ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n}ω_{N}^{nk}

= \frac{1}{N} ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} (∑_{l=0}^{N−1} u_{l}ω_{N}^{−nl})ω_{N}^{nk}

= \frac{1}{N} ∑_{l=0}^{N−1} u_{l}( ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} ω_{N}^{nk−nl})

= \frac{1}{N} ∑_{l=0}^{N−1} Nδ(k − l)u_{l} = u_{k},

où on a not δ la fonction dfinie sur les entiers, valant 1 en 0, et 0 ailleurs. L’unicit provient du fait que toute application linaire surjective de C^{N} dans lui-mme est aussi injective.

Corollary 4.2. polynme trigonomtrique Si u est un polynme trigonomtrique u(x) = ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n} exp ( \frac{2iπnx}{a} ) , les coefficients ũ_{n} sont obtenus partir des chantillons de u par la formule (4.2). Ce sont les coefficients de Fourier de u.

On note u un vecteur de C^{N} et TFD(u) = ũ sa transforme de Fourier discrte. Vrifier que \sqrt{N} TFD est unitaire et calculer TFD^{−1} grce la formule TFD^{−1} = N ^{t}\overline{TFD}.

On rappelle d’autre part que si u ∈ L^{2}(0, a), les coefficients de la srie de Fourier de u sont dfinis, pour n ∈ Z, par

c_{n}(u) = \frac{1}{a} ∫_{0}^{a} u(x) exp (\frac{−2iπnx}{a}) .  (4.4)

“JMMBookOct04” 30/10/2009 page 67

[figure] [figure]

4.1. TRANSFORME DE FOURIER DISCRTE, APPLICATIONS 67

Les coefficients ũ_{n} de la transforme de Fourier discrte sont approchs par les termes de la TFD de (u_{k}) au sens suivant:

Proposition 4.3. Soit u continue et a-priodique. Alors les ũ_{n} sont des approximations des c_{n}(u) par la formule des trapzes, pour n = \frac{−N}{2} , ..., \frac{N}{2} − 1.

Dmonstration Il suffit d’crire l’approximation de l’intgrale (4.4) par la mthode des trapzes en tenant compte du fait que u(a) = u(0) pour une fonction a-priodique.

Proposition 4.4. On suppose que les chantillons u_{k} sont rels. Alors ũ_{0} et ũ_{− \frac{N}{2}} sont rels, et pour k = 1... \frac{N}{2} − 1, ũ_{k} = \overline{ũ_{−k}}.

Dmonstration ũ_{0} = \frac{1}{N} ∑_{k} u_{k}, et ũ_{− \frac{N}{2}} = \frac{1}{N} ∑(−1)^{k}u_{k} ; ces deux coefficients sont donc rels. D’autre part

ũ_{−n} = \frac{1}{N} ∑_{k=0}^{N−1} u_{k}ω_{N}^{kn} = \frac{1}{N} ∑_{k=0}^{N−1} u_{k}\overline{ω_{N}^{−nk}} = \overline{ũ_{n}}.

Remarquons le rôle particulier jou par le terme ũ_{− \frac{N}{2}} , qui n’a pas de terme conjugu lui correspondant.

Proposition 4.5. si u est un polynme trigonomtrique rel dont les frquences sont parmi − \frac{N}{2} , ..., \frac{N}{2} − 1, le terme ũ_{− \frac{N}{2}} est nul.

Dmonstration En effet, en regroupant les termes conjugus, on a, pour le polynme trigonomtrique P dont les coefficients sont les ũ_{n} :

P (x) = ũ_{0} + ∑_{n=1}^{\frac{N}{2} −1}(ũ_{n}e^{\frac{2inπx}{a}} + ũ_{−n}e^{\frac{−2inπx}{a}}) + ũ_{− \frac{N}{2}} e^{\frac{−iNπx}{a}} .

Tous les termes de la somme sont rels sauf le dernier, qui ne l’est que si ũ_{− \frac{N}{2}} = 0. □

La Figure ?? montre un exemple de signal (reprsentant le son A) et le module de sa TFD.

4.1.2 La dimension 2

On considre un rel a, une fonction u de IR^{2} dans IR, telle que u(x + a, y + a) = u(x, y). On fixe nouveau un entier N, et l’on pose u_{k,l} = u (\frac{ka}{N}, \frac{la}{N}). On dfinit la TFD des u_{k,l} comme la suite des coefficients, pour m, n ∈ \{−\frac{N}{2} , ..., \frac{N}{2} − 1\},

ũ_{m,n} = \frac{1}{N^{2}} ∑_{k=0}^{N−1} ∑_{l=0}^{N−1} u_{k,l}ω_{N}^{−mk}ω_{N}^{−nl}.  (4.5)

Montrer que la transformation ainsi dfinie est “sparable”, c’est--dire que le passage des u_{k,l} aux ũ_{m,n} s’effectue par deux TFDs une dimension successives.

De même qu’en dimension 1, nous avons la proprit d’interpolation suivante:

“JMMBookOct04” 30/10/2009 page 68

[figure] [figure]

68 CHAPTER 4. LE CAS DISCRET

Proposition 4.6. Soient les coefficients ũ_{m,n} dfinis, pour m, n = − \frac{N}{2} , ..., \frac{N}{2} − 1, par (5.3). Considrons le polynme trigonomtrique

P (x, y) = ∑_{m,n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{m,n} exp( \frac{2iπmx}{a}) exp( \frac{2iπny}{a}).

Les coefficients ũ_{m,n} sont les seuls nombres complexes tels que, pour tout k, l ∈ \{0, ..., N − 1\}, on ait P ( \frac{ka}{N} , \frac{la}{N} ) = u ( \frac{ka}{N} , \frac{la}{N} ). Par consquent, la transforme discrte inverse de u_{k,l} → ũ_{m,n} est donne par le calcul du polynme aux chantillons ( \frac{ka}{N} , \frac{la}{N}), 0 ≤ k, l ≤ N −1 :

u(k, l) = P ( \frac{ka}{N} , \frac{la}{N}) = ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{m,n}ω_{N}^{km+ln}.

Montrer la proposition prcdente. Le calcul est exactement le même qu’en dimension 1. De même qu’en dimension 1, nous pouvons identifier un certain nombre de symtries des ũ_{m,n} si l’image est valeurs relles. On suppose nouveau que N est pair. Montrer galement la proposition suivante.

Proposition 4.7. Supposons que les chantillons u_{k,l} soient rels. Alors les coefficients ũ_{0,0}, ũ_{0,− \frac{N}{2}} , ũ_{− \frac{N}{2} ,0}, et ũ_{− \frac{N}{2} ,− \frac{N}{2}} sont rels; de plus

∀m, n ∈ \{− \frac{N}{2} + 1, ..., \frac{N}{2} − 1\}  ũ_{m,n} = \overline{ũ_{−m,−n}}

A nouveau, comme en dimension 1, les coefficients (ũ_{m,n}) correspondent aux frquences de l’image u, ordonnes des ngatives aux positives. Plus prcisment, si u ∈ L^{1} et que l’on dfinit les coefficients de la srie de Fourier de u par

c_{m,n} = \frac{1}{4π^{2}} ∫_{[0, 2π]^{2}} u(x, y) exp (\frac{−2iπmx}{a}) exp (\frac{−2iπny}{a} ) ,

alors, pour m, n = −\frac{N}{2} , ..., \frac{N}{2} − 1 les ũ_{m,n} sont des approximations des c_{m,n} par la mthode des trapzes. La figure ?? prsente une image et le logarithme du module de sa transforme de Fourier discrte (le logarithme est utilis car le module des TFD des images usuelles dcroit trs vite lorsque l’on s’loigne des basses frquences).

4.1.3 Le phnomne du repliement de spectre ou aliasage

repliement de spectre

Le but de ce paragraphe est de calculer les perturbations auxquelles est expose la transforme de Fourier discrte d’un signal lorsque celui-ci est souschantillonn. On vient de voir que la transforme de Fourier discrte calculait exactement les coefficients de Fourier d’un polynme trigonomtrique de degr \frac{N}{2} , P (x) = ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n} exp (\frac{2iπnx}{a}), dont on connaissait N chantillons u(\frac{ka}{N}),

N = 0, ..., N −1. Dans cette section, on considre une fonction u ∈ L^{2}(0, a) et sa srie de Fourier

u(x) = ∑_{n∈I Z} c_{n}(u)e^{\frac{2inπx}{a}} .

Dans toute la suite, on supposera que ∑_{n∈I Z} |c_{n}(u)| < +∞, ce qui implique que u est continue et a-priodique. Cette hypothse n’est pas irraliste. En effet, tant “JMMBookOct04” 30/10/2009 page 69

[figure] [figure]

4.1. TRANSFORME DE FOURIER DISCRTE, APPLICATIONS 69

donn un signal v rgulier (C^{2} par exemple) sur [0, a/2], on peut le rendre pair en posant u(−x) = ṽ(x) pour x ∈ [−a/2, 0], u(x) = v(x) sur [0, a]. On voit que la a-priodise de cette extension u reste Lipschitz et C^{2} par morceaux et on peut en dduire (exercice !) que la srie des coefficients de Fourier de u est convergente. On suppose galement, ce qui est raliste, qu’un signal u n’est en fin de compte connu que par ses chantillons sur [0, a], u(0), ..., u(\frac{N−1}{N}a).

Soit u dfinie sur [0, a], vrifiant ∑_{n} |c_{n}(u)| < +∞. Shannon !thorme discret de Alors la transforme de Fourier discrte des N chantillons de u sur [0, a] est la N-priodise de la suite des coefficients de Fourier de u :

ũ_{n} = ∑_{q=−∞}^{+∞} c_{n+qN}(u), n = −\frac{N}{2} , ..., \frac{N}{2} − 1.  (4.6)

Dmonstration On rappelle la notation ω_{N} = e^{\frac{2iπ}{N}} et (ω_{N})^{N} = 1. Comme

u(x) = ∑_{m∈I Z} c_{m}(u)e^{\frac{2imπx}{a}} ,

on a

u(\frac{ka}{N}) = ∑_{m∈I Z} c_{m}(u)ω_{N}^{mk}.

On pose pour m ∈ IZ, m = qN + n, −\frac{N}{2} ≤ n ≤ \frac{N}{2} − 1. En regroupant les termes de la srie de Fourier on obtient

u(\frac{ka}{N}) = ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} ( ∑_{q=−∞}^{+∞} c_{n+qN}(u)) ω_{N}^{nk}, k = 0, ..., N −1.

Mais on a aussi (formule d’inversion de la transforme de Fourier discrte):

u(\frac{ka}{N}) = ∑_{n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n}ω_{N}^{nk}, k = 0, ..., N −1.

Ces deux dernires formules dfinissent toutes deux la transforme de Fourier discrte et par identification on obtient la formule de ”repliement de spectre” (4.6).

□

On a le rsultat analogue en deux dimensions:

Soit u dfinie sur [0, a]^{2}, vrifiant ∑_{n} |c_{m,n}(u)| < +∞. Shannon !thorme discret de Alors la transforme de Fourier discrte de u est la (N, N)-priodise de la suite des coefficients de Fourier de u :

ũ_{m,n} = ∑_{p,q=−∞}^{+∞} c_{m+pN,n+qN}(u), m, n = −\frac{N}{2} , ..., \frac{N}{2} − 1.  (4.7)

Montrer le thorme 4.1.3.

Ce thorme va nous permettre d’interprter les effets de moir visibles dans beaucoup d’images digitales ou de films digitaliss (DVD). Ces effets de moir sont ds un ”repliement de spectre”, ou ”aliasage”. Le repliement de spectre provient “JMMBookOct04” 30/10/2009 page 70

[figure] [figure]

70 CHAPTER 4. LE CAS DISCRET

d’un sous-chantillonnage abusif. Le terme aliasage se rfre la prsence des coefficients parasites c_{m+pN,n+qN}, pour (p, q) ≠ (0, 0) dans le calcul du coefficient de la frquence (m, n), ũ_{m,n}. Quand la transforme de Fourier discrte fait correctement son travail, qui est de retrouver le coefficient c_{m,n} de la frquence (m, n) de u, on doit avoir ũ_{m,n} = c_{m,n}. Les coefficients c_{m+pN,n+qN} qui s’y ajoutent dans (4.7) sont des rpliques, ou ”alias” de coefficients correspondant aux frquences plus grandes (m + pN, n + qN), (p, q) ≠ (0, 0). D’o le terme d’aliasage. Dans une exprience sur ordinateur, les signaux et images sont toujours dj chantillonns. Voyons donc maintenant comment interprter l’exprience pratique suivante: un signal est donn par ses chantillons, et on le sous-chantillonne.

sous-chantillonnage d’un signal Soit un signal chantillonn (u_{k}), k = 0, ..., N− 1, et soit p un entier divisant N. On dfinit l’oprateur “sous-chantillonnage d’ordre p” comme suit:

S_{p} : ℝ^{N} ⟶ ℝ^{N/p}
(u_{k})_{k=0, ..., N−1} ⟶ (v_{k}) = (u_{kp})_{k=0,...,N/p}.

Le signal (v_{k}) est dit sous-chantillonn d’un facteur p. Nous commençons par le cas, technologiquement classique, o p = 2.

Corollary 4.8. Soit (v_{k}) = S_{2}((u_{k})) (on suppose que \frac{N}{2} est pair). Alors (ṽ_{n}), la transforme de Fourier discrte de (v_{k}), s’crit, pour n = − \frac{N}{4} , ..., \frac{N}{4} − 1,

ṽ_{n} = ũ_{n} + ũ_{n− \frac{N}{2}} + ũ_{n+ \frac{N}{2}} ,  (4.8)

le deuxime terme tant par ailleurs nul si n < 0 et le troisime tant nul si n ≥ 0.

Dmonstration Appliquons le thorme 4.1.3 l’unique polynme trigonomtrique P N coefficients qui a pour chantillons les u_{k}. Alors par dfinition de la transforme de Fourier discrte, ũ_{n} = c_{n}(P). On a donc pour \frac{N}{4} ≤ n ≤ \frac{N}{4} − 1,

ṽ_{n} = ∑_{q∈I Z} c_{n+q \frac{N}{2}} (P) = ũ_{n} + ũ_{n− \frac{N}{2}} + ũ_{n+ \frac{N}{2}} .

Remarquons que si n ≥ 0 cela donne ṽ_{n} = ũ_{n} + ũ_{n− \frac{N}{2}} , l’autre coefficient tant nul. De mme, si n < 0, on obtient ṽ_{n} = ũ_{n} + ũ_{n+ \frac{N}{2}} .

□

Cette proposition indique que le spectre du signal sous-chantillonn d’un facteur deux s’obtient en superposant lui-même le spectre du signal original avec un dcalage de \frac{N}{2} . On dit qu’il y a repliement de spectre. Ainsi, le spectre du signal sous-chantillonn contient gnralement des informations non prsentes dans le spectre du signal de dpart, ce qui se traduit sur le signal sous-chantillonn par l’apparition de structures priodiques n’ayant pas de lien direct avec le contenu du signal. Ceci est particulirement frappant dans le cas des signaux bidimensionnels, pour lesquels on a un rsultat identique celui de la proposition 4.8.

Corollary 4.9. Soit (v_{k,l}) = S_{2}((u_{k,l})) la sous-chantillonne d’un facteur 2 d’une image discrte. (on suppose que \frac{N}{2} est pair). Alors (ṽ_{m,n}), la transforme de Fourier discrte de (v_{k,l}), s’crit, pour m, n = − \frac{N}{4} , ..., \frac{N}{4} − 1,

ṽ_{m,n} = ∑_{(ε_{1},ε_{2})∈\{0,1,−1\}} ũ_{m+ε_{1} \frac{N}{2} ,n+ε_{2} \frac{N}{2}} .  (4.9)

“JMMBookOct04” 30/10/2009 page 71

[figure] [figure]

4.1. TRANSFORME DE FOURIER DISCRTE, APPLICATIONS 71

Montrer le corollaire 4.9. Parmi les neuf termes de la somme de droite de (4.9), il y a en gnral quatre termes non nuls: lesquels? Faire un dessin.

Pour illustrer ce rsultat, nous montrons deux exemples d’images sous-chantillonnes aux figures ?? (image synthtique) et ??, exemple où l’apparition de structures priodiques est due la superposition, lors du sous-chantillonnage, des hautes frquences de l’image. La manipulation numrique faire pour crer des effets de moir dans une image est aussi simple que son interprtation est subtile : il suffit de prendre ”un point sur deux” de l’image. L’interprtation de l’opration se fait en Fourier : on a cr de basses frquences parasites en c_{n} qui correspondent au ”repliement” de hautes frquences c_{n+ \frac{N}{2}} . D’o l’apparition de sinusodes qui n’ont rien voir avec le signal original et qui crent des effets de moir.

Le rsultat de la proposition 4.8 se gnralise dans le cas d’un sous-chantillonnage d’ordre plus lev, comme le montre la proposition suivante:

Proposition 4.10. Soit (v_{k}) = S_{p}((u_{k})) (on suppose que N = pM, pour un certain entier M). Alors (ṽ_{k}), la transforme de Fourier discrte de (v_{k}), s’crit, pour k = 1...M − 1,

ṽ_{k} = ∑_{a=−p+1}^{p−1} ũ_{k+ \frac{aN}{p}} .  (4.10)

Dmonstration Appliquer de nouveau le thorme 4.1.3 l’unique polynme trigonomtrique N coefficients qui a pour chantillons les u_{k}. Ce polynme vrifie c_{n}(P) = ũ_{n}. □

On peut comparer les propositions 4.8 et 4.10 et le corollaire 4.9 aux thormes 4.1.3 et 4.1.3. Ces thormes nous donnent les conditions gnrales de Shannon et Whittaker pour qu’un signal ou une image soient correctement chantillonns : ces conditions sont que le spectre soit born (nombre fini N ou N^{2} de coefficients de Fourier) et que l’on dispose d’au moins N (ou N^{2}) chantillons. Les propositions 4.8 et 4.10 sont plus pratiques : elles ne donnent aucune hypothse sur le signal ou l’image qui ont t chantillonns et ont l’avantage de s’appliquer donc un signal ou une image discrets, quelconques, qu’ils soient ou non issus d’un bon chantillonnage.

4.1.4 Manipulations propres et simples et leur visualisation

Corollary 4.11. (Reprsentation d’une image partir de ses coefficients de Fourier) Les ũ_{m,n} tant donns pour m, n = − \frac{N}{2} , …, \frac{N}{2} − 1, on dsire reprsenter une image qui a ces coefficients de Fourier. Pour cela, on fixe le domaine de l’image comme tant [0, b]^{2} et on pose donc

u(x) = ∑_{m∈\{− \frac{N}{2} ,..., \frac{N}{2} −1\}^{2}} ũ_{m}e^{\frac{2iπm.x}{b}} .

On peut donc construire une image ayant des coefficients de Fourier donns sur un domaine carr de largeur arbitraire b, et l’image reconstruite est donc b-priodique. Ses chantillons u( \frac{kb}{N}), k ∈ \{0, …, N − 1\}^{2} redonnent par DFT les ũ_{m}, et inversement ces chantillons sont obtenus par DFT inverse partir des ũ_{m}. Il en rsulte que, les coefficients de Fourier ũ_{m} tant donns l’on doit toujours fixer le domaine de reconstruction en donnant b. Les images reconstruites partir de diffrents b se dduisent l’une de l’autre par simple homothtie.

“JMMBookOct04” 30/10/2009 page 72

[figure] [figure]

72 CHAPTER 4. LE CAS DISCRET

On part d’une image digitale donne par ses chantillons u = (u(k, l)), k, l ∈ [0, N − 1]^{2}. Donc l’image qui apparat sur l’cran est de taille N × N. Si on procde par exemple un sous-chantillonnage de deux: v := S_{2}u est une image avec quatre fois moins d’chantillons, et donc de dimensions deux fois plus petites sur l’cran. Pour visualiser v et u la mme taille, on procde par zero-padding. Soit v l’unique polynme trigonomtrique N-priodique et de degr N/2 passant par les chantillons pairs de u. On a donc v = S_{2}v. Les coefficients ṽ_{m} de v sont obtenus par la formule d’aliasage ?? partir de ceux de u. Ces coefficients sont en nombre (\frac{N}{2} )^{2}. Donc pour comparer visuellement l’image sous-chantillonne v u, il faut zoomer v pour en montrer autant d’chantillons. Pour cela on procde par zro-padding, en posant ṽ_{m} = 0 pour m ∈ \{−\frac{N}{2} , …, \frac{N}{2} −1\}^{2}\\{−\frac{N}{4} , …, \frac{N}{4} −1\}^{2}. Alors on voit autant d’chantillons (N^{2}) pour v que pour u et on peut donc les comparer sur un cran la mme taille. On notera ZP_{M} l’opration qui consiste border de zros un spectre de manire passer d’une largeur N une largeur M +N, et donc (M + N)^{2} chantillons.

Pour avoir une notation computationnellement complte, il faut prciser donc quand on fait une DFT le nombre d’chantillons N^{2}. On notera donc DF T_{N} pour la DFT N chantillons si c’est un signal et DF T_{N} galement si c’est la DFT N^{2} chantillons d’une image. Avec ce formalisme simple et qui correspond ce qui est programm, les oprations d’intrt mener sont:

• Partant de N^{2} chantillons u = u(k), faire DF T_{N}^{−1}DF T (u) (simple vrification: on doit retrouver u);

• Faire v = DF T_{N}^{−1}ZP_{\frac{N}{2}} DF T_{\frac{N}{2}} S_{2}u: met en vidence les effets nfastes de l’aliasage;

• Faire w = DF T_{N}^{−1}ZP_{\frac{N}{2}} (DF T_{N}u)_{|\{− \frac{N}{4} ,... \frac{N}{4} −1\}}). En d’autres termes on fait les choses correctement: on annule les coefficients de Fourier plus grands que N/4, ce qui autorise un sous-chantillonnage de 2.

• Comparer v et w

4.1.5 L’quation de la chaleur

Consider a digital image u(k) with k = (k, l) ∈ [0, N − 1]^{2} and its DFT interpolate

u(x) =  ∑_{m=(m,n)∈[−N/2,N/2−1]^{2}} ũ_{m}e^{\frac{2iπm.x}{N}} ,  (4.11)

where ũ_{m} with m = (m, n) are the Discrete Fourier Transform (DFT) coefficients of the N^{2} samples u(k). The scale space convolves u with a Gaussian

G_{σ}(x) = \frac{1}{2πσ^{2}} e^{− \frac{m^{2}}{2σ^{2}}} . A direct calculation shows that the result of the convolution of u with G_{σ} is

v(x) := (G_{σ} ∗ u)(x) = ∑_{m∈[− \frac{N}{2} , \frac{N}{2} −1]^{2}} ũ_{m}Ĝ_{σ} (\frac{2mπ}{N} ) e^{\frac{2iπm.x}{N}}  (4.12)

with Ĝ_{σ}(ξ) = e^{\frac{−σ^{2}ξ^{2}}{2}} . Indeed,

G_{σ}∗e^{\frac{2iπm.x}{N}} = ∫_{IR^{2}} G_{σ}(y)e^{\frac{2iπm.(x−y)}{N}} dy = e^{\frac{2iπm.x}{N}} ∫_{IR^{2}} G_{σ}(y)e^{− \frac{2iπm.y}{N}} dy = e^{\frac{2iπm.x}{N}} Ĝ_{σ}(\frac{2πm}{N}).

“JMMBookOct04” 30/10/2009 page 73

[figure] [figure]

4.1. TRANSFORME DE FOURIER DISCRTE, APPLICATIONS 73

The convolved digital image is

v(k) = ∑_{m∈[− \frac{N}{2} , \frac{N}{2} −1]^{2}} ṽ_{m}e^{\frac{2iπm.k}{N}} .  (4.13)

Thus, the algorithm giving the exact solution u(σ, x) of the heat equation starting from an initial trigonometric polynomial u_{0}(x) = u(0, x) will be a simple formula: Denote by u_{0} the N^{2} samples of u_{0} and ũ_{m} = DF T_{N}(u_{0}) its DFT coefficients. Call u(σ) the N^{2} samples of u(σ, x). Then

u(σ) = DF T_{N}^{−1}((Ĝ_{σ}(\frac{2m.x}{N})ũ_{m})_{m∈\{− \frac{N}{2} ,... \frac{N}{2} −1\}}).

We denote this discrete filter by G_{σ}: u(σ) = G_{σ}u_{0}.

Finally we can make the experiment checking on what kind of Gaussian convolution is enough to make a subsampling. Trying t = 2, 4, 8 and taking an initial digital image u_{0} with N^{2} samples, we just have to compare for several values of c the digital images

u_{1} = S_{1}G_{tc}u and u_{2} = S_{\frac{1}{t}} S_{t}G_{tc}u_{0}.

Notice that S_{\frac{1}{t}} is an oversampling, obtained therefore by zero padding. More precisely,

S_{\frac{1}{t}} =: DF T_{N}^{−1}ZP_{N(1− \frac{1}{t})}.

Thus it is not true in general that S_{\frac{1}{t}} S_{t} is the identity. The difference between u_{1} and u_{2} can be visualized, but can also be reliably computed as the root mean square error per pixel (RMSE). If the RMSE is lower than 1 or 2, the visual difference is tiny, and if the RMSE is lower than, say, 0.1, it becomes outright negligible.

4.1.6 L’utilisation de la transforme de Fourier discrte pour dfinir zoom, translations et rotations des images

Le zoom Nous prsentons une mthode d’interpolation reposant sur une extension de la TFD d’un signal ou d’une image. Nous dtaillons la mthode, dite du “prolongement par des zros (“zro-padding”), en une dimension, le principe se gnralisant sans mal pour une image. Comme prcdemment, considrons des chantillons u_{k}, k variant de 0 N−1, et ũ_{n} = \frac{1}{N} ∑_{k=0}^{N−1} u_{k}ω_{N}^{−kn}. On suppose que N est pair et que l’on veut zoomer d’un facteur 2, c’est dire que l’on veut construire un signal de taille deux fois plus grande (avec deux fois plus d’chantillons) que le signal de dpart. On dfinit un nouveau signal v, de taille 2N comme tant la TFD inverse de ṽ, donn par

ṽ_{n} = ũ_{n} si −\frac{N}{2} ≤ n ≤ \frac{N}{2} −1, ṽ_{n} = 0 si n ∈ [−N, −\frac{N}{2} −1]∪[\frac{N}{2} , N−1]. (4.14)

Proposition 4.12. zoom discret par Fourier Le signal v dont la TFD est donne par la formule (4.14) vrifie v_{2k} = u_{k}, pour k = 0, ..., N −1.

“JMMBookOct04” 30/10/2009 page 74

[figure] [figure]

74 CHAPTER 4. LE CAS DISCRET

Dmonstration On a

v_{2k} = ∑_{−N}^{N−1} ṽ_{n}ω_{2N}^{2nk} = ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n}ω_{N}^{nk} = u_{k}.

En effet, ω_{2N}^{2nk} = ω_{N}^{nk}.

Remark 4.13. Ce rsultat est vident sans dmonstration : en effet, on peut considrer l’unique polynme trigonomtrique de degr \frac{N}{2} passant par les chantillons u_{k}. Les chantillons v_{k} s’interprtent immdiatement comme des chantillons de ce mme polynme. Remark 4.14. On remarquera que les nouveaux chantillons obtenus par cette mthode peuvent être complexes, même lorsque le signal original est rel (ceci tant dû au terme d’aliasage u_{− \frac{N}{2}} ). Aussi, on ne retient que la partie relle des effets de visualisation.

La mthode se gnralise aux cas des images. Nous considrons une image numrique (u_{k,l}), et nous dfinissons une image zoome (v_{i,j})_{i,j=0, ..., 2N−1} comme tant la transforme de Fourier discrte inverse de ṽ_{i,j} dfinie pour i, j = −N, ..., N −1 par

ṽ_{m,n} = ũ_{m,n} si − \frac{N}{2} ≤ m, n ≤ \frac{N}{2} − 1, ṽ_{m,n} = 0 sinon.  (4.15)

Ecrire l’quivalent de la proposition 5.4 et la montrer en dimension 2. La figure ?? montre la partie relle d’une partie de l’image ?? zoome par zro-padding, et la compare celle obtenue par rplication des pixels (chaque pixel est remplac par quatre pixels de la même valeur). On remarque que le zoom par TFD produit une image bien plus rgulire, et vite l’effet “marche d’escalier” visible sur l’image zoome par rplication. La figure ?? illustre ce point sur un dtail. Une autre remarque concerne l’effet de Gibbs (cf. paragraphe ??). Ce phnomne produit des rebonds le long de la frontire du domaine de l’image. En effet, et comme nous l’avons dj mentionn, le calcul des coefficients de Fourier de l’image (dont les coefficients de la TFD sont une approximation) suppose l’image priodique, ce qui fait apparatre des discontinuits le long des frontires de son domaine de dfinition. Le phnomne de Gibbs est galement visible le long des discontinuits dans l’image, les contours. Le phnomne est mis en vidence sur la figure ??. Expliquons pourquoi le phnomne apparaît dans le cas du zoom: une nouvelle image v_{k,l} de taille 2N × 2N est obtenue en utilisant les valeurs prises par le polynôme P (x) entre les points dont on dispose au dpart. Cette utilisation de P fait apparaître des oscillations du polynme trigonomtrique qui taient invisibles sur les chantillons de dpart (u_{k,l}). Comme nous l’avons dj voqu, les oscillations aux frontires du domaine de l’image peuvent être supprimes par utilisation de la transforme en cosinus. En revanche, le problme subsistera le long des discontinuits prsentes l’intrieur de l’image, moins de faire un “filtre anti-aliasage”. Ce filtre consiste attnuer les frquences prs du bord, pour que leur transition zro soit moins brutale.

La translation La mthode prsente au paragraphe prcdent permet de dfinir une translation d’une quantit 1/2 (ou a/(2N) pour revenir notre dfinition premire du signal u), en ne gardant que les points d’indice impair du signal zoom v. Plus gnralement, nous pouvons dfinir une translation d’un signal d’une quantit 0 < α < 1. Comme d’habitude, l’opration de translation sur la fonction u dont nous connaissons les chantillons u_{k} se fait sous l’hypothse que celle-ci

“JMMBookOct04” 30/10/2009 page 75

[figure] [figure]

4.1. TRANSFORME DE FOURIER DISCRTE, APPLICATIONS 75

est un polynme trigonomtrique. En d’autres termes, on translate le polynme d’interpolation, la ”vraie” fonction u tant inconnue en dehors des chantillons. Le polynme d’interpolation est

P (x) = ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n}e^{\frac{2iπnx}{a}} .

En translatant de α, on obtient

τ_{α}P (x) = P (x − α) = ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n}e^{− \frac{2iπnα}{a}} e^{\frac{2iπnx}{a}} .

On a donc :

Proposition 4.15. La TFD (ṽ_{n}) de P (x − α) s’obtient partir de la TFD de P (x), ũ_{n}, par

ṽ_{n} = ũ_{n}e^{− \frac{2iπnα}{a}} .

Cette mthode de translation se gnralise sans mal au cas des images, en remarquant qu’une translation deux dimensions peut se dcomposer en deux translations, une selon les lignes et une selon les colonnes.

Donner les formules de la translation par TFD en dimension 2.

La rotation Dcrivons maintenant une mthode pour implmenter une rotation discrte, due L. Yaroslavsky. En bref, cette mthode rduit une rotation des translations en ligne ou en colonne de l’image. Commençons par remarquer que

R(−θ) := ( \begin{matrix}cos(θ) & sin(θ) \\ − sin(θ) & cos(θ) \end{matrix} )=( \begin{matrix}1 & tan( \frac{θ}{2} ) \\ 0 & 1 \end{matrix} )( \begin{matrix}1 & 0 \\ − sin(θ) & 1 \end{matrix} )( \begin{matrix}1 & tan( \frac{θ}{2} ) \\ 0 & 1 \end{matrix} ):= T (θ)S(θ)T (θ)

(4.16)

(sauf si θ = π auquel cas il suffit de retourner l’image).

Une rotation d’angle θ de l’image discrte u(i, j) consiste calculer u(R(−θ)(i, j)) que l’on notera (R(−θ)u)(i, j). Mais on a R(−θ)u = T (θ)S(θ)T (θ)u. Donc il suffit d’expliquer comment calculer T (θ)u et S(θ)u. Or ces deux oprations ont la mme structure, savoir une translation ligne par ligne ou une translation colonne par colonne. Traitons par exemple le cas de T (θ). On a (T (θ)u)(i, j) = u(i + j tan(\frac{θ}{2}), j). Donc partant de la matrice u_{i,j}, on translate sa premire ligne de tan(\frac{θ}{2}), la deuxime de 2 tan(\frac{θ}{2}), etc. Appliquer S(θ) revient faire une opration similaire sur les colonnes. Enfin on rapplique T (θ) et on fait donc nouveau une translation sur les lignes. Or comme on vient de le voir ces translations ligne ligne ou colonne colonne se font en temps N log N en utilisant la TFD une dimension.

La figure ?? montre une image aprs une rotation de π/4 par la mthode dcrite ci-dessus. Puis, pour illustrer la stabilit de la mthode, nous montrons figure ?? le rsultat de l’application successive de douze rotations de π/4, et, titre de comparaison, le rsultat de ces douze rotations successives implments par interpolation bilinaire (les valeurs aux nouveaux points sont des combinaisons linaires des quatre points coordonnes entires les plus proches). Cette figure illustre clairement la supriorit de la mthode par FFT dans le cas de rotations multiples.

“JMMBookOct04” 30/10/2009 page 76

[figure] [figure]

76 CHAPTER 4. LE CAS DISCRET

Remark 4.16. Du fait que l’on manipule des fonctions priodiques, une translation conduirait faire sortir une partie de l’image par un bord pour la faire entrer par l’autre. Ce qui conduirait l’apparition, sur les bords de l’image d’un certain nombre de dtails qui sont en fait mal placs. On se dbarrasse facilement de ce problme en insrant l’image dans un cadre deux fois plus grand…

Remark 4.17. La mthode de rotation n’est pas parfaite. En effet, l’image u continue associe u(i, j) est dans l’interpolation suppose implicitement N-priodique, ce qui revient dire qu’elle est de la forme (pour une image carre)

u(x, y) = ∑_{k,l=0}^{N−1} c_{i,j}e^{2i \frac{π}{N} (kx+ly)}.

Mais, si on lui applique une ”translation” suivant l’axe des x de valeur λy, la formule devient

u_{1}(x, y) = ∑_{k,l=0}^{N−1} c_{i,j}e^{2i \frac{π}{N} (kx+(l−λk)y)}.

La fonction u_{1} n’est pas (pour λ ∉ ℤ) N-priodique en y. Or, aprs la premire translation on ne dispose plus que des chantillons du signal u_{1} sur une grille carre N × N. D’aprs la thorie de Shannon un tel ensemble de donnes ne permet pas de capturer toute l’information sur u_{1} ( la seconde tape on effectue des translations suivant y qui est justement l’axe qui pose problme). On rencontre encore ce problme la troisime translation. Le seul moyen d’avoir une rotation exacte serait d’valuer u aux points de l’image de [0, N −1] × [0, N −1] par une rotation d’angle −θ, mais cette mthode est en N^{4} ce qui la rend inoprante…

“JMMBookOct04” 30/10/2009 page 77

[figure] [figure]

Chapter 5

From continuous to digital images, and back

Consider a continuous and bounded image u(x) defined for every x = (x, y) ∈ ℝ^{2}. All continuous image operators including the sampling will be written in capital letters A, B and their composition as a mere juxtaposition AB. For any affine map A of the plane consider the affine transform of a continuous image u defined by Au(x) =: u(Ax). For instance H_{λ}u(x) =: u(λx) denotes an expansion of u by a factor λ^{−1}. In the same way if R is a rotation, Ru =: u ∘ R is the image rotation by R^{−1}.

Sampling and interpolation

Digital images, only defined for (n_{1}, n_{2}) ∈ IZ^{2}, will be denoted by u(n_{1}, n_{2}). The δ-sampled image u = S_{δ}u is defined on IZ^{2} by

u(n_{1}, n_{2}) = (S_{δ})u(n_{1}, n_{2}) =: u(n_{1}δ, n_{2}δ);  (5.1)

Conversely, the Shannon interpolate of a digital image u is defined as follows [91]. Let u be a digital image, defined on IZ^{2} and such that ∑_{n∈I Z^{2}} |u(n)|^{2} < ∞ and ∑_{n∈I Z^{2}} |u(n)| < ∞. (Of course, these conditions are automatically satisfied if the digital has a finite number of non-zero samples, which is the case here.) We call Shannon interpolate Iu of u the only L^{2}(ℝ^{2}) function u having u as samples and with spectrum support contained in (−π, π)^{2}. u = Iu is defined by the Shannon-Whittaker formula

Iu(x, y) =: ∑_{(n_{1},n_{2})∈I Z^{2}} u(n_{1}, n_{2})sinc(x − n_{1})sinc(y − n_{2}),

where sinc x =: \frac{sin πx}{πx}. The Shannon interpolation has the fundamental property S_{1}Iu = u. Conversely, if u is L^{2} continuous image, band-limited in (−π, π)^{2}, then

IS_{1}u = u.  (5.2)

In that case we simply say that u is band-limited. We shall also say that a digital image u = S_{1}u is well-sampled if it was obtained from a band-limited image u.

77 “JMMBookOct04” 30/10/2009 page 78

[figure] [figure]

78CHAPTER 5. FROM CONTINUOUS TO DIGITAL IMAGES, AND BACK

5.0.7 The practical Shannon interpolation: zero-padding

Of course, the Shannon interpolate is unpractical in that it assumes the knowledge of infinitely many samples. In practice image samples and image interpolation will be performed on rectangle. For a sake of simplicity we describe here what happens on a square. Let a > 0 and consider a function u from [0, a]^{2} to IR such that u(x + a, y + a) = u(x, y). Fix an integer N, and consider the N^{2} samples of u, u_{k,l} = (Su) (\frac{ka}{N}, \frac{la}{N}) on [0, a]^{2}.

Definition 5.1. The discrete Fourier transform (DFT) of the N^{2} samples u = (u_{k,l})_{k,l=0,1,...,N−1} is the double sequence of discrete Fourier coefficients for m, n ∈ \{−\frac{N}{2} , ..., \frac{N}{2} − 1\} defined by

DF T (u)_{m,n} = ũ_{m,n} = \frac{1}{N^{2}} ∑_{k=0}^{N−1} ∑_{l=0}^{N−1} u_{k,l}ω_{N}^{−mk}ω_{N}^{−nl},  (5.3)

where ω_{N} =: e^{\frac{2iπ}{N}} is the first N-root of 1.

Proposition 5.2. Consider the trigonometric polynomial

Iu(x, y) = ∑_{m,n=− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{m,n} exp(\frac{2iπmx}{a}) exp(\frac{2iπny}{a}).  (5.4)

Then its coefficients ũ_{m,n} are the only complex numbers such that for every k, l ∈ \{0, ..., N−1\}, Iu (\frac{ka}{N}, \frac{la}{N}) = u (\frac{ka}{N}, \frac{la}{N}). In consequence, the discrete inverse transform of the DFT u → ũ is nothing but the calculation of the value of the polynomial at the samples (\frac{ka}{N}, \frac{la}{N}), 0 ≤ k, l ≤ N −1. In other terms, setting ũ = DF T (u), the inverse transform DF T^{−1} is given by

u(k, l) = ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{m,n}ω_{N}^{km+ln},

for every k, l = 0, 1, … N − 1.

Exercise 5.1. Recall that ω_{N} = exp ( \frac{2iπ}{N} ), N-th root of 1. Show that ∑_{k=0}^{N−1} ω_{N}^{k} = 0, and that ∑_{k=0}^{N−1} ω_{N}^{kl} = 0 for l ≠ 0 modulo N and finally that for every k_{0}, ∑_{k=k_{0}}^{k_{0}+N−1} ω_{N}^{kl} = 0 for all l ≠ 0 modulo N. Using these relation show the above proposition, namely that DF T (DF T^{−1}) = Id.

In conclusion, the interpolation and sampling operators we shall consider both in theory and practice are the usual sampling S, implicitly restricted to a square. The inverse interpolation operator I is defined by (5.4), and Proposition 5.2 tells us that SIu = u. The next statement gives the converse statement.

Proposition 5.3. If u(x, y) is a a-periodic band-limited function, then it is a trigonometric polynomial. If its highest degree is \frac{N}{2} − 1, with N even, then its coefficients ũ_{m,n} are obtained by DFT from the samples u(k, l) = u(\frac{ka}{N}, \frac{la}{N}). In consequence for such functions we have ISu = u.

Exercise 5.2. Give a detailed proof of Proposition 5.3. It is a direct consequence of Proposition 5.2.

In the rest of this chapter and of the book, we shall always take the functional setting of Proposition 5.3.

“JMMBookOct04” 30/10/2009 page 79

[figure] [figure]

79

Zoom in by zero-padding

Let (u_{k,l}) be a digital image and define its zoomed version (v_{i,j})_{i,j=0, ..., 2N−1} as the inverse discrete Fourier transform of ṽ_{i,j} defined for i, j = −N, ..., N−1 by

ṽ_{m,n} = ũ_{m,n} if − \frac{N}{2} ≤ m, n ≤ \frac{N}{2} − 1, ṽ_{m,n} = 0 otherwise.  (5.5)

Proposition 5.4. The image v whose Discrete Fourier Transform is given by (5.5) satisfies v_{2k,2l} = u_{k,l}, for k, l = 0, ..., N −1.

Proof. Here is the proof in dimension 1:

v_{2k} = ∑_{−N}^{N−1} ṽ_{n}ω_{2N}^{2nk} = ∑_{− \frac{N}{2}}^{\frac{N}{2} −1} ũ_{n}ω_{N}^{nk} = u_{k}.

Indeed, ω_{2N}^{2nk} = ω_{N}^{nk}. □

Exercise 5.3. Prove Proposition (5.4) in two dimensions.

5.0.8 Shannon or zero-padding?

In what follows, most of the formal reasoning will be done with the Shannon interpolation of the digital image on IR^{2}. Thus, the operators I and S_{1} refer to the Shannon framework developed above. However, the practical interpolation is made by the discrete Fourier transform. Thus the practical DFT interpolation from a digital image u is NOT the Shannon interpolation, simply because the extension of the samples outside the square is not the same. In the Shannon-Whittaker interpolation the extension is made with zero samples while in the DFT interpolation, the sampled are periodized and the image becomes a trigonometric polynomial, not even belonging to L^{2}(IR^{2}). We shall therefore denote this second interpolation periodic of a digital image by Ĩu. Of course the values of both interpolations only make sense inside the image domain. Outside, Iu has zero samples and oscillates around the zero value, and Ĩu is simply a periodic repetition of the restriction of Ĩu in the image domain. Thus, it is crucial for the forthcoming reasoning to prove by numerical and theoretical arguments that Iu and ũ are in fact extremely close to each other, so that their difference is actually negligible with respect to other image perturbations such as noise and quantization.

To that effect, an obvious experimental device is to compare Iu and ũ on generic digital images, and to compute their quadratic distance per pixel. Even if Iu cannot be computed exactly, it can be approximated by extending u by zero samples on a white band around the image domain before applying Ĩ to the resulting image. Another more theoretical way is to give a formula for the quadratic distance of Iu and Ĩu in the case where u is a white noise. The white noise is here the most difficult case because it has a flat spectrum, far less regular than any typical digital image.

(...) The above table and formula lead to the conclusion that the difference

between Iu(x) and Ĩu(x) is negligible in general, and still more negligible when “JMMBookOct04” 30/10/2009 page 80

[figure] [figure]

80CHAPTER 5. FROM CONTINUOUS TO DIGITAL IMAGES, AND BACK

x stands at a reasonable and fixed distance from the image boundary. Thus, in the sequel, we shall reason with operators I and Ĩ depending on what kind of property for image operators we have in mind. If for example geometric invariance properties are at stake, the operator I will be preferred.

5.1 The Gaussian semigroup

For a sake of simple notation, G_{σ} denotes the convolution operator on ℝ^{2}

with the gauss kernel G_{σ}(x_{1}, x_{2}) = \frac{1}{2π(σ)^{2}} e^{− \frac{x^{2}_{1}+x^{2}_{2}}{2(σ)^{2}}} , namely G_{σ}u(x, y) =: (G_{σ} ∗ u)(x, y). Notice that the paramaterization of the gaussian is not the same as the parameterization used for the heat equation. To make a difference in notation, we use Greek letters for the new parameter. It is easily checked that G_{σ} satisfies the semigroup property

G_{σ}G_{β} = G\sqrt{σ^{2}+β^{2}}.  (5.6)

Exercise 5.4. Prove (10.1).

The proof of the next formula is a mere change of variables in the integral defining the convolution.

G_{σ}H_{γ}u = H_{γ}G_{σγ}u.  (5.7)

Exercise 5.5. Prove (10.2).

Discrete Gaussians

Many algorithms in computer vision and image processing assume that all blurs can be assumed gaussian. Thus, it will be crucial to prove that gaussian blur gives in practice well-sampled images. Thus, in all that follows, we are dealing with initial digital images obtained by sampling a continuous image with gaussian blur, u = S_{1}G_{c}u_{0};

Another question we need to deal with is how to perform a gaussian convolution on a discrete (digital) image. This is valid if and only if a discrete convolution can give an account of the underlying continuous one.

Definition 5.5. The discrete gaussian convolution applied to a digital image u is defined as a digital operator by

G_{δ}u =: S_{1}G_{δ}Iu.  (5.8)

Proposition 5.6. This definition maintains the gaussian semi-group property,

G_{δ}G_{β} = G\sqrt{δ^{2}+β^{2}}.  (5.9)

Proof Indeed, using twice (5.8) and once (10.1) and (5.2),

G_{δ}G_{β}u = S_{1}G_{δ}IS_{1}G_{β}Iu = S_{1}G_{δ}G_{β}Iu = S_{1}G\sqrt{δ^{2}+β^{2}}Iu = G\sqrt{δ^{2}+β^{2}}u.

□

The SIFT method that we will study in detail uses repeatedly the semigroup formula and a 2-sub-sampling of images with a gaussian blur larger than “JMMBookOct04” 30/10/2009 page 81

[figure] [figure]

5.2. THE RIGHT GAUSSIAN BLUR FOR WELL-SAMPLING 81

1.6. These SIFT sampling manoeuvres are valid if and only if the empirical proposition below is true.

Proposition 5.7. For every σ larger than 0.8 and every continuous and bounded image u_{0}, the gaussian blurred image G_{σ}u_{0} is well sampled, namely IS_{1}G_{σ}u_{0} = G_{σ}u_{0}.

This proposition is not a mathematical statement, but it will be checked experimentally in the next section, where we shall see that a 0.6 blur is enough to ensure good sampling in practice.

5.2 The right gaussian blur for well-sampling

Images need to be blurred before they are sampled. In principle gaussian blur cannot lead to a good sampling because it is not stricto sensu band limited. Therefore the Shannon-Whittaker formula does not apply. However, in practice it does. The aim in this section is to define a procedure that checks that a gaussian blur works and to fix the minimal variance of the blur ensuring wellsampling (up to a minor mean square and visual error).

One must distinguish two types of blur: The absolute blur with standard deviation c_{a} is the one that must be applied to an ideal infinite resolution (blur free) image to create an approximately band-limited image before 1-sampling. The relative blur σ = c_{r}(t) is the one that must be applied to a well-sampled image before a sub-sampling by a t factor. In the case of gaussian blur, because of the semi-group formula (10.1), the relation between the absolute and relative blur is

t^{2}c_{a}^{2} = c_{r}^{2}(t) + c_{a}^{2},

which yields

c_{r}(t) = c_{a}\sqrt{t^{2} − 1}.  (5.10)

In consequence, if t ≫ 1, then c_{r}(t) ≈ c_{a}t.

Two experiments have been designed to calculate the anti-aliasing absolute gaussian blur c_{a} ensuring that an image is approximately well-sampled. The first experiment compares for several values of c_{r}(t) the digital images

u_{1} =: S_{1}G_{c_{r}(t)}Iu and
u_{2} =: (S_{1/t}I)S_{t}G_{c_{r}(t)}Iu,

where u is an initial digital image that is (intuitively) well-sampled, S_{t} is a t subsampling operator, S_{\frac{1}{t}} a t over-sampling operator, and I a Shannon-Whitakker interpolation operator. The discrete convolution by a gaussian is defined in (5.8). Since t is an integer, the t sub-sampling is trivial.

If the anti-aliasing filter size c_{r}(t) is too small, u_{1} and u_{2} can be very different. The right value of c_{r}(t) should be the smallest value permitting u_{1} ≈ u_{2}. Fig. 5.1 shows u_{1} and u_{2} with t = 4 and plots their mean square error MSE(u_{1}, u_{2}). An anti-aliasing filter with c_{r}(4) = 1.2 is clearly not broad enough: u_{2} presents strong ringing artifacts. The ringing artifact is instead hardly noticeable with c_{r}(4) = 2.4. The value c_{r}(4) ≃ 2.4 is a good visual candidate, and this choice is confirmed by the curve showing that MSE(u_{1}, u_{2})

“JMMBookOct04” 30/10/2009 page 82

[figure] [figure]

82CHAPTER 5. FROM CONTINUOUS TO DIGITAL IMAGES, AND BACK

Figure 5.1: Top left: u. Top right: MSE(u_{1}, u_{2}) vs c_{r}(4). Middle (from left to right): u_{1} and u_{2} with c_{r}(4) = 1.2. MSE(u_{1}, u_{2})=17.5. Bottom (from left to right): u_{1} and u_{2} with c_{r}(4) = 2.4. MSE(u_{1}, u_{2})=0.33. Digital images are always displayed by coloring each square pixel with its central sample value.

“JMMBookOct04” 30/10/2009 page 83

[figure] [figure]

5.2. THE RIGHT GAUSSIAN BLUR FOR WELL-SAMPLING 83

decays rapidly until c_{r}(4) gets close to 2.4, and is stable and small thereafter. By (5.10), this value of c_{r} yields c_{a} = 0.62. This value has been confirmed by experiments on ten digital images. A doubt can be cast on this experiment, however: Its result slightly depends on the assumption that the initial blur on u is equal to c_{a}.

In a second experiment, c_{a} has been evaluated directly by using a binary image u_{0} that does not contain any blur. As illustrated in Fig. 5.2, u_{0} is obtained by binarizing the digital test image Lena (Fig. 5.1), the threshold being the median value of Lena. Since u_{0} is now blur-free, we can compare for several values of c_{a} and for t = 4, which is large enough, the digital images

u_{1} =: G_{tc_{a}}Iu = S_{1}G_{tc_{a}}Iu and u_{2} =: (S_{1/t}I)S_{t}G_{tc_{a}}Iu.

As shown in Fig. 5.2, c_{a} = 0.6 is the smallest value ensuring no visual ringing in u_{2}. Under this value, for example for c_{a} = 0.3, clear ringing artifacts are present in u_{2}. That c_{a} = 0.6 is the correct value is confirmed by the MSE(u_{1}, u_{2}) curve showing that the mean square error decays rapidly until c_{a} goes down to 0.6, and is stable and small thereafter. The result, confirmed in ten experiments with different initial images, is consistent with the value obtained in the first experimental setting.

5.2.1 Discrete sampling

If u is digital and therefore only defined on IZ^{2} and if δ is an integer, then one can define any sub- or over-sampling operations on u. But this requires interpolating u first.

Definition 5.8. Thus we define a digital re-sampling operator by

S_{δ}u =: S_{δ}Iu.  (5.11)

S_{δ} is a discrete filter. If δ < 1 S_{δ} is an over-sampling, and it is invertible. If δ > 1 it is an sub-sampling, and may be not invertible.

Exercise 5.6. Show that if δ < 1, then S_{δ^{−1}}S_{δ} = Id. What can happen if δ > 1?

Over-sampling can be interpreted as a zoom in. A zoom in is not the same as a blow up. Blow up is a photographic term involving the use of a system of lenses increasing the image resolution : it permits to see more details of the observed object. Zoom in is instead is a digital term. Being just an interpolation, it adds no detail to the image. The next proposition confirms that it is just an image enlargement.

Proposition 5.9. For every γ ≤ 1,

IS_{γ}u = H_{γ}Iu.  (5.12)

Proof. Iu is well sampled, with spectrum in [0, 2π]^{2}. Thus since γ < 1, IS_{γ}u is over-sampled: it has spectrum in [0, 2πγ]^{2}. Thus IS_{γ}u is band limited, as H_{γ}Iu. Since

IS_{γ}u(n_{1}, n_{2}) = S_{γ}u(n_{1}, n_{2}) = Iu(n_{1}γ, n_{2}γ) and H_{γ}Iu(n_{1}, n_{2}) = Iu(n_{1}γ, n_{2}γ),

both functions have the same IZ^{2} samples and therefore coincide. □

“JMMBookOct04” 30/10/2009 page 84

[figure] [figure]

84CHAPTER 5. FROM CONTINUOUS TO DIGITAL IMAGES, AND BACK

Figure 5.2: Top left: u. Top right: MSE(u_{1}, u_{2}) vs c_{a}. Middle (from left to right): u_{1} and u_{2} with c_{a} = 0.3. MSE(u_{1}, u_{2})=7.46. Bottom (from left to right): u_{1} and u_{2} with c_{a} = 0.6. MSE(u_{1}, u_{2})=0.09.

“JMMBookOct04” 30/10/2009 page 85

[figure] [figure]

5.2. THE RIGHT GAUSSIAN BLUR FOR WELL-SAMPLING 85

Corollary 5.10. If γ ≤ 1, then

S_{β}S_{γ} = S_{βγ}.  (5.13)

Proof. using once (5.12) and twice (5.11),

S_{β}S_{γ}u = S_{β}IS_{γ}u = S_{β}H_{γ}Iu = S_{βγ}Iu = S_{βγ}u.
□

Proposition 5.11. A discrete commutation formula : Assume u is a digital image. Then for γ < 1,

G_{β}S_{γ}u = S_{γ}G_{βγ}u.  (5.14)

Proof.

G_{β}S_{γ}u =^{(5.8)} S_{1}(G_{β}IS_{γ}u) =^{(5.12)} S_{1}(G_{β}H_{γ}Iu)

=^{(10.2)} S_{1}(H_{γ}(G_{βγ}Iu)) =^{(5.1)} S_{γ}(G_{βγ}Iu) =^{(5.2)} S_{γ}IS_{1}(G_{βγ}Iu) =^{(5.8,5.11)} S_{γ}G_{βγ}u.

Notice that we use IS_{1}u = u with u = G_{βγ}Iu. Indeed, this last function is well sampled, because Iu is. □

“JMMBookOct04” 30/10/2009 page 86

[figure] [figure]

“JMMBookOct04” 30/10/2009 page 87

[figure] [figure]

Chapter 6

The SIFT Method

This chapter is devoted to Lowe’s Scale-Invariant Feature Transform (SIFT [151]), a very efficient image comparison method. The initial goal of the SIFT method is to compare two images (or two image parts) that can be deduced from each other (or from a common one) by a rotation, a translation, and a zoom. The method turned out to be also robust to large enough changes in view point angle, which explains its success. This method uses as fundamental tool the heat equation or, in other terms, the linear scale space. The heat equation is used to simulate all zooms out of both images that have to be compared. Indeed, these images may contain similar objects taken at different distances. But at least two of the simulated zoomed in images should contain these objects at the same apparent distance. This is the principal ingredient of the SIFT method, but other invariance requirements must be addressed as well.

Sect. 6.2 gives a detailed description of the SIFT shape encoding method. Sect. 6.4 proves mathematically that the SIFT method indeed computes translation, rotation and scale invariants. This proof is correct under the main assumption that image blur can be assumed to be gaussian, and that images with a gaussian blur larger than 0.6 (SIFT takes 0.8) are approximately (but accurately) well-sampled and can therefore be interpolated. Chapter. 5.2 checked the validity of this crucial gaussian blur assumption.

6.1 Introduction

Image comparison is a fundamental step in many computer vision and image processing applications. A typical image matching method first detects points of interest, then selects a region around each point, and finally associates with each region a descriptor. Correspondences between two images may then be established by matching the descriptors of both images.

In the SIFT method, stable points of interest are supposed to lie at extrema of the Laplacian of the image in the image scale-space representation. The scale-space representation introduces a smoothing parameter σ. Images u_{0} are smoothed at several scales to obtain w(σ, x, y) =: (G_{σ} ∗ u_{0})(x, y), where we use the parameterization of the gaussian by its standard deviation σ,

G_{σ}(x, y) = G(σ, x, y) = \frac{1}{2πσ^{2}} e^{−(x^{2}+y^{2})/2σ^{2}}.

87 “JMMBookOct04” 30/10/2009 page 88

[figure] [figure]

88 CHAPTER 6. THE SIFT METHOD

Figure 6.1: A result of the SIFT method, using an outliers elimination method [206]. Pairs of matching points are connected by segments.

Taking apart all sampling issues and several thresholds whose aim it is to eliminate unreliable features, the whole method can be summarized in one single sentence:

One sentence description The SIFT method computes scale-space extrema (σ_{i}, x_{i}, y_{i}) of the space Laplacian of w(σ, x, y), and then samples for each one of these extrema a square image patch whose origin is (x_{i}, y_{i}), whose x-direction is one of the dominant gradients around (x_{i}, y_{i}), and whose sampling rate is \sqrt{σ_{i}^{2} + c^{2}}.

The constant c ≃ 0.8 is the tentative standard deviation of the image blur. The resulting samples of the digital patch at scale σ_{i} are encoded by their gradient direction, which is invariant under nondecreasing contrast changes. This accounts for the robustness of the method to illumination changes. In addition, only local histograms of the direction of the gradient are kept, which accounts for the robustness of the final descriptor to changes of view angle (see Fig. 6.5).

Figs 6.1 and 6.6 show striking examples of the method scale invariance. Lowe claims that 1) his descriptors are invariant with respect to translation, scale and rotation, and that 2) they provide a robust matching across a substantial range of affine distortions, change in 3D viewpoint, addition of noise, and change in illumination. In addition, being local, they are robust to occlusion. Thus they match all requirements for shape recognition algorithms except one: they are not really affine invariant but only robust to moderate affine distortions.

6.2 A Short Guide to SIFT Encoding

The SIFT encoding algorithm consists of four steps: detection of scale-space extrema (Sect. 6.2.1), accurate localization of key points (Sect. 6.2.2), and descriptor construction (Sect. 6.2.3).

“JMMBookOct04” 30/10/2009 page 89

[figure] [figure]

6.2. A SHORT GUIDE TO SIFT ENCODING 89

Figure 6.2: Gaussian pyramid for key points extraction (from [151])

6.2.1 Scale-Space Extrema

Following a classical paradigm, stable points of interest are supposed to lie at extrema of the Laplacian of the image in the image scale-space representation. We recall that the scale-space representation introduces a smoothing parameter σ, the scale, and convolves the image with Gaussian functions of increasing standard deviation σ. By a classical approximation inspired from psychophysics [164], the Laplacian of the Gaussian is replaced by a Difference of Gaussians at different scales (DOG). Extrema of the Laplacian are then replaced by extrema of DOG functions: 𝔻(σ, x, y) = w(kσ, x, y) − w(σ, x, y), where k is a constant multiplicative factor. Indeed, it is easy to show that 𝔻(σ, x, y) is an approximation of the Laplacian:

𝔻(σ, x, y) ≈ (k − 1)σ^{2}(ΔG_{σ} ∗ u_{0})(x, y).

In the terms of David Lowe:

The factor (k − 1) in the equation is constant over all scales and therefore does not influence extrema location. The approximation error will go to zero as k goes to 1, but in practice we have found that the approximation has almost no impact on the stability of extrema detection or localization for even significant differences in scale, such as k = \sqrt{2}.

To be more specific, quoting Lowe again:

𝔻(σ, x, y) =: (G(kσ, x, y)−G(σ, x, y))∗u_{0}(x, y) = w(kσ, x, y)−w(σ, x, y)

The relationship between D and σ^{2}ΔG can be understood from the heat diffusion equation (parameterized in terms of σ rather than

“JMMBookOct04” 30/10/2009 page 90

[figure] [figure]

90 CHAPTER 6. THE SIFT METHOD

Figure 6.3: Neighborhood for the location of key points (from [151]). Local extrema are detected by comparing each sample point in 𝔻 with its eight neighbors at scale σ and its nine neighbors in the scales above and below.

the more usual t = σ^{2}):

\frac{∂G}{∂σ} = σΔG.

From this, we see that ΔG can be computed from the finite difference approximation to ∂G/∂σ, using the difference of nearby scales at kσ and σ:

σΔG = \frac{∂G}{∂σ} ≈ \frac{G(kσ, x, y) − G(σ, x, y)}{kσ − σ}

and therefore,

G(kσ, x, y) − G(σ, x, y) ≈ (k − 1)σ^{2}ΔG.

This shows that when the difference-of-Gaussian function has scales differing by a constant factor it already incorporates the σ^{2} scale normalization required for the scale-invariant Laplacian.

This leads to an efficient computation of local extrema of 𝔻 by exploring neighborhoods through a Gaussian pyramid ; see Figs. 6.2 and 6.3.

Exercise 6.1. Show that the gaussian G_{σ} parameterized by its standard deviation σ satisfies as stated by Lowe the time-dependent heat equation \frac{∂G}{∂σ} = σΔG.

6.2.2 Accurate Key Point Detection

In order to achieve sub-pixel accuracy, the interest point position is slightly corrected thanks to a quadratic interpolation. Let us call x_{0} =: (σ_{0}, x_{0}, y_{0}) the current detected point in scale space, which is known up to the (rough) sampling accuracy in space and scale. Notice that all points x = (σ, x, y) here are scale-space coordinates. Let us call x_{1} = x_{0} + y the real extremum of the DOG function. The Taylor expansion of 𝔻 yields

𝔻(x_{0} + y) = 𝔻(x_{0}) + (D𝔻) (x_{0}) · y + \frac{1}{2} (D^{2}𝔻) (x_{0})(y, y) + o(∥y∥^{2}),

where 𝔻 and its derivatives are evaluated at an interest point and y denotes an offset from this point. Since interest points are extrema of 𝔻 in scale space, setting the derivative to zero gives:

y = − (D^{2}𝔻(x_{0}))^{−1} (D𝔻(x_{0})) ,  (6.1)

“JMMBookOct04” 30/10/2009 page 91

[figure] [figure]

6.2. A SHORT GUIDE TO SIFT ENCODING 91

which is the sub-pixel correction for a more accurate position of the key point of interest.

Exercise 6.2. Check that (6.1) is a point where the gradient of 𝔻 vanishes.

Since points with low contrast are sensitive to noise, and since points that are poorly localized along an edge are not reliable, a filtering step is called for. Low contrast points are handled through a simple thresholding step. Edge points are swept out following the Harris and Stephen’s interest points paradigm. Let H be the following Hessian matrix:

H = ( \begin{matrix}𝔻_{xx} & 𝔻_{xy} \\ 𝔻_{xy} & 𝔻_{yy} \end{matrix} ) .

The reliability test is simply to assess whether the ratio between the larger eigenvalue and the smaller one is below a threshold r. This amounts to check:

\frac{Tr(H)^{2}}{Det(H)} < \frac{(r + 1)^{2}}{r} .  (6.2)

This rules out standard edge points and puts points of interest at locations which are strong enough extrema, or saddle points.

Exercise 6.3. Explain why (6.2) is equivalent imposing that the ratio between the smaller eigenvalue and the larger eigenvalue of H is smaller than r. These eigenvalues are assumed to have the same sign. Why?

6.2.3 Construction of the SIFT descriptor

In order to extract rotation-invariant patches, an orientation must be assigned to each key point. Lowe proposes to estimate a semi-local average orientation for each key point. From each sample image L_{σ}, gradient magnitude and orientation is pre-computed using a 2 × 2 scheme. An orientation histogram is assigned to each key point by accumulating gradient orientations weighted by 1) the corresponding gradient magnitude and by 2) a Gaussian factor depending on the distance to the considered key point and on the scale. The precision of this histogram is 10 degrees. Peaks simply correspond to dominant directions of local gradients. Key points are created for each peak with similar magnitude, and the assigned orientation is refined by local quadratic interpolation of the histogram values.

Once a scale and an orientation are assigned to each key point, each keypoint is associated a square image patch whose size is proportional to the scale and whose side direction is given by the assigned direction. The next step is to extract from this patch robust information. Gradient samples are accumulated into orientation histograms summarizing the contents over 4 × 4 subregions surrounding the key point of interest. Each of the 16 subregions corresponds to a 8-orientations bins histogram, leading to a 128 element feature for each key point (see Fig. 6.5). Two modifications are made in order to reduce the effects of illumination changes: histogram values are thresholded to reduce importance of large gradients (in order to deal with a strong illumination change such as camera saturation), and feature vectors are normalized to unit length (making them invariant to affine changes in illumination).

“JMMBookOct04” 30/10/2009 page 92

[figure] [figure]

92 CHAPTER 6. THE SIFT METHOD

Figure 6.4: SIFT key points. The arrow starting point, length and the orientation signify respectively the key point position, scale, and dominant orientation. These features are covariant to any image similarity.

6.2.4 Final matching

The outcome is for each image, a few hundreds or thousands SIFT descriptors associated with as many key points. The descriptors of any image can be compared to the descriptors of any other image, or belonging to a database of descriptors built up from many images. The only remaining question is to decide when two descriptors match, or not. In the terms of Lowe again:

The best candidate match for each keypoint is found by identifying its nearest neighbor in the database of keypoints from training images. The nearest neighbor is defined as the keypoint with minimum Euclidean distance for the invariant descriptor vector. However, many features from an image will not have any correct match in the training database because they arise from background clutter or were not detected in the training images. Therefore, it would be useful to have a way to discard features that do not have any good match to the database. A global threshold on distance to the closest feature does not perform well, as some descriptors are much more discriminative than others. A more effective measure is obtained by comparing the distance of the closest neighbor to that of the secondclosest neighbor. (...) This measure performs well because correct matches need to have the closest neighbor significantly closer than the closest incorrect match to achieve reliable matching. For false matches, there will likely be a number of other false matches within similar distances due to the high dimensionality of the feature space. We can think of the second-closest match as providing an estimate of the density of false matches within this portion of the feature space “JMMBookOct04” 30/10/2009 page 93

[figure] [figure]

6.3. IMAGE ACQUISITION MODEL UNDERLYING SIFT 93

Figure 6.5: Example of a 2×2 descriptor array of orientation histograms (right) computed from an 8 × 8 set of samples (left). The orientation histograms are quantized into 8 directions and the length of each arrow corresponds to the magnitude of the histogram entry. (From [151])

and at the same time identifying specific instances of feature ambiguity. (...) For our object recognition implementation, we reject all matches in which the distance ratio is greater than 0.8, which eliminates 90% of the false matches while discarding less than 5% of the correct matches.

6.3 Image acquisition model underlying SIFT

6.3.1 The camera model

We always work on the camera CCD plane, whose mesh unit is taken to be 1. We shall always assume that the camera pixels are indexed by IZ^{2}. The image sampling operator is therefore always S_{1}. Our second assumption is that the digital initial image is well-sampled and obtained by a gaussian kernel. Thus, the digital image is u = S_{1}G_{δ}Au_{0}, where δ ≥ c, c ≃ 0.6 ensures wellsampling (see Chapter 5.2), and A is a similarity with positive determinant. (In fact Lowe’s original paper assumes c ≃ 0.5, which amounts to assume a slight under-sampling of the original image).

Definition 6.1. We model all digital frontal images obtained from a given ideal planar object whose frontal infinite resolution image is u_{0} as

u_{0} =: S_{1}G_{δ}Au_{0}  (6.3)

where δ ≥ c and A is a A = RH_{λ}T is the composition of a translation and of a similarity.

So the possibility of aliasing (under-sampling, δ < c is discarded). Taking into account the way the digital image is blurred and sampled in the SIFT method, we can now list the SIFT assumptions and formalize the method itself. The description is by far simpler if we do it without mixing in sampling issues. We need not mix them in, since the fact that images are well-sampled at all stages permits equivalently to describe all operations with the continuous images “JMMBookOct04” 30/10/2009 page 94

[figure] [figure]

94 CHAPTER 6. THE SIFT METHOD

directly, and to deduce afterwards the discrete operators on samples. We refer to section 5.2.1 for this passage from continuous to discrete operations in the well-sampled world.

6.3.2 Condensed description of the SIFT method

1. There is an underlying infinite resolution bounded planar image u_{0};

2. The initial digital image is S_{1}G_{δ}Au_{0} where δ ≥ c, and A = RH_{λ}T is the composition of a rotation, a zoom, and a translation;

3. the SIFT method computes a sufficient scale-space sampling of u(σ, x) = (G_{σ}G_{δ}Au_{0})(x), and deduces by the Newton method the accurate location or key points defined as extrema in scale-space of the spatial image Laplacian, Δu(σ; x);

4. The blurred u(σ, ·) image is then re-sampled around each characteristic point with sampling mesh \sqrt{σ^{2} + c^{2}};

5. the directions of the sampling axes are fixed by a dominant direction of the gradient of u(σ, ·) in a neighborhood, with size proportional to \sqrt{σ^{2} + c^{2}} around the characteristic point;

6. the rest of the operations in the SIFT method is a contrast invariant encoding of the samples around each characteristic point. It is not needed for the discussion to follow.

6.4 Scale and SIFT: consistency of the method

In this section, in conformity with the SIFT model of Sect. 6.3.2, the digital image is a frontal view of an infinite resolution ideal image u_{0}. In that case, A = HT R is the composition of a homothety H, a translation T and a rotation R. Thus the digital image is u = S_{1}G_{δ}HT Ru_{0}, for some H, T , R as above. Assuming that the image is not aliased boils down, by the experimental results of Sect. 5.2, to assuming δ ≥ 0.8.

Consider T an arbitrary image translation, R an arbitrary image rotation, H_{λ} an arbitrary image homothety, G an arbitrary gaussian convolution, D the gradient and Δ the Laplacian, all applied to continuous images. We say that there is strong commutation of two of these operators if we can exchange the order of their application to any image. We say that there is weak commutation between two of these operators if we can exchange their order by changing one of the parameters of one of the operators. For example we have RT = T′R, meaning that given R and T there is T′ such that the former relation occurs. The next lemma is straightforward.

Lemma 6.2. All of the aforementioned operators weakly commute, with the following exceptions: R and G commute strongly, DH_{λ} = λH_{λ}D, ΔH_{λ} = λ^{2}H_{λ}Δ, and D and Δ do not commute.

Exercise 6.4. Check all of the mentioned commutations and give their exact formula. There are six kinds of operators: translations, rotations, homotheties, gaussian convolutions, gradients, and Laplacians. Thus, there are 15 verifications to make.

“JMMBookOct04” 30/10/2009 page 95

[figure] [figure]

6.4. SCALE AND SIFT: CONSISTENCY OF THE METHOD 95

Lemma 6.3. For any rotation R and any translation T , the SIFT descriptors of S_{1}G_{δ}HT Ru_{0} are identical to those of S_{1}G_{δ}Hu_{0}.

Proof. By the weak commutation of translations and rotations with all other operators (Lemma 6.2), the SIFT descriptors of a rotated or translated image are identical to those of the original. Indeed, the set of scale space Laplacian extrema is covariant to space translations and rotations. The normalization process for each SIFT descriptor situates the origin at each extremum in turn, thus canceling the translation. The local sampling grid defining the SIFT patch has axes given by peaks in its gradient direction histogram. Such peaks are translation invariant and rotation covariant. Thus, the normalization of the direction also cancels the rotation. □

Lemma 6.4. Let u and v be two digital images that are frontal snapshots of the same continuous flat image u_{0}, u = S_{1}G_{β}H_{λ}u_{0} and v =: S_{1}G_{δ}H_{μ}u_{0}, taken at different distances, with different gaussian blurs and possibly different sampling rates. Let w(σ, x) = (G_{σ}u_{0})(x) denote the scale space of u_{0}. Then the scale spaces of u = G_{β}H_{λ}u_{0} and v = G_{δ}H_{μ}u_{0} are

u(σ, x) = w(λ\sqrt{σ^{2} + β^{2}}, λx) and v(σ, x) = w(μ\sqrt{σ^{2} + δ^{2}}, μx).

If (s_{0}, x_{0}) is a key point of w satisfying s_{0} ≥ max(λβ, μδ), then it corresponds to a key point of u at the scale σ_{1} such that λ\sqrt{σ_{1}^{2} + β^{2}} = s_{0}, whose SIFT descriptor is sampled with mesh \sqrt{σ_{1} + c^{2}}. In the same way (s_{0}, x_{0}) corresponds to a key point of v at scale σ_{2} such that s_{0} = μ\sqrt{σ_{2}^{2} + δ^{2}}, whose SIFT descriptor is sampled with mesh \sqrt{σ_{2}^{2} + c^{2}}.

Proof. The interpolated initial images are by (5.2)

u =: IS_{1}G_{β}H_{λ}u_{0} = G_{β}H_{λ}u_{0} and v =: IS_{1}G_{δ}H_{μ}u_{0} = G_{δ}H_{μ}u_{0}.

Computing the scale-space of these images amounts to convolve these images for every σ > 0 with G_{σ}, which yields, using the commutation relation (10.2) and the semigroup property (10.1):

u(σ, ·) = G_{σ}G_{β}H_{λ}u_{0} = G\sqrt{σ^{2}+β^{2}}H_{λ}u_{0} = H_{λ}G_{λ\sqrt{σ^{2}+β^{2}}}u_{0}.

By the same calculation, this function is compared by SIFT with

v(σ, ·) = H_{μ}G_{μ\sqrt{σ^{2}+δ^{2}}}u_{0}.

Set w(s, x) =: G_{s}u_{0}. Then the scale spaces compared by SIFT are

u(σ, x) = w(λ\sqrt{σ^{2} + β^{2}}, λx) and v(σ, x) = w(μ\sqrt{σ^{2} + δ^{2}}, μx).

Let us consider an extremal point (s_{0}, x_{0}) of the Laplacian of the scale space function w. If s_{0} ≥ max(λβ, μδ), an extremal point occurs at scales σ_{1} for the Laplacian of u(σ, x) and at scale σ_{2} for the Laplacian of v(σ, x) satisfying

s_{0} = λ\sqrt{σ_{1}^{2} + β^{2}} = μ\sqrt{σ_{2}^{2} + δ^{2}}.  (6.4)

□

“JMMBookOct04” 30/10/2009 page 96

[figure] [figure]

96 CHAPTER 6. THE SIFT METHOD

Theorem 6.5. Let u and v be two digital images that are frontal snapshots of the same continuous flat image u_{0}, u = S_{1}G_{β}H_{λ}T Ru_{0} and v =: S_{1}G_{δ}H_{μ}u_{0}, taken from arbitrary distances, with possibly different camera gaussian blurs, with an arbitrary camera translation parallel to its focal plane, and an arbitrary rotation around its optical axe. Without loss of generality, assume λ ≤ μ. Then if the camera blurs are standard (β = δ = c), each SIFT descriptor of u = Iu is identical to some SIFT descriptor of v = Iv. If β ≠ δ (or β = δ ≠ c), the SIFT descriptors of u become (quickly) similar to SIFT descriptors of v when their scales grow, namely as soon as \frac{σ_{1}}{max(c,β)} ≫ 1 and \frac{σ_{2}}{max(c,δ)} ≫ 1.

Proof. By the result of Lemma 6.3, we can neglect the effect of translations and rotations. Therefore assume w.l.o.g. that the images under comparison are as in Lemma 6.4. Assume a key point (s_{0}, x_{0}) of w has scale s_{0} ≥ max(λβ, μδ). This key point has a sampling rate proportional to s_{0}. There is a corresponding key point (σ_{1}, \frac{x_{0}}{λ}) for u with sampling rate \sqrt{σ_{2}^{2} + c^{2}} and a corresponding key point (σ_{2}, \frac{x_{0}}{μ}) with sampling rate \sqrt{σ_{2}^{2} + c^{2}} for v. To have a common reference for these sampling rates, it is convenient to refer to the corresponding sampling rates for w(s_{0}, x), which are λ\sqrt{σ_{1}^{2} + c^{2}} for the SIFT descriptors of u at scale σ_{1}, and μ\sqrt{σ_{2}^{2} + c^{2}} for the descriptors of v at scale σ_{2}. Thus the SIFT descriptors of u and v for x_{0} will be identical if and only if λ\sqrt{σ_{1}^{2} + c^{2}} = μ\sqrt{σ_{2}^{2} + c^{2}}. Now, we have λ\sqrt{σ_{1}^{2} + β^{2}} = μ\sqrt{σ_{2}^{2} + δ^{2}}, which implies λ\sqrt{σ_{1}^{2} + c^{2}} = μ\sqrt{σ_{2}^{2} + c^{2}} if and only if

λ^{2}β^{2} − μ^{2}δ^{2} = (λ^{2} − μ^{2})c^{2}.  (6.5)

Since λ and μ are proportional to camera distances to the observed object u_{0}, they are arbitrary and generally different. Thus, the only way to ensure (6.5) is to have β = δ = c, which means that the blurs of both images (or of both cameras) are ideal and gaussian. In any case, β = δ = c does imply that the SIFT descriptors of both images are identical.

The second statement is straightforward: If σ_{1} and σ_{2} are large enough with respect to β, δ and c, the relation λ\sqrt{σ_{1}^{2} + β^{2}} = μ\sqrt{σ_{2}^{2} + δ^{2}}, implies λ\sqrt{σ_{1}^{2} + c^{2}} ≃ μ\sqrt{σ_{2}^{2} + c^{2}}. □

The almost perfect scale invariance of SIFT stated in Theorem 6.5 is illustrated by the striking example of Fig. 6.6. The 28 SIFT key points of a very small digital image u are compared to the 86 key points obtained by zooming in u by a 32 factor: The resulting digital image is the digital image v = S_{\frac{1}{32}} Iu, again obtained by zero-padding. For better observability, both images are displayed with the same size by enlarging the pixels of u. Almost each key point (22 out of 28) of u finds its counterpart in v. 22 matches are detected between the descriptors as shown on the right. If we trust Theorem 6.5, all descriptors of u should have been retrieved in v. This does not fully happen for two reasons. First, the SIFT method thresholds (not taken into account in the theorem) eliminate many potential key points. Second, the zero-padding interpolation giving v is imperfect near the image boundaries.

“JMMBookOct04” 30/10/2009 page 97

[figure] [figure]

6.5. EXERCISES 97

Figure 6.6: Scale invariance of SIFT, an illustration of Theorem 6.5. Left: a very small digital image u with its 28 key points. For the conventions to represent key points and matches, see the comments in Fig. 6.4. Middle: this image is over sampled by a 32 factor to v = S_{\frac{1}{32}} Iu. It has 86 key points. Right: 22 matches found between u and S_{\frac{1}{32}} Iu.

6.5 Exercises

Exercise 6.5. The aim of the exercise is to explain why the experiment of Fig. 6.6 works, and to illustrate Theorem 6.5. The digital zoom in by a factor λ is nothing but the discrete over-sampling operator S_{\frac{1}{λ}} with sampling step \frac{1}{λ}, defined in (5.11). Here, λ = 32. In the experiment an original digital image u = S_{1}G_{δ}u is zoomed into v = S_{\frac{1}{λ}} u.

1) Using the definition of the discrete zoom and the right commutation relations given in this chapter and in the former one (give their numbers), show that

v = S_{\frac{1}{λ}} G_{δ}Iu = S_{1}G_{λδ}H_{\frac{1}{λ}} Iu.

2) Is v well-sampled if u was?

3) By applying carefully Theorem 6.5, assuming that δ ≃ c, discuss why SIFT manages to match SIFT descriptors of u and v.

6.6 Comments and references

Many variations exist on the computation of interest points, following the pioneering work of Harris and Stephens [105]. The Harris-Laplace and Hessian-Laplace region detectors [172, 175] are invariant to rotation and scale changes. Some moment-based region detectors [146, 25] including Harris-Affine and Hessian-Affine region detectors [173, 175], an edge-based region detector [245], an intensitybased region detector [245], an entropy-based region detector [124], and two independently developed level line-based region detectors MSER (“maximally stable extremal region”) [168] and LLD (“level line descriptor”) [187, 189, 191] are designed to be invariant to affine transformations. These two methods stem from the Monasse image registration method [179] that used well contrasted extremal regions to register images. MSER is the most efficient one and has shown better performance than other affine invariant detectors [177]. However, as pointed out in [151], no known detector is actually fully affine invariant: All “JMMBookOct04” 30/10/2009 page 98

[figure] [figure]

98 CHAPTER 6. THE SIFT METHOD

of them start with initial feature scales and locations selected in a non-affine invariant manner. The difficulty comes from the scale change from an image to another: This change of scale is actually an under-sampling, which means that the images differ by a blur.

In his milestone paper [151], Lowe has addressed this central problem and has proposed the so called scale-invariant feature transform (SIFT) descriptor, that is invariant to image translations and rotations, to scale changes (blur), and robust to illumination changes. It is also surprisingly robust to large enough orientation changes of the viewpoint (up to 60 degrees). Based on the scale-space theory [145], the SIFT procedure simulates all gaussian blurs and normalizes local patches around scale covariant image key points that are Laplacian extrema. A number of SIFT variants and extensions, including PCA-SIFT [126] and gradient location-orientation histogram (GLOH) [176], that claim to have better robustness and distinctiveness with scaled-down complexity have been developed ever since [85, 143]. Demonstrated to be superior to other descriptors [174, 176], SIFT has been popularly applied for scene recognition [79, 182, 217, 251, 96, 226] and detection [86, 194], robot localization [28, 196, 120], image registration [264], image retrieval [104], motion tracking [247, 128], 3D modeling and reconstruction [211, 252], building panoramas [3, 34], or photo management [263, 141, 51].

As pointed out by several benchmarks, the robustness and repeatability of the SIFT descriptors outperforms other methods. However, such benchmarks mix three very different criteria that, in our opinion, should have been discussed separately. The first one is the formal real invariance of each method when all thresholds have been eliminated. This real invariance has been proved here for SIFT. The second criterion is the practical validity of the sampling method used in SIFT, that has been again checked in Chapter 5.2. The last criterion is the clever fixing of several thresholds in the SIFT method ensuring robustness, repeatability, and a low false alarm rate. This one has been extensively tested and confirmed in previous benchmark papers (see also the very recent and complete report [80]). We think, however, that the success of SIFT in these benchmarks is primarily due to its full scale invariance.

“JMMBookOct04” 30/10/2009 page 99

[figure] [figure]

Chapter 7

Linear Scale Space and Edge Detection

The general analysis framework in which an image is associated with smoothed versions of itself at several scales is called scale space. Following the results of Chapter 3, a linear scale space must be performed by applying the heat equation to the image. The main aim of this smoothing is to find out edges in the image. We shall first explain this doctrine. In the second section, we discuss experiments and several serious objections to such an image representation.

7.1 The edge detection doctrine

One of the uses of linear theory in two dimensions is edge detection. The assumption of the edge detection doctrine is that relevant information is contained in the traces produced in an image by the apparent contours of physical objects. If a black object is photographed against a white background, then one expects the silhouette of the object in the image to be bounded by a closed curve across which the light intensity u_{0} varies strongly. We call this curve an edge. At first glance, it would seem that this edge could be detected by computing the gradient Du_{0}, since at a point x on the edge, |Du_{0}(x)| should be large and Du(x) should point in a direction normal to the boundary of the silhouette. It would therefore appear that finding edges amounts to computing the gradient of u_{0} and determining the points where the gradient is large. This conclusion is unrealistic for two reasons:

(a) There may be many points where the gradient is large due to small oscillations in the image that are not related to real objects. Recall that digital images are always noisy, and thus there is no reason to assume the existence or computability of a gradient.

(b) The points where the gradient exceeds a given threshold are likely to form regions and not curves.

As we emphasized in the Introduction, objection (a) is dealt with by smoothing the image. We associate with the image u_{0} smoothed versions u(t, ·), where the scale parameter t indicates the amount of smoothing. In the classical linear theory, this smoothing is done by convolving u_{0} with the Gaussian G_{t}.

99 “JMMBookOct04” 30/10/2009 page 100

[figure] [figure]

100 CHAPTER 7. LINEAR SCALE SPACE AND EDGE DETECTION

One way that objection (b) has been approached is by redefining edge points. Instead of just saying an edge point is a point x where |Du_{0}(x)| exceeds a threshold, one requires the gradient to satisfy a maximal property. We illustrate this in one dimension. Suppose that u ∈ C^{2}(ℝ) and consider the points where |u′(x)| attains a local maximum. At some of these points, the second derivative u′′ changes sign, that is, sign(u′′(x − h)) ≠ sign(u′′(x + h)) for sufficiently small h. These are the points where u′′ crosses zero, and they are taken to be the edge points. Note that this criterion avoids classifying a point x as an edge point if the gradient is constant in an interval around x. Marr and Hildreth generalized this idea to two dimensions by replacing u′′ with the Laplacian Δu, which is the only isotropic linear differential operator of order two that generalizes u′′ [165]. Haralick’s edge detector is different but in the same spirit [101]. Haralick gives up linearity and defines edge points as those points where the gradient has a local maximum in the direction of the gradient. In other words, an edge point x satisfies g′(0) = 0, where g(t) = |Du(x + tDu(x)|/|Du(x)|. This implies that D^{2}u(x)(Du(x), Du(x)) = 0 (see Exercise 7.2). We are now going to state these two algorithms formally. They are illustrated in Figures 7.2 and 7.3, respectively.

Algorithm 7.1 (Edge detection: Marr–Hildreth zero-crossings).

(1) Create the multiscale images u(t, ·) = G_{t} ∗ u_{0} for increasing values of t.

(2) At each scale t, compute all the points where Du ≠ 0 and Δu changes sign. These points are called zero-crossings of the Laplacian, or simply zero-crossings.

(3) (Optional) Eliminate the zero-crossings where the gradient is below some prefixed threshold.

(4) track back from large scales to fine scales the “main edges” detected at large scales.

Algorithm 7.2 (Edge detection: The Haralick–Canny edge detector).

(1) As before, create the multiscale images u(t, ·) = G_{t} ∗u_{0} for increasing values of t.

(2) At each scale t, find all points x where Du(x) ≠ 0 and D^{2}u(x)(z, z) crosses zero, z = Du/|Du|. At such points, the function s ↦ u(x + sz) changes from concave to convex, or conversely, as s passes through zero.

(3) At each scale t, fix a threshold θ(t) and retain as edge points at scale t only those points found above that satisfy |Du(x)| > θ(t). The backtracking step across scales is the same as for Marr–Hildreth.

In practice, edges are computed for a finite number of dyadic scales, t = 2^{n}, n ∈ ℤ.

7.1.1 Discussion and critique

The Haralick–Canny edge detector is generally preferred for its accuracy to the Marr–Hildreth algorithm. Their use and characteristics are, however, essentially “JMMBookOct04” 30/10/2009 page 101

[figure] [figure]

7.1. THE EDGE DETECTION DOCTRINE 101

Figure 7.1: A three-dimensional representation of the Laplacian of the Gaussian. This convolution kernel, which is a wavelet, is used to estimate the Laplacian of an image at different scales of linear smoothing.

the same. There are also many variations—attempted improvements—of the algorithms we have described, and the following discussion adapts easily to these related edge detection schemes. The first thing to notice is that, by Proposition 2.5, u(t, ·) = G_{t} ∗ u_{0} is a C^{∞} function for each t > 0 if u_{0} ∈ F. Thus we can indeed compute second order differential operators applied to u(t, ·) = G_{t} ∗ u_{0}, t > 0. In the case of linear operators like the Laplacian or the gradient, the task is facilitated by the formula proved in the mentioned proposition. For example, we have Δu(t, x) = Δ(G_{t} ∗ u_{0})(x) = (ΔG_{t}) ∗ u_{0}(x), where in dimension two (Figure 7.1),

ΔG_{t}(x) = \frac{|x|^{2} − 4t}{16πt^{3}}e^{−|x|^{2}/4t}.

In the same way, Haralick’s edge detector makes sense, because u is C^{∞}, at all points where Du(x) ≠ 0. If Du(x) = 0, then x cannot be an edge point, since u is “flat” there. Thus, thanks to the filtering, there is no theoretical problem with computing edge points. There are, however, practical objections to these methods, which we will now discuss.

Linear scale space

The first serious problems are associated with the addition of an extra dimension: Having many images u(t, ·) at different scales t confounds our understanding of the image and adds to the cost of computation. We no longer have an absolute definition of an edge. We can only speak of edges at a certain scale. Conceivably, a way around this problem would be to track edges across scales. In fact, it has been observed in experiments that the “main edges” persist under convolution as t increases, but they lose much of their spatial accuracy. On the other hand, filtering with a sharp low-pass filter, that is, with t small, keeps these edges in their proper positions, but eventually, as t becomes very small, even these main edges can be lost in the crowd of spurious edge signals due to noise and texture. The scale space theory of Witkin proposes to identify the main edges at some scale t and then to track them backward as t decreases [262]. In theory, it would seem that this method could give an accurate location of the main edges. In practice, any implementation of these ideas is computationally costly due to the problems involved with multiple thresholdings and following edges across scales. In fact, tracking edges across scales is incompatible with having thresholds for the gradients, since such thresholds may remove edges at “JMMBookOct04” 30/10/2009 page 102

[figure] [figure]

102 CHAPTER 7. LINEAR SCALE SPACE AND EDGE DETECTION

Figure 7.2: Zero-crossings of the Laplacian at different scales. This figure illustrates the original scale space theory as developed by David Marr [163]. To extract more global structure, the image is convolved with Gaussians whose variances are powers of two. One computes the Laplacian of the smoothed image and displays the lines along which this Laplacian changes sign: the zerocrossings of the Laplacian. According to Marr, these zero-crossings represent the “raw primal sketch” of the image, or the essential information on which further vision algorithms should be based. Above, left to right: the results of smoothing and the associated Gaussian kernels at scales 1, 2, and 4. Below, left to right: the zero-crossings of the Laplacian and the corresponding kernels, which are the Laplacians of the Gaussians used above.

certain scales and not at others. The conclusion is that one should trace all zero-crossings across scales without considering whether they are true edges or not. This makes matching edges across scales very difficult. For example, experiments show that zero-crossings of sharp edges that are sparse at small scales are no longer sparse at large scales. (Figure 7.4 shows how zero-crossings can be created by linear smoothing.) The Haralick–Canny detector suffers from the same problems, as is well demonstrated by experiments.

Other problems with linear scale space are illustrated in Figures 7.5 and 7.6. Figure 7.5 illustrates how linear smoothing can create new gray levels and new extrema. Figure 7.6 shows that linear scale space does not maintain the inclusion between objects. The shape inclusion principal will be discussed in Chapter 21.

We must conclude that the work on linear edge detection has been an attempt to build a theory that has not succeeded. After more than thirty years of activity, it has become clear that no robust technology can be based on these ideas. Since edge detection algorithms depend on multiple thresholds on the gradient, followed by “filling-the-holes” algorithms, there can be no scientific agreement on the identification of edge points in a given image. In short, the problems associated with linear smoothing followed by edge detection have not been resolved by the idea of chasing edges across scales.

“JMMBookOct04” 30/10/2009 page 103

[figure] [figure]

7.1. THE EDGE DETECTION DOCTRINE 103

Figure 7.3: Canny’s edge detector. These images illustrate the Canny edge detector. Left column: result of the Canny filter without the threshold on the gradient. Middle column: result with a visually “optimal” scale and an imagedependent threshold (from top to bottom: 15, 0.5, 0.6). Right column: result with a fixed gradient threshold equal to 0.7. Note that such an edge detection theory depends on no fewer than two parameters that must be fixed by the user: smoothing scale and gradient threshold .

Figure 7.4: Zero-crossings of the Laplacian of a synthetic image. Left to right: the original image; the image linearly smoothed by convolution with a Gaussian; the sign of the Laplacian of the filtered image (the gray color corresponds to values close to 0, black to clear-cut negative values, white to clear-cut positive values); the zero-crossings of the Laplacian. This experiment clearly shows a drawback of the Laplacian as edge detector.

Contrast invariance

The use of contrast-invariant operators can solve some of the technical problems associated with linear smoothing and other linear image operators. An (image) operator u ↦ T u is contrast invariant if T commutes with all nondecreasing

“JMMBookOct04” 30/10/2009 page 104

[figure] [figure]

104 CHAPTER 7. LINEAR SCALE SPACE AND EDGE DETECTION

Figure 7.5: The heat equation creates structure. This experiment shows that linear scale space can create new structures and thus increase the complexity of an image. Left to right: The original synthetic image (a) contains three gray levels. The black disk is a regional and absolute minimum. The “white” ring around the black disk is a regional and absolute maximum. The outer gray ring has a gray value between the other two and is a regional minimum. The second image (b) shows what happens when (a) is smoothed with the heat equation: New local extrema have appeared. Image (c) illustrates the action on (a) of a contrast-invariant local filter, the iterated median filter, which is introduced in Chapter 17.

functions g, that is, if

g(T u) = T (g(u)).  (7.1)

If image analysis is to be robust, it must be invariant under changes in lighting that produce contrast changes. It must also be invariant under the nonlinear response of the sensors used to capture an image. These, and perhaps other, contrast changes are modeled by g. If g is strictly increasing, then relation (7.1) ensures that the filtered image T u = g^{−1}(T (g(u))) does not depend on g. A problem with linear theory is that linear smoothing, that is, convolution, is not generally contrast invariant:

g(k ∗ u) ≠ k ∗ (g(u)).

In the same way, the operator T_{t} that maps u_{0} into the solution of the heat equation, u(t, ·) is not generally contrast invariant. In fact, if g is C^{2}, then

\frac{∂(g(u))}{∂t} = g′(u)\frac{∂u}{∂t}

and

Δ(g(u)) = g′(u)Δu + g′′(u)|Du|^{2}.

Exercise 7.1. Prove this last relation. Prove that if g(s) = as + b then g(u) satisfies the heat equation if u does.

“JMMBookOct04” 30/10/2009 page 105

[figure] [figure]

7.2. EXERCISES 105

Figure 7.6: Violation of the inclusion by the linear scale space. Top, left: an image that contains a black disk enclosed by a white disk. Top, right: At a certain scale, the black and white circles mix together. Bottom, left: The boundaries of the two circles. Bottom, right: After smoothing with a certain value of t, the inclusion that existed for very small t in no longer preserved. We display the level lines of the image at levels multiples of 16.

7.2 Exercises

Exercise 7.2. Define an edge point x in a smooth image u as a point x at which g(t) attains a maximum, where

g(t) = |Du (x + t \frac{Du(x)}{|Du(x)|} ) |.

Prove by differentiating g(t) that edge points satisfy D^{2}u(x)(Du(x), Du(x)) = 0 Exercise 7.3. Construct simple functions u, g, and k such that g(k ∗ u) ≠ k ∗ (g(u)).

Exercise 7.4. Consider the Perona–Malik equation in divergence form:

\frac{∂u}{∂t} = div(g(|Du|)Du),  (7.2)

where g(s) = 1/(1 + λ^{2}s^{2}). It is easily checked that we have a diffusion equation when λ|Du| ≤ 1 and an inverse diffusion equation when λ|Du| > 1. To see this, consider the second derivative of u in the direction of Du,

u_{ξξ} = D^{2}u( \frac{Du}{|Du|} , \frac{Du}{|Du|} ),

“JMMBookOct04” 30/10/2009 page 106

[figure] [figure]

106 CHAPTER 7. LINEAR SCALE SPACE AND EDGE DETECTION

and the second derivative of u in the orthogonal direction,

u_{ηη} = D^{2}u( \frac{Du^{⊥}}{|Du|} , \frac{Du^{⊥}}{|Du|} ),

where Du = (u_{x}, u_{y}) and Du^{⊥} = (−u_{y}, u_{x}). The Laplacian can be rewritten in the intrinsic coordinates (ξ, η) as Δu = u_{ξξ} + u_{ηη}. Prove that the Perona–Malik equation then becomes

\frac{∂u}{∂t} = \frac{1}{1 + λ^{2}|Du|^{2}} u_{ηη} + \frac{1 − λ^{2}|Du|^{2}}{(1 + λ^{2}|Du|^{2})^{2}} u_{ξξ}.

Interpret the local behavior of the equation as a heat equation or a reverse heat equation according to the size of |Du| compared to λ^{−1}.

7.3 Comments and references

Scale space. The term “scale space” was introduced by Witkin in 1983. He suggested tracking the zero-crossings of the Laplacian of the smoothed image across scales [262]. Yuille and Poggio proved that these zero-crossings can be tracked for one-dimensional signals [268]. Hummel and Moniot [113, 116] and Yuille and Poggio [269] analyzed the conjectures of Marr and Witkin according to which an image is completely recoverable from its zero-crossings at different scales. Mallat formulated Marr’s conjecture as an algorithm in the context of wavelet analysis. He replaced the Gaussian with a two-dimensional cubic spline, and he used both the zero-crossings of the smoothed images and the nonzero values of the gradients at these points to reconstruct the image. This algorithm works well in practice, and the conjecture was that these zero-crossings and the values of the gradients determined the image. A counterexample given by Meyer shows that this is not the case. Perfect reconstruction is possible in the one-dimensional case for signals with compact support if the smoothing kernel is the Tukey window, k(x) = 1 + cos x for |x| ≤ π and zero elsewhere. An account of the Mallat conjecture and these examples can be found in [119]. Koenderink presents a general and insightful theory of image scale space in [134].

Gaussian smoothing and edge detection. The use of Gaussian filtering in image analysis is so pervasive that it is impossible to point to a “first paper.” It is, however, safe to say that David Marr’s famous book, Vision [163], and the original paper by Hildreth and Marr [165] have had an immeasurable impact on edge detection and image processing in general. The term “edge detection” appeared as early as 1959 in connection with television transmission [121]. The idea that the computation of derivatives of an image necessitates a previous smoothing has been extensively developed by the Dutch school of image analysis [30, 84]. See also the books by Florack [83], Lindeberg [144], and Romeny [243], and the paper [78]. Haralick’s edge detector [101], as implemented by Canny [38], is probably the best known image analysis operator. A year after Canny’s 1986 paper, Deriche published a recursive implementation of Canny’s criteria for edge detection [69].

“JMMBookOct04” 30/10/2009 page 107

[figure] [figure]

Chapter 8

Four Algorithms to Smooth a Shape

In this short but important chapter, we discuss algorithms whose aim it is to smooth shapes. Shape must be understood as a rough data which can be extracted from an image, either a subset of the plane, or the curve surrounding it. Shape smoothing is directed at the elimination of spurious, often noisy, details. The smoothed shape can then be reduced to a compact and robust code for recognition. The choice of the right smoothing will make us busy throughout the book. A good part of the solution stems from the four algorithms we describe and their progress towards more robustness, more invariance and more locality. What we mean by such qualities will be progressively formalized. We will discuss two algorithms which directly smooth sets, and two which smooth Jordan curves. One of the aims of the book is actually to prove that both approaches, different though they are, eventually yield the very same process, namely a curvature motion.

8.1 Dynamic shape

In 1986, Koenderink and van Doorn defined a shape in ℝ^{N} to be a closed subset X of ℝ^{N} [135]. They then proposed to smooth the shape by applying the heat equation ∂u/∂t − Δu = 0 directly to 1_{X}, the characteristic function of X. Of course, the solution G_{t} ∗1_{X} is not a characteristic function. The authors defined the evolved shape at scale t to be

X_{t} = {x | u(t, x) ≥ 1/2}.

The value 1/2 is chosen so the following simple requirement is satisfied: Suppose that X is the half-plane X = {(x, y) | (x, y) ∈ ℝ^{2}, x ≥ 0}. The requirement is that this half plane doesn’t move,

X = X_{t} = {(x, y) | G_{t} ∗ 1_{X}(x, y) ≥ λ},

and this is true only if λ = 1/2. There are at least two problems with dynamic shape evolution for image analysis. The first concerns nonlocal interactions, as illustrated in Figure 8.1. Here we have two disks that are near one another.

107 “JMMBookOct04” 30/10/2009 page 108

[figure] [figure]

108 CHAPTER 8. FOUR ALGORITHMS TO SMOOTH A SHAPE

Figure 8.1: Nonlocal interactions in the dynamic shape method. Left to right: Two close disks interact as the scale increases. This creates a new, qualitatively different, shape. The change of topology, at the scale where the two disks merge into one shape, also entails the appearance of a singularity (a cusp) on the shape(s) boundaries.

The evolution of the union of both disks, considered as a single shape, is quite different from the evolution of the disks separately. A related problem, also illustrated in Figure 8.1, is the creation of singularities. Note how a singularity in orientation and the curvature of the boundary of the shape develops at the point where the two disks touch. Figure 8.2 further illustrates the problems associated with the dynamic shape method.

8.2 Curve evolution using the heat equation

We consider shapes in ℝ^{2} whose boundaries can be represented by a finite number of simple closed rectifiable Jordan curves. Thus, each curve we consider can be represented by a continuous mapping f : [0, 1] → ℝ^{2} such that f is one-to-one on (0, 1) and f(0) = f(1), and each curve has a finite length. We also assume that these curves do not intersect each other. We will focus on smoothing one of these Jordan curves, which we call C_{0}. We assume that C_{0} is parameterized by s ∈ [0, L], where L is the length of the curve. Thus, C_{0} is represented as x_{0}(s) = (x(s), y(s)), where s is the length of the curve between x_{0}(0) and x_{0}(s).

At first glance, it might seem reasonable to smooth C_{0} by smoothing the coordinate functions x and y separately. If this is done linearly, we have seen from Theorem 2.3 that the process is asymptotic to smoothing with the heat equation. Thus, one is led naturally to consider the vector heat equation

\frac{∂x}{∂t}(t, s) = \frac{∂^{2}x}{∂s^{2}}(t, s)  (8.1)

with initial condition x(0, s) = x_{0}(s). If x(t, s) = (x(t, s), y(t, s)) is the solution of (8.1), then we know from Proposition 1.9 that

inf_{s∈[0,L]} x_{0}(s) ≤ x(t, s) ≤ sup_{s∈[0,L]} x_{0}(s),
inf_{s∈[0,L]} y_{0}(s) ≤ y(t, s) ≤ sup_{s∈[0,L]} y_{0}(s),

for s ∈ [0, L] and t ∈ [0, +∞). Thus, the evolved curves C_{t} remain in the rectangle that held C_{0}. Also, we know from Proposition 2.5 that the coordinate functions x(t, ·) and y(t, ·) are C^{∞} for t > 0. There are, however, at least two reasons that argue against smoothing curves this way:

“JMMBookOct04” 30/10/2009 page 109

[figure] [figure]

8.3. RESTORING LOCALITY AND CAUSALITY 109

Figure 8.2: Nonlocal behavior of shapes with the dynamic shape method. This image displays the smoothing of two irregular shapes by the dynamic shape method (Koenderink–van Doorn). Top left: initial image, made of two irregular shapes. From left to right, top to bottom: dynamic shape smoothing with increasing Gaussian variance. Notice how the shapes merge more and more. We do not have a separate analysis of each shape but rather a “joint analysis” of the two shapes. The way the shapes merge is of course sensitive to the initial distance between the shapes. Compare with Figure 8.4.

(1) When t > 0, s is no longer a length parameter for the evolved curve C_{t}.

(2) Although x(t, ·) and y(t, ·) are C^{∞} for t > 0, this does not imply that the curves C_{t} have similar smoothness properties. In fact, it can be seen from Figure 8.3 that it is possible for an evolved curve to cross itself and it is possible for it to develop singularities.

How is this last mentioned phenomenon possible ? It turns out that one can parameterize a curve with corners or cusps with a very smooth parameterization: see Exercise 8.1.

In image processing, we say that a process that introduces new features, such as described in item (2) above, is not causal. ^{1}

8.3 Restoring locality and causality

Our main objective is to redefine the smoothing processes so they are local and do not create new singularities. This can be done by alternating a small-scale linear convolution with a natural renormalization process.

1This informal definition should not be confused with the use of “causality,” as it is used, for example, when speaking about filters: A filter F is said to be causal, or realizable, if the equality of two signals s_{0} and s_{1} up to time t_{0} implies that F s_{0}(t) = F s_{1}(t) for the same period.

“JMMBookOct04” 30/10/2009 page 110

[figure] [figure]

110 CHAPTER 8. FOUR ALGORITHMS TO SMOOTH A SHAPE

Figure 8.3: Curve evolution by the heat equation. The coordinates of the curves are parameterized by the arc length and then smoothed as real functions of the length using the heat equation. From A to D: the coordinates are smoothed with an increasing scale. Each coordinate function therefore is C^{∞}; the evolving curve can, however, develop self-crossings (as in C) or singularities (as in D).

8.3.1 Localizing the dynamic shape method

In the case of dynamic shape analysis, we define an alternate dynamic shape algorithm as follows:

Algorithm 8.1 (The Merriman–Bence–Osher algorithm).

(1) Convolve the characteristic function of the initial shape X_{0} with G_{h}, where h is small.

(2) Define X_{1} = {x | G_{h} ∗ 1_{X_{0}} ≥ 1/2}.

(3) Set X_{0} = X_{1} and go back to (1).

This is an iterated dynamic shape algorithm. The dynamic shape method itself is an example of a median filter, which will be defined in Chapter 17. The Merriman–Bence–Osher algorithm is thus an iterated median filter (see Figure 8.4). We will see in Chapters ?? and ?? that median filters have asymptotic properties that are similar to those expressed in Theorem 3.3. In the case of median filters, the associated partial differential equation will be a curvature motion equation (defined in Chapter 19).

8.3.2 Renormalized heat equation for curves

In 1992, Mackworth and Mokhtarian noticed the loss of causality when the heat equation was applied to curves [154]. Their method to restore causality looks, at least formally, like the remedy given for the nonlocalization of the dynamic “JMMBookOct04” 30/10/2009 page 111

[figure] [figure]

8.3. RESTORING LOCALITY AND CAUSALITY 111

Figure 8.4: The Merriman–Bence–Osher shape smoothing method is a localized and iterated version of the dynamic shape method. A convolution of the binary image with small-sized Gaussians is alternated with mid-level thresholding. It uses the same initial data (top, left) as in Figure 8.2. From left to right, top to bottom: smoothing with increasing scales. Notice that the shapes remain separate. In fact, their is no interaction between the evolving shapes. Each one evolves as if the other did not exist.

shape method. Instead of applying the heat equation for relatively long times (or, equivalently, convolving the curve x with the Gaussian G_{t} for large t), they use the following algorithm:

Algorithm 8.2 (Renormalized heat equation for curves).

(1) Convolve the initial curve x_{0}, parameterized by its length parameter s_{0} ∈ [0, L_{0}], with the Gaussian G_{h}, where h is small.

(2) Let L_{n} denote the length of the curve x_{n} obtained after n iterations and let s_{n} denote its length parameter. For n ≥ 1, write x̃_{n+1}(s_{n}) = G_{h} ∗ x_{n}(s_{n}). Then reparameterize x̃_{n+1} by its length parameter s_{n+1} ∈ [0, L_{n+1}], and denote it by x_{n+1}.

(3) Iterate.

This algorithm is illustrated in Figure 8.5. It should be compared with Figure 8.3.

Theorem 8.1. Let x be a C^{2} curve parameterized by its length parameter s ∈ [0, L]. Then for small h,

G_{h} ∗ x(s) − x(s) = h\frac{∂^{2}x}{∂s^{2}} + o(h).  (8.2)

“JMMBookOct04” 30/10/2009 page 112

[figure] [figure]

112 CHAPTER 8. FOUR ALGORITHMS TO SMOOTH A SHAPE

Figure 8.5: Curve evolution by the renormalized heat equation (Mackworth– Mokhtarian). After each smoothing step, the coordinates of the curve are reparameterized by the arc length of the smoothed curve. From A to D: the curve is smoothed with an increasing scale. Note that, in contrast with the linear heat equation (Figure 8.3), the evolving curve shows no singularities and does not cross itself.

This theorem is easily checked, see Exercise 8.2

In view of (8.2) and what we have seen regarding asymptotic limits in Theorem 3.3 and Exercise 3.5, it is reasonable to conjecture that, in the asymptotic limit, Algorithm 8.2 will yield the solution of following evolution equation:

\frac{∂x}{∂t} = \frac{∂^{2}x}{∂s^{2}},  (8.3)

where x_{0} = x(0, ·). It is important to note that (8.3) is not the heat equation (8.1). Indeed, from Algorithm 8.2 we see that s must denote the length parameter of the evolved curve x(t, ·) at time t. In fact ∂^{2}x/∂s^{2} has a geometric interpretation as a curvature vector. We will study this nonlinear curve evolution equation in Chapter 19.

8.4 Exercises

Exercise 8.1. Construct a C^{∞} mapping f : [0, 1] → ℝ^{2} such that the image of [0, 1] is a square. This shows that a curve can have a C^{∞} parameterization without being smooth.

Exercise 8.2. Prove Theorem 8.1. If x is a C^{3} function of s, then the result follows directly from Theorem 3.2. The result holds, however, for a C^{2} curve.

“JMMBookOct04” 30/10/2009 page 113

[figure] [figure]

8.5. COMMENTS AND REFERENCES 113

8.5 Comments and references

Dynamic shape, curve evolution, and restoring causality. Our account of the dynamic shape method is based on the well-known paper by Koenderink and van Doorn in which they introduced this notion [135]. The curve evolution by the heat equation is from the first 1986 version of curve analysis proposed by Mackworth and Mokhtarian [153]. See also the paper by Horn and Weldon [109]. There were model errors in the 1986 paper [153] that were corrected by the authors in their 1992 paper [154]. There, they also proposed the correct intrinsic equation. However, this 1992 paper contains several inexact statements about the properties of the intrinsic equation. The correct theorems and proofs can be found in a paper by Grayson written in 1987 [98]. The algorithm that restores causality and locality to the dynamic shape method was discovered by Merriman, Bence, and Osher, who devised this algorithm for a totally different reason: They were looking for a clever numerical implementation of the mean curvature equation [170].

Topological change under smoothing. We have included several figures that illustrate how essential topological properties of an image change when the image is smoothed with the Gaussian. Damon has made a complete analysis of the topological behavior of critical points of an image under Gaussian smoothing [63]. This analysis had been sketched in [267].

“JMMBookOct04” 30/10/2009 page 114

[figure] [figure]

“JMMBookOct04” 30/10/2009 page 115

[figure] [figure]

Chapter 9

Affine Invariant Image Comparison

If a physical object has a smooth or piecewise smooth boundary, its images obtained by cameras in varying positions undergo smooth apparent deformations. These deformations are locally well approximated by affine transforms of the image plane.

In consequence the solid object recognition problem has often been led back to the computation of affine invariant image local features. Such invariant features could be obtained by normalization methods, but no fully affine normalization method exists for the time being. As a matter of fact, the scale invariance, which actually means invariance to blur, is only dealt with by methods inspired from the scale space theory, like the SIFT method. By simulating zooms out, this method normalizes the four translation, rotation and scale (blur) parameters, out of the six parameters of an affine transform. Affine normalization methods like MSER or Hessian Affine normalize with respect to all six parameters of the affine transform, but this normalization is imperfect, not dealing rigorously with blur for MSER, or not starting with affine invariant scale space extrema for Hessian Affine.

The method proposed in this chapter, affine SIFT (A-SIFT), simulates all image views obtainable by varying the two camera parameters left over by the SIFT method. Then it normalizes the other four parameters by simply using the SIFT method itself. The two additional parameters are the angles (a longitude and a latitude) defining the camera axis orientation. Mathematical arguments will be given in Chapter 10 to prove that the resulting method is fully affine invariant, up to an arbitrary precision.

Against any prognosis, simulating all views depending on the two camera orientation parameters is feasible with no dramatic computational load. The method permits to reliably identify features that have undergone tilts of large magnitude, up to 30 and more, while state-of-the-art methods do not exceed tilts of 2.5 (SIFT) or 4.5 (MSER). This chapter puts in evidence the role of high transition tilts: while a tilt from a frontal to an oblique view exceeding 6 is rare, higher transition tilts are common as soon as two oblique views of an object are compared (see Fig. 9.1). Thus, a fully affine invariance is required for 3D scene analysis. This fact is substantiated by many experiments.

115 “JMMBookOct04” 30/10/2009 page 116

[figure] [figure]

116 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON

Figure 9.1: High transition tilts

Section 9.1 gives the main decomposition formula of affine maps used throughout the paper and its geometric interpretation in terms of cameras at infinity. Section 11.1 describes and discusses a method that attempts affine invariance by normalization: MSER. Section 9.2 describes the A-SIFT algorithm and discusses precursors. Section 9.4 addresses the critical sampling issues for the new-simulated parameters in A-SIFT (tilt and longitude). It then provides a complexity analysis and a fast version of the method. Section 9.3 presents and experiments the crucial notion of transition tilt.

9.1 The affine camera model

The general (solid) shape recognition problem starts with several photographs of a physical object, possibly taken with different cameras and view points. These digital images are the query images. Given other digital images, the search images, the question is whether some of them contain, or not, a view of the object taken in the query image. A solid object’s view can deform from an image to another for two obvious reasons: First, because it underwent some physical deformation, and second, because the change of camera position induced an apparent deformation.

Image distortions arising from viewpoint changes can be locally modeled by affine planar transforms, provided the object’s boundaries are piecewise smooth. In other terms, a perspective effect can be modeled by a combination of several different affine transforms in different image regions (see Fig. 9.3). Indeed, by first order Taylor formula, any planar smooth deformation (x, y) → (X, Y) = (F_{1}(x, y), F_{2}(x, y)) can be locally approximated around each point (x_{0}, y_{0}) → (X_{0}, Y_{0}) by the affine map

( \begin{matrix}X−X_{0} \\ Y−Y_{0} \end{matrix})=[\begin{matrix}\frac{∂F_{1}}{∂x} (x_{0}, y_{0}) & \frac{∂F_{1}}{∂y} (x_{0}, y_{0}) \\ \frac{∂F_{2}}{∂x} (x_{0}, y_{0}) & \frac{∂F_{2}}{∂y} (x_{0}, y_{0}) \end{matrix}] (\begin{matrix}x−x_{0} \\ y−y_{0} \end{matrix}) + O (\begin{matrix}(x−x_{0})^{2} + (y−y_{0})^{2} \\ (x−x_{0})^{2} + (y−y_{0})^{2} \end{matrix}) .

(9.1)

“JMMBookOct04” 30/10/2009 page 117

[figure] [figure]

9.1. THE AFFINE CAMERA MODEL 117

Figure 9.2: Geometric interpretation of the Taylor formula (9.1): Although the global deformation of each wall is strongly projective (a rectangle becomes a trapezoid), the it local deformation is affine: each tile on the pavement is almost a parallelogram. Indeed, projective maps are C^{1} and therefore locally affine. The painting, due to Uccello, is one of the first Renaissance paintings with a correct geometric perspectives following the rules invented by Brunelleschi.

Figure 9.3: Another way to understand why the local object apparent deformations are affine. Local planar homographies are equivalent to multiple local cameras at infinity. Cameras at infinity generate affine deformations of planar objects. This is true even if the object under observation is curved, because it is then locally planar. Thus, the overall apparent deformation of the object is C^{1}, and Formula (9.1) applies.

“JMMBookOct04” 30/10/2009 page 118

[figure] [figure]

118 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON

Thus, all object deformations and all camera motions are locally approximated by affine transforms. For example, in the case of a flat object, the deformation induced by a camera motion is a planar homographic transform, which is smooth and therefore locally tangent to affine transforms.

The converse statement is true: any affine transform with positive determinant can be interpreted as the apparent deformation induced on a planar object by a camera motion, the camera being assumed far away from the object. Thus, under the local smoothness assumption of the object’s boundary, the (local) deformation model of an image u(x, y) under a deformation of the object or under a camera motion is

u(x, y) → u(ax + by + e, cx + dy + f),

where the mapping

( \begin{matrix}x \\ y \end{matrix} ) → [\begin{matrix}a & b \\ c & d \end{matrix} ] ( \begin{matrix}x \\ y \end{matrix} ) + ( \begin{matrix}e \\ f \end{matrix} )

is any affine transform of the plane with positive determinant. The above statements rely on the next crucial following decomposition formula.

Theorem 9.1. Any linear planar map whose matrix A has strictly positive determinant, and which is not a similarity, has a unique decomposition

A=H_{λ}R_{1}(ψ)T_{t}R_{2}(ϕ)=λ [\begin{matrix}cos ψ & − sin ψ \\ sin ψ & cos ψ \end{matrix}] [ \begin{matrix}t & 0 \\ 0 & 1 \end{matrix} ] [\begin{matrix}cos ϕ & − sin ϕ \\ sin ϕ & cos ϕ \end{matrix}]  (9.2)

where λ > 0, λt is the determinant of A, R_{i} are rotations, ϕ ∈ [0, π[, and T_{t} is a tilt, namely a diagonal matrix with a first eigenvalue equal to t ≥ 1 and the second one equal to 1.

Proof. Consider the real symmetric positive semi-definite matrix A^{t}A, where A^{t} denotes the transposed matrix of A. By classic spectral theory there is an orthogonal transform O such that A^{t}A = ODO^{t} where D a diagonal matrix with ordered eigenvalues λ_{1} ≥ λ_{2}. Set O_{1} = AOD^{− \frac{1}{2}} . Then

O_{1}O_{1}^{t} = AOD^{− \frac{1}{2}} D^{− \frac{1}{2}} O^{t}A^{t} = AOD^{−1}O^{t}A^{t} = A(A^{t}A)^{−1}A^{t} = I.

Thus, there are orthogonal matrices O_{1} and O such that

A = O_{1}D^{\frac{1}{2}} O^{t}.  (9.3)

Since the determinant of A is positive, the product of the determinants of O and O_{1} is positive. If both determinants are positive, then O and O_{1} are rotations and we can write A = R(ψ)DR(ϕ). If ϕ is not in [0, π[, changing ϕ into ϕ−π and ψ into ψ + π ensures that ϕ ∈ [0, π[. If the determinants of O and O_{1} are both

negative, replacing O and O_{1} respectively by ( \begin{matrix}−1 & 0 \\ 0 & 1 \end{matrix} ) O and ( \begin{matrix}−1 & 0 \\ 0 & 1 \end{matrix} ) O_{1}

makes them into rotations without altering (9.3), and we can as above ensure ϕ ∈ [0, π[ by adapting ϕ and ψ. The final decomposition is obtained by taking for λ the smaller eigenvalue of D^{\frac{1}{2}} . □

“JMMBookOct04” 30/10/2009 page 119

[figure] [figure]

9.1. THE AFFINE CAMERA MODEL 119

Exercise 9.1. The aim of the exercise is to show the uniqueness of the decomposition

(9.2). Assume there are two decompositions λR_{1} [ \begin{matrix}t & 0 \\ 0 & 1 \end{matrix} ] R_{2} = λ′R′_{1} [ \begin{matrix}t′ & 0 \\ 0 & 1 \end{matrix} ] R′_{2}. Using

the uniqueness of the eigenvalues of a matrix show first that λ = λ′, t = t′. You will obtain a relation of the form R_{1}DR_{2} = D where D is diagonal and R_{1} and R_{2} are rotations. Deduce from this relation that R_{1}D^{2}R_{1}^{t} = D^{2}. Deduce from this last relation the form of R_{1}, conclude carefully.

Figure 9.4: Geometric interpretation of the decomposition formula (9.2).

Exercise 9.2. Consider two cameras looking at a flat square piece of landscape which is assimilated to an infinite resolution image u_{0}(x, y) (See Fig. 9.4). The first camera is very far above the landscape and looking down perpendicularly to the landscape.

(i) Assuming the first camera is pin-hole, show that the generated image is a square image u_{0}(μR(ψ)(x, y)). Consider the coordinate system (O, i⃗, j⃗, k⃗) such that (i⃗, j⃗) are the coordinate vectors in the square image u_{0}, parallel to the image sides, and O is the image center.

(ii) Assume a second pinhole camera has its optical axis pointing down to O. Assume its optical axis is supported by the unit vector with coordinates (sin θ cos ϕ, sin θ sin ϕ, cos θ). Assume again that this camera is very far from the square piece of landscape, so the light rays coming from the landscape to the camera are almost parallel. Thus the image formation on this second camera is assimilated to an orthogonal projection of the landscape u_{0} onto a plane passing by the camera center C and orthogonal to the optical axis. Taking adequate coordinates on this coordinate plane, show that the generated image is

u_{0} (R(ψ_{1})T_{t_{1}}R(ϕ_{1}) ( \begin{matrix}x \\ y \end{matrix} )) for some values of ψ_{1}, ϕ_{1}, t, that you will relate to

ϕ, ψ, and θ.

Fig. 9.4 shows a camera motion interpretation of this affine decomposition: ϕ and θ = arccos 1/t are the viewpoint angles and ψ parameterizes the camera spin. Thus, this figure illustrates the four main parameters in the affine image deformation caused by a camera motion, starting from a frontal view u. The camera is assumed to stay far away from the image. The camera can first move parallel to the object’s plane: this motion induces a translation T that is not represented here. The camera can rotate around its optical axis (rotation parameter ψ). Its optical axis can take a θ angle with respect to the normal to the image plane u. This parameter is called latitude. The plane containing the normal and the new position of the optical axis makes an angle ϕ with a fixed “JMMBookOct04” 30/10/2009 page 120

[figure] [figure]

120 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON

vertical plane. This angle is called longitude. Last but not least, the camera can move forward or backward. This is the zoom parameter λ. The motion of a frontal view λ = 1, t = 1, ϕ = ψ = 0 to a slanted view corresponds to the image deformation u(x, y) → u(A(x, y)) given by (9.2).

9.2 A-SIFT : combining simulation and normalization

The idea of combining simulation and normalization is the main successful ingredient of the SIFT method. This method normalizes rotations and translations, but simulates all zooms out of the query and of the search images. Because of the feature, it is the only fully scale invariant method.

A-SIFT simulates with enough accuracy all distortions caused by a variation of the direction of the optical axis of a camera (two parameters). Then it normalizes the other four by the SIFT method, or any other method that is rotation, translation, and scale invariant. More specifically, the method proceeds by the following steps. (See Fig. 9.5.)

A-SIFT algorithm

1. Each image is transformed by simulating all possible affine distortions caused by the change of orientation of the camera axis of camera from a frontal position. These distortions depend upon two parameters: the longitude ϕ and the latitude θ. The images undergo ϕ-rotations followed by tilts with parameter t = |\frac{1}{cos θ} | (a tilt by t in the direction of x is the operation u(x, y) → u(tx, y)). For digital images, the tilt is performed as t-subsampling, and therefore requires the previous application of an antialiasing filter in the direction of x, namely the convolution by a gaussian with standard deviation c\sqrt{t^{2} − 1}. For good antialiasing, c ≃ 0.8, see Chapter 5.2.

2. These rotations and tilts are performed for a finite and small number of latitudes and longitudes, the sampling steps of these parameters ensuring that the simulated images keep close to any other possible view generated by other values of ϕ and θ.

3. All simulated images are compared by SIFT.

4. To be more specific, the latitudes θ are such that the associated tilts follow a geometric series 1, a, a^{2}, , …, a^{n}, with a > 1. The choice a = \sqrt{2} is a good compromise between accuracy and sparsity. The value n can go up to 6 or more, if the tilts are simulated on the query and the searched image, and up to 10 and more if the tilts are simulated on one image only. That way, transition tilts going up to 64 and more can be explored.

5. The longitudes ϕ are for each tilt an arithmetic series 0, b/t, …, kb/t, where b ≃ 72^{∘} seems again a good compromise, and k is the last integer such that kb/t < 180^{∘}.

“JMMBookOct04” 30/10/2009 page 121

[figure] [figure]

9.3. HIGH TRANSITION TILTS 121

Figure 9.5: Overview of the A-SIFT algorithm. The square images A and B represent the compared images u and v. A-SIFT simulates arbitrary camera changes of direction by applying rotations followed by a tilts to both images. The simulated images, represented by the parallelograms, are then compared with an image matching algorithm like SIFT, that is invariant to similarity transformations, i.e., invariant to scale change, rotation and translation.

9.3 High transition tilts

Equation (9.2) and its geometric interpretation in Fig. 9.4 are crucial to the scopes of this study. This last figure associates any linear map A with positive singular eigenvalues with the planar deformation u(A(x, y)) of a frontal view u(x, y), when the camera changes position. The parameter λ corresponds to a change of scale. The non critical translation parameter has been eliminated by assuming that the camera axis meets the image plane at a fixed point. Let us now consider the case where two camera positions, not necessarily frontal are at stake, corresponding to two different linear maps A and B. (Again, the translation parameter is left out of the discussion by fixing the intersection of the camera axis with image plane.) This physical situation is the generic one; when taking several snapshots of a scene, there is no particular reason why objects would be taken frontally. The resulting images are u_{1}(x, y) = u(A(x, y)) and u_{2}(x, y) = u(B(x, y)). Let us now take one of these images as reference image, and the other one as search image.

Definition 9.2. Given two views of a planar image, u_{1}(x, y) = u(A(x, y)) and u_{2}(x, y) = u(B(x, y)), we call transition tilt τ(u_{1}, u_{2}) and transition rotation ϕ(u_{1}, u_{2}) the unique parameters such that

BA^{−1} = H_{λ}R_{1}(ψ)T_{τ} R_{2}(ϕ),  (9.4)

with the notation of Formula (9.2).

It is an easy check that the transition tilt is symmetric, namely τ(u_{1}, u_{2}) = τ(u_{2}, u_{1}). Fig. 9.6 illustrates the affine transition between two images taken “JMMBookOct04” 30/10/2009 page 122

[figure] [figure]

122 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON

Figure 9.6: Illustration of the difference between absolute tilt and transition tilt.

from different viewpoints, and in particular the difference between absolute tilt and transition tilt. The camera is first put in two positions corresponding to absolute tilts t and t′, but with ϕ = ϕ′. The transition tilt between the resulting images v and v′ is τ = t′/t, assuming t′ = max(t′, t). On the second illustration of Fig. 9.6, the tilts are made in two orthogonal directions: ϕ = ϕ′ + π/2. Then an easy calculation shows that the transition tilt between v and v′ is the product τ(v, v′) = tt′. Thus, two moderate absolute tilts can lead to a large transition tilt! In the first case considered in the figure, the transition tilt is \sqrt{3} and therefore smaller than the absolute tilts. In the second case, the tilt is tt′ = 8. Since in realistic cases the tilt can go up to 6 or even 8, is easily understood that the transition tilt can go up to 36, 84, and more.

Exercise 9.3. The aim of the exercise is to prove that, given two views u = Au_{0} and v = Bu_{0} of a same image, with absolute tilts s and t satisfying t ≥ s, the transition tilt τ = τ(BA^{−1}) between u and v satisfies \frac{t}{s} ≤ τ ≤ st.

(i) Set A = H_{λ}Q_{1}T_{s}Q_{2} and B = H_{μ}R_{1}T_{s}R_{2}, where Q_{1}, Q_{2}, R_{1}, R_{2} are rotations, H_{λ} and H_{μ} homotheties and T_{s} and T_{t} tilts with t ≥ s. Show first that τ(BA^{−1}) = τ(T_{t}R_{2}Q_{2}^{−1}T_{s}^{−1}).

(ii) Deduce that if Q_{2} = R_{2}, then τ(BA^{−1}) = \frac{t}{s} . (iii) Deduce also that if R_{2}Q_{2}^{−1} = R( \frac{π}{2} ) then τ(BA^{−1}) = st. (iv) Set R(ϕ) = R_{2}Q_{2}^{−1}. Thus τ(BA^{−1}) = τ(C), with C =: T_{t}R(ϕ)T_{s^{−1}} Compute

the matrix C^{t}C. Check that its determinant is det(C^{t}C) = t^{2}s^{−2} and that its trace is tr(C^{t}C) = (s^{−2} − 1)(t^{2} − 1) cos^{2} φ + s^{−2} + t^{2}.

(v) Show that the eigenvalues λ_{1} ≥ λ_{2} of C^{t}C satisfy \frac{t^{2}}{s^{2}} ≤ \frac{λ_{1}}{λ_{2}} ≤ t^{2}s^{2}. (vi) Conclude.

Each recognition method can be characterized by its transition tilt, namely the variation in relative observation angle compatible with recognition. Fig. 9.13 shows the regions of the observation half sphere that can be attained for a given transition tilt, from a fixed viewpoint with various tilts. This figure “JMMBookOct04” 30/10/2009 page 123

[figure] [figure]

9.3. HIGH TRANSITION TILTS 123

shows perspective and zenith views of the observation half sphere. The central point of the bright region on the observation sphere is the original viewpoint from which an image has been taken. The rest of the bright region depicts the attainable observation region, namely the subset of the observation half sphere for which recognition succeeds for the given transition tilt. The latitude angle of the first image is respectively θ = 45, 60, 70, 80^{∘} that correspond respectively to the absolute tilts t = \sqrt{2}, 2, 2.9, 5.8. The three columns show the attainable regions on the half sphere for transition tilts t < 2.5, 5, , and 40. From the strong 80^{∘}, latitude, it needs a τ = 40 transition tilt to attain the rest of the sphere! SIFT and MSER only attain small regions.

Fig. 9.7 shows the A-SIFT results for a pair of images under orthogonal viewpoints (transition rotation ϕ = 90^{∘}) that leads to an extreme transition tilt t ≈ 37. This is not at all an exceptional situation. It just so happens that the object’s planar surface is observed at the same latitude by both views with a tilt t ≃ t′ ≃ 6. This figure shows two snapshots of a magazine lying on a table, not even really flat, and with a non lambertian surface plagued with reflections. The difference of longitudes being about 90 degrees, the transition tilt between both images is surprisingly high: τ = tt′ ≃ 37. Thus, it is many times larger than the transition tilt attainable with SIFT or MSER. A-SIFT finds 120 matches out of which only 4 are wrong.

Figure 9.7: Top: Image pair with transition tilt t ≈ 37. (SIFT, Harris-Affine, Hessian-Affine and MSER fail completely.) Bottom: A-SIFT finds 120 matches out which 4 are false. See comments in text.

The relevance of the notion of transition tilt is corroborated by the fact that the highest transition tilt τ_{max} permitting to match two images with absolute tilts t and t′ is fairly independent from t and t′. It has been experimentally checked that for SIFT τ_{max} ≃ 2.5 and for MSER τ_{max} ≃ 4.

To demonstrate this for SIFT, the transition tilts attainable by SIFT have “JMMBookOct04” 30/10/2009 page 124

[figure] [figure]

124 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON

been explored by systematic tilt simulations and tests. The experiments have been performed in the most favorable conditions for SIFT. The seed image u_{0} is a high quality frontal view of the Graffiti series. Tilted views from this frontal view were simulated by subsampling the image in one direction by a factor \sqrt{t}, and oversampling the image in the orthogonal direction by the same factor. That way, the absolute tilt is t, but the image area is not decreased. A set of tiltedrotated images u_{1} = u_{0}(t_{1}, 0) and u_{2} = u(t_{2}, ϕ) was generated by this method from u with absolute tilts t_{1} = (\sqrt{2})^{k}, k = 1, 2, …, 5, t_{2} = (2^{\frac{1}{4}} )^{l}, l = 1, 2, …, 14, and ϕ_{2} in a dense subset of [0, 90^{∘}]. The table shows for each pair t_{1}, t_{2} the maximal longitude ϕ_{max} ensuring that u_{1}(t_{1}, 0^{∘}) and u_{2}(t_{2}, ϕ_{max}) match. On the right of ϕ_{max}, the table displays in each box the corresponding transition tilt τ(t_{1}, 0, t_{2}, φ_{max}). Conspicuously enough, τ_{max} is most of the time close to 2.5. This experiment, and other similar ones, substantiate the empirical law that SIFT works for comparing images with transition tilts smaller than 2.5. In all of these tests, success with SIFT means that at least 20 correct SIFT descriptors, or SIFs, have been found.

Figure 9.8: Top and bottom: SIFT detects respectively 234 and 28 matches between a frontal image and two images with tilts t ≈ 2 and t ≈ 2.3. This latter value is close to the limiting tilt for SIFT to work.

“JMMBookOct04” 30/10/2009 page 125

[figure] [figure]

9.4. PARAMETER SAMPLING AND COMPLEXITY 125

t_{1} = \sqrt{2} t_{1} = 2 t_{1} = 2\sqrt{2} t_{1} = 4 t_{1} = 4\sqrt{2}
t_{2} = 2^{1/4}  90^{∘}/1.7 60^{∘}/2.2 0^{∘}/2.4
t_{2} = 2^{1/2}  90^{∘}/2.0 56^{∘}/2.4 11^{∘}/2.1
t_{2} = 2^{3/4}  90^{∘}/2.4 50^{∘}/2.6 20^{∘}/2.1 0^{∘}/2.4
t_{2} = 2  63^{∘}/2.6 36^{∘}/2.4 20^{∘}/2.1 9^{∘}/2.2
t_{2} = 2 × 2^{1/4} 37^{∘}/2.4 30^{∘}/2.3 23^{∘}/2.3 9^{∘}/1.9
t_{2} = 2 × 2^{1/2} 18^{∘}/2.6 22^{∘}/2.2 24^{∘}/2.6 12^{∘}/2.0 0^{∘}/1.4
t_{2} = 2 × 2^{3/4} 6^{∘}/2.4 16^{∘}/2.2 21^{∘}2.6 16^{∘}/2.5 5^{∘}/1.4
t_{2} = 4  0^{∘}/2.8 9^{∘}/2.2 18^{∘}/2.6 14^{∘}/2.4 9^{∘}/1.8
t_{2} = 4 × 2^{1/4}  4^{∘}/2.4 11^{∘}/2.2 12^{∘}/2.3 8^{∘}/2.0
t_{2} = 4 × 2^{1/2}  6^{∘}/2.2  7^{∘}/1.9  8^{∘}/2.3
t_{2} = 4 × 2^{3/4}  0^{∘}/2.4  5^{∘}/2.0  8^{∘}/2.5
t_{2} = 8  0^{∘}/2.0  7^{∘}/2.5
t_{2} = 8 × 2^{1/4}  4^{∘}/2.6
t_{2} = 8 × 2^{1/2}  3^{∘}/2.9

Table 9.1: m/n in each entry means: maximal longitude angle ϕ giving at least 20 matches by SIFT / corresponding transition tilt τ(t_{1}, t_{2}, ϕ). This table shows that SIFT covers a transition tilt τ ≈ 2.5.

9.4 Parameter sampling and complexity

9.4.1 Sampling ranges

The camera motion depicted in Fig. 9.4 shows that ϕ should naturally cover all the directions from 0 to 2π. But, by Theorem 9.1, it is enough to simulate ϕ from 0 to π to cover all possible linear transforms.

The sampling range of the tilt parameter t determines the degree of the tilt invariance the algorithm can achieve. Image recognition under a remarkable viewpoint change in practice requires that the scene is planar and Lambertian and its structures are not squashed when observed from an oblique viewpoint. Due to these physical limitations, affine image recognition is impractical under too big a tilt t. The physical upper bound t_{max} can be obtained experimentally using some images taken from indoor and outdoor scenes, each image pair being composed of a frontal view and an oblique view.

The images used in the experiments satisfy as much as possible the physical conditions mentioned above. The indoor scene is a magazine placed on a table with the artificial illumination coming from the ceiling as shown in Fig. 9.9. The outdoor scene is a building façade with some graffiti as illustrated in Fig. 9.10. For each pair of images, the true tilt parameter t between them is obtained by manual measurement. A-SIFT is applied with very large parameter sampling ranges and small sampling steps, so that the simulated views cover accurately the true affine distortion. The A-SIFT matching results depicted in Figs. 9.9 and 9.10 show that the limit is t_{max} ≈ 5.6 that corresponds to a view angle θ_{max} = arccos 1/t_{max} ≈ 80^{∘}. A-SIFT finds a large number of matches when the tilt between the frontal image and the oblique image is smaller than about 5.6. Therefore we set the tilt simulation range t_{max} = 4\sqrt{2}.

Let us emphasize that when the two images under comparison are taken from “JMMBookOct04” 30/10/2009 page 126

[figure] [figure]

126 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON

orthogonal longitude angles (see Fig. 9.7 as an example), i.e., ϕ = ϕ′ + π/2, the maximum tilt invariance A-SIFT with t_{max} = 4\sqrt{2} can achieve in theory is about t_{max}^{2} = 32.

However, these experiments only fix reasonable bounds for all purpose algorithms. For high resolution images, for very flat lambertian surfaces, larger tilts might be recognizable.

Figure 9.9: A-SIFT on an indoor scene. From top to bottom: tilt distortion t between the two images are respectively t ≈ 3, 5.2, 8.5; the number of matches are respectively 107 (3 false), 25 (7 false), and 7 (all false).

9.4.2 Sampling steps

In order to have A-SIFT invariant to any affine transform, one needs to sample the tilt t and angle ϕ with a high enough precision. The sampling steps △t and △ϕ must be fixed experimentally by testing several natural images.

The camera motion model illustrated in Fig. 9.4 indicates that the sampling precision of the latitude angle θ = arccos 1/t should increase with θ. A geometric sampling for t satisfies this requirement. Naturally, the sampling ratio △t = t_{k+1}/t_{k} should be independent of the angle ϕ. In the sequel, the tilt sampling step is experimentally fixed to △t = \sqrt{2}.

As can be observed from the camera motion model in Fig. 9.4, one needs a finer ϕ sampling when θ = arccos 1/t increases: the image distortion caused “JMMBookOct04” 30/10/2009 page 127

[figure] [figure]

9.4. PARAMETER SAMPLING AND COMPLEXITY 127

Figure 9.10: A-SIFT on an outdoor scene. From top to bottom: tilt distortion t between the two images are respectively t ≈ 3.8, 5.6, 8; the number of matches are respectively 71 (4 false), 33 (4 false), 10 (all false).

“JMMBookOct04” 30/10/2009 page 128

[figure] [figure]

128 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON

by a fixed longitude angle displacement △ϕ, is much more drastic when the latitude angle θ increases. The longitude sampling step in the sequel will be △ϕ = 2 × \frac{36^{∘}}{t} = \frac{72^{∘}}{t}.

Fig. 9.11 illustrates the sampling of the parameters θ = arccos 1/t and ϕ. At bigger θ the sampling of θ as well as the sampling of ϕ are denser.

Figure 9.11: Sampling of the parameters θ = arccos 1/t and ϕ. Black dots represent the sampling. Left: perspective illustration (only t = 2, 2\sqrt{2}, 4 are shown). Right: zenith view of the observation half sphere. The values of θ are indicated on the figure.

9.4.3 Acceleration with multi-resolution

The multi-resolution procedure accelerates A-SIFT by selecting the transforms that yield SIFT matches on low-resolution (LR) versions of the compared images. In case of success only, the procedure simulates the identified affine transforms on the query, and applies SIFT to compare them to the targets.

The multi-resolutions A-SIFT is summarized as follows.

1. Down-sample all compared digital images u and v by a K × K factor: u′ = S_{K}G_{K}u and v′ = S_{K}G_{K}v, where G_{K} is an anti-aliasing gaussian discrete filter.

2. Low-resolution (LR) A-SIFT: perform A-SIFT between u′ and v′.

3. Identify the M affine transforms yielding the biggest numbers of matches between u′ and v′. They are retained only if the matches are meaningful. In practice, it is enough to put a threshold on the number k of matches, and k = 15 seems to be a good choice.

4. High-resolution (HR) A-SIFT: apply A-SIFT between u and v by simulating only the affine transforms previously identified.

Fig. 9.12 shows an example. The low-resolution A-SIFT that is applied on the 3 × 3 sub-sampled images finds 12 correspondences and identifies the 5 best affine transforms. The high-resolution A-SIFT finds 133 matches.

“JMMBookOct04” 30/10/2009 page 129

[figure] [figure]

9.4. PARAMETER SAMPLING AND COMPLEXITY 129

Figure 9.12: “77 Mass Ave”. Left: low-resolution A-SIFT, 12 matches, three of which are wrong. Right: high-resolution A-SIFT, 133 matches. Due to the lack of details in these images, the number of matches at low resolution is critically low.

9.4.4 A-SIFT Complexity

The complexity of the A-SIFT algorithm will be estimated under the recommended baseline configuration: The tilt and angle ranges are [t_{min}, t_{max}] = [1, 4\sqrt{2}] and [ϕ_{min}, ϕ_{max}] = [0^{∘}, 180^{∘}], and the sampling steps are △t = \sqrt{2}, △ϕ = 36^{∘} × \frac{t}{2}. Each t tilt is simulated by image sub-sampling in one direction by a t factor. All images are sub-sampled by a K × K = 3 × 3 factor for the low-resolution A-SIFT. Finally, the high-resolution A-SIFT simulates the M = 5 best affine transformations that are identified, but only in case they contain enough matches. When matching an image to a large database, the most common event is failure. Thus, the final high-resolution step is only to be taken into account when comparing images of the same scene.

The complexity of the descriptor computation is proportional to the input image area. This area is proportional to the number of simulated tilts t. Indeed, the number of ϕ simulations is proportional to t for each t, but the t sub-sampling for each tilt simulation divides the area by t. More precisely, the image area input to low-resolution A-SIFT is

\frac{1 + (|Γ_{t}| − 1)\frac{180^{∘}}{2×36^{∘}}}{K^{2}} = \frac{1 + 5 × 2.5}{9} = 1.5

times as large as that of the original images, where |Γ_{t}| is the number of tilt simulations. Thus, the complexity of the low-resolution A-SIFT is 1.5 times as much as that of a single SIFT routine, and generates 1.5 as many SIFs. Here we must distinguish two cases:

1. If the comparisons involve a large database (where most comparisons will be failures), the complexity is propositional to the number of SIFs in the queries multiplied by the number of SIFs in the targets. Since A-SIFT “JMMBookOct04” 30/10/2009 page 130

[figure] [figure]

130 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON

introduces a a 1.5 area factor, the final complexity is simply 1.5^{2} = 2.25 times the SIFT complexity.

2. If the comparisons involve a set of images with high match likeliness, then the high resolution step is no more negligible. Then, it can only be asserted that the complexity will be less than 6.5 + 2.5 = 9 times a SIFT routine on the same images. However, in that case, A-SIFT ensures many more detections than SIFT, because it explores many more viewpoint angles. Thus, the complexity rate per detected SIF might be much closer to, or even smaller than the per detection complexity in a SIFT routine.

For the high-resolution A-SIFT, this factor is M = 5. Therefore the total complexity of the A-SIFT is 6.5 times a SIFT routine.

The SIFT subroutines can be implemented in parallel in A-SIFT (for both the low-resolution and the high-resolution A-SIFT). Recently many authors have investigated SIFT accelerations [126, 85, 143]. A realtime SIFT implementation has been proposed in [238]. Obviously, all of these accelerations directly apply to A-SIFT.

“JMMBookOct04” 30/10/2009 page 131

[figure] [figure]

9.4. PARAMETER SAMPLING AND COMPLEXITY 131

Figure 9.13: Each recognition method can be characterized by its transition tilt, namely the variation in relative observation angle compatible with recognition. This figure shows perspective and zenith views of the observation sphere. The central point of the bright region on the observation sphere is the original viewpoint from which an image has been taken. The rest of the bright region depicts, for several positions of this original point and several transition tilts, the attainable observation region, namely all other view angles in the observation sphere for which recognition succeeds for the given transition tilt. From top to bottom: latitude angle of the first image θ = 45, 60, 70, 80^{∘} that correspond respectively to the absolute tilts t = \sqrt{2}, 2, 2.9, 5.8. From left to right: transition tilt < 2.5, 5, 40.

“JMMBookOct04” 30/10/2009 page 132

[figure] [figure]

132 CHAPTER 9. AFFINE INVARIANT IMAGE COMPARISON “JMMBookOct04” 30/10/2009 page 133

[figure] [figure]

Chapter 10

The mathematical justification

This chapter gives the mathematical formalism and a mathematical proof that A-SIFT is fully affine invariant, up to sampling errors. The next chapter 11.3 is devoted to many comparative experiments where all mentioned state-of-the art algorithms are compared for their scale and tilt invariance.

In this chapter, to lighten the notation of the gaussian, G_{σ} will denote the

convolution operator on ℝ^{2} with the gauss kernel G_{σ}(x, y) = \frac{1}{2π(cσ)^{2}} e^{− \frac{x^{2}+y^{2}}{2(cσ)^{2}}} , namely Gu(x, y) =: (G ∗ u)(x, y). The constant c ≥ 0.8 is large enough to ensure that all considered images can be sampled with sampling mesh 1 after convolution with G_{1}. The one dimensional gaussians will be denoted by

G_{σ}^{x}(x, y) = \frac{1}{\sqrt{2π}cσ} e^{− \frac{x^{2}}{2(cσ)^{2}}} and G_{σ}^{y}(x, y) = \frac{1}{\sqrt{2π}cσ} e^{− \frac{y^{2}}{2(cσ)^{2}}} . G_{σ} still satisfies the semigroup property

G_{σ}G_{β} = G\sqrt{σ^{2}+β^{2}}.  (10.1)

The proof of the next formula is a mere change of variables in the integral defining the convolution.

G_{σ}H_{γ}u = H_{γ}G_{σγ}u.  (10.2)

Exercise 10.1. Prove (10.2).

Using the above notation, the next paragraph formalizes the SIFT method.

10.0.5 The image formation model

As developed in Section 9.1, the whole image comparison process, based on local features, can proceed as though images where (locally) obtained by using digital cameras that stand far away, at infinity. The geometric deformations induced by the motion of such cameras are affine maps. A model is also needed for the two main camera parameters not deducible from its position, namely sampling and blur. The digital image is defined on the camera CCD plane. The pixel width can be taken as length unit, and the origin and axes chosen so that the camera pixels are indexed by IZ^{2}. The associated image sampling operator will be denoted by S_{1}. The digital initial image is always assumed well-sampled and obtained by a gaussian blur with standard deviation 0.6. (See

133 “JMMBookOct04” 30/10/2009 page 134

[figure] [figure]

134 CHAPTER 10. THE MATHEMATICAL JUSTIFICATION

[184] for a detailed analysis of why this model is sufficient and coherent for most digital images, and compatible with the SIFT method.) In all that follows, u_{0} denotes the (theoretical) infinite resolution image that would be obtained by a frontal snapshot of a plane object with infinitely many pixels. The digital image obtained by any camera at infinity is u = S_{1}G_{1}AT u_{0}, where A is any linear map with positive singular values and T any plane translation. Thus we can summarize the general image formation model with cameras at infinity as follows.

Figure 10.1: The projective camera model u = S_{1}G_{1}Au_{0}. A is a planar projective transform (a homography). G_{1} is an anti-aliasing gaussian filtering. S_{1} is the CCD sampling.

Definition 10.1. Image formation model. Digital images of a planar object whose frontal infinite resolution image is u_{0}, obtained by a digital camera far away from the object, satisfy

u =: S_{1}G_{1}AT u_{0}  (10.3)

where A is any linear map and T any plane translation. G_{1} denotes a gaussian kernel broad enough to ensure no aliasing by 1-sampling, namely IS_{1}G_{1}AT u_{0} = G_{1}AT u_{0}.

The formal description of A-SIFT will be by far simpler if sampling issues do not interfere. All operations and all reasoning will be made with continuous well sampled images. It is easy to deduce afterwards the discrete operators acting on samples. T denotes an arbitrary translation, R an arbitrary rotation, H_{λ} an arbitrary homothety, and G an arbitrary gaussian convolution, all applied to continuous images. In the particular case in the digital image formation model (10.3) where A is a frontal view of u_{0}, A = HRT is the composition of a translation T , a homothety H, and a rotation R. Thus the digital image is u = S_{1}G_{1}HT Ru_{0}.

10.0.6 Inverting tilts

We shall denote by ∗_{y} the 1-D convolution convolution operator in the ydirection. When we write G∗_{y}, we mean that G is a one-dimensional gaussian, “JMMBookOct04” 30/10/2009 page 135

[figure] [figure]

135

depending on y, and the 1-D convolution means

G ∗_{y} u(x, y) =: ∫ G^{y}(z)u(x, y − z)dz.

There are three different notions of tilt, that we must carefully distinguish.

Definition 10.2. Given t > 1, the tilt factor, define

• the absolute tilt : T_{t}^{x}u_{0}(x, y) =: u_{0}(tx, y). In case this tilt is made in the y direction. It will be denoted by T_{t}^{y}u_{0}(x, y) =: u_{0}(x, ty);

• the continuous tilt (taking into account camera blur): 𝕋_{t}^{x}v =: T_{t}^{x}G_{√\overline{t^{2}−1}}^{x} ∗_{x} v. In case the simulated tilt is done in the x direction, it is denoted 𝕋_{t}^{y}v =: T_{t}^{y}G_{√\overline{t^{2}−1}}^{y} ∗_{y} v.

• the digital tilt (transforming a digital image u into a digital image) : u → S_{1}𝕋_{t}^{x}Iu. This is the one that is used in the algorithm. It is correct because, as we shall see, the simulated tilt yields a blur permitting S_{1}-sampling.

If u_{0} is an infinite resolution image observed with a t camera tilt in the x direction, the observed image is G_{1}T_{t}^{x}u_{0}. Our main problem is to reverse such tilts. This operation is in principle impossible, because absolute tilts do not commute with blur. However, the next lemma shows that 𝕋_{t}^{y} is actually a pseudo inverse to T_{t}^{x}.

Lemma 10.3. One has

𝕋_{t}^{y} = H_{t}G_{√\overline{t^{2}−1}}^{y} ∗_{y} (T_{t}^{x})^{−1}.

Proof Since (T_{t}^{x})^{−1}u(x, y) = u(\frac{x}{t}, y),

(G_{√\overline{t^{2}−1}}^{y} ∗_{y} (T_{t}^{x})^{−1}u) (x, y) = ∫ G\sqrt{t^{2}−1}(z)u(\frac{x}{t}, y − z)dz.

Thus

H_{t} (G_{√\overline{t^{2}−1}}^{y} ∗_{y} (T_{t}^{x})^{−1}u) (x, y) = ∫ G\sqrt{t^{2}−1}(z)u(x, ty − z)dz =
(G_{√\overline{t^{2}−1}}^{y} ∗_{y} u) (x, ty) = (T_{t}^{y}G_{√\overline{t^{2}−1}}^{y} ∗_{y} u) (x, y).

□

The meaning of the next result is that a tilted image G_{1}T_{t}^{x}u can be tilted back by tilting in the orthogonal direction. The price to pay is a t zoom out. The second relation in the theorem means that the application of the simulated tilt to an image that can be well sampled by S_{1} yields an image that keeps that well sampling property.

Theorem 10.4. Let t ≥ 1. Then

𝕋_{t}^{y}(G_{1}T_{t}^{x}) = G_{1}H_{t};  (10.4)
𝕋_{t}^{y}G_{1} = G_{1}T_{t}^{y}.  (10.5)

“JMMBookOct04” 30/10/2009 page 136

[figure] [figure]

136 CHAPTER 10. THE MATHEMATICAL JUSTIFICATION

Proof By Lemma 10.3, we recall that

𝕋_{t}^{y} = H_{t}G_{√\overline{t^{2}−1}}^{y} ∗_{y} (T_{t}^{x})^{−1}.

Thus,

𝕋_{t}^{y}(G_{1}T_{t}^{x}) = H_{t}G_{√\overline{t^{2}−1}}^{y} ∗_{y} ((T_{t}^{x})^{−1}G_{1}T_{t}^{x}).  (10.6)

By a variable change in the integral defining the convolution, it is an easy check that

(T_{t}^{x})^{−1}G_{1}T_{t}^{x}u = (\frac{1}{t}G_{1}(\frac{x}{t}, y)) ∗ u,  (10.7)

and by the separability of the 2D gaussian in two 1D gaussians,

\frac{1}{t}G_{1}(\frac{x}{t}, y) = G_{t}(x)G_{1}(y).  (10.8)

From (10.7) and (10.8) one obtains

(T^{x})^{−1}G_{1}T_{t}^{x}u = ((G_{t}^{x}(x)G_{1}^{y}(y)) ∗ u = G_{t}^{x}(x) ∗_{x} G_{1}^{y}(y) ∗_{y} u,

which implies

G_{√\overline{t^{2}−1}}^{y} ∗_{y} (T^{x})^{−1}G_{1}T_{t}^{x}u = G_{√\overline{t^{2}−1}}^{y} ∗_{y} (G_{t}^{x}(x) ∗_{x} G_{1}^{y}(y) ∗_{y} u) = G_{t}u.

Indeed, the 1D convolutions in x and y commute and G_{t}^{y} ∗ G_{√\overline{t^{2}−1}}^{y} = G_{t}^{y} by the Gaussian semigroup property (10.1). Substituting the last proven relation in (10.6) yields

𝕋_{t}^{y}G_{1}T_{t}^{x}u = H_{t}G_{t}u = G_{1}H_{t}u.

The second relation (10.5) follows immediately by noting that H_{t} = T_{t}^{y}T_{t}^{x}. □

Exercise 10.2. Prove (10.7).

10.0.7 Proof that A-SIFT works

The meaning of Theorem 10.4 is that we can design an exact algorithm that simulates all inverse tilts for comparing two images. After interpolation, A-SIFT handles two images u = G_{1}AT_{1}u_{0} and v = G_{1}BT_{2}u_{0} that are two snapshots from different view points of a flat object whose front infinite resolution image is denoted by u_{0}. For a sake of simplicity, we break the symmetry, and set ũ_{0} =: AT_{1}u_{0}, so that u = G_{1}ũ_{0} and v = G_{1}BT_{2}T_{1}^{−1}A^{−1}ũ_{0} = G_{1}BA^{−1}T ũ_{0} for a translation T that depends on T_{1}, T_{2}, and A. Let us use the decomposition given by (9.2),

BA^{−1} = R_{1}T_{t}^{x}H_{λ}R_{2},

where R_{1}, R_{2} are rotations, H_{λ} a zoom, and T_{t}^{x}(x, y) = (tx, y) is the transition tilt from u to v. In summary A-SIFT has to compare the interpolated images

v = G_{1}R_{1}T_{t}^{x}H_{λ}R_{2}T ũ_{0} and u = G_{1}ũ_{0}.

“JMMBookOct04” 30/10/2009 page 137

[figure] [figure]

137

The A-SIFT formal algorithm

The following algorithm, where image sampling issues are eliminated by interpolation, is actually a proof that A-SIFT manages to compare u and v obtained from u_{0} by arbitrary camera positions at infinity. In this ideal algorithm, a “‘dense enough” set of rotations and tilts is applied to v, so that each one of the simulated rotation-tilts is “close enough” to any other rotation-tilt. In the mathematical setting, this approximation must be infinitesimal. In the practical empirical setting, we’ll have to explore how dense the sets of rotations and tilts must be (see Section 9.4).

A-SIFT Algorithm (formal)

1. Apply a dense set of all possible rotations (and therefore also a rotation close to R_{1}^{−1}) to v. Thus, some of the simulated images will be arbitrary close to v → R_{1}^{−1}G_{1}R_{1}H_{λ}T_{t}^{x}R_{2}T ũ_{0} = G_{1}T_{t}^{x}H_{λ}R_{2}T ũ_{0};

2. apply in continuation a dense set of simulated tilts 𝕋_{t}^{y}, and therefore also one arbitrary close to the right one 𝕋_{t}^{y} = T_{t}^{y}G_{√\overline{t^{2}−1}}^{y}∗_{y}, to R_{1}^{−1}v = G_{1}T_{t}^{x}H_{λ}R_{2}T ũ_{0}. By Theorem 10.4 we have the commutation T_{t}^{y}G_{1} = G_{1}T_{t}^{y}, which yields

𝕋_{t}^{y}R_{1}^{−1}v = G_{1}H_{t}H_{λ}R_{2}T ũ_{0} = G_{1}H_{tλ}R_{2}T ũ_{0};

3. perform a SIFT comparison of G_{1}H_{tλ}R_{2}T ũ_{0}, which is a frontal view of ũ_{0}, with u = G_{1}ũ_{0} which also is a frontal view of ũ_{0}.

The above algorithm description is also a proof of the following consistency theorem, since the SIFT method finds all SIFs common to two frontal views (Theorem 6.5).

Theorem 10.5. Let u = G_{1}AT_{1}u_{0} and v = BT_{2}u_{0} be two images obtained from an infinite resolution image u_{0} by cameras at infinity with arbitrary position and focal lengths. Then A-SIFT, applied with a dense set of tilts and longitudes, simulates two views of u and v that are obtained from each other by a translation, a rotation, and a camera zoom. As a consequence, these images match by the SIFT algorithm.

Remark 10.6. Even if the above proof, and the statement of Lemma ??, deal with asymptotic statements when the sampling steps tend to infinity or when the SIFT scales tend to infinity, the approximation rate is very quick, a fact that can only be checked experimentally. This fact is actually extensively verified by the huge amount of experimental evidence on SIFT, that shows first that the recognition of scale invariant features (SIFs) is robust to a substantial variation of latitude and longitude, and second that the scale invariance is quite robust to moderate errors on scale. Section 9.4 has evaluated the adequate sampling rates and ranges for tilts and longitudes.

Simulating midway tilts

The algorithm of Section 10.0.7 can be implemented in several ways. In the above description, the transition tilt T_{t}^{x} is directly inverted on one of the images.

“JMMBookOct04” 30/10/2009 page 138

[figure] [figure]

138 CHAPTER 10. THE MATHEMATICAL JUSTIFICATION

This strategy is consistent, but not optimal. As we have seen, the transition tilt can be very large. It is preferable to simulate moderate tilts on two images that large tilts on one of them. To this aim a midway image can be reached from both images by applying a \sqrt{t} tilt to one of them and a \sqrt{t} tilt to the other one. The only change to the formal algorithm will be that rotations and tilts are applied to both images, not just to one of them.

Midway A-SIFT (formal)

1. Apply a dense set of all possible rotations to both images, and therefore R_{2} to u and R_{1}^{−1} to v;

2. apply in continuation a dense set of simulated tilts 𝕋_{t}^{x} in a fixed [0, t_{max}] range;

3. perform a SIFT comparison of all pairs of resulting images.

Let us now prove that this algorithm works, namely that two of the simulated images are deduced from each other by a similarity. The query and target images are u = G_{1}AT_{1}u_{0} and v = G_{1}BT_{2}u_{0}. By the usual decomposition of a linear map (9.2),

BA^{−1} = R_{1}T_{t}^{x}R_{2}H_{λ} = (R_{1}T\sqrt{t}^{x})(T_{√\overline{t}}^{x}R_{2}H_{λ}).

Notice that by the relation

𝕋_{t}^{x}R(−\frac{π}{2} ) = R(\frac{π}{2} )𝕋_{t}^{y},  (10.9)

the algorithm also simulates tilts in the y direction, up to R(\frac{π}{2} ) rotation. In particular, the above algorithm applies:

1. 𝕋_{√\overline{t}}^{x}R_{2} to G_{1}AT_{1}u_{0}, which by (10.5) yields ũ = G_{1}T_{√\overline{t}}^{x}R_{2}AT_{1}u_{0} =: G_{1}ÃT_{1}u_{0};

2. R(\frac{π}{2} )𝕋_{√\overline{t}}^{y}R_{1}^{−1} to G_{1}BT_{2}u_{0}, which by (10.5) yields G_{1}R(\frac{π}{2} )T_{√\overline{t}}^{y}R_{1}^{−1}BT_{2}u_{0} =:

G_{1}B̃T_{2}u_{0}.

Let us show that Ã and B̃ only differ by a similarity. Indeed,

B̃^{−1}R(\frac{π}{2} )H\sqrt{t}Ã = B^{−1}R_{1}T_{\sqrt{t}^{−1}}^{y}T_{√\overline{t}}^{x}H\sqrt{t}R_{2}A =

B^{−1}R_{1}T_{\sqrt{t}^{−1}}^{y}T_{√\overline{t}}^{x}H\sqrt{t}R_{2}A = B^{−1}R_{1}T_{t}^{x}R(\frac{π}{2} )R_{2}A = B^{−1}(BA^{−1})A = I,

where I is the identity. It follows that B̃ = R(\frac{π}{2} )H\sqrt{t}Ã. Thus,

ũ = G_{1}ÃT_{1}u_{0} and ṽ = G_{1}R(\frac{π}{2} )H\sqrt{t}ÃT_{2}u_{0},

that are two of the simulated images, are deduced from each other by a rotation and a \sqrt{t} zoom. It follows that their SIFs are identical as soon as the scale of the SIF exceeds \sqrt{t}.

□

Exercise 10.3. There is a (non crucial) error in the above proof. Read this proof carefully, find the error, and adjust the proof.

“JMMBookOct04” 30/10/2009 page 139

[figure] [figure]

139

10.0.8 Conclusion on the algorithms

The above descriptions have neglected the sampling issues, but care was taken that input images and output images be always written in the G_{1}u form. For the digital input images, that always have the form u = S_{1}G_{1}u_{0}, the Shannon interpolation algorithm is I is first applied, to give back IS_{1}G_{1}u_{0} = G_{1}u_{0}. For the output images, that always have the form G_{1}v, the sampling S_{1} gives back a digital image.

Thus, the descriptions of the formal algorithm A-SIFT and of its “midway” version are changed into a digital algorithm by:

• replacing everywhere the inputs G_{1}u by their digital version S_{1}G_{1}u;

• by applying digital rotations to digital images : u → Ru =: S_{1}RIu;

• by applying digital tilts as defined in Def. 10.2, namely u → S_{1}T_{t}^{x}Iu.

That way, the formal algorithms are transformed into digital algorithms. The proofs need not be repeated, since by Shannon interpolation and sampling, it is equivalent to talk about S_{1}G_{1}u_{0} or about G_{1}u_{0}.

Clearly the midway algorithm is better, because it only needs simulating tilts that are square roots of the real transition tilts. Thus, all of the forthcoming discussion will focus on the midway version, that we’ll simply call A-SIFT.

“JMMBookOct04” 30/10/2009 page 140

[figure] [figure]

140 CHAPTER 10. THE MATHEMATICAL JUSTIFICATION “JMMBookOct04” 30/10/2009 page 141

[figure] [figure]

Chapter 11

Experiments on affine invariant methods

11.1 Affine normalization methods: are they fully affine invariant?

Since the affine transform depends upon six parameters, it is out of the question to just simulate all of them and compare the original image to all deformed images by all possible affine deformations. However, simulation can be a solution for a few parameters: the SIFT method actually simulates zooms out.

The other way that has been tried by many authors is normalization. Normalization is a magic method that, given a patch that has undergone an unknown affine transform, transforms the patch into a standardized one, where the effect of the affine transform has been eliminated (see Fig. 11.1). Normalization by translation is easily achieved: A patch around (x_{0}, y_{0}) is translated back to a patch around (0, 0). A rotational normalization requires a circular patch. In this patch, a principal direction is found, and the patch is rotated so that this principal direction coincides with a fixed direction. Thus, of the six parameters in an affine transform, at least three are easily eliminated by normalization.

Figure 11.1: Normalization methods can eliminate the effect of a a class of affine transforms by associating the same standard patch to all transformed patches.

However, when it comes to the other three parameters, things get difficult and controversial. Two methods have been recently proposed to perform a level lines based full affine normalization: MSER [168] and LLD [189]. Both of them apply to image level lines, or to image pieces of level lines, an affine normalization

141 “JMMBookOct04” 30/10/2009 page 142

[figure] [figure]

142 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

in the spirit of the translation and rotation normalization explained above. We shall focus on MSER, but the discussion applies to LLD as well.

11.2 Global Normalization and Encoding

11.2.1 Global Affine Normalization

Classical shape normalization methods are based on the inertia matrix normalization. We shall use Cohignac’s presentation of this method, given in [41]. Denote by 1_{F} the indicator function of a shape domain F. The shape is usually associated a weight function defined on the shape domain, u(x, y) = φ(x, y)1_{F}. Classically, φ(x, y) is the restriction of an image to the shape extracted from it, or the restriction to the shape of the image gradient, or the restriction to the shape of the image gradient direction, etc. Thus when we talk about “the shape F”, we actually talk about u(x, y). Of course, when φ(x, y) = 1, the whole analysis of this section boils down to the analysis of the proper shape u = 1_{F}. In all that follows, a convenient abbreviation is F for u(x, y). If A is linear or affine map, AF denotes the function Au defined as usual by Au(x, y) = u(A(x, y)). In order to achieve translation invariance of the normalized representation, it may be assumed that F has been previously translated so that its barycenter weighted by u(x, y) is at the origin of the image plane.

Exercise 11.1. Show that this last assumption amounts to assume that

μ_{1,0}(F) =: ∫_{IR^{2}} xu(x, y)dxdy = 0 and μ_{0,1}(F) =: ∫_{IR^{2}} yu(x, y)dxdy = 0.

More precisely, give a formula for the weighted barycenter b(F) of F and show that the barycenter is covariant by any affine transform, namely b(AF) = Ab(F) for every linear transform A. Show that the weighted area of F defined by μ_{0,0}(F) =: ∫_{IR^{2}} u(x, y)dxdy satisfies μ_{0,0}(AF) = |det(A)|μ_{0,0}(F).

The moment of order (p, q) (p and q natural integers) of the shape F (weighted by u) is defined by

μ_{p,q}(F) = ∫_{IR^{2}} x^{p}y^{q}u(x, y)dxdy.

Let S_{F} be the following 2 × 2 positive-definite, symmetric matrix

S_{F} = \frac{1}{μ_{0,0}} ( \begin{matrix}μ_{2,0} & μ_{1,1} \\ μ_{1,1} & μ_{0,2} \end{matrix} ) ,

where μ_{i,j} = μ_{i,j}(F). By the uniqueness of Cholesky factorization [94], S_{F} may be uniquely decomposed as S_{F} = B_{F}B_{F}^{t} where B_{F} is a lower-triangular real matrix with positive diagonal entries.

Definition 11.1. The pre-normalized shape associated to F is the shape F′ = B_{F}^{−1}(F).

The aim is to prove that the pre-normalized solid shape is invariant to affine transformations, up to a rotation.

Lemma 11.2. Let A be a non-singular 2 × 2 matrix. Then S_{AF} = AS_{F}A^{t}.

“JMMBookOct04” 30/10/2009 page 143

[figure] [figure]

11.2. GLOBAL NORMALIZATION AND ENCODING 143

Proof. Let a, b, c and d be real numbers such that:

A = ( \begin{matrix}a & b \\ c & d \end{matrix} ) .

The moment of order (2, 0) associated to the solid shape AF is

μ_{2,0}(AF) = det(A) ∫_{IR^{2}}(ax + by)^{2}u(x, y)dxdy
= det(A)(a^{2}μ_{2,0} + 2abμ_{1,1} + b^{2}μ_{0,2}).

The same computation for moments of order (0, 2) and (1, 1) yields

μ_{0,2}(AF) = det(A)(c^{2}μ_{2,0} + 2cdμ_{1,1} + d^{2}μ_{0,2}),
μ_{1,1}(AF) = det(A)(acμ_{2,0} + bdμ_{0,2} + (ad + bc)μ_{1,1}).

Since μ_{0,0}(AF) = det(A)μ_{0,0}, one can easily check that S_{AF} = AS_{F}A^{t}. □

Exercise 11.2. Make the calculation proving that S_{AF} = AS_{F}A^{t}.

Lemma 11.3. Let X_{0} be a 2 × 2 invertible matrix. Then, for any 2 × 2 matrix X: XX^{t} = X_{0}X_{0}^{t} if and only if there exists an orthogonal matrix Q such that X = X_{0}Q.

Proof. Since X_{0} is invertible, XX^{t} = X_{0}X_{0}^{t} iff X_{0}^{−1}X(X_{0}^{−1}X)^{t} = Id_{2}. Letting Q = X_{0}^{−1}X yields the result. □

Proposition 11.4. The pre-normalized solid shape is invariant to any invertible, planar, linear transformation (x, y)^{t} ↦ A(x, y)^{t}, up to an orthogonal transformation. Moreover, if det(A) > 0, the invariance holds up to a rotation.

Proof. Since A is a 2 × 2 non singular matrix, following Lemma 11.2, S_{AF} = AS_{F}A^{t}. By letting B_{F} be the lower-triangular matrix of Cholesky’s decomposition of B_{F}, it follows that S_{AF} = AB_{F}(AB_{F})^{t}. Now, since S_{AF} is a 2 × 2 positive-definite, symmetric matrix, Cholesky factorization yields S_{AF} = B_{AF}B_{AF}^{t}, where B_{AF} is a 2 × 2 non-singular, lower-triangular real matrix. Then, by Lemma 11.3, B_{AF} = AB_{F}Q, where Q is a 2 × 2 orthogonal matrix. Hence, B_{AF}^{−1}AF = (AB_{F}Q)^{−1}AF = Q^{−1}B_{F}^{−1}A^{−1}AF = Q^{−1}B_{F}^{−1}F, which proves the invariance of F′ = B_{F}^{−1}F to planar isomorphisms, up to an orthogonal transformation. Finally, notice that if det(A) > 0, then det(Q) > 0. □

Exercise 11.3. Prove that a closed form for B_{F}^{−1} in terms of the moments of F can be computed by taking the inverse of B_{F}, the lower-triangular matrix given by the Cholesky decomposition of S_{F},

B_{F}^{−1} = \sqrt{μ_{0,0}} ( \begin{matrix}\frac{1}{\sqrt{μ_{2,0}}} & 0 \\ −_{\frac{μ_{1,1}}{μ_{2,0}\sqrt{μ_{0,2}− \frac{μ{2 \atop 1,1}}{μ_{2,0}}}}} & \frac{1}{\sqrt{μ_{0,2}− \frac{μ{2 \atop 1,1}}{μ_{2,0}}}} \end{matrix} ) .

“JMMBookOct04” 30/10/2009 page 144

[figure] [figure]

144 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

The pre-normalized solid shape F′ = B_{F}^{−1}F is then an affine invariant representation of F modulo a rotation. In order to obtain a full affine invariant representation, only a reference angle is needed. This can be achieved, for instance, by computing in polar coordinates

φ = Arg (∫_{0}^{2π} ∫_{0}^{+∞}(B_{F}^{−1}u)(r, θ)e^{iθ}rdrdθ) ,

then rotating F′ by −φ.

Putting all the steps together, the support of the affine invariant normalization of F is the set of points (x_{N}, y_{N}) given by

( \begin{matrix}x_{N} \\ y_{N} \end{matrix} ) = ( \begin{matrix}cos φ & sin φ \\ − sin φ & cos φ \end{matrix} ) B_{F}^{−1} ( \begin{matrix}x − μ_{1,0} \\ y − μ_{0,1} \end{matrix} ) ,

for all (x, y) ∈ F.

11.2.2 Maximally Stable Extremal Regions (MSER)

The MSER method introduced by Matas et al. [168] attempts to achieve affine invariance by selecting the most robust connected components of upper and lower level sets as image features.

Extremal regions is the name given by the authors to the connected components of upper or lower level sets. Maximally stable extremal regions, or MSERs, are defined as maximally contrasted regions in the following way. let Q_{1}, ..., Q_{i−1}, Q_{i}, ... be a sequence of nested extremal regions, i.e. Q_{i} ⊂ Q_{i+1} where Q_{i} is defined by a threshold at level i or, in other terms, Q_{i} is an upper (resp. lower) level set at level i. An extremal region in the list Q_{i_{0}} is said to be maximally stable if the area variation q(i) =: |Q_{i+1} \ Q_{i−1}|/|Q_{i}| has a local minimum at i_{0}, where |Q| denotes the area of a region |Q|. Clearly the above measure is a measure of contrast along the boundary ∂Q_{i} of Q_{i}. Indeed, assuming that u is C^{1} and that the grey level increment between i and i + 1 is infinitesimal, the area |Q_{i+1} \Q_{i−1}| varies least when ∫_{∂Q_{i}} |∇u| is maximal. The MSER extraction is a first step of image matching. Once MSERs are computed, the affine normalization of Section 11.2.1 is performed on the MSERs before they can be compared. The fact that the method is not fully scale invariant is easily explained with the experiment of Fig. 11.2. In MSER the scale normalization is based on the size (area) of the detected extremal regions. However, scale change is not just a homothety: it involves a blur followed by sub-sampling. The blur changes drastically the size of the regions. As pointed out in [42] this entails a strong lack of scale invariance. It could only be compensated by simulating actual blur on the images, as made by the SIFT method.

11.3 Experiments

A-SIFT image matching performance will be compared with the state-of-the-art approaches with the detectors DoG [151], Hessian-Affine, Harris-Affine [172, 175] and MSER [168] all coded by the most popular SIFT descriptor [151]. The MSER detector followed by the correlation descriptor as proposed in the original work [168] is also included in the comparison, whose performance will be shown “JMMBookOct04” 30/10/2009 page 145

[figure] [figure]

11.3. EXPERIMENTS 145

Figure 11.2: Shapes change with distance: The level lines not stable by down-sampling. This is the main problem with level lines methods (MSER).

slightly worse than that of the MSER detector followed by the SIFT descriptor. For simplicity, in the text the methods will be named respectively SIFT, Harris-Affine, Hessian-Affine and MSER for short. (The MSER detector followed by the SIFT and the correlation descriptors are sometimes denoted as MSER+SIFT and MSER+Corr. By MSER alone, we mean the MSER detector followed by either of the two descriptors.)

The Lowe [150] reference software was used for DoG with SIFT. For all the other methods we used the Hessian-Affine, Harris-Affine and MSER descriptor code provided by the authors and combined them with the SIFT descriptor implemented by Mikolajczyk, all downloadable from [171].

Applications of A-SIFT and comparisons with the other methods will also be performed for video object tracing and symmetry detection.

The experiments will show images taken from different viewpoints with varying tilts, zooms, and transition tilts. Correspondences will be connected by white segments. Note that the parallelism or coherent directions of the connecting lines usually indicates that most correspondences are correct.

All images under comparison have a low resolution 600 × 450. As reported under each figure, A-SIFT applied an image sub-sampling of factor K × K with K = 3 for most images. A very few cases where objects of interest are too small will be shown. In those cases A-SIFT only works with a more conservative subsampling K = 2, and in one case only with K = 1.

11.3.1 Exploring tilts and zooms

Fig. 11.3 illustrates the two settings that we have adopted to make systematic comparisons respectively for evaluating the maximum absolute tilt and transition tilt attained by each algorithm. A magazine and a painting shown in Fig. 11.4 were photographed for the experiments. Unlike SIFT and A-SIFT, the Hessian-Affine, Harris-Affine and MSER detectors are not robust to scale change as shown in Fig. 11.6. Therefore the pairs of images under comparison were chosen free of scale change so that the evaluation is focused on the tilt invariance.

“JMMBookOct04” 30/10/2009 page 146

[figure] [figure]

146 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

a  b

Figure 11.3: The settings adopted to systematic comparison. Left: absolute tilt test. An object is photographed with a latitude angle θ (between the camera axis and the normal to the object) that varies from 0 degree (frontal view) to 80 degrees, from distances varying between 1 and 10, which is the maximum focal distance change. Right: transition tilt test. An object is photographed with a longitude angle ϕ (between the camera axis projected on the object plane and a fixed direction thereon) that varies from 0 degree to 90 degrees, from a fixed distance.

Figure 11.4: The magazine cover and the painting are photographed in the experiments.

“JMMBookOct04” 30/10/2009 page 147

[figure] [figure]

11.3. EXPERIMENTS 147

11.3.2 Absolute Tilt Tests

The painting illustrated in Fig. 11.4 is photographed with a reflex camera, with distances varying between ×1 and ×10, which is the maximum focal distance change, and with viewpoint angles between the camera axis and the normal to the poster that varies from 0 degree (frontal view) to 80 degrees. It is clear that beyond 80 degrees, to establish a correspondence between the frontal image and the extreme viewpoint becomes absolutely haphazard. Even when the photo acquisition conditions and the image resolution are excellent, with such a big view angle change the observed surface becomes in general reflective, and the image in the resulting photo is totally different from the frontal view. Nevertheless, A-SIFT works until 80 degrees, and it would be unrealistic to insist on bigger angles.

Table 11.1 summarizes performance of each algorithm in terms of number of correct matches. Some matching results are illustrated in Figs. 11.7 to 11.10.

One remarks first that MSER, which uses maximally stable level lines as features, obtains for most time much less correspondences than other methods whose features are based on local maxima in the scale-space. This has been confirmed by LLD, a novel image matching approach independently developed at ENS Cachan that applies also level lines as features [191, 189]. Let us recall that robust image matching requires a sufficiently big number of correspondences.

For images taken at short distance as illustrated in Figs. 11.7 and 11.8, tilt varies on the same flat object because of the perspective effect, as illustrated in Fig. 11.5. The number of SIFT correspondences drops dramatically when the angle is bigger than 65 degrees (that corresponds to a tilt t ≈ 2.3) and it fails completely when the angle exceeds 75 degrees (tilt t ≈ 3.8). At 65 and 75 degrees, as shown in Fig. 11.8, most matches are located on the side closer to the camera where the actual tilt is smaller. The performance of Harris-Affine and Hessian-Affine degrades considerably when the angle goes over 75 degrees (tilt t ≈ 3.8). The MSER correspondences remain at a small number with a noticeable decline over 65 degrees (tilt t ≈ 2.4). A-SIFT works perfectly until 80 degrees (tilt t ≈ 5.8).

Images taken at a camera-object distance multiplied by 10, as shown in Figs. 11.9 and 11.10, exhibits less perspective effects but contains less meaningful pixels at big angles. For these images the SIFT performance drops considerably: recognition is possible only with angles smaller than 45 degrees. The performance of Harris-Affine and Hessian-Affine declines clear when the angle goes from 45 to 65 degrees and beyond 65 degrees they fail completely. MSER struggles at the angle of 45 degrees and fails at 65 degrees. A-SIFT again functions perfectly until 80 degrees.

Rich in highly contrasted regions, the magazine shown in Fig. 11.4 is more favorable to MSER. Table 11.2 shows the result of a similar experiment performed with the magazine, with the latitude angles from 50 to 80 degrees on one side and with the camera focus distance ×4. Fig. 11.11 shows the result with 80-degree angle. The performance of SIFT, Harris-Affine and Hessian-Affine drops dramatically with the angle going from 50 to 60 degrees (tilt t from 1.6 to 2). Beyond 60 degrees (tilt t = 2) they all fail completely. MSER finds many correspondences until 70 degrees (tilt t ≈ 2.9). The number drops considerably when the angle exceeds 70 degrees and becomes too small at 80 degrees (tilt t ≈ 5.8) for robust recognition. A-SIFT works perfectly until 80 “JMMBookOct04” 30/10/2009 page 148

[figure] [figure]

148 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

Figure 11.5: When the camera view angle is large, the absolute tilt of a plane object can vary considerably in the same image.

Figure 11.6: Robustness to scale change. A-SIFT (shown), SIFT (shown), Harris-Affine (shown), Hessian-Affine, MSER+Corr and MSER+SIFT find respectively 221, 86, 4, 3, 3 and 4 correct matches. Harris-Affine, Hessian-Affine and MSER are not robust to scale change. A-SIFT is implemented with K = 2, which means that K = 3 doesn’t work.

“JMMBookOct04” 30/10/2009 page 149

[figure] [figure]

11.3. EXPERIMENTS 149

Figure 11.7: Correspondences between the painting images taken from short distance (zoom ×1) at frontal view and at −45^{∘} angle. The absolute tilt varies: t = 2 (middle), t < 2 (left part), t > 2 (right part). A-SIFT (shown), SIFT (shown), Harris-Affine (shown), Hessian-Affine, MSER+Corr and MSER+SIFT find respectively 624, 236, 28, 15, 7 and 11 correct matches. A-SIFT is implemented with K = 3.

Figure 11.8: Correspondences between the painting images taken from short distance (zoom ×1) at frontal view and at 75^{∘} angle. The local absolute tilt varies: t = 4 (middle), t < 4 (right part), t > 4 (left part). A-SIFT (K = 3, shown), SIFT (shown), Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT (shown) find respectively 202, 15, 3, 1, 5 and 5 correct matches.

“JMMBookOct04” 30/10/2009 page 150

[figure] [figure]

150 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

Figure 11.9: Correspondences between long distance snapshots (zoom ×10) at frontal and 65^{∘} angle, absolute tilt t = 2.4. A-SIFT (K = 3, shown), SIFT (shown), Harris-Affine (shown), Hessian-Affine, MSER+Corr and MSER+SIFT find respectively 341, 5, 3, 0, 3 and 4 correct matches.

Figure 11.10: Correspondences between long distance views (zoom ×10), frontal view and 80^{∘} angle, absolute tilt t = 5.8. A-SIFT (K = 3, shown), SIFT, Harris-Affine, Hessian-Affine, MSER+Corr (shown) and MSER+SIFT (shown) find respectively 75, 1, 1, 0, 2 and 2 correct matches.

“JMMBookOct04” 30/10/2009 page 151

[figure] [figure]

11.3. EXPERIMENTS 151

Z × 1 SIFT Haraff Hesaff MSER+Corr MSER+SIFT A-SIFT
−80^{∘}/5.8  1  16  1  3  4  110
−75^{∘}/3.9 24  36  7  4  3  281
−65^{∘}/2.3 117  43  36  6  5  483
−45^{∘}/1.4 245  83  51  9  13  559
45^{∘}/1.4 195  86  26  10  12  428
65^{∘}/2.3  92  58  32  10  11  444
75^{∘}/3.9  15  12  7  7  7  203
80^{∘}/5.8  2  6  6  5  5  204
Z × 10 SIFT Haraff Hesaff MSER+Corr MSER+SIFT A-SIFT
−80^{∘}/5.8  1  3  0  4  4  116
−75^{∘}/3.9  0  3  0  6  6  265
−65^{∘}/2.3 10  22  16  7  10  542
−45^{∘}/1.4 182  68  45  18  19  722
45^{∘}/1.4 171  54  26  14  15  707
65^{∘}/2.3  5  12  5  5  6  468
75^{∘}/3.9  2  1  0  4  4  152
80^{∘}/5.8  3  0  0  4  2  110

Table 11.1: Absolute tilt invariance comparison. Summary of the results of the experiments that compare A-SIFT with SIFT, Harris-Affine (HarAff), Hessian-Affine (HesAff), MSER coded by the correlation descriptor (MSER+Corr) and by the SIFT descriptor (MSER+SIFT) for viewpoint angles between 45 and 80 degrees. Top: images taken with zoom ×1. Bottom: images taken with zoom ×10. (The camera-object distance is 10 times bigger.) The latitude angles and the absolute tilts are listed in the left column. With zoom ×1, the actual tilt varies on the same object varies around the marked value due to the perspective effect.

degrees.

The above experiments lead us to the following conclusion of the maximum absolute tilts of the approaches under comparison. SIFT exceeds hardly an absolute of 2 and the limit is about 2.5 for Harris-Affine and Hessian-Affine. The performance of MSER depends heavily on the types of image. For images with highly contrasted regions, MSER reaches an absolute tilt about 5. However if the images do not contain highly contrasted regions, the performance of MSER is very limited even under small tilts. For A-SIFT, an absolute tilt of 5.8 that corresponds to an extreme viewpoint angle of 80 degrees does not pose any difficulty to achieve robust recognition.

11.3.3 Transition Tilt Tests

The magazine shown in Fig. 11.4 is place face-up and photographed by a reflex camera and makes two sets of images. As illustrated in Fig. 11.3-b, for each image set, the camera with a fixed latitude angle θ, i.e., a fixed absolute tilt t of respectively 2 and 4, circles around with the longitude angle ϕ going from 0 to 90 degrees. The camera focus distance is ×4. In each set the images have “JMMBookOct04” 30/10/2009 page 152

[figure] [figure]

152 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

SIFT Haraff Hesaff MSER+Corr MSER+SIFT A-SIFT
50^{∘}/1.6 267  131  144  129  150  1692
60^{∘}/2.0 20  29  39  88  117  1012
70^{∘}/2.9  1  2  2  48  69  754
80^{∘}/5.8  0  0  0  10  17  267

Table 11.2: Absolute tilt invariance comparison. Summary of the results of the experiments that compare A-SIFT with SIFT, Harris-Affine (HarAff), Hessian-Affine (HesAff), MSER coded by the correlation descriptor (MSER+Corr) and by the SIFT descriptor (MSER+SIFT) for viewpoint angles between 50 and 80 degrees. The latitude angles and the absolute tilts are listed in the left column.

the same absolute tilt t while the transition tilt τ (with respect to the image taken at ϕ = 0 degree) goes from 1 to t^{2} when ϕ goes from 0 to 90 degrees. To evaluate the maximum transition tilt, the images taken at ϕ ≠ 0 are matched against the one taken at ϕ = 0.

Table 11.3 compares the performance of the algorithms. The number of matches under the absolute tilt t = 2 shows clearly that performance of SIFT drops dramatically when the transition tilt goes from 1.3 to 1.7. With a transition tilt over 2.1, SIFT fails completely. Similarly a considerable performance decline is observed for Harris-Affine and Hessian-Affine when the transition tilt goes from 1.3 to 2.1. Hessian-Affine slightly outperform Harris-Affine but both methods fail completely when the transition tilt goes above 3. MSER and A-SIFT works stably until the transition tilt goes to 4. A-SIFT distinguishes itself by finding more than 10 times as many as those of MSER that cover a much larger area, as illustrated in Fig. 11.12.

Under an absolute tilt t = 4, SIFT, Harris-Affine and Hessian-Affine struggle at a transition tilt of 1.9 having comparable number of matches. They fail completely when the transition tilt goes bigger. MSER works stably until a transition tilt of 7.7. Over this value, the number of correspondences is too small for reliable recognition. A-SIFT works perfectly.

The transition tilt is a crucial notion that evaluates the degree of affine invariance of the image comparison algorithms. With the ordinary viewpoint changes the transition tilt goes easily to a big value (above 16 for example). The experiments above show that the maximum transition tilt, about 2 for SIFT and 2.5 for Harris-Affine and Hessian-Affine, is by far not enough. MSER enables reliable recognition until a transition tilt of about 10, under the condition that the images under comparison are free of scale change and contain highly contrasted regions. The limit of A-SIFT goes beyond 16 largely. Images that have undergone transition tilts up to 30 and more can be reliably recognized by A-SIFT, an example being illustrated in Fig. 9.7.

11.3.4 Comparative experiments

Fig. 11.18 compares the A-SIFT image matching with SIFT, Harris-Affine, Hessian-Affine and MSER. Table 11.4 summarizes the results. Fig. 11.13 shows images of a building facade taken from very different viewpoints. The transformation of the rectangle facade on the left to a trapezia on the right indicates “JMMBookOct04” 30/10/2009 page 153

[figure] [figure]

11.3. EXPERIMENTS 153

Figure 11.11: Correspondences between images taken with zoom ×4, frontal view and 80^{∘} angle, absolute tilt t = 5.8. A-SIFT (shown, K = 3), SIFT, Harris-Affine, Hessian-Affine, MSER+Corr (shown) and MSER+SIFT (shown) find respectively 349, 0, 0, 0, 10 and 17 correct matches.

Figure 11.12: Correspondences with absolute tilts t_{1} = t_{2} = 2 and longitude angles ϕ_{1} = 0^{∘} and ϕ_{2} = 50^{∘}, transition tilt τ = 3. A-SIFT (K = 3, shown), SIFT, Harris-Affine, Hessian-Affine, MSER+Corr (shown) and MSER+SIFT (shown) find respectively 881, 2, 0, 2, 70 and 87 correct matches.

“JMMBookOct04” 30/10/2009 page 154

[figure] [figure]

154 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

ϕ_{2}/τ t = 2 SIFT Haraff Hesaff MSER+Corr MSER+SIFT A-SIFT
10^{∘}/1.3  408  233  176  110  124  1213
20^{∘}/1.7  49  75  84  96  122  1173
30^{∘}/2.1  5  24  32  85  103  1048
40^{∘}/2.5  3  13  29  71  88  809
50^{∘}/3.0  3  1  3  70  87  745
60^{∘}/3.4  2  0  1  50  62  744
70^{∘}/3.7  0  0  0  34  51  557
80^{∘}/3.9  0  0  0  40  51  589
90^{∘}/4.0  0  0  1  41  56  615
ϕ_{2}/τt = 4 SIFT Haraff Hesaff MSER+Corr MSER+SIFT A-SIFT
10^{∘}/1.9  22  32  14  38  49  1054
20^{∘}/3.3  4  5  1  32  39  842
30^{∘}/5.3  3  2  1  24  32  564
40^{∘}/7.7  0  0  0  22  28  351
50^{∘}/10.2  0  0  0  15  19  293
60^{∘}/12.4  1  0  0  12  17  145
70^{∘}/14.3  0  0  0  6  13  90
80^{∘}/15.6  0  0  0  6  12  106
90^{∘}/16.0  0  0  0  5  9  88

Table 11.3: Transition tilt invariance comparison. Summary of the results of the experiments that compare A-SIFT with SIFT, Harris-Affine (HarAff), Hessian-Affine (HesAff), MSER coded by the correlation descriptor (MSER+Corr) and by the SIFT descriptor (MSER+SIFT). The affine parameters of the two images are ϕ_{1} = 0^{∘}, t_{1} = t_{2} = 2 (above), t_{1} = t_{2} = 4 (below). ϕ_{2} and the transition tilts τ are in the left column.

that the transformation is not affine, but strongly perspective. Nevertheless, since a projective transformation can be locally modeled by affine transforms, a large number of correspondences is established by A-SIFT. All the other methods fail. Fig. 11.14 shows the results of the standard test pair Graffiti 1 and Graffiti 6 proposed by Mikolajczyk [171]. A-SIFT finds 724 correspondences, out of which 3 are false. SIFT, Harris-Affine and Hessian-Affine find respectively 0, 3 and 1 correct correspondences: the τ = 3.2 transition tilt is just a bit too large. MSER+Corr and MSER+SIFT find respectively 50 and 70 correct correspondences. Proposed by Matas et al. in their online demo [167] as a standard image to test MSER [168], the images in Fig. 11.15 show a number of containers placed on a desktop ^{1}. A-SIFT finds 194 correct correspondences. SIFT, Harris-Affine, Hessian-AFfine, MSER+Corr and MSER+SIFT find respectively 10, 23, 11, 16 and 22 correct correspondences. Let us note that images in Figs. 11.14 and 11.15 provide optimal conditions for MSER: the camera-object distances are similar and well contrasted shapes are present. But let us recall that MSER fails under large scale changes or when well contrasted shapes are not present. Fig. 11.16 contains two orthogonal road signs taken under a view change that makes a transition tilt τ ≈ 2.6. A-SIFT successfully matches the two signs “JMMBookOct04” 30/10/2009 page 155

1We thank Michal Perdoch for having kindly provided us with the images.

[figure] [figure]

11.3. EXPERIMENTS 155

finding 50 correspondences while all the other methods totally fail. The monument shown in Fig. 11.17 has undergone a viewpoint change of latitude angle 65 degrees (tilt t=2.4). A-SIFT works perfectly and finds 101 correct correspondences. SIFT struggles by establishing 13 correspondences. Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT fail finding only 2, 2, 5 and 4 matches. In Fig. 11.18 is the stump taking from different viewpoints which makes a transition tilt τ ≈ 2.6. A-SIFT achieved success finding 168 correct correspondences while all the other methods fail.

Fig. 11.19 illustrates a complex scene in a coffee room in which divers objects are presented. A-SIFT recognizes the scene by finding 125 correspondences over six non-coplanar objects. SIFT finds 11 correspondence on the wall and 2 on the box over the fridge. Other methods fail finding at most a few correspondences on the wall. Fig. 11.20 shows a coffee can that has been rotated for about 120 degrees. Over the cylinder transition tilt varies continuously and reaches big values. A-SIFT identifies 287 correspondences that cover almost all visible common areas. SIFT fails completely due to the too large viewpoint change. Harris-Affine and Hessian Affine fail by finding 3 and 6 correspondences. MSER finds a small number of correspondences that cover only part of letters that provide highly contrasted regions but it does not catch anything on the image part on the lower half of the can. The Palace of Versaille in Fig. 11.19 undertakes a viewpoint change of about 50 degrees. A-SIFT detect 67 matches uniformly distributed on the facade that can be viewed in the two images. SIFT finds 26 correspondences, mostly located in the closer end where the transition tilt is smaller due to the perspective effect. The other methods fail completely by finding zero or sporadic correspondences.

Fig./τ  SIFT Haraff Hesaff MSER1 MSER2 A-SIFT
Fig. A/3.0  0  0  1  3  3  58
Fig. 11.13/3.8  0  0  1  0  2  68
Fig. 11.14/3.2  0  3  1  50  50  721
Fig. 11.15/[1.6,3.0]  10  23  11  16  22  254
Fig. 11.16/2.6  0  0  0  0  1  50
Fig. B/15.0  0  0  0  0  0  78
Fig. 11.17/2.4  13  2  2  5  4  101
Fig. 11.18/2.6  1  2  1  6  6  168
Fig. C/[1.6, ∞]  26  7  2  3  4  143
Fig. 11.19/[1.5, 3.3] 13  0  3  5  2  125
Fig. 11.20/[2.3, ∞]  0  6  3  12  22  287
Fig. D/[2, ∞]  19  5  7  7  13  123
Fig. 11.21/1.8  67  26  2  1  0  4

Table 11.4: Summary of the results of the experiments that compare A-SIFT with SIFT, Harris-Affine (HarAff), Hessian-Affine (HesAff), MSER coded by the correlation descriptor (MSER+Corr) and by the SIFT descriptor (MSER+SIFT). The transition tilts or their ranges are listed in the left column. The figures with Latin numbers are not shown in the text.

“JMMBookOct04” 30/10/2009 page 156

[figure] [figure]

156 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

Figure 11.13: Image matching: Facade. Absolute (and transition) tilt t = 3.8 (θ = 75^{∘}). A-SIFT, SIFT, Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT find respectively 68, 0, 1, 1, 0 and 2 correct matches. A-SIFT is implemented with K = 2, which means that K = 3 doesn’t work Results shown: A-SIFT and MSER+SIFT

Figure 11.14: Image matching between Graffiti 1 and Graffiti 6. Transition tilt: τ ≈ 3.2. From top to bottom, left to right: A-SIFT, SIFT, Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT find respectively 721, 0, 3, 1, 50 and 70 correct matches. A-SIFT is implemented with K = 3. Results shown: A-SIFT and MSER+SIFT

Figure 11.15: Image matching (images proposed by Matas et al [167]). Transition tilt: τ ∈ [1.6, 3.0]. From top to bottom, left to right: A-SIFT (shown), SIFT (shown), Harris-Affine, Hessian-Affine (shown), MSER+Corr and MSER+SIFT (shown) find respectively 254, 10, 23, 11, 16 and 22 correct matches. A-SIFT is implemented with K = 3.

“JMMBookOct04” 30/10/2009 page 157

[figure] [figure]

11.3. EXPERIMENTS 157

Figure 11.16: Image matching: road signs. Transition tilt τ ≈ 2.6. A-SIFT (shown), SIFT, Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT (shown) find respectively 50, 0, 0, 0, 0 and 1 correct matches. A-SIFT is implemented with K = 3.

Figure 11.17: Image matching: Ecole Polytechnique. Absolute (and transition) tilt t ≈ 2.4 (θ = 65^{∘}). A-SIFT (shown), SIFT, Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT (shown) find respectively 101, 13, 2, 2, 5 and 4 correct matches. A-SIFT is implemented with K = 3.

“JMMBookOct04” 30/10/2009 page 158

[figure] [figure]

158 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

Figure 11.18: Image matching: stump. Transition tilt τ ≈ 2.6. A-SIFT (shown), SIFT, Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT (shown) find respectively 168, 1, 2, 1, 6 and 6 correct matches. A-SIFT is implemented with K = 3.

Figure 11.19: Image matching: coffee room. Transition tilt τ ∈ [1.5, 3.3]. A-SIFT (shown), SIFT, Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT (shown) find respectively 125, 13, 0, 3, 5 and 2 correct matches. A-SIFT is implemented with K = 1, which gives one of the very few examples where details in one of the images are so small that K = 2 and K = 3 do not work.

“JMMBookOct04” 30/10/2009 page 159

[figure] [figure]

11.3. EXPERIMENTS 159

Figure 11.20: Image matching: can. Transition tilt τ ∈ [2.3, ∞]. A-SIFT (shown), SIFT, Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT (shown) find respectively 287, 0, 6, 3, 12 and 22 correct matches. A-SIFT is implemented with K = 2, which means that K = 3 doesn’t work

Figure 11.21: Image matching: Palace of Versailles. Transition tilt τ = 1.8. A-SIFT (shown), SIFT (shown), Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT (shown) find respectively 67, 26, 2, 1, 0 and 4 correct matches. A-SIFT is implemented with K = 3.

“JMMBookOct04” 30/10/2009 page 160

[figure] [figure]

160 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

11.3.5 Symmetry detection in perspective

Symmetry detection has drawn considerable attention in computer vision and has been used for numerous applications such as image indexing, completion of occluded shapes, object detection, facial image analysis and visual attention (see, for example, [56] for a survey). The image projection is usually approximated by plane affine transforms for symmetry detection in perspective [185]. Some recent works apply SIFT, MSER and other affine-invariant detectors and descriptors to detect bilateral symmetry [152, 56]. Conversely, symmetry has been used to extract affine-invariant image features [15].

The image matching algorithm can be used to detect bilateral symmetry in an image u, by simply looking for correspondences between u(x, y) and its flipped version u(−x, y). After being flipped, symmetric structures become either identical if taken in frontal view, or identical up to an oblique view otherwise. A correspondence between u(x, y) and u(−x, y) therefore connects a pair of bilateral symmetrical points in u(x, y). Fig. 11.22 shows some examples of bilateral symmetry detection obtained by A-SIFT, SIFT, Harris-Affine, Hessian-Affine, MSER+Corr and MSER+SIFT. A-SIFT that has the best performance on affine invariant image matching, results in the best symmetry detection in perspective.

Figure 11.22: Symmetry detection in perspective. From left to right: detection results by A-SIFT, SIFT, Hessian-Affine, MSER+SIFT.

11.4 Comments and references

David Pritchard’s master thesis was a first step toward A-SIFT. Quoting [205] in his 2003 master thesis on cloth parameters and motion capture:

Cloth strongly resists stretching, but permits substantial bending; folds and wrinkles are a distinctive characteristic of cloth. This “JMMBookOct04” 30/10/2009 page 161

[figure] [figure]

11.4. COMMENTS AND REFERENCES 161

behaviour means that sections of the cloth are often seen at oblique angles, leading to large affine distortions of features in certain regions of the cloth. Unfortunately, SIFT features are not invariant to large affine distortions.(...) To compensate for this, we use an expanded set of reference features. We generate a new reference image by using a 2 x 2 transformation matrix T to scale the reference image by half horizontally. We repeat three more times, scaling vertically and along axes at 45 degrees, as shown in Figure 5.3. This simulates different oblique views of the reference image. For each of these scaled oblique views, we collect a set of SIFT features. Finally, these new SIFT features are merged into the reference feature set. When performing this merge, we must adjust feature positions, scales and orientations by using T-1. This approach is compatible with the recommendations made by Lowe for correcting SIFT’s sensitivity to affine change.

In recent years local image detectors have bloomed. They can be classified by their incremental invariance properties. All of them are translation invariant. The Harris point detector [105] is also rotation invariant. The Harris-Laplace, Hessian-Laplace and the DoG (Difference-of-Gaussian) region detectors [172, 175, 151, 80] are invariant to rotations and changes of scale. Some momentbased region detectors [146, 25] including the Harris-Affine and Hessian-Affine region detectors [173, 175], an edge-based region detector [246, 245], an intensitybased region detector [244, 245], an entropy-based region detector [124], and two level line-based region detectors MSER (“maximally stable extremal region”) [168] and LLD (“level line descriptor”) [187, 189, 191] are designed to be invariant to affine transformations. MSER, in particular, has been demonstrated to have often better performance than other affine invariant detectors, followed by Hessian-Affine and Harris-Affine [177].

The mentioned methods have a varying complexity. Measured in terms of their processing times, the fastest is MSER, followed by Harris-Affine and Hessian-Affine. SIFT is ten times slower, but a recent acceleration has been proposed in [97], that equals it to the other detectors.

However, the mentioned affine invariant detectors aren’t fully affine invariant. As pointed out in [151], they start with initial feature scales and locations selected in a non-affine invariant manner. As shown in [191], MSER and LLD are not fully scale invariant, because they do not take into account the drastic changes of level lines due to blur. This is also the case for other image local descriptors, such as the distribution-based shape context [27], the geometric histogram [18] descriptors, the derivative-based complex filters [25, 224], and the moment invariants [250].

There is, however, at least one method dealing with scale (and therefore with blur) in a fully satisfactory way. In his milestone paper [151], Lowe has proposed a scale-invariant feature transform (SIFT) descriptor that is invariant to image scaling and rotation and partially invariant to illumination and viewpoint changes. Although SIFT is a priori less invariant to affine transforms than other descriptors such as Hessian-Affine and Harris-Affine [172, 175], its performance turns out to be comparable, as we shall see in many experiments. Furthermore, SIFT is really scale invariant (a mathematical proof of this fact is given in [184]). A number of SIFT variants and extensions, in- “JMMBookOct04” 30/10/2009 page 162

[figure] [figure]

162 CHAPTER 11. EXPERIMENTS ON AFFINE INVARIANT METHODS

cluding PCA-SIFT [126], GLOH (gradient location-orientation histogram) [176] and SURF (speeded up robust features) [26], that claim to have improved robustness and distinctiveness with scaled-down complexity have been developed ever since [85, 143]. Demonstrated to be superior to many other descriptors [174, 176], SIFT and its variants have been popularly applied for scene recognition [79, 182, 217, 251, 96, 226, 265, 183] and detection [86, 194], robot localization [28, 227, 196, 120], image registration [264], image retrieval [104], motion tracking [247, 128], 3D modeling and reconstruction [211, 252], building panoramas [3, 34], photo management [263, 141, 235, 51], as well as symmetry detection [152].

“JMMBookOct04” 30/10/2009 page 163

[figure] [figure]

Part II

Contrast-Invariant Image Analysis

163 “JMMBookOct04” 30/10/2009 page 164

[figure] [figure]

“JMMBookOct04” 30/10/2009 page 165

[figure] [figure]

Chapter 12

Contrast-Invariant Classes of Functions and Their Level Sets

This chapter is about one of the major technological contributions of mathematical morphology, namely the representation of images by their upper level sets. As we shall see in this chapter, this leads to a handy contrast invariant representation of images.

Definition 12.1. Let u ∈ F. The level set of u at level 0 ≤ λ ≤ 1 is denoted by X_{λ}u and defined by

X_{λ}u = {x | u(x) ≥ λ}.

Strictly speaking, we have called level sets what should more properly be called upper level sets. Several level sets of a digital image are shown in Figure 12.1 and all of the level sets of a synthetic image are illustrated in Figure 12.2. The reconstruction of an image from its level sets is illustrated in Figure 12.3. Two important properties of the level sets of a function follow directly from the definition. The first is that the level sets provide a complete description of the function. Indeed, we can reconstruct u from its level sets X_{λ}u by the formula

u(x) = sup{λ | x ∈ X_{λ}u}.

This formula is called superposition principle as u is being reconstructed by “superposing” its level sets.

Exercise 12.1. Prove the superposition principle.

The second important property is that level sets of a function are globally invariant under contrast changes. We say that two functions u and v have the same level sets globally if for every λ there is μ such that X_{μ}v = X_{λ}u, and conversely. Now suppose that a contrast change g : ℝ → ℝ is continuous and increasing. Then it is not difficult to show that v = g(u) and u have the same level sets globally.

Exercise 12.2. Check this last statement for any function u and any continuous increasing contrast change g.

165 “JMMBookOct04” 30/10/2009 page 166

[figure] [figure]

166 CHAPTER 12. CONTRAST-INVARIANCE AND LEVEL SETS

Figure 12.1: Level sets of a digital image. Left to right, top to bottom: We first show an image with range of gray levels from 0 to 255. Then we show eight level sets in decreasing order from λ = 225 to λ = 50, where the grayscale step is 25. Notice how essential features of the shapes are contained in the boundaries of level sets, the level lines. Each level set (which appears as white) is contained in the next one, as guaranteed by Proposition 12.2.

Conversely, we shall prove that if the level sets of a function v ∈ F are level sets of u, then there is a continuous contrast change g such that v = g(u). This justifies the attention we will dedicate to level sets, as they turn out to contain all of the contrast invariant information about u.

12.1 From an image to its level sets and back

In the next proposition, for a sake of generality, we consider bounded measurable functions on S_{N}, not just functions in F.

“JMMBookOct04” 30/10/2009 page 167

[figure] [figure]

12.1. FROM AN IMAGE TO ITS LEVEL SETS AND BACK 167

Figure 12.2: A simple synthetic image and all of its level sets (in white) with decreasing levels, from left to right and from top to bottom.

Proposition 12.2. Let X_{λ} denote the level sets X_{λ}u of a bounded measurable function u : S_{N} → ℝ. Then the sets X_{λ} satisfy the following two structural properties:

(i) If λ > μ, then X_{λ} ⊂ X_{μ}. In addition, there are two real numbers λ_{max} ≥ λ_{min} so that X_{λ} = S_{N} for λ < λ_{min}, X_{λ} = ∅ for λ > λ_{max}.

(ii) X_{λ} = ⋂_{μ<λ} X_{μ} for every λ ∈ ℝ.

Conversely, if (X_{λ})_{λ∈ℝ} is a family of sets of M that satisfies (i) and (ii), then the level sets of the function u defined by superposition principle,

u(x) = sup{λ | x ∈ X_{λ}}  (12.1)

satisfy X_{λ}u = X_{λ} for all λ ∈ ℝ and λ_{min} ≤ u ≤ λ_{max}.

Proof. The first part of Relation (i) follows directly from the definition of upper level sets. The second part of (i) works with λ_{min} = inf u and λ_{max} = sup u. The relation (ii) follows from the equivalence u(x) ≥ λ ⇔ u(x) ≥ μ for every μ < λ.

Conversely, take a family of subsets (X_{λ})_{λ∈ℝ} satisfying (i) and (ii) and define u by the superposition principle. Let us show that X_{λ} = X_{λ}u. Take first x ∈ X_{λ}. Then it follows from the definition of u that u(x) ≥ λ, and hence x ∈ X_{λ}u. Thus, X_{λ} ⊂ X_{λ}u. Conversely, let x ∈ X_{λ}u. Then u(x) = sup{ν | x ∈ X_{ν}} ≥ λ. Consider any μ < λ. Then there exists a μ′ such that μ < μ′ ≤ sup{ν | x ∈ X_{ν}} and x ∈ X_{μ′}. It follows from (i) that x ∈ X_{μ}. Since μ was any number less that λ, we conclude by using (ii) that x ∈ ⋂_{μ<λ} X_{μ} = X_{λ}. It is easily checked that λ_{min} ≤ u ≤ λ_{max}. □

Exercise 12.3. Check the last statement of the preceding proof, that λ_{min} ≤ u ≤ λ_{max}.

“JMMBookOct04” 30/10/2009 page 168

[figure] [figure]

168 CHAPTER 12. CONTRAST-INVARIANCE AND LEVEL SETS

Figure 12.3: Reconstruction of an image from its level sets: an illustration of Proposition 3.2. We use four different subsets of the image’s level sets to give four reconstructions. Top, left: all level sets; top, right: all level sets whose gray level is a multiple of 8; bottom, left: multiples of 16; bottom, right: multiples of 32. Notice the relative stability of the image shape content under these drastic quantizations of the gray levels.

12.2 Contrast changes and level sets

Practical aspects of contrast changes are illustrated in Figures 12.4, 12.5, 12.6, and 12.7, which illustrate how insensitive our perception of images is to contrast changes, even when they are flat on some interval. When this happens, some information on the image is even lost, as several grey levels melt together.

Definition 12.3. Any nondecreasing continuous surjection g : ℝ → ℝ will be called a contrast change.

Exercise 12.4. Remark that g(s) → ±∞ as s → ±∞. Check that if u ∈ F and g is a contrast change, then g(u) ∈ F.

In case g is increasing, g has an inverse contrast change g^{−1}. In case g is flat on some interval, we shall be happy with a pseudo-inverse for g.

Definition 12.4. The pseudo-inverse of any contrast change g : ℝ → ℝ is defined by

g^{(−1)}(λ) = inf{r ∈ ℝ | g(r) ≥ λ}.

Exercise 12.5. Check that g^{−1} is finite on ℝ and tends to ±∞ as s → ±∞. Give an example of g such that g^{−1} is not continuous.

Exercise 12.6. Compute and draw g^{(−1)} for the function g(s) = max(0, s). Notice that such a function is ruled out by our conditions at infinity for contrast changes.

“JMMBookOct04” 30/10/2009 page 169

[figure] [figure]

12.2. CONTRAST CHANGES AND LEVEL SETS 169

Figure 12.4: The histogram of the image Bird. For each i ∈ {0, 1, …, 255}, we display (above, right) the function h(i) = Card {x | u(x) = i}. The function below is the cumulative histogram, namely the primitive of h defined by H_{u}(i) = Card {x | u(x) ≤ i}. The shape of h provides an indication about the overall contrast of the image and about the contrast change imposed by the sensors. See Chap. 13 for manipulations of the cumulative histogram.

Lemma 12.5. Let g : ℝ → ℝ be a contrast change. Then for every λ ∈ ℝ, g(g^{(−1)})(λ) = λ and

g(s) ≥ λ if and only if s ≥ g^{(−1)}(λ).  (12.2)

Proof. The first relation follows immediately from the continuity of g. If g(s) ≥ λ, then s ≥ g^{(−1)}(λ) by the definition of g^{(−1)}(λ). Conversely, if s ≥ g^{(−1)}(λ), then g(s) ≥ g(g^{(−1)}(λ)) = λ and thus g(s) ≥ λ. □

Theorem 12.6. Let u ∈ F and g be a contrast change. Then any level set of g(u) is a level set of u. More precisely, for λ ∈ ℝ,

X_{λ}g(u) = X_{g^{(−1)}(λ)}u.  (12.3)

Proof. The proof is read directly from Lemma 12.5 by taking s = u. □

The next result is a converse statement to Theorem 12.6.

“JMMBookOct04” 30/10/2009 page 170

[figure] [figure]

170 CHAPTER 12. CONTRAST-INVARIANCE AND LEVEL SETS

Figure 12.5: Contrast changes and an equivalence class of images. The three images have exactly the same level sets and level lines, but their level sets are mapped onto three different gray-level scales. The graphs on the right are the graphs of the contrast changes u ↦ g(u) that have been applied to the initial gray levels. The first one is concave; it enhances the darker parts of the image. The second one is the identity; it leaves the image unaltered. The third one is convex; it enhances the brighter parts of the image. Software allows one to manipulate the contrast of an image to obtain the best visualization. From the image analysis viewpoint, image data should be considered as an equivalence class under all possible contrast changes.

Theorem 12.7. Let u and v ∈ F such that every level set of v is a level set of u. Then v = g(u) for some contrast change g.

Proof. One can actually give an explicit formula for g, namely, for every μ ∈ u(S_{N}),

g(μ) = sup{λ ∈ v(S_{N}) | X_{μ}u ⊂ X_{λ}v}.  (12.4)

For μ ∉ u(S_{N}), we can easily extend g into an nondecreasing function such that g(±∞) = ±∞). (Take (e.g.) g piecewise affine). Note that ν > μ implies that g(ν) ≥ g(μ). Let us first show that inf v ≤ g(μ) ≤ sup v. Set

Λ := {λ | X_{μ}u ⊂ X_{λ}v}.

“JMMBookOct04” 30/10/2009 page 171

[figure] [figure]

12.2. CONTRAST CHANGES AND LEVEL SETS 171

Figure 12.6: The two images (left) have the same set of level sets. The contrast change that maps the upper image onto the lower image is displayed on the right. It corresponds to one of the possible g functions whose existence is stated in Corollary 3.14. The function g may be locally constant on intervals where the histogram of the upper image is zero (see top, middle graph). Indeed, on such intervals, the level sets are invariant.

Λ is not empty because X_{inf v} = S_{N} and therefore inf v ∈ Λ. Thus g(μ) = sup Λ ≥ inf v. On the other hand X_{sup v+ε}v = ∅ for every ε > 0. Since μ ∈ u(S_{N}), X_{μ}u ≠ ∅ and therefore g(μ) = sup Λ ≤ sup v.

Step 1: Proof that v(x) ≥ g(u(x)). By Proposition 12.2(i) Λ has the form (−∞, sup Λ) or (−∞, sup Λ]. But by Proposition 12.2(ii), X_{sup Λ}v = ⋂_{λ<sup Λ} X_{λ}v, and this implies by the definition of Λ that g(μ) = sup Λ ∈ Λ. Thus,

X_{μ}u ⊂ X_{g(μ)}v.  (12.5)

Given x ∈ S_{N}, let μ = u(x) in (12.5). Then,

X_{u(x)}u ⊂ X_{g(u(x))}v.

Since x ∈ X_{u(x)}u, we conclude that x ∈ X_{g(u(x))}v = {y | v(y) ≥ g(u(x))}.

Step 2: Proof that v(x) ≤ g(u(x)). Given x ∈ S_{N}, we translate the assumption with λ = v(x) as follows: There exists a μ(x) ∈ ℝ such that

X_{v(x)}v = {y | u(y) ≥ μ(x)} = X_{μ(x)}u.  (12.6)

Since x ∈ X_{v(x)}v, we know that x ∈ X_{μ(x)}u. Thus, u(x) ≥ μ(x), and X_{u(x)}u ⊂ X_{μ(x)}u = X_{v(x)}v. This last relation implies by the definition of g that v(x) ≤ g(u(x)).

Step 3: Proof that g is continuous. Recall that the image of a connected set by a continuous function is connected. Thus u(S_{N}) is an interval of ℝ and so is v(S_{N}). Since g(u) = v, g(u(S_{N})) = v(S_{N}) is an interval. Now, a nondecreasing function is continuous on an interval if and only if its range is connected. Thus g is continuous on u(S_{N}) and so is its extension to ℝ. □

“JMMBookOct04” 30/10/2009 page 172

[figure] [figure]

172 CHAPTER 12. CONTRAST-INVARIANCE AND LEVEL SETS

Figure 12.7: The original image (top, left) has a strictly positive histogram (all gray levels between 0 and 255 are represented). Therefore, if any contrast change g that is not strictly increasing is applied, then some data will be lost. Every level set of the transformed image g(u) is a level set of the original image; however, the original image has more level sets than the transformed image.

Exercise 12.7. Prove the last statement in the theorem, namely that “a nondecreasing function is continuous on an interval if and only if its range is connected”.

Exercise 12.8. By reading carefully the steps 1 and 2 of the proof of Theorem 12.7, check that this theorem applies with u and v just bounded and measurable on S_{N}. Then one has still has v = g(u) with g defined in the same way. Of course g is still nondecreasing but not necessarily continuous. Find a simple example of functions u and v such that g is not continuous.

12.3 Exercises

Exercise 12.9. This exercise gives a way to compute the function g such that v = g(u) defined in the proof of Theorem 12.7 in terms of the repartition functions of u and v. Let G be a Gauss function defined on ℝ^{N} such that ∫_{ℝ^{N}} G(x)dx = 1. For every measurable subset of ℝ^{N}, set |A|_{G} := ∫_{A} G(x)dx. Let u be a bounded continuous function on ℝ^{N}. We can associate with u its repartition function h_{u}(λ) := |X_{λ}u|_{G}. Show that h_{u} : λ ∈ [inf u, sup u] → h_{u}(λ) is strictly decreasing. Show that it can have jumps but is left-continuous, that is h_{u}(λ) = lim_{μ↑λ} h_{u}(μ). Define for every non increasing function h a pseudo inverse by h^{((−1))}(μ) := sup{λ | h(λ) ≥ μ}. Show that h^{((−1))} is non increasing and that h^{((−1))} ∘ h(μ) ≥ μ, and that if h is left-continuous, h ∘ h^{((−1))}(μ) ≥ μ. Using (12.4) prove that g = h_{v}^{((−1))} ∘ h_{u}. Hint: prove that g(μ) = sup{λ | |X_{μ}u|_{G} ≤ |X_{λ}v|_{G}}.

Exercise 12.10. Let u be a real-valued function. If (μ_{n})_{n∈ℕ} is an increasing sequence that tends to λ, prove that

X_{λ}u = ⋂_{n∈ℕ}X_{μ_{n}}u  (12.7)

{x | u(x) > λ} = ⋃_{μ>λ} X_{μ}u.  (12.8)

“JMMBookOct04” 30/10/2009 page 173

[figure] [figure]

12.4. COMMENTS AND REFERENCES 173

12.4 Comments and references

Contrast invariance and level sets. It was Wertheimer who noticed that the actual local values of the gray levels in an image could not be relevant information for the human visual system [260]. Contrast invariance is one of the fundamental model assumptions in mathematical morphology. The two basic books on this subject are Matheron [169] and Serra [228, 230]. See also the fundamental paper by Serra [229]. Ballester et al. defined an “image intersection” whose principle is to keep all pieces of bilevel sets common to two images [22]. (A bilevel set is of the form {x | λ ≤ u(x) ≤ μ}.) Monasse and Guichard developed a fast level set transform (FLST) to associate with every image the inclusion tree of connected components of level sets [181]. They show that the inclusion trees of connected upper and lower level sets can be fused into a single inclusion tree; among other applications, this tree can be used for image registration. See Monasse [180].

Contrast changes. The ability to vary the contrast (to apply a contrast change) of a digital image is a very useful tool for improving image visualization. Professional image processing software has this capability, and it is also found in popular software for manipulating digital images. For more about contrast changes that preserve level sets, see [48]. Many reference on contrast-invariant operators are given at the end of Chapter 14.

“JMMBookOct04” 30/10/2009 page 174

[figure] [figure]

“JMMBookOct04” 30/10/2009 page 175

[figure] [figure]

Chapter 13

Specifying the contrast of images

Midway image equalization means any method giving to a pair of images a similar histogram, while maintaining as much as possible their previous grey level dynamics. The comparison of two images is one of the main goals of computer vision. The pair can be a stereo pair, two images of the same object (a painting for example), multi-channel images of the same region, images of a movie, etc. Image comparison is perceptually greatly improved if both images have the same grey level dynamics (which means, the same grey level histogram). Many image comparison algorithms are based on grey level and take as basic assumption that intensities of corresponding points in both images are equal. However, this assumption is generally false for stereo pairs, and deviations from this assumption cannot even be modeled by affine transforms [58]. Consequently, if we want to compare visually and numerically two images, it is useful to give them first the same dynamic range and luminance.

In all of this applicative chapter the images u(x) and v(x) are defined on a domain which is the union of M pixels. The area of each pixel is equal to 1. The images are discrete in space and values: they attain values in a finite set 𝕃 and they are constant on each pixel of the domain. We shall call such images discrete images. The piecewise constant interpolation is a very bad image interpolation. It is only used here for a fast handling of image histograms. For other scopes, better interpolation methods are of course necessary.

Definition 13.1. Let u be a discrete image. We call cumulative histogram of u the function H_{u} : 𝕃 → 𝕄 := [0, M] ∩ ℕ defined by

H_{u}(l) =: meas({x | u(x) ≤ l}).

This cumulative histogram is a primitive of the histogram of the image h(l) = meas({x | u(x) = l}). Figures 12.4, 12.6 and the first line of Figure 13.1. show the histograms of some images and their cumulative histograms. In fact Figure 12.7 shows first the histogram and then the modified histogram after a contrast change has been applied. These experiments illustrate the robustness of image relevant information to contrast changes and even to the removal of some level sets, when the contrast change is flat on an interval. Such experiments suggest

175 “JMMBookOct04” 30/10/2009 page 176

[figure] [figure]

176 CHAPTER 13. SPECIFYING THE CONTRAST OF IMAGES

that one can specify the histogram of a given image by applying the adequate contrast change. Before proceeding, we have to define the pseudo-inverses of a discrete function.

Proposition 13.2. Let φ : 𝕃 → 𝕄 be a nondecreasing function from a finite set of values into another. Define two pseudo-inverse functions for φ :

φ^{(−1)}(l) := inf{s | φ(s) ≥ l} and φ^{((−1))}(l) := sup{s | φ(s) ≤ l.}

Then one has the following equivalences:

φ(s) ≥ l ⇔ s ≥ φ^{(−1)}(l), φ(s) ≤ l ⇔ s ≤ φ^{((−1))}(l)  (13.1)

and the identity

(φ^{(−1)})^{((−1))} = φ.  (13.2)

Proof. The implication φ(s) ≥ l ⇒ s ≥ φ^{(−1)}(l) is just the definition of φ^{(−1)}. The converse implication is due to the fact that the infimum on a a finite set is attained. Thus φ(φ^{(−1)}(l)) ≥ l and therefore s ≥ φ^{(−1)}(l) ⇒ φ(s) ≥ l. The identity (13.2) is a direct consequence of the equivalences (13.1). Indeed,

s ≤ (φ^{(−1)})^{((−1))}(l) ⇔ φ^{(−1)}(s) ≤ l ⇔ s ≤ φ(l).
□

Exercise 13.1. Prove that if φ is increasing, φ^{(−1)} ∘ φ(l) = l and φ^{((−1))} ∘ φ(l) = l. If φ is surjective, φ ∘ φ^{(−1)} = l and φ ∘ φ^{((−1))}(l) = l.

Proposition 13.3. Let φ be a discrete contrast change and u a digital image. Then

H_{φ(u)} = H_{u} ∘ φ^{((−1))}.

Proof. By (13.1), φ(u) ≤ l ⇔ u ≤ φ^{((−1))}(l). Thus by the definitions of H_{u} and H_{φ(u)},

H_{φ(u)}(l) = meas({x | φ(u) ≤ l}) = meas({x | u(x) ≤ φ^{((−1))}(l)}) = H_{u}∘φ^{((−1))}(l).

□

Let G : 𝕃 → 𝕄 := [0, 1, …, M] be any discrete nondecreasing function. Can we find a contrast change φ : 𝕃 → 𝕃 such that the cumulative histogram of φ(u), H_{φ(u)} becomes equal to G? Not quite: if for instance u is constant its cumulative histogram is a one step function and Proposition 13.3 implies that H_{φ(u)} will also be a one step function. More generally if u attains k values, then φ(u) attains k values or less. Hence its cumulative histogram is a step function with k + 1 steps. Yet, at least formally, the functional equation given by Proposition 13.3, H_{u} ∘ φ^{−1} = G, leads to φ = G^{−1} ∘ H_{u}. We know that we cannot get true inverses but we can involve pseudo-inverses. Thus, we are led to the following definition:

“JMMBookOct04” 30/10/2009 page 177

[figure] [figure]

13.1. MIDWAY EQUALIZATION 177

Definition 13.4. Let G : 𝕃 → 𝕄 be a nondecreasing function. We call specification of u on the cumulative histogram G the image

ũ := G^{((−1))} ∘ H_{u}(u).

Exercise 13.2. Prove that if G and H_{u} are one to one, then the cumulative histogram of ũ is G. Is it enough to assume that H_{u} is one to one?

Definition 13.5. Let, for l ∈ [0, L] ∩ IN, G(l) = ⌊\frac{M}{L} l⌋, where ⌊r⌋ denotes the largest integer smaller than r. Then ũ := G^{((−1))} ∘ H_{u}(u) is called the uniform equalization of u. If v is another discrete image and one takes G = H_{v}, ũ := H_{v}^{((−1))} ∘ H_{u}(u) is called the specification of u on v.

When H_{u} is one to one, one can reach by applying a contrast change to u any specified cumulative histogram G. Otherwise, the above definitions do the best that can be expected and are actually quite efficient. For instance in the “marshland experiment” (Figure 13.1) the equalized histogram and its cumulative histogram are displayed on the second row. The cumulative histogram is very close to its goal, the linear function. The equalized histogram does not look flat but a sliding average of it would look almost flat.

Yet it is quite dangerous to specify the histogram of an image with an arbitrary histogram specification. This fact is illustrated in Figures 13.1 and 13.2 where a uniform equalization erases existing textures by making them too flat (Figure 13.1) but also enhances the quantization noise in low contrasted regions and produces artificial edges or textures (see Figure 13.2).

13.1 Midway equalization

We have seen that if one specifies u on v, then u inherits roughly the histogram of v. It is sometimes more adequate to bring the cumulative histograms of u and v towards a cumulative histogram which would be “midway” between both. Indeed, if we want to compare visually and numerically two images, it is useful to give them first the same dynamic range and luminance. Thus we wish:

• From two images u and v, construct by contrast changes two images ũ and ṽ, which have a similar cumulative histogram.

• This common cumulative histogram h should stand “midway” between the previous cumulative histograms of u and v, and be as close as possible to each of them. This treatment must avoid to favor one cumulative histogram rather than the other.

Definition 13.6. Let u and v be two discrete images. Set

Φ := \frac{1}{2} (H_{u}^{(−1)} + H_{v}^{(−1)}) .

We call midway cumulative histogram of u and v the function

G := Φ^{((−1))} = (\frac{1}{2}(H_{u}^{(−1)} + H_{v}^{(−1)}))^{((−1))}  (13.3)

and “midway specifications” of u and v the functions ũ := Φ ∘ H_{u}(u) and ṽ := Φ ∘ H_{v}(v).

“JMMBookOct04” 30/10/2009 page 178

[figure] [figure]

178 CHAPTER 13. SPECIFYING THE CONTRAST OF IMAGES

Figure 13.1: First row: Image u, the corresponding grey level histogram h_{u}, and the cumulative histogram H_{u}. Second row: Equalized image H_{u}(u), its histogram and its cumulative histogram. In the discrete case, histogram equalization flattens the histogram as much as possible. We see on this example that image equalization can be visually harmful. In this marshland image, after equalization, the water is no more distinguishable from the vegetation. The third row shows a zoom on the rectangular zone, before and after equalization.

Exercise 13.3. Let u and v be two constant images, whose values are a and b. Prove that their “midway” function is the right one, namely a function w which is constant and equal to \frac{a+b}{2} .

Exercise 13.4. Prove that if we take as a definition of the midway histogram

G := ( \frac{1}{2} (H_{u}^{((−1))} + H_{v}^{((−1))}))^{(−1)} ,

then for two constant images u = a and v = b the midway image is constant and equal to [1/2(a + b) − 1]. This proves that Definition 13.6 is better.

Exercise 13.5. Prove that if u is a discrete image and f and g two nondecreasing functions, then the midway image of f(u) and g(u) is \frac{f(u)+g(u)}{2} .

Exercise 13.6. If we want the “midway” cumulative histogram H to be a compromise between H_{u} and H_{v}, the most elementary function that we could imagine is their average, which amounts to average their histograms as well. However, the following example proves that this idea is not judicious at all.

Consider two images whose histograms are “crenel” functions on two disjoint intervals, for instance u(x) := ax, v(x) = bx + c. Compute a, b, c in such a way that h_{u} and h_{v} have disjoint supports. Then compute the specifications of u and v on the

“JMMBookOct04” 30/10/2009 page 179

[figure] [figure]

13.2. MIDWAY EQUALIZATION ON IMAGE PAIRS 179

Figure 13.2: Effect of histogram equalization on the quantization noise. On the left, the original image. On the right, the same image after histogram equalization. The effect of this equalization on the dark areas (the piano, the left part of the wall), which are low contrasted, is perceptually dramatic. We see many more details but the quantization noise has been exceedingly amplified.

mean cumulative histogram G := \frac{H_{u}+H_{v}}{2} . Compare with their specifications on the midway cumulative histogram.

13.2 Midway equalization on image pairs

Results on a stereo pair

The top of Figure 13.3 shows a pair of aerial images in the region of Toulouse. Although the angle variation between both views is small, and the photographs are taken at nearly the same time, we see that the lightning conditions vary significantly (the radiometric differences can also come from a change in camera settings). The second line shows the result of the specification of the histogram of each image on the other one. The third line shows both images after equalization.

If we scan some image details, as illustrated on Figure 13.4, the damages caused by a direct specification become obvious. Let us specify the darker image on the brightest one. Then the information loss, due to the reduction of dynamic range, can be detected in the brightest areas. Look at the roof of the bright building in the top left corner of the image (first line of Figure 13.4): the chimneys project horizontal shadows on the roof. In the specified image, these shadows have almost completely vanished, and we cannot even discern the presence of a chimney anymore. In the same image after equalization, the shadows are still entirely recognizable, and their size reduction remains minimal. The second line of Figure 13.4 illustrates the same phenomenon, observed in the bottom center of the image. The structure present at the bottom of the image has completely disappeared after specification and remains visible after midway equalization. These examples show how visual information can be lost by specification and how midway algorithms reduce significantly this loss.

“JMMBookOct04” 30/10/2009 page 180

[figure] [figure]

180 CHAPTER 13. SPECIFYING THE CONTRAST OF IMAGES

Figure 13.3: Stereo pair: two pieces of aerial images of a region of Toulouse. Same images after specification of their histograms on each other (left: the histogram of the first image has been specified on the second, and right: the histogram of the second image has been specified on the first). Stereo pair after midway equalization.

“JMMBookOct04” 30/10/2009 page 181

[figure] [figure]

13.2. MIDWAY EQUALIZATION ON IMAGE PAIRS 181

Figure 13.4: Extracts from the stereo pair shown on Figure 13.3. From left to right: in the original image, in the specified one, in the original image after midway equalization. Notice that no detail is lost in the midway image, in contrast with the middle image.

Figure 13.5: First line: two images of Toulouse (blue and infrared channel). Second line: same images after midway equalization.

Multi-Channel images

The top of Figure 13.5 shows two pieces of multi-channel images of Toulouse. The first one is extracted from the blue channel, and the other one from the infrared channel. The second and third line of the same figure show the same images after midway equalization. The multichannel images have the peculiarity to present contrast inversions : for instance, the trees appear to be darker than the church in the blue channel, and are naturally brighter than the church in the infrared channel. The midway equalization being limited to increasing contrast changes, it obviously cannot handle these contrast inversions. In spite of these contrast inversions, the results remain visually good, which underlines the robustness of the method gives globally a good equalization.

“JMMBookOct04” 30/10/2009 page 182

[figure] [figure]

182 CHAPTER 13. SPECIFYING THE CONTRAST OF IMAGES

Photographs of the same painting

The top of Figure 13.6 shows two different snapshots of the same painting, Le Radeau de la Méduse^{1}, by Théodore Géricault (small web public versions). The second one is brighter and seems to be damaged at the bottom left. The second line shows the same couple after midway equalization. Finally, the last line of Figure 13.6 shows the difference between both images after equalization. We see clear differences around the edges, due to the fact that the original images are not completely similar from the geometric point of view.

13.2.1 Movie equalization

One can define a midway cumulative histogram to an arbitrary number of images. This is extremely useful for the removal of flicker in old movies. Flicker has multiple causes, physical, chemical or numerical. The overall contrast of successive images of the same scene in a movie oscillates, some images being dark and others bright. Our main assumption is that image level sets are globally preserved from one image to the next, even if their level evolves. This leads to the adoption of a movie equalization method preserving globally all level sets of each image. We deduce from Theorem 12.7 in the previous chapter that the correction must be a global contrast change on each image. Thus the only left problem is to specify a common cumulative histogram (and therefore a common histogram) to all images of a given movie scene. Noticing that the definition of G in (13.3) for two images simply derives from a mean, its generalization is easy. Let us denote u(t, x) the movie (now a discrete time variable has been added) and by H^{t} the cumulative histogram function of x → u(t, x) at time t. Since flicker is localized in time, the idea is to define a time dependent cumulative histogram function K_{t}^{h} which will be the “midway” cumulative histogram of the cumulative histograms in an interval [t − h, t + h]. Of course the linear scale space theory of Chapter 3 applies here. The ideal average is gaussian. Hence the following definition.

Definition 13.7. Let u(t, x) be a movie and denote by H_{t} the cumulative histogram of u(t) : x → u(t, x). Consider a discrete version of the 1-D gaussian G_{h}(t) = \frac{1}{(4πh)^{\frac{1}{2}}} e^{− \frac{t^{2}}{4h}}. Set

Φ_{(t,l)} := ∫ G_{h}(t − s)(H_{s}^{(−1)})(l)ds.

We call “midway gaussian cumulative histogram at scale h” of the movie u(t, x) the time dependent cumulative histogram

𝔾_{(t,l)} := Φ_{(t,l)}^{((−1))} = (∫ G_{h}(t − s)(H_{s}^{(−1)})(l)ds)^{((−1))}  (13.4)

and “midway specification” of the movie u(t) the function ũ(t) := Φ∘H_{u(t)}(u(t)). If H_{u(t)} is surjective, then ũ(t) has G_{(t,l)} as common cumulative histogram.

Notice that this is a straightforward extension of Definition 13.6.

1Muse du Louvre, Paris.

“JMMBookOct04” 30/10/2009 page 183

[figure] [figure]

13.2. MIDWAY EQUALIZATION ON IMAGE PAIRS 183

Figure 13.6: Two shots of the Radeau de la Méduse, by Géricault. The same images after midway equalization. Image of the difference between both images after equalization. The boundaries appearing in the difference are mainly due to the small geometric distortions between the initial images.

“JMMBookOct04” 30/10/2009 page 184

[figure] [figure]

184 CHAPTER 13. SPECIFYING THE CONTRAST OF IMAGES

Figure 13.7: (a) Three images of Chaplin’ s film His New Job, taken at equal intervals of time. This extract of the film suffers from a severe real flicker. (b) Same images after the Scale-Time Equalization at scale s = 100. The flicker observed before has globally decreased. (c) Evolution of the mean of the current frame in time and at three different scales. The most oscillating line is the mean of the original sequence. The second one is the mean at scale s = 10. The last one, almost constant, corresponds to the large scale s = 1000. As expected the mean function is smoothed by the heat equation.

“JMMBookOct04” 30/10/2009 page 185

[figure] [figure]

13.3. COMMENTS AND REFERENCES 185

The implementation and experimentation is easy. We simply show in Figure 13.7 three images of Chaplin’ s film His New Job, taken at equal intervals of time. This extract of the film suffers from a severe real flicker. This flicker is corrected at the scale where, after gaussian midway equalization, the image mean becomes nearly constant through the sequence. The effects of this equalization are usually excellent. They are easily extended to color movies by processing each channel independently.

13.3 Comments and references

Histogram specification As we have seen histogram specification [95] can be judicious if both images have the same kind of dynamic range. For the same reason as in equalization, this method can also product contouring artifacts. The midway theory is essentially based on Julie Delons’ PhD and papers [67],

[68] where she defines two midway histogram interpolation methods. One of them, the square root method involves the definition of a square root of any nondecreasing function g, namely a function g such that f ∘f = g. Assume that u and v come from the same image (this intermediate image is unknown), up to two contrast changes f and f^{−1}. The function f is unknown, but satisfies formally the equality H_{u} ∘ f = H_{v} ∘ f^{−1}. Thus

H_{u}^{−1} ∘ H_{v} = f ∘ f.

It follows that the general method consists in building an increasing function f such that f ∘ f = H_{u}^{−1} ∘ H_{v} and replacing v by f(v) and u by f^{−1}(u). This led Delon [?] to call this new histogram midway method, the “square root” equalization. The midway interpolation developed in this chapter uses mainly J. Delon’s second definition of the midway cumulative histogram as the harmonic mean of the cumulative histograms of both images. This definition is preferable to the square root. Indeed, both definitions yield very similar results but the harmonic mean extends easily to an arbitrary number of images and in particular to movies [68]. The Cox, Roy and Hingorani algorithm defined in [58] performs a midway equalization. They called their algorithm “Dynamic histogram warping” and its aim is to give a common cumulative histogram (and therefore a common histogram) to a pair of images. Although their method is presented as a dynamic algorithm, there is a very simple underlying formula, which is the harmonic mean of cumulative histograms discovered by Delon [67].

“JMMBookOct04” 30/10/2009 page 186

[figure] [figure]

“JMMBookOct04” 30/10/2009 page 187

[figure] [figure]

Chapter 14

Contrast-Invariant Monotone Operators

A function operator T is monotone if u ≥ v ⇒ T u ≥ T v. A set operator T is monotone if X ⊂ Y implies T X ⊂ T Y. We are mainly interested in monotone function operators, since they are nonlinear generalizations of linear smoothing using a nonnegative convolution kernel. We have already argued that for image analysis to be robust, the operators must also be contrast invariant. The overall theme here will be to develop the equivalence between monotone contrast-invariant function operators and monotone set operators. This equivalence is based on one of the fundamentals of mathematical morphology described in Chapter 12: A real-valued function is completely described by its level sets.

This allows one to process an image u by processing separately its level sets by some monotone set operator T and defining the processed image by the superposition principle

T u = sup{λ, x ∈ T (X_{λ}u)}.

Such an operator is called in digital technology a stack filter, since it processes an image as a stack of level sets. Conversely, we shall associate with any contrast invariant monotone function operator T a monotone set operator by setting

T (X_{λ}u) = X_{λ}(T u).

Such a construction is called a level set extension of T.

Several questions arise, which will be all answered positively once the functional framework is fixed: Are stack filters contrast invariant? Conversely, is any monotone contrast invariant operator a stack filter? Is any monotone set operator the level set extension of its stack filter?

In Section 14.1 we shall make definitions precise and give some remarkable conservative properties of contrast invariant monotone operators. Section 14.2 is devoted to stack filters and shows that they are monotone and contrast invariant. Section 14.3 defines the level set extension and shows the converse statement: Any contrast invariant monotone operator is a stack filter. Section 14.4 applies this construction to a remarkable denoising stack filter due to Vincent and Serra, the area opening.

187 “JMMBookOct04” 30/10/2009 page 188

[figure] [figure]

188 CHAPTER 14. CONTRAST-INVARIANT MONOTONE OPERATORS

14.1 Contrast-invariance

14.1.1 Set monotone operators

We will be mostly dealing with function operators T defined on F and set operators T defined on L, but sometimes also defined on M. We denote by D(T ) the domain of T . Now, all set operators we shall consider in practice are defined first on subsets of ℝ^{N}.

Definition 14.1. Let T a monotone operator defined on a set of subsets of ℝ^{N}. We call standard extension of T to S_{N} the operator, still denoted by T , defined by

T (X) = T (X \ {∞}) ∪ (X ∩ {∞}).

In other terms if X doesn’t contain ∞, T (X) is already defined and if X contains ∞, T (X) contains it too. Thus a standard extension satisfies ∞ ∈ T X ⇔ ∞ ∈ X.

Remark 14.2. Let us examine the case where T is initially defined on C, the set of all closed subsets of ℝ^{N}. There are only two kinds of sets in L, namely

• compact sets of ℝ^{N}

• sets of the form X = C ∪ {∞}, where C is a closed set of ℝ^{N}.

Thus the standard extension of T extends T to L, the set of all closed (and therefore compact) subsets of S_{N}.

All of the usual monotone set operators used in shape analysis satisfy a small list of standard properties which it is best to fix now. Their meaning will come obvious in examples.

Definition 14.3. We say that a set operator T defined on its domain D(T ) is standard monotone if

• X ⊂ Y ⟹ T X ⊂ T Y;

• ∞ ∈ T X ⟺ ∞ ∈ X;

• T (∅) = ∅, T (S_{N}) = S_{N};

• T (X) is bounded in ℝ^{N} if X is;

• T (X)^{c} is bounded in ℝ^{N} if X^{c} is.

Definition 14.4. Let T be a monotone set operator on its domain D(T ). We call dual domain the set

D(T̃) := {X ⊂ S_{N} | X^{c} ∈ D(T )}.

We call dual of T the operator X → T̃X = (T (X^{c}))^{c}, defined on D(T̃).

Proposition 14.5. T is a standard monotone operator if and only if T̃ is.

Exercise 14.1. Prove it!

“JMMBookOct04” 30/10/2009 page 189

[figure] [figure]

14.1. CONTRAST-INVARIANCE 189

14.1.2 Monotone function operators

Function operators are usually defined on F, the set of continuous functions having some limit u(∞) at infinity. We shall always assume that this limit is preserved by T, that is, T u(∞) = u(∞). Think that images are usually compactly supported. Thus u(∞) is the “color of the frame” for a photograph. There is no use in changing this color.

Definition 14.6. We say that a function operator T : F → F is standard monotone if for all u, v ∈ F,

u ≥ v ⟹ T u ≥ T v; T u(∞) = u(∞).  (14.1)

Exercise 14.2. Is the operator T defined by (T u)(x) = u(x) + 1 standard monotone?

Recall from Chapter 12 that any nondecreasing continuous surjection g : ℝ → ℝ is called a contrast change.

Definition 14.7. A function operator T: F → F is said to be contrast invariant if for every u ∈ F and every contrast change g,

g(T u) = T g(u).  (14.2)

Checking contrast invariance with increasing contrast changes will make our life simpler.

Lemma 14.8. A monotone operator is contrast invariant if and only if it commutes with strictly increasing contrast changes.

Proof. Let g be a contrast change. We can find strictly increasing continuous functions g_{n} and h_{n} : ℝ → ℝ such that g_{n}(s) → g(s), h_{n}(s) → g(s) for all s and g_{n} ≤ g ≤ h_{n} (see Exercise 14.12.) Thus, by using the commutation of T with increasing contrast changes, we have

T (g(u)) ≥ T (g_{n}(u)) = g_{n}(T u) → g(T u) and
T (g(u)) ≤ T (h_{n}(u)) = h_{n}(T u) → g(T u),

which yields T (g(u)) = g(T u). □ Let us give some notable properties entailed by the monotonicity and the contrast invariance.

Lemma 14.9. Let T be standard monotone contrast invariant operator. Then for every constant function u ≡ c one has T u ≡ c.

Proof. Let g be a contrast change such that g(s) = s for inf T u ≤ s ≤ sup T u. Since T u(∞) = u(∞) = c, this implies that inf T u ≤ c ≤ sup T u and therefore g(c) = c, which means g(u) = u.. By the contrast invariance we therefore obtain T u = T g(u) = g(T u) ≡ c. □

“JMMBookOct04” 30/10/2009 page 190

[figure] [figure]

190 CHAPTER 14. CONTRAST-INVARIANT MONOTONE OPERATORS

We have indicated several times the importance of image operators being contrast invariant. In practice, image operators are also translation invariant. For x ∈ ℝ^{N} we are going to use the notation τx to denote the translation operator for both sets and functions: For X ∈ M, τxX = {x + y | y ∈ X}, and for u ∈ F, τxu is defined by τxu(y) = u(y − x). Since elements of M can contain ∞, we specify that ∞ ± x = ∞ when x ∈ ℝ^{N}. This implies that τxu(∞) = u(∞).

Definition 14.10. A set operator T is said to be translation invariant if its domain is translation invariant and if for all X ∈ D(T ) and x ∈ ℝ^{N},

τxT X = T τxX.

A function operator T is said to be translation invariant if for all u ∈ F and x ∈ ℝ^{N},

τxT u = T τxu.

We say that a function operator T commutes with the addition of constants if u ∈ F and c ∈ ℝ imply T (u + c) = T u + c.

Contrast-invariant operators clearly commute with the addition of constants: Consider the contrast change defined by g(s) = s + c.

Lemma 14.11. Let T be a translation-invariant monotone function operator on F that commutes with the addition of constants. If u ∈ F is K-Lipschitz on ℝ^{N}, namely |u(x) − u(y)| ≤ K|x − y| for all x, y in ℝ^{N}, then so is T u.

Proof. For any x ∈ ℝ^{N}, y ∈ ℝ^{N}, and z ∈ S_{N}, we have

u(y + z) − K|x − y| ≤ u(x + z) ≤ u(y + z) + K|x − y|.  (14.3)

These inequalities work for z = ∞ because u(y+∞) = u(x+∞) = u(∞). Thus we can write them as inequalities between functions on S_{N}:

τ_{−y}u − K|x − y| ≤ τ_{−x}u ≤ τ_{−y}u + K|x − y|.  (14.4)

Since T is monotone, we can apply T to the functions in (14.4) and preserve the inequalities, which yields

T (τ_{−y}u − K|x − y|) ≤ T (τ_{−x}u) ≤ T (τ_{−y}u + K|x − y|).

Now use the fact that T commutes with the addition of constants the translation invariance of T to obtain

τ_{−y}(T u) − K|x − y| ≤ τ_{−x}(T u)) ≤ T (τ_{−y}u) + K|x − y|).

Taking the values of these functions at 0 yields

T u(y) − K|x − y| ≤ T u(x) ≤ T u(y) + K|x − y|,

which is the announced result. □

We say that an operator is monotone on a set of functions if u ≥ v ⇒ T u ≥ T v. Clearly all above proofs do not depend upon the fact that the operator is standard, but just upon its translation invariance and monotonicity. Thus, by considering the proof of Lemma 14.11 and the definition of uniform continuity (Definition 1.3), one obtains the following generalizations.

“JMMBookOct04” 30/10/2009 page 191

[figure] [figure]

14.2. STACK FILTERS 191

Corollary 14.12. Assume that T is a translation-invariant monotone operator on a set of uniformly continuous functions, that commutes with the addition of constants. Then T u is uniformly continuous on ℝ^{N} with the same modulus of continuity. In particular if u is L-Lipschitz on ℝ^{N}, then so is T u.

Exercise 14.3. Prove corollary 14.12.

14.2 Stack filters

Definition 14.13. We say that a function operator T is obtained from a monotone set operator T as a stack filter if

T u(x) = sup{λ | x ∈ T (X_{λ}u)}  (14.5)

for every x ∈ S_{N}.

The relation (14.5) has practical implications. It means that T u can be computed by applying T separately to each characteristic function of the level sets X_{λ}u. This leads to the following stack filter algorithm.

u X_{λ}u → T (X_{λ}u) ; ↗  ↘ ; ⋮ ; ↘  ↗ ; X_{μ}u → T (X_{μ}u) T u(x) = sup{λ | x ∈ T (X_{λ}u)}.

The image u is decomposed into the stack of level sets. Each level set is processed independently by the monotone operator T . This yields a new stack of sets T (X_{λ}u) and Formula (14.5) always defines a function T u. Now, this construction will be perfect only if

X_{λ}(T u) = T (X_{λ}u).  (14.6)

Definition 14.14. When (14.6) holds, we say that T “commutes with thresholds”, or that T and T satisfy the “commutation with threshold” property.

Of course, this commutation can hold only if T sends L into itself. A further condition which turns out to be necessary is introduced in the next definition.

Definition 14.15. We say that a monotone set operator T : L → L is upper semicontinuous if for every sequence of compact sets X_{n} ∈ D(T ) = L such that X_{n+1} ⊂ X_{n}, we have

T (⋂_{n} X_{n}) = ⋂_{n} T (X_{n}).  (14.7)

Exercise 14.4. Show that a monotone operator T : L → L is upper semicontinuous if and only if it satisfies, for every family (X_{λ})_{λ∈ℝ} ⊂ L such that X_{λ} ⊂ X_{μ} for λ > μ, the relation T (⋂_{λ} X_{λ}) = ⋂_{λ} T (X_{λ}).

“JMMBookOct04” 30/10/2009 page 192

[figure] [figure]

192 CHAPTER 14. CONTRAST-INVARIANT MONOTONE OPERATORS

Exercise 14.5. Show that a monotone operator on L is upper semicontinuous if and only if it satisfies (14.7) for every sequence of compact sets X_{n} such that X_{n+1} ⊂ X_{n}^{∘}. Hint: Since S_{N} is the unit sphere in ℝ^{N+1}, one can endow it with the euclidian distance d in ℝ^{N+1}. Given a nondecreasing sequence Y_{n} in L, set X_{n} = \{x, d(x, Y_{n}) ≤ \frac{1}{n} \}. Then apply (14.7) to X_{n} and check that ⋂_{n} X_{n} = ⋂_{n} Y_{n}.

Exercise 14.6. Show that a monotone operator T : L → L is upper semicontinuous if and only if it satisfies, for every family (X_{λ})_{λ∈ℝ} ⊂ L such that X_{λ} ⊂ X_{μ}^{∘} for λ > μ, the relation T (⋂_{λ} X_{λ}) = ⋂_{λ} T (X_{λ}).

Theorem 14.16. Let T : L → M be a translation invariant standard monotone set operator. Then the associated stack filter T is translation invariant, contrast invariant and standard monotone from F into itself. If, in addition, T is upper semicontinuous, then T commutes with thresholds.

Proof that T is monotone. One has

u ≤ v ⇔ (∀λ, X_{λ}u ⊂ X_{λ}v).

Since T is monotone, we deduce that

∀λ, T (X_{λ}u) ⊂ T (X_{λ}v)

which by (14.5) implies T u ≤ T v.

Proof that T is contrast invariant. By Lemma 14.8 we can take g strictly increasing and therefore a bijection from ℝ to ℝ. We notice that : For λ > g(sup u), X_{λ}g(u) = ∅ and therefore T (X_{λ}g(u)) = ∅. For λ < g(inf u), X_{λ}g(u) = S_{N} and therefore T (X_{λ}g(u)) = S_{N}. Thus using (14.5) we can restrict the range of λ in the definition of T (g(u))(x) :

T (g(u))(x) = sup{λ, g(inf u) ≤ λ ≤ g(sup u), x ∈ T (X_{λ}g(u))}
= sup{g(μ), x ∈ T (X_{g(μ)}g(u))}
= sup{g(μ), x ∈ T (X_{μ}u)} = g(T u(x)).

Proof that T u belongs to F. T is by construction translation invariant. By Corollary 14.12, T u is uniformly continuous on ℝ^{N}. Let us prove that T u(x) → u(∞) as x → ∞. We notice that for λ > u(∞), X_{λ}u is bounded. Since T is standard monotone T (X_{λ}u) is bounded too. Now, by (14.5), T u(x) ≤ λ if x ∈ T (X_{λ}u)^{c}. This last condition is satisfied if x is large enough and we deduce that lim sup_{x→∞} T u(x) ≤ u(∞). In the same way notice that (X_{λ}u)^{c} is bounded if λ < u(∞). So by the same argument, we also get lim inf_{x→∞} T u(x) ≥ u(∞). T being standard, it is easily checked using (14.5) that T u(∞) = u(∞). Thus, T u is continuous on ℝ^{N} and at ∞ and therefore on S_{N}.

Proof that T commutes with thresholds, when T is upper semicontinuous. Let us show that the sets Y_{λ} = T (X_{λ}u) satisfy the properties (i) and (ii) in “JMMBookOct04” 30/10/2009 page 193

[figure] [figure]

14.3. THE LEVEL SET EXTENSION 193

Proposition 12.2. By the monotonicity of T , Y_{λ} ⊂ Y_{μ} for λ > μ. Since T (∅) = ∅, we have

Y_{λ} = T (X_{λ}u) = T (∅) = ∅

for λ > max u and, in the same way Y_{λ} = S_{N} for λ < min u. So T u has the same bounds as u. This proves Property (i). As for Property (ii), we have for every λ, using the upper semicontinuity and exercise 14.4,

Y_{λ} = T (X_{λ}u) = T ( ⋂_{μ<λ} X_{μ}u) = ⋂_{μ<λ} T (X_{μ}u) = ⋂_{μ<λ} Y_{μ}.

So by applying the converse statement of Proposition 12.2, we deduce that

X_{λ}(T u) = T (X_{λ}u).
□

Exercise 14.7. Check that T u(∞) = u(∞), as claimed in the former proof.

The upper semicontinuity of T is necessary to ensure the commutation with thresholds. See Exercise 14.21. The assumption that T sends bounded sets of ℝ^{N} on bounded sets of ℝ^{N} and complementary sets of bounded sets onto complementary sets of bounded sets also is necessary to ensure that T u is continuous at ∞: see Exercise 14.16.

14.3 The level set extension

Our aim here is just the converse as in the former section. We wish to associate a standard monotone set operator T from L to L with any contrast invariant standard monotone function operator T, in such a way that the whole machinery works, namely both operators satisfy the commutation with threshold property T (X_{λ}u) = X_{λ}(T u) and T is the stack filter of T .

Lemma 14.17. Let u ≤ 0 and v ≤ 0 ∈ F and assume that X_{0}u = X_{0}v (≠ ∅). Then there is a contrast change h such that h(0) = 0 and u ≥ h(v).

Proof. Define

h̃(r) = \{ \begin{matrix}min\{u(x) | x ∈ X_{r}v\} & if min v ≤ r ≤ 0; \\ r & if r > 0; \\ min u − min v + r & if r ≤ min v. \end{matrix}

Notice that h̃(0) = 0 and that h̃ is nondecreasing. The following relation holds for all x ∈ ℝ^{N} by the definition of h̃ and because u(x) belongs to the set {u(y) | v(y) ≥ v(x)}:

u(x) ≥ min{u(y) | v(y) ≥ v(x)} = h̃(v(x)).

We now use the compactness in S_{N} of the level sets of v to show that h̃ is continuous at zero. Let (r_{k})_{k∈ℕ} be an arbitrary increasing sequence tending to zero. Choose x_{k} ∈ X_{r_{k}}v such that h̃(r_{k}) = u(x_{k}). This is possible because u is “JMMBookOct04” 30/10/2009 page 194

[figure] [figure]

194 CHAPTER 14. CONTRAST-INVARIANT MONOTONE OPERATORS

continuous and the X_{r_{k}}v are compact and nonempty. Since h̃ is nondecreasing, h̃(r_{k}) → h̃^{−}(0).

Let x be any accumulation point of the set {x_{k}}_{k∈ℕ}. Since the X_{r_{k}}v are compact, all the accumulation points of the set {x_{k}}_{k∈ℕ} are contained in X_{0}v = ⋂_{k∈ℕ} X_{r_{k}}v. This means that u(x) = 0. But lim u(x_{k}) = u(x) by the continuity of u, and we conclude that h̃^{−}(0) = 0. At this point h̃ satisfies the announced requirements for h, except that it is not always continuous for all r < 0. This is easily fixed by choosing a continuous nondecreasing function h such that h̃ ≥ h and h(0) = 0. One way to do this is to take h(r) = (1/|r|) ∫_{2r}^{r} h̃(s) ds for r < 0. Then u(x) ≥ h̃(v(x)) ≥ h(v(x)) as announced. □

Exercise 14.8. Prove that h(r) = (1/|r|) ∫_{2r}^{r} h̃(s) ds is indeed nondecreasing and continuous for r ≤ 0 and that h̃ ≥ h. Find examples of functions u and v defined on S_{1} for which h̃ is not continuous.

Definition 14.18 (and proposition (Evans-Spruck)). ^{1} Given a contrast invariant monotone operator T on F, we call level set extension of T the set operator defined in the following way : for any X ∈ L, take u ≤ 0 such that X_{0}u = X and set

T (X) = X_{0}T (u).

Then T (X) does not depend upon the particular choice of u.

Proof. The proof follows directly from Lemma 14.17: Take u and v ∈ F such that u ≤ 0, v ≤ 0, and X_{0}u = X_{0}v. Let h be a contrast change such that h(0) = 0 and u ≥ h(v). Since T is monotone and contrast invariant one has by Lemma 14.1.2 T u ≤ 0, and T u ≥ T h(v) = h(T v). Using the fact that h(0) = 0, we obtain that T v(x) = 0 implies that T u(x) = 0. By interchanging the roles of u and v, T u(x) = 0 implies that T v(x) = 0. We conclude that X_{0}T u = X_{0}T v. □

Exercise 14.9. Definition 14.18 would’nt be complete if we did not prove that for any X ∈ L we can find u ≤ 0 in F such that X_{0}u = X. Hint: Since S_{N} is the unit sphere in ℝ^{N+1}, one can endow it with the euclidian distance d in ℝ^{N+1}. Use the distance function d(x, X) to define u. This distance function is continuous: see Exercise 14.18.

Theorem 14.19 (Evans–Spruck). Let T be a contrast-invariant monotone operator on F and T its level set extension on L. Then T is monotone, T and T satisfy the commutation with thresholds T X_{λ}u = X_{λ}T u for all λ ∈ ℝ, T is the stack filter associated with T and T is upper semicontinuous on L. In addition, if T is standard, then so is T .

Proof. Commutation with thresholds: Given u and λ, let g be a continuous contrast change such that g(s) = min(s, λ) − λ on the range of u, which is a compact interval of ℝ. We then have X_{0}g(u) = X_{λ}u. Using this relation, the level set extension and the contrast invariance of T,

T (X_{λ}u) = T (X_{0}g(u)) = X_{0}(T (g(u))) = X_{0}(g(T u)) = X_{λ}(T u).

1What we are doing here is related to the scheme originally introduced by Osher and Sethian as a numerical method for front propagation [199]. We briefly described their work in the Introduction (see page 26).

“JMMBookOct04” 30/10/2009 page 195

[figure] [figure]

14.4. A FIRST APPLICATION: THE EXTREMA KILLER 195

Proof of the stack filter property: This is an immediate consequence of the superposition principle and the commutation with thresholds :

T u(x) = sup{λ | x ∈ X_{λ}T u} = sup{λ | x ∈ T (X_{λ}u)}.

Proof that T is upper semicontinuous on L: By the result of Exercise 14.5, it is enough to consider a sequence (X_{n})_{n≥1} in L such that X_{n+1} ⊂ X_{n}^{∘}. By Lemma 14.20 below there is a function u ∈ F such that X_{1− \frac{1}{n}} u = X_{n} and X_{1}u = ⋂_{n} X_{n}. Finally, using twice the just proven commutation of thresholds,

T (⋂_{n} X_{n}) = T (X_{1}u) = X_{1}(T u) = ⋂_{n} X_{1− \frac{1}{n}} T u = ⋂_{n} T (X_{1− \frac{1}{n}} u) = ⋂_{n} T (X_{n}).

Proof that T is standard if T is: Recall that T is standard if T u(∞) = u(∞). By using the commutation with thresholds, all of the standard properties for T are straightforward. For instance, taking some u ∈ F,

T (∅) = T (X_{max u+1}u) = X_{max u+1}T u = ∅.

Indeed, by the monotonicity, the contrast invariance, and Lemma 14.1.2, u ≤ C ⇒ T u ≤ C. In the same way, let X ∈ L and u a function such that X_{0}u = X. If X is bounded, then u(∞) < 0, so that T u(∞) = u(∞) < 0. Thus T (X) = X_{0}T u is bounded. If X^{c} = {x | u(x) < 0} is bounded, then T u(∞) = u(∞) ≥ 0. Thus T (X)^{c} = (X_{0}T u)^{c} is bounded. Finally by the commutation with thresholds,

∞ ∈ X ⇔ u(∞) ≥ 0 ⇔ T u(∞) ≥ 0 ⇔ ∞ ∈ X_{0}(T u) = T (X).

□

Exercise 14.10. Prove that the level set extension T is monotone. The argument is not given in the above proof.

Lemma 14.20. Let (X_{n})_{n≥1} be a sequence in L such that X_{n+1} ⊂ X_{n}^{∘}. There is a function u ∈ F such that X_{1− \frac{1}{n}} u = X_{n} for n ≥ 1 and X_{1}u = ⋂_{n≥1} X_{n}.

Proof. Let us use the euclidian distance d of ℝ^{N+1} restricted to S_{N} considered as a subset of ℝ^{N+1}. Set u(x) = 1 if x ∈ ⋂_{n} X_{n},

u(x) = (1 − \frac{1}{n})\frac{d(x, X_{n+1})}{d(x, X_{n}^{c}) + d(x, X_{n+1})} + (1 − \frac{1}{n + 1})\frac{d(x, X_{n}^{c})}{d(x, X_{n}^{c}) + d(x, X_{n+1})}

for x ∈ X_{n} \ X_{n+1} and n ≥ 1, u(x) = − sup(−1, −d(x, X_{1})) if x ∉ X_{1}. It is easily checked that u belongs in F and satisfies the announced properties. □

14.4 A first application: the extrema killer

This section is devoted to the study of operators that remove “peaks,” or extreme values, from an image. Such peaks are often created by impulse noise, “JMMBookOct04” 30/10/2009 page 196

[figure] [figure]

196 CHAPTER 14. CONTRAST-INVARIANT MONOTONE OPERATORS

that is, local destruction of pixel values and their replacement by a random value. Old movies present this kind of noise and it also occurs by transmission failure in satellite imaging. The operators we study are called area opening, or extrema killer operators, and they have been shown to be very effective at removing this kind of noise. The action of these operators is illustrated in Figures 14.1 and 14.2.

The following definitions are standard, but we include them here for completeness.

Definition 14.21. Consider a closed subset X of S_{N}. X is disconnected if it can be written as X = (A ∩ X) ∪ (B ∩ X), where A and B are disjoint open sets and both A ∩ X and B ∩ X are not empty. X is connected if it is not disconnected. The connected component of x in X, denoted by cc(x, X), is the maximal connected subset of X that contains x.

We wish to define a denoising operator on L ; since some sets therein contain ∞, we need an extension of the Lebesgue measure on ℝ^{N} to S_{N}. This is immediately fixed by setting meas({∞}) = +∞. The only property of this extended measure that we need to check is following:

Lemma 14.22. if Y_{n} is a nonincreasing sequence of compact sets of S_{N}, then meas(∩_{n}Y_{n}) = lim_{n} meas(Y_{n}).

Proof. If the compact sets Y_{n} do not contain ∞ for n large enough, then they are bounded in ℝ^{N} for n large and the result just follows from Lebesgue theorem. If instead the sets Y_{n} all contain ∞, then ∩_{n}Y_{n} contains it too and all sets have infinite measure. □

Definition 14.23. Let a > 0 a scale parameter and denote for every X ∈ L by X_{i} its connected components, so that X = ⋃_{i} X_{i}. We call small component killer the operator on L which removes from X all connected components with area stricly less than a :

T_{a}X = ⋃_{meas (X_{i})≥a} X_{i}.  (14.8)

Theoretically, X can have an uncountable number of components; take, for example, the Cantor set. However, X can have only a countable number of components with positive measure. The assumption meas({∞}) = +∞ implies that all connected components of X containing ∞ stay in T_{a}X. We are going to prove that the small component killer is upper semicontinuous and this uses some elementary topological lemmas.

Lemma 14.24. Consider an arbitrary nonincreasing sequence of nonempty compact sets (Y_{n})_{n∈ℕ} of S_{N} and its limit Y = ⋂_{n∈ℕ} Y_{n}. Then Y is not empty and compact. In addition, for any open set Z that contains Y, there is an index n_{0} such that Y_{n} ⊂ Z for all n ≥ n_{0}.

“JMMBookOct04” 30/10/2009 page 197

[figure] [figure]

14.4. A FIRST APPLICATION: THE EXTREMA KILLER 197

Proof. The first property is a classical property of compact sets. Assume by contradiction that the second property is not true. Then Y_{n} ∩ (S_{N} \ Z) ≠ ∅ infinitely often. This implies that (Y_{n}∩(S_{N} \Z))_{n∈ℕ} is a nonincreasing sequence of nonempty compact sets. But this means that Y ∩ (S_{N} \ Z) ≠ ∅, which is a contradiction. □

Lemma 14.25. Let (Y_{n})_{n∈ℕ} be a nonincreasing sequence of nonempty compact subsets of S_{N} and consider the intersection Y = ⋂_{n∈ℕ} Y_{n}. If the Y_{n} are connected, then Y is connected.

Proof. We know that Y is not empty and compact. Suppose, by contradiction, that Y is not connected. Then we can represent Y by Y = (Y ∩ Z_{1}) ∪ (Y ∩ Z_{2}), where Z_{1} and Z_{2} are disjoint open sets, Y ∩ Z_{1} ≠ ∅, and Y ∩ Z_{1} ≠ ∅. Since Y ⊂ Z_{1} ∪ Z_{2}, by Lemma 14.24 there exists an n_{0} such that Y_{n} ⊂ Z_{1} ∪ Z_{2} for all n ≥ n_{0}, and for these n we have

Y_{n} = Y_{n} ∩ (Z_{1} ∪ Z_{2}) = (Y_{n} ∩ Z_{1}) ∪ (Y_{n} ∩ Z_{2}).

Furthermore, Y_{n} ∩ Z_{1} ≠ ∅ and Y_{n} ∩ Z_{1} ≠ ∅. This contradicts the fact that the Y_{n} are connected. □

Exercise 14.11. Show that T_{a} is idempotent: T_{a}^{2}X = T_{a}X and that it is a contraction mapping: T_{a}X ⊂ X.

With the extrema killer we have a prime example of a theory that begins with a set operator T_{a} defined on L.

Lemma 14.26. The small component killer T_{a} is upper semicontinuous on L.

Proof. We first prove that T_{a} is monotone. Assume X ⊂ Y. Then for every x ∈ X, cc(x, X) ⊂ cc(x, Y). If meas (cc(x, X)) ≥ a, then meas (cc(x, Y)) ≥ a, and we conclude that T_{a}X ⊂ T_{a}Y. Now let (X_{n})_{n} be any nonincreasing sequence of nonempty compact sets and X = ∩_{n}X_{n}. We wish to show that T_{a}X = ⋂_{n} T_{a}X_{n}. By monotonicity of T_{a},

T_{a}X ⊂ ⋂_{n} T_{a}(X_{n}).

Let us show the converse inclusion. Let x ∈ ∩_{n}T_{a}(X_{n}). Then Y_{n} := cc(x, X_{n}) has measure larger than a for all n. In addition if m < n then Y_{n} ⊂ Y_{m}. By Lemmas 14.24 and 14.25, Y := ∩_{n}Y_{n} is a connected compact set that contains x. In addition by Lemma 14.22, measure(Y) = lim_{n} measure(Y_{n}) ≥ a. Since Y = ∩_{n}Y_{n} ⊂ ∩_{n}X_{n} = X, we have cc(x, X) ⊇ Y and therefore x ∈ T_{a}(X). □

We can now build a stack filter from T_{a}.

Definition 14.27 (and proposition). The stack filter T_{a} of T_{a} is called a maxima killer. T_{a} and T_{a} satisfy the commutation with thresholds. As a consequence, no connected component of a level set of T_{a}u has measure less than a. Furthermore, T_{a} is standard monotone, translation and contrast invariant from F into F.

“JMMBookOct04” 30/10/2009 page 198

[figure] [figure]

198 CHAPTER 14. CONTRAST-INVARIANT MONOTONE OPERATORS

Figure 14.1: Extrema killer: maxima killer followed by minima killer. The extrema killer removes all connected components of upper and lower level sets with area less than some threshold, which here equals 20 pixels. Notice how texture disappears in the second image. All other features seem preserved. On the second row, we see for both the original and the processed image the level lines at 16 equally spaced levels. The level lines on the right hand side are a subset of the level lines of the left hand. All level lines surrounding extremal regions with area smaller than 20 have been removed and the other ones are untouched.

Proof. We just have to check that all assumptions of Theorem 14.16 are satisfied. T_{a} is obviously translation invariant, monotone and is upper semicontinuous by Lemma 14.26. It satisfies T_{a}(∅) = ∅, T_{a}(S_{N}) = S_{N}. T_{a}(E) is compact if E is. Indeed, it is the union of a finite set of compact connected components. If E is bounded in ℝ^{N}, then so is T_{a}E ⊂ E. (T_{a}E)^{c} is bounded in S_{N} if E^{c} is. Indeed, if E^{c} is bounded, then E has a connected component Y containing B(0, R)^{c} for some R > 0. This connected component has infinite measure. Then T_{a}(E) still contains Y and T_{a}(E)^{c} is contained in B(0, R). By construction, ∞ belongs to T_{a}X if and only if it belongs to X. Thus, T_{a} is standard monotone. □

“JMMBookOct04” 30/10/2009 page 199

[figure] [figure]

14.5. EXERCISES 199

A maxima killer T_{a} cuts off the maxima of continuous functions, but it does nothing for the minima. We can immediately define a minima killer T_{a}^{−} as the dual operator of T_{a},

T_{a}^{−}u = −T_{a}(−u).

A good denoising process is to alternate T_{a} and T_{a}^{−}, as illustrated in Figures 14.1 and 14.2 . We note, however, that T_{a} and T_{a}^{−} do not necessarily commute, as is shown in Exercise 14.17.

14.5 Exercises

Exercise 14.12. Let g : ℝ → ℝ be a contrast change. Construct increasing contrast changes g_{n} and h_{n} such that g_{n}(s) → g(s), h_{n}(s) → g(s) for all s and g_{n} ≤ g ≤ h_{n}. Hint : define first an increasing continuous function f(s) on ℝ such that f(−∞) = 0 and f(+∞) = \frac{1}{n}.

Exercise 14.13. Let u : ℝ^{N} → ℝ. Show that τxX_{λ}u = X_{λ}τxu, x ∈ ℝ^{N}.

Exercise 14.14. Prove that a translation invariant operator T from L to L satisfies one of the three possibilities : T ({∞}) = {∞}, T ({∞}) = S_{N} or T ({∞}) = ∅.

Exercise 14.15. Let T be a translation invariant standard monotone operator on F. Prove the following statements:

(i) T u = c for every constant function u : S_{N} → c.

(ii) u ≥ c implies T u ≥ c, and u ≤ c implies T u ≤ c.

(iii) If in addition T commutes with the addition of constants, sup_{x∈ℝ^{N}} |T u(x) − T v(x)| ≤ sup_{x∈ℝ^{N}} |u(x) − v(x)|. (Hint: Write − sup |u(x) − v(x)| ≤ u(x) − v(x) ≤ sup |u(x) − v(x)|.)

Exercise 14.16.

1) In dimension 1, consider the set operator defined on L by T X = [inf X, ∞] if inf(X ∩ ℝ) ∈ ℝ, T X = S_{1} if inf(X ∩ ℝ) = −∞, T ({∞}) = {∞}, T (∅) = ∅. Check that T satisfies all assumptions of Theorem 14.16 except one. Compute the stack filter associated with T and show that it satisfies all conclusions of the mentioned theorem except one : T u does not belong to F and more specifically T u(x) is not continuous at ∞.

2) Consider the function operator on F, T u(x) = sup_{x∈S_{N}} u(x). Check that T is monotone, contrast invariant, and sends F to F. Compute the level set extension T of T.

Exercise 14.17. Let N = 1 and take u(x) = sin x for |x| ≤ 8π, u(x) = 0 otherwise. Compute T_{a}u and T_{a}^{−}u and show that they commute on u if a ≤ π and do not commute if a > π. Following the same idea, construct a function u ∈ F in dimension two such that T_{a}T_{a}^{−}u ≠ T_{a}^{−}T_{a}u.

Exercise 14.18. Let X be a closed subset of a metric space endowed with a distance d and consider the distance function to X,

d(y) = d(y, X) = inf_{x∈X} d(x, y).

Show that d is 1-Lipschitz, that is, |d(x, X) − d(y, X)| ≤ d(x, y).

“JMMBookOct04” 30/10/2009 page 200

[figure] [figure]

200 CHAPTER 14. CONTRAST-INVARIANT MONOTONE OPERATORS

Exercise 14.19. In the following questions, we explain the necessity of the assumptions T (∅) = ∅, T (S_{N}) = S_{N} for defining function monotone operators from F to F.

1) Set T (X) = X_{0} for all X ∈ L, where X_{0} ≠ ∅ is a fixed set. Check that the associated stack filter satisfies T u(x) = +∞ if x ∈ X_{0}, T u(x) = −∞ otherwise.

2) Let T be a monotone set operator, without further assumption. Show that its associated stack filter T is, however, monotone and commutes with all contrast changes. (We extend each contrast change g by setting g(±∞) = ±∞.)

Exercise 14.20. Take an operator T satisfying the same assumptions as in Theorem 14.16, but defined on M and apply the arguments of the proof of Theorem 14.16. Check that the stack filter associated with T is a contrast invariant, translation invariant monotone operator on the set of all bounded measurable functions, L^{∞}(ℝ^{N}). If in addition T is upper semicontinuous on M, then the commutation with thresholds holds.

Exercise 14.21. The upper semicontinuity is necessary to ensure that a monotone set operator defines a function operator such that the commutation with thresholds X_{λ}(T u) = T (X_{λ}(u)) holds for every λ. Let us choose for example the following set operator T ,

T (X) = X if meas(X) > a and T (X) = ∅ otherwise .

(We use the Lebesgue measure on ℝ^{N}, with the completion meas({∞}) = +∞)

1) Prove that T is standard monotone.

2) Let u be the function from S_{1} into S_{1} defined by u(x) = max(−|x|, −2a) for some a > 0, with u(∞) = −2a. Check that u belongs to F. Then, applying the stack filter T of T , check that

T (u)(x) = sup{λ, x ∈ T (X_{λ}u)} = max(min(−|x|, −a/2), −2a).

3) Deduce that X_{−a/2}T (u) = [−a/2, a/2], X_{−a/2}u = [−a/2, a/2] and therefore

T (X_{−a/2}u) = ∅ ≠ X_{−a/2}T (u),

which means that T does not commute with thresholds.

Exercise 14.22. Like in the preceding exercise, we consider here contrast invariant operators defined on all measurable bounded functions of ℝ^{N}. The aim of the exercise is to show that such operators send images with finite range into images with finite range. More precisely, denote by R(u) = u(ℝ^{N}) the range of u. Then we shall prove that for every u, R(T u) ⊂ \overline{Ru}. In particular, if R(u) is finite, then the range of T u is a finite subset of Ru. If u is binary, T u is, etc. This shows that contrast invariant operators preserve sharp contrasts. A binary image is transformed into a binary image. So contrast invariant operators create no blur, as opposed to linear operators, which always create new intermediate grey levels.

1) Consider

g(s) = s + \frac{1}{2} d(s, \overline{Ru})

where d(s, X) denotes the distance from s to X, that is, d(s, X) = inf_{x∈X} |s−x|. Show that g is a contrast change satisfying g(s) = s for s ∈ \overline{Ru} and g(s) > s otherwise.

2) Check that g(s) = s if and only if s ∈ \overline{Ru}. In particular, g(u) = u. Deduce from this and from the contrast invariance of T that for every x ∈ ℝ^{N}, T u(x) is a fixed point of g. Conclude.

“JMMBookOct04” 30/10/2009 page 201

[figure] [figure]

14.6. COMMENTS AND REFERENCES 201

Figure 14.2: Extrema killer: maxima killer followed by minima killer. Above, left: original image. Above, right: image after extrema killer removed connected components of 20 pixels or less. Below: level lines (levels of multiples of 16) of the image before and after the application of the extrema killer.

14.6 Comments and references

Contrast invariance and stack filters. Image operators that commute with thresholds have been popular because, among other reasons, they are easily implemented in hardware (VLSI). This led to very simple patents being awarded in signal and image processing as late as 1987 [59]. These operators have been given four different names, although operators are equivalent: stack filters [37, 103, 259]; threshold decomposition [108]; rank filters [52, 129, 261]; and order filters [241]. The best known of these are the sup, inf, and median operators. The implementation of the last named has received much attention because of its remarkable denoising properties [82, 197, 266].

Maragos and Shafer [160, 161] and Maragos and Ziff [162] introduced the functional notation and established the link between stack filters and the Matheron formalism in “flat” mathematical morphology. The complete equivalence between contrast-invariant operators and stack filters, as developed in this chapter, does not seem to have appeared elsewhere; at least we do not know of other references. A related classification of rank filters with elegant and useful generalizations to the so-called neighborhood filters can be found in [129].

The extrema killer. The extrema killer is probably the most efficient denoising filter for images degraded by impulse noise, which is manifest by small spots. In spite of its simplicity, this filter has only recently seen much use. This is undoubtedly due to the nontrivial computations involved in searching for the connected components of upper and lower level sets. The first reference to the “JMMBookOct04” 30/10/2009 page 202

[figure] [figure]

202 CHAPTER 14. CONTRAST-INVARIANT MONOTONE OPERATORS

extrema killer that we know is [55]. The filter in its generality was defined by Vincent in [253]. This definition fits into the general theory of connected filters developed by Salembier and Serra [219]. Masnou defined a variant called the grain filter that is both contrast invariant and invariant under reverse contrast changes [166]. Monasse and Guichard developed a fast implementation of this filter based on the so-called fast level set transform [181].

We will develop in Chapter 19 a theory of scale space that is based on a family of image smoothing operators T_{t}, where t is a scale parameter. We note here that the family (T_{a})_{a∈ℝ^{+}} of extrema killers does not constitute a scale space because it does not satisfy one of the conditions, namely, what we call the local comparison principle. That this is so, is the content of Exercise 19.1.

“JMMBookOct04” 30/10/2009 page 203

[figure] [figure]

Chapter 15

Sup-Inf Operators

The main contents of this chapter are two representation theorems: one for translation-invariant monotone set operators and one for functions operators that are monotone, contrast invariant, and translation invariant. If T is a function operator satisfying these three conditions, then it has a “sup-inf” representation of the form

T u(x) = sup_{B∈B} inf_{y∈B} u(x + y),

where B is a family of subsets of M(S_{N}), the set of all measurable subsets of S_{N}. This theorem is a nonlinear analogue of the Riesz theorem that states that a continuous linear translation-invariant operator from L^{2}(ℝ^{N}) to C^{0}(ℝ^{N}) can be represented as a convolution

T u(x) = ∫_{ℝ^{N}} u(x − y)k(y) dy.

In this case, the kernel k ∈ L^{2}(ℝ^{N}) is called the impulse response. In the same way, B is an impulse response for the nonlinear operator.

15.1 Translation-invariant monotone set operators

Recall that a set of M can contain ∞. We have specified that x + ∞ = ∞ for every x ∈ S_{N}. As a consequence, for any subset B of S_{N}, ∞ + B = {∞}. In this chapter we shall associate with any nonempty subset B of M a set operator defined by

T X = {x ∈ S_{N} | x + B ⊂ X for some B ∈ B}.  (15.1)

Definition 15.1. We say that a subset B of M is standard if it is not empty and if its associated operator satisfies

(i) ∀R > 0, ∃R′ > 0, T (B(0, R)) ⊂ B(0, R′);

(ii) ∀R > 0, ∃R′ > 0, B(0, R′)^{c} ⊂ T (B(0, R)^{c}).

203 “JMMBookOct04” 30/10/2009 page 204

[figure] [figure]

204 CHAPTER 15. SUP-INF OPERATORS

Exercise 15.1. Conditions (i) and (ii) on B are easily satisfied. Check that Condition (i) is equivalent to

∀R > 0, ∃C > 0, (B ∈ B, and diameter(B) ≤ R) ⇒ B ⊂ B(0, C).

Check that this condition is achieved (e.g.) if all elements of B contain 0. Check that Condition (ii) is achieved if B contains at least one bounded element B. Exercise 15.2. Show that if B contains ∅, then B is not standard.

Theorem 15.2 (Matheron). Let T be a translation-invariant and standard monotone set operator. Consider the subset of D(T ),

B = {B ∈ D(T ) | 0 ∈ T B}  (15.2)

Then B is standard and the operator T is associated with B by (15.1). Conversely, if B is any standard subset of M, then (15.1) defines a translationinvariant standard monotone set operator on M.

Definition 15.3. In Mathematical Morphology, a set B such that (15.1) holds is called a set of structuring elements of T and B = {X ∈ D(T ) | 0 ∈ T X} is called the canonical set of structuring elements of T .

Proof of Theorem 15.2.

Proof of (15.1). Let B = {X ∈ D(T ) | 0 ∈ T X}. Then for any x ∈ ℝ^{N},

x ∈ T X ⟺^{(1)} 0 ∈ T X − x ⟺^{(2)} 0 ∈ T (X − x) ⟺^{(3)} X − x ∈ B

⟺^{(4)} X − x = B for some B ∈ B ⟺^{(5)} x + B ⊂ X for some B ∈ B.

The equivalence (2) follows from the translation invariance of T X; (3) is just the definition of B; and (4) is a restatement of (3). The implication from left to right in (5) is obvious. The implication from right to left in (5) is the point where the monotonicity of T is used: Since B ⊂ X − x, it follows from the monotonicity of T that X − x ∈ B. Let now x = ∞. Since T is standard, B is not empty (it contains S_{N} because T (S_{N}) = S_{N}) and we have

∞ ∈ T X ⇔ ∞ ∈ X ⇔ ∃B ∈ B, ∞ + B ⊂ X,

because ∞ + S_{N} = {∞}.

Proof that B is standard if T is standard monotone. Since T (S_{N}) = S_{N}, B contains S_{N} and is therefore not empty. The other properties are straightforward.

Proof that (15.1) defines a standard monotone set operator if B is standard. Using (15.1), it is a straightforward calculation to check that T is monotone and translation invariant, and that T (S_{N}) = S_{N}, T (∅) = ∅. The equivalence ∞ ∈ T X if and only if ∞ ∈ X follows from the fact that B is not empty. T sends bounded sets onto bounded sets and complementary sets of bounded sets onto complementary sets of bounded sets by definition of a standard set B. □

“JMMBookOct04” 30/10/2009 page 205

[figure] [figure]

15.2. THE SUP-INF FORM 205

Exercise 15.3. Check that if T is standard monotone, then its canonical set of structuring elements satisfies (ii).

B_{0} = {X | 0 ∈ T X} is not the only set that can be used to represent T . A monotone operator T can have many such sets and here is their characterization.

Proposition 15.4. Let T be a translation invariant standard monotone set operator and let B_{0} its canonical set of structuring elements. Then B_{1} is another standard set of structuring elements for T if and only if it satisfies

(i) B_{1} ⊂ B_{0},

(ii) for all B_{0} ∈ B_{0}, there is B_{1} ∈ B_{1} such that B_{1} ⊂ B_{0}.

Proof. Assume that T is obtained from some set B_{1} by (15.1) and let B_{0} be the canonical set of structuring elements of T . Then for every B_{1} ∈ B_{1}, T B_{1} = {x | x + B ⊂ B_{1} for some B ∈ B_{1}}. It follows that 0 ∈ T B_{1} and therefore B_{1} ∈ B_{0}. Thus B_{1} ⊂ B_{0}. In addition, if B_{0} ∈ B_{0}, then 0 ∈ T B_{0}, which means that 0 ∈ {x | x + B_{1} ⊂ B_{0} for some B_{1} ∈ B_{1}}, that is B_{1} ⊂ B_{0} for some B_{1} ∈ B_{1}.

Conversely, let B_{1} satisfy (i) and (ii) and let

T_{1}X = {x | ∃B_{1} ∈ B_{1}, x + B_{1} ⊂ X}.

Using (i), one deduces that T_{1}X ⊂ T X for every X and using (ii) yields the converse inclusion. Thus B_{1} is a structuring set for T . The fact that B_{1} is standard is an obvious check using (i) and (ii). □

15.2 The Sup-Inf form

Lemma 15.5. Let T : F → F be a standard monotone function operator, T a standard monotone translation invariant set operator and B a set of structuring elements for T . If T and T satisfy the commutation of thresholds T X_{λ}u = X_{λ}T u, then T has the “sup-inf” representation

T u(x) = sup_{B∈B} inf_{y∈x+B} u(y).  (15.3)

Proof. For u ∈ F, set T̃u(x) = sup_{B∈B} inf_{y∈x+B} u(y). We shall derive the identity T = T̃ from the equivalence

T̃u(x) ≥ λ ⟺ T u(x) ≥ λ.  (15.4)

Assume first that x ∈ ℝ^{N}. Then

T u(x) ≥ λ ⟺^{(1)} T u(x) ≥ μ for all μ < λ ⟺^{(2)} x ∈ X_{μ}T u for all μ < λ

⟺^{(3)} x ∈ T X_{μ}u for all μ < λ ⟺^{(4)} ∃B ∈ B, x + B ⊂ X_{μ}u for all μ < λ

⟺^{(5)} There is a B ∈ B such that inf_{y∈x+B} u(y) ≥ μ for all μ < λ

⟺^{(6)} sup_{B∈B} inf_{y∈x+B} u(y) ≥ λ ⟺^{(7)} T̃u(x) ≥ λ.

“JMMBookOct04” 30/10/2009 page 206

[figure] [figure]

206 CHAPTER 15. SUP-INF OPERATORS

Equivalence (1) is just a statement about real numbers and (2) is the definition of a level set. It is at (3) that we replace X_{μ}T u with T X_{μ}u. Equivalence (4) follows by the definition of T from B by (15.1). The equivalence (5) is the definition of the level set X_{μ}u. Equivalence (6) is another statement about real numbers, and (7) is the definition of T̃. Assume now that x = ∞. Since for all B ∈ L, ∞ + B = {∞}, one obtains T̃u(∞) = u(∞). By assumption T u(∞) = u(∞). This completes the proof of (15.3). □

From the preceding result, we can easily derive a general form for translation and contrast invariant standard monotone operators.

Theorem 15.6. Let T : F → F be a translation and contrast invariant standard monotone operator. Then it has a “sup-inf” representation (15.3) with a standard set of structuring elements.

Proof. By the level set extension (Theorem 14.19), T defines a unique standard monotone set operator T : L ↦ L. T is defined by the commutation of thresholds, T X_{λ}u = X_{λ}T u. By Lemma 15.5, the commutation with thresholds is enough to ensure that T has the sup-inf representation (15.3) for any set of structuring elements B of T . □

Definition 15.7. As a consequence of the preceding theorem, the canonical set of structuring elements of T will also be called canonical set of structuring elements of T.

The next theorem closes the loop.

Theorem 15.8. Given any standard subset B of M, Equation (15.3),

T u(x) = sup_{B∈B} inf_{y∈x+B} u(y),

defines a contrast and translation invariant standard monotone function operator from F into itself.

Proof. By Theorem 14.16, it is enough to prove that T is the stack filter of T , the standard monotone set operator associated with B. Let us call T′ this stack filter and let us check that T u(x) ≥ λ ⇔ T′u(x) ≥ λ. we have T′u = sup{λ, x ∈ T (X_{λ}u)}. Thus by (15.1),

T′u(x) ≥ λ ⇔ ∀μ < λ, ∃B, x + B ⊂ X_{μ}u.

On the other hand,

T u(x) = sup_{B∈B} inf_{y∈x+B} u ≥ λ ⇔
∀μ < λ, ∃B ∈ B, inf_{y∈x+B} u ≥ μ ⇔
∀μ < λ, ∃B ∈ B, x + B ⊂ X_{μ}u.

“JMMBookOct04” 30/10/2009 page 207

[figure] [figure]

15.3. LOCALITY AND ISOTROPY 207

Thus, T = T′. □

We end this section by showing that sup-inf operators can also be represented as inf-sup operators,

T u(x) = inf_{B∈B} sup_{y∈x+B} u(y).

This is done, in the mathematical morphology terminology, by “duality”. The dual operator of a function operator is defined by T̃u = −T (−u). Notice that ˜T̃ = T.

Proposition 15.9. If T is a standard monotone, translation invariant and contrast invariant operator, then so is T̃. As a consequence, T has a dual “infsup” form

T u = inf_{B∈B̃} sup_{y∈x+B} u(y),

where B̃ is any set of structuring elements for T̃

Proof. Setting g̃(s) = −g(−s), it is easily checked that g̃ is a contrast change if and only if g is. One has by the contrast invariance of T,

T̃(g(u)) = −T (−g(u)) = −T (g̃(−u)) = −g̃(T (−u)) = g(−T (−u)) = g(T̃u).

Thus, T̃ is contrast invariant. The standard monotonicity and translation invariance of T̃ are obvious. Finally, if we have T̃u(x) = sup_{B∈B̃} inf_{y∈x+B} u(y), then

T u = − sup_{B∈B̃} inf_{y∈x+B}(−u(y)) = − sup_{B∈B̃}(− sup_{y∈x+B} u(y)) = inf_{B∈B̃} sup_{y∈x+B} u(y).

□

Exercise 15.4. Check the standard monotonicity and translation invariance of T̃.

15.3 Locality and isotropy

For linear filters, locality can be defined by the fact that the convolution kernel is compactly supported. This property is important, as it guarantees that the smoothed image is obtained by a local average. Morphological filters may need a locality property for the same reason.

Definition 15.10. We say that a translation invariant function operator T on F is local if there is some M ≥ 0 such that

(u = u′ on B(0, M)) ⇒ T u(0) = T u′(0).

The point 0 plays no special role in the definition. By translation invariance it is easily deduced from the definition that for x ∈ ℝ^{N}, the values of T u(x) only depend upon the restriction of u to B(x, M).

“JMMBookOct04” 30/10/2009 page 208

[figure] [figure]

208 CHAPTER 15. SUP-INF OPERATORS

Proposition 15.11. Let T : F → F be a contrast and translation invariant standard monotone operator and B a set of structuring elements for T. If T is local, then B_{M} = \{B ∈ B | B ⊂ \overline{B(0, M)}\} also is a set of structuring elements for T. Conversely, if all elements of B are contained in \overline{B(0, M)}, then T is local.

Proof. We prove the statement with the sup-inf form for T given by Theorem 15.6,

T u(x) = sup_{B∈B} inf_{y∈B} u(x + y).  (15.5)

Consider the new function u_{ε}(x) = u(x) − \frac{1}{ε}d(x, B(0, M)), where we take for d a distance function on S_{N} ⊂ ℝ^{N+1}, so that u_{ε} ∈ F. Take any B ∈ B containing a point z ∉ \overline{B(0, M)} and therefore not belonging to B_{M}. Then inf_{y∈B} u_{ε}(y) ≤ u(z) − \frac{1}{ε}d(z, B(0, M)) < T u(0) for ε small enough. So we can discard such B’s in the computation of T u(0) by (15.5). Since by the locality assumption T u(0) = T u_{ε}(0), we obtain

T u(0) = T u_{ε}(0) = sup_{B∈B_{M}} inf_{y∈B} u(y).

By the translation invariance of all all considered operators, this proves the direct statement. The converse statement is straightforward. □

We end this paragraph with a definition and an easy characterization of isotropic operators in the sup-inf form. In the next proposition, we actually consider a more general setting, namely the invariance of T under some geometric group G of transformations of ℝ^{N}, for example the isometries. Since we use to extend the set and function operators to S_{N}, we must also extend such transforms by setting g(∞) = ∞ for g ∈ G. Also, define the operator I_{g} on functions u : S_{N} → ℝ by I_{g}u(x) = u(gx).

Definition 15.12. • We say that B is invariant under a group G of transformations of S_{N} onto S_{N} if, for all g ∈ G, B ∈ B implies gB ∈ B.

• If, for all g ∈ G, T I_{g} = I_{g}T (resp. T g = gT ), we say that T (resp. T ) is invariant under G.

• In particular, we say that T (resp. T ) is isotropic if it commutes with all linear isometries R of ℝ^{N}, and affine invariant if it commutes with all linear maps A with determinant 1.

Proposition 15.13. Let G be any group of linear maps : g : ℝ^{N} → ℝ^{N} extended to S_{N} by setting g(∞) = ∞. If T (resp. T ) is translation invariant and invariant under G and B is a standard set of structuring elements for T (resp T ), then GB = {gB | g ∈ G, B ∈ B} is another, G-invariant, standard set of structuring elements. Conversely, if B is a standard and G-invariant set of structuring elements for T (resp. T ), then this operator is G-invariant (and translation invariant.)

Proof. All the verifications are straightforward. The only point to mention is that the considered groups are made of transforms sending bounded sets onto “JMMBookOct04” 30/10/2009 page 209

[figure] [figure]

15.4. THE OPERATORS WHO’S WHO 209

bounded sets and complementary sets of bounded sets onto complementary sets of bounded sets. □

Exercise 15.5. Prove carefully Proposition 15.13.

Some terminology. It would be tedious to state theorems on operators on F with such a long list of requirements as Standard Monotone, Translation and Contrast Invariant, Isotropic. We shall call such operators Morpho operators because they retain the essential requirements of morphological smoothing filters . All the examples we consider in this book are actually Morpho operators. Not all are local, so we will specify it when needed. Operators can be still more invariant, in fact affine invariant, and we will specify it as well. Since all of these operators T have an inf-sup or a sup-inf form, we always take for B a standard structuring set reflecting the properties of T, that is, bounded in B(0, M) when T is local and invariant by the same group as T. A last thing to specify is this: We have restricted our analysis to operators defined on F. On the other hand, their inf-sup form permits to extend them to all measurable functions and we shall still denote the resulting operator by T. T u can then assume the −∞ and +∞ values. All the same, it is an immediate check to see that this extension still is monotone and commutes with contrast changes:

Proposition 15.14. Let T be a function operator in the inf-sup or sup-inf form associated with a standard set of structuring elements B ⊂ M. Then T is monotone and contrast invariant on the set of all bounded measurable functions of S_{N}.

Exercise 15.6. Prove Proposition 15.14.

15.4 The who’s who of monotone contrast invariant operators

The aim of this short section is to draw a synthetic picture of an equivalence chain built up in this chapter and in Chapter 14. We have constructed three kinds of objects,

• contrast and translation invariant standard monotone function operators T : F → F;

• translation invariant standard monotone set operators T defined on L;

• standard sets of structuring elements B.

The results proven so far can be summarized in the following theorem.

Theorem 15.15. Given any of the standard objects T, T and B mentioned above, one can pass to any other one by using one of the six formulae given “JMMBookOct04” 30/10/2009 page 210

[figure] [figure]

210 CHAPTER 15. SUP-INF OPERATORS

below.

B → T,  T u(x) = sup_{B∈B} inf_{y∈x+B} u(y);
B → T ,  T X = {x | ∃B ∈ B, x + B ⊂ X};
T → T,  T u(x) = sup{λ | x ∈ T X_{λ}u};
T → T ,  T (X_{0}u) = X_{0}(T u);
T → B,  B = {B ∈ L | 0 ∈ T B};
T → B,  by T → T and T → B.

In addition, B can be bounded in some B(0, M) if and only if T is local; T or T is G-invariant, for instance isotropic, if and only if it derives from some G-invariant (isotropic) B. If an operator has the inf-sup or sup-inf form for some B, it can be extended to all measurable functions on ℝ^{N} into a monotone and contrast invariant operator.

Proof. Theorem 15.2 yields T → B and B → T ; Theorem 14.16 yields T → T; Theorem 15.6 yields T → T → B; Theorem 14.19 yields T → T . The final statements come from Propositions 15.11, 15.13 and 15.14. □

So we get a full equivalence between all objects, but we have left apart the commutation with thresholds property. When we define a set operator T from a function operator T by the level set extension, we know that T : L → L is upper semicontinuous and that the commutation with thresholds X_{λ}(T u) = T (X_{λ}u) holds. Conversely, if we define a function operator T as the stack filter of a standard monotone set T , we do not necessarily have the commutation of thresholds; this is true only if T is upper semicontinuous on L (see Theorem 14.16) and this upper semicontinuity property is not always granted for interesting monotone operators, particularly when they are affine invariant. Fortunately enough, the commutation with thresholds is “almost” satisfied for any stack filter as we state in Proposition 15.18 in the next section.

15.4.1 Commutation with thresholds almost everywhere

In this section we always assume the considered sets to belong to M and the considered functions to be Lebesgue measurable. We say that a set X is contained in a set Y almost everywhere if

measure(X \ Y) = 0,

where measure denotes the usual Lebesgue measure in ℝ^{N}. We say that X = Y almost everywhere if X ⊂ Y and Y ⊂ X almost everywhere. We say that two functions u and v are almost everywhere equal if measure({x, u(x) ≠ v(x)}) = 0.

Lemma 15.16. Let (X_{λ})_{λ∈ℝ} be a nonincreasing family of sets of M, that is X_{λ} ⊂ X_{μ} if λ ≥ μ. Then, for almost every λ in ℝ,

X_{λ} = ⋂_{μ<λ} X_{μ},  almost everywhere.  (15.6)

“JMMBookOct04” 30/10/2009 page 211

[figure] [figure]

15.5. EXERCISES 211

Proof. Consider an integrable and strictly positive continuous function h ∈ L^{1}(ℝ^{N}) (for instance, the gaussian.) Set m(X) = ∫_{X} h(x)dx. We notice that m(X) = 0 if and only if measure(X) = 0. The function λ → m(X_{λ}) is nonincreasing. Thus, it has a countable set of jumps. Since every countable set has zero Lebesgue measure, we deduce that for almost every λ,

lim_{μ→λ} m(X_{μ}) = m(X_{λ}).

As a consequence, for those λ’s, m(⋂_{μ<λ} X_{μ} \ X_{λ}) = 0, which implies (15.6). □

Corollary 15.17. Let (X_{λ})_{λ∈ℝ} be a family of measurable subsets of S_{N} such that X_{λ} ⊂ X_{μ} for λ ≥ μ, X_{λ} = ∅ for λ ≥ λ_{0}, X_{λ} = S_{N} for λ ≤ μ_{0}. Then the function u defined on S_{N} by the superposition principle

u(x) = sup{λ | x ∈ X_{λ}}

is bounded and satisfies for almost every λ, X_{λ} = X_{λ}u almost everywhere.

Proof. It is easily checked that μ_{0} ≤ u ≤ λ_{0}. We have

X_{λ}u = {x | sup{μ, x ∈ X_{μ}} ≥ λ}

Now, if x ∈ X_{λ}, we have sup{μ | x ∈ X_{μ}} ≥ λ which implies x ∈ X_{λ}u. Thus, X_{λ} ⊂ X_{λ}u. Conversely, let λ be chosen so that X_{λ} = ∩_{μ<λ}X_{μ} almost everywhere. This is by Lemma 15.16 true for almost every λ ∈ ℝ. Then if x ∈ X_{λ}u, we have by definition of u, x ∈ X_{μ} for every μ < λ. Thus x ∈ ⋂_{μ<λ} X_{μ}. We conclude that X_{λ}u ⊂ ⋂_{μ<λ} X_{μ} and therefore X_{λ}u ⊂ X_{λ} almost everywhere. □

Exercise 15.7. By using Corollary 15.17 show that if two measurable functions u and v are such that X_{λ}u = X_{λ}v almost everywhere for almost every λ, then u and v are almost everywhere equal.

Proposition 15.18. Let T : L → M be a standard monotone set operator and T its stack filter. If u ∈ F then for almost every level λ ∈ ℝ,

X_{λ}(T u) = T (X_{λ}(u)) almost everywhere.

Proof. Since T u is obtained from the sets T (X_{λ}u) by superposition principle, this is an immediate consequence of Corollary 15.17. □

15.5 Exercises

Exercise 15.8. It is useful to have a test for B to determine whether or not the operator T can be expected to be upper semicontinuous on L. Prove that the translationinvariant monotone operator in Theorem 15.2 defined by a given set B is upper semicontinuous on L if and only if the following condition holds: If ⋂_{n∈ℕ} T X_{n} ≠ ∅, then there is a B ∈ B such that x + B ⊂ ⋂_{n∈ℕ} X_{n}, where x ∈ ⋂_{n∈ℕ} T X_{n} and (X_{n})_{n∈ℕ} is any nonincreasing sequence in L.

“JMMBookOct04” 30/10/2009 page 212

[figure] [figure]

212 CHAPTER 15. SUP-INF OPERATORS

Exercise 15.9. Suppose that B ⊂ L contains exactly one set. Show that T is u.s.c. Generalize this to the case where B contains a finite number of sets. Exercise 15.10. Use Theorem 15.6 and Proposition 15.4 to show that the extrema killer T_{a} can be represented as a sup-inf function operator with the structuring elements

B_{a} = {B | B is compact, connected, meas (B) = a, and 0 ∈ B} ∪ {∞}.

Check that B_{a} is standard. Exercise 15.11. Let B = {{x} | x ∈ D(0, 1)}, D(0, 1) = {x | |x| ≤ 1} and consider the associated set operator T and the associated function operator T, defined on all measurable sets and functions of ℝ^{N} by formulas (15.1) and (15.3).

1) Check that T u(x) = sup_{y∈x+D} u(y).

2) Let (q_{n})_{n∈ℕ} be a countable dense set in ℝ^{N} and consider u defined by u(x) = 1−1/n if x = q_{n} and u(x) = 0 otherwise. Show that T X_{1}u ≠ X_{1}T u. The operator T in this exercise is one of the classic image operators called a dilation. Check that T commutes with thresholds when its domain of definition is restricted to F and the domain of T to L. This example shows that this restriction is useful to get a simple theory. Exercise 15.12. Show the following property used in the proof of Lemma : if h is a positive continuous integrable function on ℝ^{N} and if we set m(X) = ∫_{X} h(x)dx, then for every measurable set X, m(X) = 0 if and only if measure(X) = 0.

15.6 Comments and references

The formalism presented in this chapter is due to Matheron [169] in the case of set operators and to Serra [228] and Maragos [157] in the case of function operators. Serra’s formalism is actually more general than the one presented here; it will be developed in Chapter ??, which is about “nonflat” morphology. Our presentation relating the sup-inf form of the operator directly to contrast invariance and establishing the full equivalence between sup-inf operators and contrast-invariant monotone operators is original. The fact, proven in Proposition 15.18 that commutation with thresholds occurs almost everywhere without further assumption was proven in [100].

The mysterious “set of structuring elements” has received a great deal of attention in the literature. Here are a few references: on finding the right set of structuring elements [218, 240]; on simplifying them [225]; on decomposing them into simpler ones as one does with linear filters [200, 270, 271]; on reducing the number [209].

“JMMBookOct04” 30/10/2009 page 213

[figure] [figure]

Chapter 16

Erosions and Dilations

We are going to study in detail two of the simplest operators of mathematical morphology, the erosions and dilations. In fact, there will be essentially four operators: two set operators and the two related function operators. These operators will depend on a scale parameter t. We will also study the underlying PDEs ∂u/∂t = c|Du|, where c = 1 for dilations and c = −1 for erosions.

16.1 Set and function erosions and dilations

We saw in Chapter 15 that every contrast-invariant monotone function operator has a sup-inf and an inf-sup representation in terms of some set of structuring elements. This is the point of view we take here, and furthermore, we assume that the set of structuring elements B has the simplest possible form, namely, B = {B}. We actually introduce a parameter t scaling the size of B and therefore consider the two operators of the next definition.

Definition 16.1. For u ∈ F, define D_{tB}u = D_{t}u by

D_{t}u(x) = sup_{y∈tB} u(x − y),  (16.1)

the “dilation of u by tB. In the same way, define E_{tB}u = E_{t}u, the “erosion of u by −tB”, by

E_{t}u(x) = inf_{y∈−tB} u(x − y).  (16.2)

These function operators have associated set operators.

Definition 16.2. Let B be a non empty subset of ℝ^{N} and let t ≥ 0 be a scale parameter. The set operators D_{tB} and E_{tB} are defined on subsets X ∈ M(ℝ^{N}) by

D_{tB}X = D_{t}X = X + tB = {x | ∃b ∈ B, x − tb ∈ X},  (16.3)
E_{tB} = E_{t}X = {x | x + tB ⊂ X},  (16.4)

and extended to M(S_{N}) by the standard extension (Definition 14.1.) D_{t}X is called the dilation of X by B at scale t. E_{t}X is called the erosion of X by B at scale t.

213 “JMMBookOct04” 30/10/2009 page 214

[figure] [figure]

214 CHAPTER 16. EROSIONS AND DILATIONS

Exercise 16.1. (Duality formulas.) Show that E_{tB}u = −D_{−tB}(−u) and E_{tB}X = (D_{−tB}X^{c})^{c}. Exercise 16.2. Show that if B is bounded, dilations and erosions are standard monotone operators. Compute their associated set of structuring elements (Proposition 15.2) and check that it is standard.

Theorem 16.3. The function erosion by tB is the stack filter of the set erosion by tB ; the function dilation by tB is the stack filter of the set dilation by t\overline{B} and the commutation with thresholds holds. In other terms for u ∈ F and all λ in ℝ, and calling \overline{D}_{t} the dilation by t\overline{B},

D_{t}u(x) = sup\{λ | x ∈ D_{t}X_{λ}u\}, \overline{D}_{t}X_{λ}u = X_{λ}D_{t}u;  (16.5)
E_{t}u(x) = sup{λ | x ∈ E_{t}X_{λ}u}, E_{t}X_{λ}u = X_{λ}E_{t}u.  (16.6)

Proof. We prove the statement for the dilations, the case of the erosions being just simpler. Consider some X ∈ L and u(x) ≤ 0 a function vanishing on X only. By the definition 14.18 of the level set extension D̃_{t} of D_{t}, D̃_{t}(X) = X_{0}D_{t}(u). Thus, using (16.3),

x ∈ D̃_{t}(X) ⇔ (D_{t}u)(x) = 0 ⇔ sup_{y∈−tB} u(x − y) = 0 ⇔

∃y ∈ t\overline{B}, x − y ∈ X ⇔ x ∈ X + t\overline{B} ⇔ x ∈ \overline{D}_{t}(X).
□

The operators D_{t} and E_{t} are in a certain sense the inverse of each other. This is clearly the case, for example, if B = {x_{0}}. Then D_{t} is just the translation by tx_{0}, and E_{t} = D_{t}^{−1} is the translation by −tx_{0}. If B is the open ball centered at zero with radius one, then D_{t}X is the set of all points whose distance from X is less than t, or the t-neighborhood of X. When B is symmetric with respect to zero, the operator D_{t}E_{t} is called an opening at scale t and E_{t}D_{t} is called a closing at scale t. These names have a topological origin. If B is the open ball centered at zero with radius one, then the opening at scale t of a set X is the union of all balls with radius t contained in X. The interior of X is the union of all open balls contained in X; it is also the largest open set contained in X. If we call the interior map T^{∘}X = X^{∘} the opening, then an opening at scale t appears as a quantified opening (see Exercise 16.6). The topological statement “the closure of the complement of X is the complement of the interior of X” has its counterpart for openings and closings at scale t, as shown in Exercise 16.6. The actions of erosions and dilations are illustrated in Figures 16.2, 16.2, and 16.2; actions of openings and closings are illustrated in Figures 16.2, 16.2, 16.2, 16.3, and 16.3.

16.2 Multiscale aspects

We say that the family of dilations {D_{t} | t > 0} associated with a structuring element B is recursive if D_{t}D_{s} = D_{t+s} for all s, t > 0, and similarly for the family {E_{t} | t > 0}. (A recursive family is also called a semigroup.) Being recursive is a very desirable property for any family of scaled operators used “JMMBookOct04” 30/10/2009 page 215

[figure] [figure]

16.2. MULTISCALE ASPECTS 215

Figure 16.1: Dilation of a set. Left to right: A set; its dilation by a ball of radius 20; the difference set.

Figure 16.2: Erosion of a set. Left to right: A set; its erosion by a ball of radius 20; the difference set.

Figure 16.3: Opening of a set as curvature threshold from above. Left to right: A set X; its opening by a ball of radius 20; the difference set. This opening transforms X into the union of all balls of radius 20 contained in it. The resulting operation can be understood as a threshold from above of the curvature of the set boundary.

for image analysis. Having D_{t} = (D_{t/n})^{n} is useful for practical computations. {D_{t} | t > 0} and {E_{t} | t > 0} will be recursive if and only if B is convex, but before proving this result we need the condition for B to be convex given in the next lemma. The proof of the next statement is an easy exercise.

Lemma 16.4. B is convex if and only if (s + t)B = sB + tB for all s, t ≥ 0.

Proposition 16.5. The dilations D_{t} and the erosions E_{t} are recursive if and only the structuring element B is convex.

“JMMBookOct04” 30/10/2009 page 216

[figure] [figure]

216 CHAPTER 16. EROSIONS AND DILATIONS

Figure 16.4: Closing of a set as a curvature threshold from below. Left to right: A set X; its closing by a ball of radius 20; the difference set. The closing of X is just the opening of X^{c}. It can be viewed as a threshold from below of the curvature of the set boundary.

Proof. Taking for simplicity B closed, we have

D_{t}D_{s}X = (X + sB) + tB = X + sB + tB

and

D_{s+t}X = X + (s + t)B.

If (t + s)B = tB + sB, then clearly D_{t}D_{s}X = D_{s+t}X. Conversely, if D_{t}D_{s}X = D_{s+t}X, then by taking X = {0} we see that (t+s)B = tB+sB. One can deduce the corresponding equivalence for erosions from the duality formula (Exercise 16.1). □

16.3 The PDEs associated with erosions and dilations

As indicated in the introduction to the chapter, scaled dilations and erosions are associated with the equations ∂u/∂t = ±|Du|. To explain this connection, we begin with a bounded convex set B that contains the origin, and we define the gauge ∥ · ∥_{B} on ℝ^{N} associated with B by ∥x∥_{B} = sup_{y∈B}(x · y). If B is a ball centered at the origin with radius one, then ∥ · ∥_{B} is the usual Euclidean norm, which we write simply as | · |.

Proposition 16.6. [Hopf–Lax formula [73, 140]]. Assume that B is a bounded convex set in ℝ^{N} that contains the origin. Given u_{0} : ℝ^{N} → ℝ, define u : ℝ^{+} × ℝ^{N} → ℝ by u(t, x) = D_{t}u_{0}(x). Then u satisfies the equation

\frac{∂u}{∂t} = ∥Du∥_{−B}

at each point (t, x) where u has continuous derivatives in t and x. One has an analogue result with E_{t} and the equation ∂u/∂t = −∥Du∥_{−B}.

Proof. We begin by proving the result for D_{t} at t = 0. Thus assume that u_{0} is C^{1} at x. Then

u_{0}(x − y) − u_{0}(x) = −Du_{0}(x) · y + o(|y|),

“JMMBookOct04” 30/10/2009 page 217

[figure] [figure]

16.3. THE PDES ASSOCIATED WITH EROSIONS AND DILATIONS 217

Figure 16.5: Erosion and dilation of a natural image. First row: a sea bird image and its level lines for all levels multiple of 12. Second row: an erosion with radius 4 has been applied. On the right, the resulting level lines where the circular shape of the structuring element (a disk with radius 4) appears around each local minimum of the original image. Erosion removes local maxima (in particular, all small white spots) but expands minima. Thus, all dark spots, like the eye of the bird, are expanded. Third row: the effect of a dilation with radius 4 and the resulting level lines. We see how local minima are removed (for example, the eye of the bird) and how white spots on the tail expand. Here, in turn, circular level lines appear around all local maxima of the original image.

and we have by applying D_{h},

u(h, x) − u(0, x) = sup_{y∈hB}(−Du_{0}(x) · y + o(|y|)).

Since B is bounded, the term o(|y|) is o(|h|) uniformly for y ∈ hB, and we get

u(h, x) − u(0, x) = h sup_{z∈B}((−Du_{0}(x) · z) + o(|h|).

“JMMBookOct04” 30/10/2009 page 218

[figure] [figure]

218 CHAPTER 16. EROSIONS AND DILATIONS

Figure 16.6: Openings and closings of a natural image. First row: the original image and its level lines for all levels multiple of 12. Second row: an opening with radius 4 has been applied. Third row: a closing with radius 4 has been applied. We can recognize the circular shape of the structuring element in the level lines displayed on the right.

We can divide both sides by h and pass to the limit as |h| → 0 to obtain

\frac{∂u}{∂t}(0, x) = ∥Du_{0}(x)∥_{−B},

which is the result for t = 0. For an arbitrary t > 0, we have D_{t+h} = D_{t}D_{h} = D_{h}D_{t}, and we can write

u(t + h, x) − u(t, x) = D_{h}u(t, ·)(x) − u(t, x).

By repeating the argument made for t = 0 with u_{0} replaced with u(t, ·), we arrive at the general result. The proof for E_{t} is similar. □

Exercise 16.3. Prove the above result for E_{t}.

“JMMBookOct04” 30/10/2009 page 219

[figure] [figure]

16.4. EXERCISES 219

Figure 16.7: Denoising based on openings and closings. First row: scanned picture of the word “operator” with black dots and a black line added; a dilation with a 2 × 5 rectangle; an erosion with the same structuring element applied to the middle image. The resulting operator is a closing. Small black structures are removed by such a process. Second row: the word “operator” with a white line and white dots inside the letters; erosion with a rectangle 2 × 5; a dilation with the same structuring element applied to the middle image. The resulting operator is an opening. This time, small white structures are removed.

16.4 Exercises

Exercise 16.4. A straightforward adaptation on a grid ℤ×ℤ of the formulas u(t, x) := sup_{y∈B(x,t)} u_{0}(y) for dilation and u(t, x) := inf_{y∈B(x,t)} u_{0}(y) for erosion leads to the zero-order schemes

u^{n+1}(i, j) = sup_{(k,l)∈B((i,j),t)∩ℤ^{2}} u^{n}(k, l)

and

u^{n+1}(i, j) = inf_{(k,l)∈B((i,j),t)∩ℤ^{2}} u^{n}(k, l), u^{0}(i, j) = u_{0}(i, j).

Unfortunately, the zero-order schemes are strongly grid dependent. They do not make any difference between two balls which contain the same discrete pixels. In particular, such schemes only permit discrete motions of the shape boundaries. Thus, they are efficient only when t is large. Section 16.3 suggests that we can implement erosion and dilations on a finite image grid by more clever numerical schemes. One can try to discretize the associated PDE’s ∂u/∂t = ±|Du| by the Rouy-Tourin scheme:

u_{ij}^{n+1} =u_{ij}^{n} +Δt (max(0, u_{i+1,j}^{n} − u_{ij}^{n},u_{i−1,j}^{n} −u_{ij}^{n})^{2}+max(0, u_{i,j+1}^{n}−u_{ij}^{n},u_{i,j−1}^{n}−u_{ij}^{n})^{2})^{\frac{1}{2}}

for dilation and

u_{ij}^{n+1} =u_{ij}^{n} −Δt (max(0, u_{ij}^{n} − u_{i+1,j}^{n},u_{ij}^{n} −u_{i−1,j}^{n})^{2}+max(0, u_{ij}^{n} −u_{i,j+1}^{n},u_{ij}^{n} −u_{i,j−1}^{n})^{2})^{\frac{1}{2}}

for erosion. In both cases if t = nΔt then u^{n}(i, j) is a discrete version of u(t, (i, j)).

1) Explain why the schemes are consistent with their underlying partial differential equation. Check that with this clever scheme local maxima of u^{n} do not go up by dilation and local minima do not go down by erosion. Show that for example the following scheme would be a catastrophe at extrema (you’ll have to try it anyway):

u_{ij}^{n+1} =u_{ij}^{n}+Δt (max(|u_{i+1,j}^{n} − u_{ij}^{n}|,|u_{i−1,j}^{n} −u_{ij}^{n}|)^{2}+max(|u_{i,j+1}^{n}−u_{ij}^{n}|,|u_{i,j−1}^{n}−u_{ij}^{n}|)^{2})^{\frac{1}{2}} .

2) Implement the schemes and compare their performance with the discrete zero order schemes for several shapes and images.

3) Compute on some well-chosen images the “top hat transforms” u − O_{t}u and F_{t}u − u. The first transform aims at extracting all structures from an image which are “JMMBookOct04” 30/10/2009 page 220

[figure] [figure]

220 CHAPTER 16. EROSIONS AND DILATIONS

Figure 16.8: Examples of denoising based on opening or closing, as in Figure 16.7. Perturbations made with both black and white lines or dots have been added to the “operator” image. First column, top to bottom: original perturbed image; erosion with a 1 × 3 rectangle; then dilation with the same structuring element. (In other words, opening with this rectangle.) Then a dilation is applied with a rectangle 3 × 1, and finally an erosion with the same rectangle. Second column: The same process is applied, but with erosions and dilations exchanging their roles. It does not work so well because closing expands white perturbations and opening expands black perturbations. These operators do not commute. See Figure ??, where an application of the median filter is more successful.

thinner than t and have brightness above the average. The second transform does the same job for dark structures. These transforms can be successfully applied on aerial images for extracting roads or rivers, and in many biological applications. Exercise 16.5. Show that E_{t}(u) = −D_{t}(−u) if B is symmetric with respect to zero.

Exercise 16.6.

(i) Let B = {x | |x| < 1}. Show that D_{t}E_{t}X is the union of all open balls with radius t contained in X.

(ii) Let B be any structuring element that is symmetric with respect to zero. Write X^{c} = ℝ^{N} \ X. Show that D_{t}X^{c} = (E_{t}X)^{c}. Use this to show that E_{t}D_{t}X^{c} =

“JMMBookOct04” 30/10/2009 page 221

[figure] [figure]

16.5. COMMENTS AND REFERENCES 221

(D_{t}E_{t}X)^{c}. Exercise 16.7. Prove that the dilation and erosion set operators associated with B are standard monotone if and only if B is bounded. If B is bounded and isotropic, prove that the associated erosion and dilation function operators are local Morpho operators.

16.5 Comments and references

Erosions and dilations. Matheron introduced dilations and erosions as useful tools for set and shape analysis in his fundamental book [169]. A full account of the properties of dilations, erosions, openings, and closings, both as set operators and function operators, can be found in Serra’s books [228, 230]. We also suggest the introductory paper by Haralick, Sternberg, and Zhuang [102] and an earlier paper by Nakagawa and Rosenfeld [192]. An axiomatic algebraic approach to erosions, dilations, openings, and closings has been developed by Heijmans and Ronse [106, 212]. We did not develop this algebraic point of view here. The obvious relations between the dilations and erosions of a set and the distance function have been exploited numerically in [110], [132], and [234]. The skeleton of a shape can be defined as the set of points where the distance function to the shape is singular. A numerical procedure for computing the skeleton this way is proposed in [133].

The PDEs. The connection between the PDEs ∂u/∂t = ±|Du| and multiscale dilations and erosions comes from the work of Lax, where it is used to give stable and efficient numerical schemes for solving the equations [140]. Rouy and Tourin [213] have shown that the distance function to a shape is a viscosity solution of 1 − |Du| = 0 with the null boundary condition (Dirichlet condition) on the boundary of the shape. To define efficient numerical schemes for computing the distance function, they actually implement the evolution equation ∂u/∂t = 1 − |Du| starting from zero and with the null boundary condition on the boundary of the shape. The fact that the multiscale dilations and erosions can be computed using the PDEs ∂u/∂t = ±|Du| has been rediscovered or revived, thirty years after Lax’s work, by several authors: Alvarez et. al. [7], van den Boomgaard and Smeulders [249], Maragos [158, 159]. See also [248] for a numerical review. For an implementation using curve evolution, see [220]. Curiously, the link between erosions, dilations, and their PDEs seems to have remained unknown or unexploited until 1992. The erosion and dilation PDEs can be used for shape thinning, which is a popular way to compute the skeleton. Pasquignon developed an erosion PDE with adaptive stopping time that allows one to compute directly a skeleton that does not look like barbed wire [201].

“JMMBookOct04” 30/10/2009 page 222

[figure] [figure]

“JMMBookOct04” 30/10/2009 page 223

[figure] [figure]

Chapter 17

Median Filters and Mathematical Morphology

This entire chapter is devoted to median filters. They are among the most characteristic and numerically efficient contrast-invariant monotone operators. The denoising effects of median filters are illustrated in Figures 17.1 and 17.2; the smoothing effect of a median filter is illustrated in Figure 17.4. They also are extremely useful in 3D-image or movie denoising.

As usual, there will be two associated operators, a set operator and a function operator. All of the median operators (or filters) will be defined in terms of a nonnegative measurable weight function k : ℝ^{N} → [0, +∞) that is normalized:

∫_{ℝ^{N}} k(y) dy = 1.

The k-measure of a measurable subset B ⊂ ℝ^{N} is denoted by |B|_{k} and defined by

|B|_{k} = ∫_{B} k(y) dy = ∫_{ℝ^{N}} k(y)1_{B}(y) dy.

Clearly, 0 ≤ |B|_{k} ≤ 1. The simplest example for k is given by the function k = c_{N}^{−1}(r)1_{B(0,r)}, where B(0, r) denotes the ball of radius r centered at the origin and c_{N}(r) is the Lebesgue measure of B(0, r). Another classical example to think of is the Gaussian.

17.1 Set and function medians

We first define the set operators, whose form is simpler. We define them on M(ℝ^{N}), the set of measurable subsets of ℝ^{N} and then apply the standard extension to M(S_{N}) given in Definition 14.1.

Definition 17.1. Let X ∈ M(ℝ^{N}) and let k be a weight function. The median set of X weighted by k is defined by

Med_{k}X = \{x | |X − x|_{k} ≥ \frac{1}{2}\}  (17.1)

223 “JMMBookOct04” 30/10/2009 page 224

[figure] [figure]

224CHAPTER 17. MEDIAN FILTERS AND MATHEMATICAL MORPHOLOGY

and its standard extension to M(S_{N}) by

Med_{k}X = \{x | |X − x|_{k} ≥ \frac{1}{2}\} ∪ (X ∩ \{∞\}).  (17.2)

The extension amounts to add ∞ to Med_{k}X if ∞ belongs to X. Note that we have already encountered the median operator in Section 8.1. Koenderink and van Doorn defined the dynamic shape of X at scale t to be the set of x such that G_{t}∗1_{X}(x) ≥ 1/2. The dynamic shape is, in our terms, a Gaussian-weighted median filter.

To gain some intuition about median filters, we suggest considering the weight k defined on ℝ^{2} by k = (1/πr^{2})1_{B(0,r)}. Then x ∈ ℝ^{2} belongs to Med_{k}X if and only if the Lebesgue measure of X ∩B(x, r) is greater than or equal to half the measure of B(0, r). Thus, x ∈ Med_{k}X if points of X are in the majority around x.

Lemma 17.2. Med_{k} is a standard monotone operator on M.

Proof. Obviously Med_{k}(∅) = ∅ and Med_{k}(S_{N}) = S_{N}. By definition, ∞ ∈ Med_{k}X ⇔ ∞ ∈ X. If X is bounded, it is a direct application of Lebesgue theorem that

|X − x|_{k} = ∫ k(y)1_{X−x}(y)dy → 0 as x → ∞.

Thus |X − x|_{k} < \frac{1}{2} for x large enough and Med_{k}X is therefore bounded. In the same way, if X^{c} is bounded |X − x|_{k} → 1 as x → ∞ and therefore (Med_{X})^{c} is bounded. □

Lemma 17.3. We can represent Med_{k} by

Med_{k}X = {x | x + B ⊂ X, for some B ∈ B},  (17.3)

where B = \{B | |B|_{k} ≥ \frac{1}{2}\} or B = \{B | |B|_{k} = \frac{1}{2}\}.

Proof. By Lemma 17.2, Med_{k} is standard monotone and it is obviously translation invariant. So we can apply Theorem 15.2. The canonical set of structuring elements of Med_{k} is

B = \{B | 0 ∈ Med_{k}B\} = \{B | |B|_{k} ≥ \frac{1}{2}\}.

The second set B mentioned in the lemma, which we call now for convenience B′, is a subset of B such that for every B ∈ B, there is some B′ ∈ B′ such that B′ ⊂ B. Thus by Proposition 15.4, Med_{k} can be defined from B′. □

The next lemma will help defining the function operator Med_{k} associated with the set operator Med_{k}.

Lemma 17.4. The set operator Med_{k} is translation invariant and upper semicontinuous on M.

“JMMBookOct04” 30/10/2009 page 225

[figure] [figure]

17.2. SELF-DUAL MEDIAN FILTERS 225

Proof. The first property is straightforward. Consider a nonincreasing sequence (X_{n})_{n∈ℕ} in M and let us show that

Med_{k} ⋂_{n∈ℕ} X_{n} = ⋂_{n∈ℕ} Med_{k}X_{n}.

Since Med_{k} is monotone, it is always true that Med_{k} ⋂_{n∈ℕ} X_{n} ⊂ ⋂_{n∈ℕ} Med_{k}X_{n}. To prove the other inclusion, assume that x ∈ ⋂_{n∈ℕ} Med_{k}X_{n}. If x ∈ ℝ^{N}, by the definition of Med_{k}, |X_{n} −x| ≥ 1/2 for all n ∈ ℕ. Since X_{n} −x ↓ ⋂_{n∈ℕ}(X_{n} −x), we deduce from Lebesgue Theorem that |X_{n} − x|_{k} ↓ | ⋂_{n∈ℕ}(X_{n} − x)|_{k}. This means that | ⋂_{n∈ℕ}(X_{n} − x)|_{k} ≥ 1/2, and hence that x ∈ Med_{k} ⋂_{n∈ℕ}(X_{n} − x). If x = ∞, it belongs to Med_{k}X_{n} for all n and therefore to X_{n} for all n. Thus, it belongs to ⋂_{n∈ℕ} X_{n} and therefore to Med_{k}(⋂_{n∈ℕ} X_{n}). □

Definition 17.5 (and proposition). Define the function operator Med_{k} from Med_{k} as a stack filter,

Med_{k}u(x) = sup{λ | x ∈ Med_{k}X_{λ}u}.

Then Med_{k} is standard monotone, contrast invariant and translation invariant from F to F. Med_{k} and Med_{k} commute with thresholds,

X_{λ}Med_{k}u = Med_{k}X_{λ}u.  (17.4)

If k is radial, Med_{k} therefore is Morpho.

Proof. By Lemma 17.4, Med_{k} is upper semicontinuous and by Lemma 17.2 it is standard monotone and translation invariant. So we can apply Theorem 14.16, which yields all announced properties for Med_{k}. □

We get a sup-inf formula for the median as a direct application of Theorem 15.6.

Proposition 17.6. The median operator Med_{k} has the sup-inf representation

Med_{k}u(x) = sup_{B∈B} inf_{y∈x+B} u(y),  (17.5)

where B = {B | B ∈ M, |B|_{k} = 1/2}.

A median value is a kind of average, but with quite different results, as is illustrated in Exercise 17.4.

17.2 Self-dual median filters

The median operator Med_{k}, as defined, is not invariant under “reverse contrast,” that is, it does not satisfy −Med_{k}u = Med_{k}(−u) for all u ∈ F. This is clear from the example in the next exercise. Self-duality is a conservative requirement which is true for all linear filters. It means that the white and black balance is respected by the operator. We have seen that dilations favor whites and erosions favor black colors: These operators are not self-dual.

“JMMBookOct04” 30/10/2009 page 226

[figure] [figure]

226CHAPTER 17. MEDIAN FILTERS AND MATHEMATICAL MORPHOLOGY

Figure 17.1: Example of denoising with a median filter. Left to right: scanned picture of the word “operator” with perturbations and noise made with black or white lines and dots; the image after one application of a median filter with a circular neighborhood of radius 2; the image after a second application of the same filter. Compare with the denoising using openings and closings (Figure 16.8).

Exercise 17.1. Consider the one-dimensional median filter with k = \frac{1}{2} 1_{[−2,−1]∪[1,2]}. Let u(x) = −1 if x ≤ −1, u(x) = 1 if x ≥ 1, u(x) = x elsewhere. Check that Med_{k}u(0) ≠ −Med(−u)(0).

As we did with erosions and dilations, one can define a dual version of the median Med_{k}^{−} by

Med_{k}^{−}u = −Med_{k}(−u), so that  (17.6)
Med_{k}^{−}u(x) = inf_{|B|_{k}≥ \frac{1}{2}} sup_{y∈x+B} u(y).  (17.7)

A quite general condition on k is sufficient to guarantee that Med_{k} and Med_{k}^{−} agree on continuous functions.

Definition 17.7. We say that k is not separable if |B|_{k} ≥ 1/2 and |B′|_{k} ≥ 1/2 imply that \overline{B} ∩ \overline{B′} ≠ ∅.

Proposition 17.8.

(i) For every measurable function u, Med_{k}u ≥ Med_{k}^{−}u.

(ii) Assume that k is not separable. Then for every u ∈ F, Med_{k}u = Med_{k}^{−}u and Med_{k} is self-dual.

Proof. Both operators are translation invariant, so without loss of generality we may assume that x = 0. To prove (i), let λ = Med_{k}u(0) = sup_{|B|_{k}≥1/2} inf_{y∈B} u(y). Take ε > 0 and consider the level set X_{λ+ε}u. Then inf_{y∈X_{λ+ε}} u(y) ≥ λ+ε. Thus |X_{λ+ε}u|_{k} < 1/2, since inf_{y∈B} ≤ λ for any set B such that |B| ≥ 1/2. Thence |(X_{λ+ε}u)^{c}|_{k} ≥ 1/2. By the definition of level sets, sup_{y∈(X_{λ+ε}u)^{c}} u(y) ≤ λ + ε. These two last relations imply that

inf_{|B|_{k}≥ \frac{1}{2}} sup_{y∈B} u(y) ≤ λ + ε.

Since ε > 0 was arbitrary, this proves (i).

The assumption that k is not separable implies that for all B and B′ having k-measure greater than or equal to 1/2, we have inf_{y∈\overline{B}} u(y) ≤ sup_{y∈\overline{B′}} u(y). Since u ∈ F is continuous, inf_{y∈B} u(y) ≤ sup_{y∈B′} u(y). Since B and B′ were arbitrary except for the conditions |B|_{k} ≥ 1/2 and |B′|_{k} ≥ 1/2, the last inequality implies that

sup_{|B|_{k}≥ \frac{1}{2}} inf_{y∈B} u(y) ≤ inf_{|B′|_{k}≥ \frac{1}{2}} sup_{y∈B′} u(y).

“JMMBookOct04” 30/10/2009 page 227

[figure] [figure]

17.3. DISCRETE MEDIAN FILTERS AND THE ”USUAL” MEDIAN VALUE227

Figure 17.2: Denoising based on a median filter. Left: an image altered on 40% of its pixels with salt and pepper noise. Right: the same image after three iterations of a median filter with a 3 × 3 square mask.

From this last inequality and (i), we conclude that Med_{k}u = Med_{k}^{−}u. □

17.2.1 Chessboard dilemma and fattening effect

In Figure 17.2.1, the median filter has been applied iteratively to a function u whose grid values are equal to 255 at the white pixels and to 0 at the black pixels. The function is continuous, being (e.g.) interpolated by standard bilinear interpolation. The iso-level set I_{127.5}u := {x | u(x) = 127.5} consists of the line segments separating the squares and has therefore zero measure. As we know, the median filter tends to smooth, to round off the level lines of the image. Yet we have with a chessboard a fundamental ambiguity : are these iso-level lines surrounding the black squares, or are they surrounding the white squares? In other terms, do we see in a chessboard a set of white squares on black background, or conversely?

Since our operator is self-dual it doesn’t favor any of the considered interpretations: it rounds off simultaneously the lines surrounding the black squares and the level lines surrounding the white squares (second image of Figure 17.2.1). This results in the “fattening” of the level lines separating white and black, which have the mid-level 127.5. Hence the appearance in the second image of a grey zone separating the smoothed out black and white squares. If we take a level set X_{ε}T u of this image with ε < 0 (third image), the fattened set joins the level set and we observe black squares on white background. Symmetrically if ε > 0 the level set shows white squares on black background.

17.3 Discrete median filters and the ”usual” median value

We define a discrete median filter by considering, instead of a function, a uniform discrete measure k = ∑_{i=1,...,N} δ_{x_{i}}, where δ_{x_{i}} denotes the Dirac mass at x_{i}.

“JMMBookOct04” 30/10/2009 page 228

[figure] [figure]

228CHAPTER 17. MEDIAN FILTERS AND MATHEMATICAL MORPHOLOGY

Figure 17.3: The chessboard dilemma. Left: a chessboard image. The next image is obtained by a self-dual median filter. Notice the expansion of the median grey level, 127.5, who was invisible in the original image and grows in the second one. This effect is called “fattening effect”. The third and fourth image show the evolution of the level sets at levels 127 and 128 respectively. This experiment illustrates a dilemma as to whether we consider the chessboard as black squares on white background, or conversely.

We could normalize k, but this is not necessary, as will become clear. Translates of the points x_{i} create the discrete neighborhood that is used to compute the median value of a function u at a point x. We denote the set of subsets of {1, …, N} by P(N) and the number of elements in P ∈ P(N) by card(P). Since card(P) = |P |_{k}, we will suppress the k-notation is favor of the more transparent “card(P),” but one should remember that the k-measure is still there. An immediate generalization of the definition of the median filters to the case where k is such a discrete measure yields

Medu(x) = sup_{{P ∈P(N) \atop card(P)≥N/2}} inf_{i∈P} u(x − x_{i}),

Med^{−}u(x) = inf_{{P ∈P(N) \atop card(P)≥N/2}} sup_{i∈P} u(x − x_{i}).

When k was continuous, we could replace “|B|_{k} ≥ 1/2” with “|B|_{k} = 1/2,” but this is not directly possible in the discrete case, since N/2 is not an integer if N is odd. To fix this, we define the function M by M(N) = N/2 if N is even and M(N) = (N/2) + (1/2) if N is odd. Now we have

Medu(x) = sup_{{P ∈P(N) \atop card(P)=M(N)}} inf_{i∈P} u(x − x_{i}),

Med^{−}u(x) = inf_{{P ∈P(N) \atop card(P)=M(N)}} sup_{i∈P} u(x − x_{i}).

The fact that we can replace “card(P) ≥ N/2” with “card(P) = M(N)” has been argued elsewhere for the continuous case; for the discrete case, it is a matter of simple combinatorics. Given any x, let y_{i} = u(x−x_{i}). After a suitable permutation of the i’s, we can order the y_{i} as follows: y_{1} ≤, ⋯, ≤ y_{M} ≤, ⋯ ≤

“JMMBookOct04” 30/10/2009 page 229

[figure] [figure]

17.4. EXERCISES 229

y_{N}. Then for N even,

{inf_{i∈P} y_{i} | card(P) ≥ N/2} = {inf_{i∈P} y_{i} | card(P) = M} = {y_{1}, …, y_{M+1}},
{sup_{i∈P} y_{i} | card(P) ≥ N/2} = {sup_{i∈P} y_{i} | card(P) = M} = {y_{M}, …, y_{N}},

and Medu(x) = y_{M+1} ≥ y_{M} = Med^{−}u(x). If N is odd, we have

{inf_{i∈P} y_{i} | card(P) ≥ N/2} = {inf_{i∈P} y_{i} | card(P) = M} = {y_{1}, …, y_{M}},
{sup_{i∈P} y_{i} | card(P) ≥ N/2} = {sup_{i∈P} y_{i} | card(P) = M} = {y_{M}, …, y_{N}},

and Medu(x) = Med^{−}u(x) = y_{M}. This shows that Med = Med^{−} if and only if N is odd. What we see here is the discrete version of Proposition 17.8. When N is odd, the measure is not separable, since two sets P and P′ with card(P) ≥ N/2 and card(P′) ≥ N/2 always have a nonempty intersection. In general, a median filter with an odd number of pixels is preferred, since Med = Med^{−} in this case.

This discussion shows that the definition of the discrete median filter Med corresponds to the usual statistical definition of the median of a set of data: If the given data consists of the numbers y_{1} ≤ y_{2} ≤ ⋯ ≤ y_{N} and N = 2n+1, them by definition, the median is y_{n+1}. In case N = 2n, the median is (y_{n} + y_{n+1})/2. In both cases, half of the terms are greater than or equal to the median and half of the terms are less than or equal to the median. The usual median minimizes the functional ∑_{i=1}^{N} |y_{i} − y|. Exercise 17.9 shows how Med and Med^{−} relate to this functional.

Finally, we wish to show that the discrete median filter Med can be a cyclic operator on discrete images. As a simple example, consider the chessboard image, where u(i, j) = 255 if i + j is even and u(i, j) = 0 otherwise. When we apply the median filter that takes the median of the four values surrounding a pixel and the pixel value, it is clear that the filter “reverses” the chessboard pattern. Indeed, any white pixel (value 255) is surrounded by four black pixels (value zero), so the median filter transforms the white pixel into a black pixel. In the same way, a black pixel is transformed into a white pixel and this can go for ever.

17.4 Exercises

Exercise 17.2. Check that Med_{k} as defined in Definition 17.1 is monotone and translation invariant. Exercise 17.3. Koenderink and van Doorn defined the dynamic shape of X at scale t to be the set of x such that G_{t} ∗ 1_{X}(x) ≥ 1/2. Check that this is a Gaussian-weighted median filter. Exercise 17.4. Consider the weighted median filter defined on S_{1} with k = (1/2)1_{[−1,1]}. Compute Med_{k}u for u(x) = \frac{1}{1+x^{2}} . Compare the result with the local average M_{1}u(x) = \frac{1}{2} ∫_{−1}^{1} u(x + y)dy. What happens on intervals where u is monotone? Exercise 17.5. Saying that k is not separable is a fairly weak assumption. It corresponds roughly to saying that the support of k cannot be split into two disjoint connected components each having k-measure 1/2. Show that if k is continuous and if its support is connected, then it is not separable.

“JMMBookOct04” 30/10/2009 page 230

[figure] [figure]

230CHAPTER 17. MEDIAN FILTERS AND MATHEMATICAL MORPHOLOGY

Exercise 17.6. Prove the following inequalities for any measurable function :

sup_{|B|_{k}≥ \frac{1}{2}} inf_{y∈x+B} u(y) ≥ sup_{|B|_{k}> \frac{1}{2}} inf_{y∈x+B} u(y) ≥ inf_{|B|_{k}≥ \frac{1}{2}} sup_{y∈x+B} u(y),

sup_{|B|_{k}≥ \frac{1}{2}} inf_{y∈x+B} u(y) ≥ inf_{|B|_{k}> \frac{1}{2}} sup_{y∈x+B} u(y) ≥ inf_{|B|_{k}≥ \frac{1}{2}} sup_{y∈x+B} u(y).

Exercise 17.7. Median filter on measurable sets and functions. The aim of the exercise is to study the properties of the median filter extended to the set M of all measurable sets of S_{N} and all bounded measurable functions (u ∈ L^{∞}(S_{N})). The definition of Med_{k} on M is identical to the current definition.

1) Using the result of Exercise 14.20, show that one can define Med_{k} from Med_{k} as a stack filter and that it is monotone, translation and contrast invariant. In addition, Med_{k} and Med_{k} still satisfy the commutation with thresholds, X_{λ}Med_{k}u = Med_{k}X_{λ}u.

2) Prove that Med_{k} maps measurable sets into closed sets. Deduce that if u is a measurable function, then Med_{k}u is upper semicontinuous and Med_{k}^{−}u is lower semicontinuous.

3) Assume that k is not separable. Check that the proof of Proposition 17.8 still applies to the more general Med_{k} and Med_{k}^{−}, applied to all measurable functions. Deduce that if k is not separable, then Med_{k}u is continuous whenever u is a measurable function.

Exercise 17.8. The discrete median filters can more generally be defined in terms of a nonuniform measure k that places different weights k_{i} on the points x_{i}, so |{x_{i}}|_{k} = k_{i}. Check that Med_{k}^{−}u ≤ Med_{k}u. Prove that Med_{k}^{−}u = Med_{k}u if and only if there is no subset of the numbers k_{1}, …, k_{N} whose sum is K/2. In particular, if the k_{i} are integers and K is odd, then Med_{k}^{−}u = Med_{k}u. Exercise 17.9. Variational interpretations of the median and the average values. Let arginf_{m}g(m) denote the value of m, if it exists, at which g attains its infimum. Consider N real numbers {x_{i} | i = 1, 2, …, N} and denote by Med((x_{i})_{i}) and Med^{−}((x_{i})_{i}) their usual lower and upper median values (we already know that both are equal if N is odd but can be different if N is even).

(i) Show that

\frac{1}{N} ∑_{i=1}^{N} x_{i} = arginf_{m} ∑_{i=1}^{N}(x_{i} − m)^{2}.

(ii) Show that

Med^{−}((x_{i})_{i}) ≤ arginf_{m} ∑_{i=1}^{N} |x_{i} − m| ≤ Med((x_{i})_{i}).

(iii) Let k = 1_{B}, where B is set with Lebesgue measure equal to one. Let Med_{B}u denote the median value of u in B, defined by Med_{B}u = Med_{k}u(0). Consider a bounded measurable function u defined on B. Show that

∫_{B} u(x) dx = arginf_{m} ∫_{B}(u(x) − m)^{2} dx

and that

Med_{B}^{−}u ≤ arginf_{m} ∫_{B} |u(x) − m| dx = \frac{Med_{B}^{−}u + Med_{B}u}{2} ≤ Med_{B}u.

(iv) Deduce from the above that the mean value is the best constant approximation in the L^{2} norm and that the median is the best constant approximation in the L^{1} norm.

“JMMBookOct04” 30/10/2009 page 231

[figure] [figure]

17.5. COMMENTS AND REFERENCES 231

Figure 17.4: Smoothing effect of a median filter on level lines. Above, left to right: original image; all of its level lines (boundaries of level sets) with levels multiple of 12; level lines at level 100. Below,left to right: result of two iterations of a median filter with a disk with radius 2; corresponding level lines (levels multiple of 12); level lines at level 100.

17.5 Comments and references

The remarkable denoising properties and numerical efficiency of median filters for the removal of all kinds of impulse noise in digital images, movies, and video signals are well known and acclaimed [66, 117, 193, 202, 207]. The last reference cited as well as the next three all propose simple and efficient implementations of the median filter [19, 64, 111]. An introduction to weighted median filters can be found in [35, 266], and information about some generalizations (conditional median filters, for example) can be found in [16, 142, 239]. The min, max, and median filters are particular instances of rank order filters; see [60] for a general presentation of these filters. There are few studies on iterated median filters. The use of iterated median filters as a scale space is, however, proposed in [23]. The extension of median filtering to multichannel (color) images is problematic, although there have been some interesting attempts [50, 208].

“JMMBookOct04” 30/10/2009 page 232

[figure] [figure]

232CHAPTER 17. MEDIAN FILTERS AND MATHEMATICAL MORPHOLOGY “JMMBookOct04” 30/10/2009 page 233

[figure] [figure]

Part III

Local Asymptotic Analysis of Operators

233 “JMMBookOct04” 30/10/2009 page 234

[figure] [figure]

“JMMBookOct04” 30/10/2009 page 235

[figure] [figure]

Chapter 18

Curves and Curvatures

This chapter contains the fundamentals of differential geometry that are used in the book. Our main aim is to define the orientation and curvatures of a curve or a surface as the main contrast invariant differential operators we shall deal with in image and curve smoothing.

18.1 Tangent, normal, and curvature

We summarize in this section the concepts and results about smooth curves that are needed in this chapter and elsewhere in the book. The curves we considered will always be plane curves.

Definition 18.1. We call simple arc or Jordan arc the image Γ of a continuous one-to-one function x : [0, 1] → ℝ^{2}, x(t) = (x(t), y(t)). We say that Γ is a simple closed curve or Jordan curve if the mapping restricted to (0, 1) is oneto-one and if x(0) = x(1). If x is continuously differentiable on [0, 1], we define the arc length of the segment of the curve between x(t_{0}) and x(t) by

L(x, t_{0}, t) = ∫_{t_{0}}^{t} |x′(τ)| dτ = ∫_{t_{0}}^{t} \sqrt{x′(τ) · x′(τ)} dτ.  (18.1)

In particular, set

L(t) = L(x, 0, t) = ∫_{0}^{t} |x′(τ)| dτ = ∫_{0}^{t} \sqrt{x′(τ) · x′(τ)} dτ.

The curves we deal with will always be smooth. Now, we want the definition of “smoothness” to describe an intrinsic property of Γ rather than a property of some parameterization x(s) of Γ. If a function x representing Γ is C^{1}, then the function L in equation (18.1) has a derivative with respect to s,

L′(t) = |x′(t)|

that is continuous. Nevertheless, the curve itself may not conform to our idea of being smooth, which at a minimum requires a tangent at every point y ∈ Γ. For example, the motion of a point on the boundary of a unit disk as it rolls along the x-axis is described by x(t) = (t − sin t, 1 − cos t), which is a C^{∞} function.

235 “JMMBookOct04” 30/10/2009 page 236

[figure] [figure]

236 CHAPTER 18. CURVES AND CURVATURES

Nevertheless, the curve has cusps at all multiples of 2π. The problem is that x′(2kπ) = 0.

Definition 18.2. We say that a curve Γ admits an arc-length parameterization s ∈ ℝ ↦ x(s) if the function x is C^{1} and L′(s) = |x′(s)| = 1 for all s. In case Γ is closed, we identify [0, l(Γ)] algebraically with the circle group by adding elements of [0, l(Γ)] modulo l(Γ). We say that Γ is C^{m}, m ∈ ℕ, m ≥ 1, if the arc-length parameterization x is a C^{m} function.

Exercise 18.1. The aim of the exercise is to give a formula transforming a C^{1} parameterization t ∈ [0, 1] → x(t) such that |x′(t)| ≠ 0 for all t into an arc-length parameterization. Notice that L : [0, 1] → [0, L(1)] is increasing. Set, for s ∈ [0, L(1)], x̃(s) = x(L^{−1}(s)) and check that x̃ is an arc-length parameterization of the curve defined by x.

An arc-length parameterization is also called a Euclidean parameterization. If a Jordan curve has an arc-length parameterization x, then the domain of definition of x on the real line must be an interval [a, b], where b − a is the length of Γ, which we denote by l(Γ). In this case, we will always take [0, l(Γ)] as the domain of definition of x.

Definition 18.3. Assume that Γ is C^{2} and let s ↦ x(s) be an arc-length parameterization. The tangent vector τ is defined as τ (s) = x′(s). The curvature vector of the curve Γ is defined by κ(s) = x′′(s). The normal vector n(s) is defined by n(s) = τ^{⊥}, where (x, y)^{⊥} = (−y, x).

One can easily describe all Euclidean parameterizations of a Jordan curve.

Proposition 18.4. Suppose that Γ is a C^{1} Jordan curve with arc-length parameterization x : [0, l(Γ)] → Γ. Then any other arc-length parameterization y : [0, l(Γ)] → Γ is of the form y(s) = x(s + σ) or y(s) = x(−s + σ) for some σ ∈ [0, l(Γ)].

Proof. Denote by C the interval [0, l(Γ)], defined as an additive subgroup of ℝ modulo l(Γ). Let x, y : C ↦ Γ be two length preserving parameterizations of Γ. Then f = x ∘ y^{−1} is a length preserving bijection of C. Using the parameterization of C, this implies f(s) = ±s + σ for some σ ∈ [0, l(Γ)] and the proof is easily concluded. (See exercise 18.7 for some more details.) □

Proposition 18.5. Let Γ be a C^{2} Jordan curve, and let x and y by any two arc-length parameterizations of Γ.

(i) If x(s) = y(t), then x′(s) = ±y′(t).

(ii) The vector κ is independent of the choice of arc-length parameterizations and it is orthogonal to τ = x′.

Proof. By Proposition 18.4, y(s) = x(±s+σ) and (i) follows by differentiation. This is also geometrically obvious: x′(s) and y′(t) are unit vectors tangent to Γ at the same point. Thus, they either point in the same direction or they point in opposite directions.

“JMMBookOct04” 30/10/2009 page 237

[figure] [figure]

18.1. TANGENT, NORMAL, AND CURVATURE 237

Using any of the above representations and differentiating twice shows that x′′ = y′′. Since x′ · x′ = 1, differentiating this expression shows that x′′ · x′ = 0. Thus, x′′ and x′ are orthogonal and x′′ and x^{′⊥} are collinear. □

It will be convenient to have a flexible notation for the curvature in the different contexts we will use it. This is the object of the next definition.

Definition 18.6 (and notation). Given a C^{2} curve Γ, which is parameterized by length as s ↦ x(s) and x = x(s) a point of Γ, we denote in three equivalent ways the curvature of Γ at x = x(s),

κ(x) = κ(x(s)) = κ(s) = x′′(s).

In the first notation, κ is the curvature of the curve Γ at a point x implicitly supposed to belong Γ. In the second notation a particular parameterization of Γ, x(s), is being used. In the third one, x is omitted.

The above notations create no ambiguity or contradiction, since by Proposition 18.5 the curvature is independent of the Euclidean parameterization. Of course, a smooth Jordan curve is locally a graph. More specifically:

Proposition 18.7. A C^{1} Jordan arc Γ can be represented around each one of its points x_{0} as the graph of a C^{1} scalar function y = f(x) such that x_{0} = (0, f(0)) = (0, 0), f′(0) = 0, and

κ(x_{0}) = (0, f′′(0)).  (18.2)

Conversely, the graph of any C^{1} function f is a C^{1} Jordan arc. If f is C^{2} the curvature of its associated Jordan curve satisfies (18.2) at each point where f′(0) = 0.

Proof. Assume we are given a C^{1} Jordan arc Γ and an arc-length parameterization c in a neighborhood of x_{0} = c(s_{0}) ∈ Γ. We assume, without loss of generality, that s_{0} = 0. Then we can establish a local coordinate system with origin x_{0} and based on the two unit vectors c′(0) and c′(0)^{⊥} where the x-axis is positive in the direction of c′(0). If we write c(s) = (x(s), y(s)) in this coordinate system, then

x(s) = c(s) · c′(0),
y(s) = c(s) · c′(0)^{⊥}.

Since dx/ds(s) = c′(s) · c′(0), dx/ds(0) = 1. Then the inverse function theorem implies the existence of a C^{1} function g and a δ > 0 such that s = g(x) for |x| < δ. This means that, for |x| < δ, Γ is represented locally by the graph of the C^{1} function f(x) = y(g(x)) = c(g(x)) · c′(0)^{⊥}. To be slightly more precise, denote the graph of f for |x| < δ by Γ_{f}. Since g is one-to-one, Γ_{f} is a homeomorphic image of the open interval (−δ, δ) and Γ_{f} ⊂ Γ. If Γ is C^{2}, then f is C^{2} and f′′(0) = c′′(0) · c′(0)^{⊥}. Thus, on the local coordinate system, the coordinates of c′′(0) = κ(x_{0}) are (0, f′′(0)).

Conversely, given a C^{1} function f, we can consider the graph Γ_{f} of f in a neighborhood of the origin. Then Γ_{f} is represented by c, where c(x) = (x, f(x)).

“JMMBookOct04” 30/10/2009 page 238

[figure] [figure]

238 CHAPTER 18. CURVES AND CURVATURES

We may assume that f(0) = 0 and f′(0) = 0 (by a translation and rotation if necessary). The arc-length along Γ is measured by

s(x) = ∫_{0}^{x} \sqrt{1 + [f′(t)]^{2}} dt,

and s′(x) = \sqrt{1 + [f′(x)]^{2}}, so that s′(0) = 1. This time there is a C^{1} function h such that h(s) = x and h′(s) = (1 + [f′(h(s))]^{2})^{−1/2}. Then Γ is represented by c̃(s) = (h(s), f(h(s))). Short computations show that |c̃′(s)| = 1. If in addition f is C^{2}, then Γ is C^{2} and it is an easy check that c̃′′(0)·c̃′(0)^{⊥} = f′′(0). □

Exercise 18.2. Make the above “short computations” and the “easy check”.

18.2 The structure of the set of level lines

We saw in Chapter 12 how an image can be represented by its level sets. The next step, with a view toward shape analysis, is the representation of an image in terms of its level lines. We rely heavily on the implicit function theorem to develop this representation. We begin with a two-dimensional version. The statement here is just a slight variation on the implicit function theorem quoted in section 1.

Theorem 18.8. Let u ∈ F be a C^{1} function such that Du(x_{0}) ≠ 0 at some x_{0} = (x_{0}, y_{0}). Let i denote the unit vector in the direction (u_{x}, u_{y}), let j denote the unit vector in the orthogonal direction (−u_{y}, u_{x}), and write x = x_{0} +xi+yj. Then there is a disk D(x_{0}, r) and a unique C^{1} function φ, φ : [−r, r] → ℝ, such that if x ∈ D(x_{0}, r), then

u(x, y) = u(x_{0}) ⟺ x = φ(y).

The following corollary is a global version of this local result.

Corollary 18.9. Assume that u ∈ F is C^{1} and let u^{−1}(λ) = {x | u(x) = λ} for λ ∈ ℝ. If λ ≠ u(∞) and Du(x) ≠ 0 for all x ∈ u^{−1}(λ), then u^{−1}(λ) is a finite union of disjoint Jordan curves.

Proof. From Theorem 18.8 we know that for each point x ∈ u^{−1}(λ) there is an open disk D(x, r(x)) such that \overline{D}(x, r(x)) ∩ u^{−1}(λ) is a C^{1} Jordan arc x(s) and we can take the endpoints of the arc on ∂D(x, r(x)). Since λ ≠ u(∞), u^{−1}(λ) is compact. Thus there is a finite number of points x_{i}, i = 1, …, m, such that u^{−1}(λ) ⊂ ⋃_{i=1}^{m} D(x_{i}, r(x_{i})). This implies that u^{−1}(λ) is a finite union of Jordan arcs which we can parameterize by length. The rest of the proof is very intuitive and is left to the reader. I consists of iteratively gluing the Jordan arcs until they close up into one or several Jordan curves. □

The next theorem is one of the few results that we are going to quote rather than prove, as we have done with the implicit function theorem.

Theorem 18.10 (Sard’s theorem). Let u ∈ F ∩ C^{1}. Then for almost every λ in the range of u, the set u^{−1}(λ) is nonsingular, which means that for all x ∈ u^{−1}(λ), Du(x) ≠ 0.

“JMMBookOct04” 30/10/2009 page 239

[figure] [figure]

18.2. THE STRUCTURE OF THE SET OF LEVEL LINES 239

Figure 18.1: Level lines as representatives of the shapes present in an image. Left: noisy binary image with two apparent shapes; right: the two longest level lines.

As a direct consequence of Sard’s Theorem and Corollary 18.9, we obtain:

Corollary 18.11. Let u ∈ F ∩ C^{1}. Then for almost every λ in the range of u, the set u^{−1}(λ) is the union of a finite set of disjoint simple closed C^{1} curves.

The sole purpose of the next proposition is to convince the reader that the level lines of a function provide a faithful representation of the function.

Proposition 18.12. Let u ∈ F ∩ C^{1}. Then u can be reconstructed from the following data: the family of all of its level lines at nonsingular levels, the level of each level line being also kept.

Proof. Let G be the closure of the union of the ranges of all level lines of u at nonsingular levels. If x ∈ G, then there are points x_{n} belonging to level lines of some levels λ_{n} such that x_{n} → x. As a consequence, λ_{n} = u(x_{n}) → u(x). So we get back the value of u(x). Let now x belong to the open set G^{c}. Let us first prove that Du(x) = 0. Assume by contradiction that Du(x) ≠ 0. By using the first order Taylor expansion of u around x, one sees that for all r > 0 the connected range u(B(x, r)) must contain some interval (u(x) − α(r), u(x) + α(r)) with α(r) → 0 as r → 0. By Sard’s theorem some of the values in this interval are nonsingular. Thus we can find nonsingular levels λ_{n} → u(x) and points x_{n} → x such that u(x_{n}) = λ_{n}. This implies that x ∈ G and yields a contradiction. Thus Du(x) = 0 on G^{c} and u is therefore constant on each connected component A of G^{c}. The value of u is then uniquely determined by the value of u on the boundary of A. This value is known, since ∂A is contained in G. □

“JMMBookOct04” 30/10/2009 page 240

[figure] [figure]

240 CHAPTER 18. CURVES AND CURVATURES

Figure 18.2: Level lines as a complete representation of the shapes present in an image. All level lines of the image of a sea bird for levels that are multiples of 12 are displayed. Notice that we do not need a previous smoothing to visualize the shape structures in an image: It is sufficient to quantize the displayed levels.

18.3 Curvature of the level lines

The intrinsic local coordinates

We continue to work in ℝ^{2}. Consider a real-valued function u that is twice continuously differentiable in a neighborhood of x_{0} ∈ ℝ^{2}. To simplify the notation, we will often write Du rather than Du(x_{0}), and so on.

Definition 18.13. If Du = (u_{x}, u_{y}) ≠ 0, then we establish a local coordinate system by letting i = Du/|Du| and j = Du^{⊥}/|Du|, where Du^{⊥} = (−u_{y}, u_{x}). Thus, for a point x near x_{0}, we write x = x_{0} + xi + yj and the local coordinates of x are (x, y). (See Figure 18.3.) Without risk of ambiguity we shall write u(x, y) for u(x) = u(x_{0} + xi + yj).

Since u is C^{2}, we can use Taylor’s formula to express u in this coordinate system in a neighborhood of x_{0}.

u(x) = u(x, y) = u(x_{0}) + px + ax^{2} + by^{2} + cxy + O(|x|^{3}),  (18.3)

where p = u_{x}(0, 0) = |Du(x_{0})| > 0 and

\begin{aligned}a & = \frac{1}{2} \frac{∂^{2}u}{∂x^{2}}(0, 0) = & \frac{1}{2}D^{2}u(\frac{Du}{|Du|}, \frac{Du}{|Du|})(x_{0}), \\ b & = \frac{1}{2} \frac{∂^{2}u}{∂y^{2}}(0, 0) = & \frac{1}{2}D^{2}u(\frac{Du^{⊥}}{|Du|} , \frac{Du^{⊥}}{|Du|} )(x_{0}), \\ c & = \frac{∂^{2}u}{∂x∂y} (0, 0) = & D^{2}u(\frac{Du^{⊥}}{|Du|} , \frac{Du}{|Du|})(x_{0}). \end{aligned}  (18.4)

Exercise 18.3. Check the three above formulas.

The implicit function theorem 18.8 ensures that in a neighborhood of x_{0} the set {x | u(x) = u(x_{0})} is a C^{2} graph whose equation can be written in the local coordinates x = φ(y), where φ is a C^{2} function in an interval I containing y = 0. In this interval, we have u(φ(y), y) = u(x_{0}). Differentiating this shows that u_{x}φ′ + u_{y} = 0 for y ∈ I. Since |Du(x_{0})| = u_{x}(0, 0) and u_{y}(0, 0) = 0 “JMMBookOct04” 30/10/2009 page 241

[figure] [figure]

18.3. CURVATURE OF THE LEVEL LINES 241

Figure 18.3: Intrinsic coordinates. Note that φ′′(0) > 0, so b < 0.

in our coordinate system, we obtain φ′(0) = 0. A second differentiation of u_{x}(φ(y), y)φ′ + u_{y}(φ(y), y) = 0 yields

(u_{xx}φ′ + u_{xy})φ′ + u_{x}φ′′ + u_{yx}φ′ + u_{yy} = 0.

Since φ′(0) = 0, we obtain φ′′(0) = −u_{yy}(0, 0)/u_{x}(0, 0). Using the notation of (18.4), one obtains

φ(y) = −\frac{b}{p}y^{2} + o(y^{2}).  (18.5)

Equation (18.5) is the representation of the level line {x | u(x) = u(x_{0})} in the intrinsic coordinates at x_{0}. Let us set |2b/p| = 1/R. If the curve is a circle, R is the radius of this circle. More generally R is called radius of the osculatory circle to the curve. See exercise 18.11.

We are now going to do another simple computation to determine the curvature vector of the Jordan arc c defined by c(y) = x_{0} + φ(y)i + yj near y = 0. Recall that we denote the curvature of a curve c by κ(c) and the value of this function at a point c(y) by κ(c)(y).) Since in the local coordinates c′(y) = (φ′(y), 1) and c′′(y) = (φ′′(y), 0), at y = 0, we have c′(0) = (0, 1) and c′′(0) = (φ′′(0), 0), so that c′′(0) · c′(0) = 0. Using this and the expression of the curvature in local graph coordinates (18.2) yields

κ(c)(0) = (φ′′(0), 0) = φ′′(0)\frac{Du}{|Du|}(x_{0}).

We now use (18.5) and (18.4) to write the last expression as

κ(c)(0) = −\frac{1}{|Du|}D^{2}u(\frac{Du^{⊥}}{|Du|} , \frac{Du^{⊥}}{|Du|} )\frac{Du}{|Du|}(x_{0})  (18.6)

This tells us that the vectors κ(c)(0) and Du(0) are collinear. Equation (18.6) also leads to the following definition and lemma introducing a scalar curvature.

Definition 18.14. Let u be a real-valued function that is C^{2} in a neighborhood of a point x ∈ ℝ^{2} and assume that Du(x) ≠ 0. The curvature of u at x, denoted by curv(u)(x), is the real number defined in the local coordinates at x by

\frac{1}{|Du|^{3}} D^{2}u(Du^{⊥}, Du^{⊥})(x) = \frac{u_{xx}u_{y}^{2} − 2u_{xy}u_{x}u_{y} + u_{yy}u_{x}^{2}}{(u_{x}^{2} + u_{y}^{2})^{3/2}} (0, 0).  (18.7)

“JMMBookOct04” 30/10/2009 page 242

[figure] [figure]

242 CHAPTER 18. CURVES AND CURVATURES

Exercise 18.4. Check the above identity.

Lemma 18.15. Assume that u : ℝ^{2} → ℝ is C^{2} in a neighborhood of a point x_{0} and assume that Du(x_{0}) ≠ 0. Let N = N(x_{0}) be a neighborhood of x_{0} in which the iso-level set of u, {x | u(x) = u(x_{0})}, is a simple C^{2} arc, which we still denote by x = x(s). Then at every point x of this arc,

κ(x) = −curv(u)(x)\frac{Du}{|Du|}(x).  (18.8)

Proof. This is an immediate consequence of (18.6) and (18.7). We need only remark that, given the hypotheses of the lemma, there is a neighborhood N of x_{0} such that Du(x) ≠ 0 for x ∈ N and such that {x | u(x) = u(x_{0})} is a simple C^{2} arc for x ∈ N. Then the argument we made to derive (18.6) holds for any point x ∈ N ∩ {x | u(x) = u(x_{0})}. □ The next exercise proposes as a sanity check a verification that the curvature thus defined is contrast invariant and rotation invariant.

Exercise 18.5. Use equation (18.7) to show that

curv(u) = \frac{∂}{∂x} ( \frac{Du}{|Du|} ) + \frac{∂}{∂y} ( \frac{Du}{|Du|} ) = div( \frac{Du}{|Du|} ).  (18.9)

Use this last relation to show that curv(g(u)) = curv(u) if g is any C^{2} function g : ℝ → ℝ such that g′(x) > 0 for all x ∈ ℝ. What happens if g′(x) < 0 for all x ∈ ℝ? Show that curv(U) = curv(u), where U(s, t) = u(x, y) and x = s cos θ−t sin θ, y = s sin θ+t cos θ. Check that curv(−u) = −curv(u) and give a geometric interpretation to this relation.

Before leaving this section, we wish to emphasize geometric aspects of the functions we have introduced. Perhaps the most important fact is that the curvature of a C^{2} Jordan arc Γ is an intrinsic property of Γ; it does not depend on the parameterization. If x is a point on Γ, then the curvature vector κ(x) points toward the center of the osculating circle. Furthermore, 1/|κ(x)| is the radius of this circle, so when |κ(x)| is large, the osculating circle is small, and the curve is ”turning a sharp corner.”

If Du(x) ≠ 0, then the vector Du(x) points in the direction of greatest increase, or steepest ascent, of u at x: Following the gradient leads uphill. The function curv(u) does not have such a clear geometric interpretation, and it is perhaps best thought of in terms of equation (18.8): curv(u)(x) is the coefficient of −Du(x)/|Du(x)| that yields the curvature vector κ(x) of the level curve through the point x. We cannot over emphasize the importance of the two operators curv and Curv for the theories that follow. In addition to (18.8), a further relation between these operators is shown in Proposition 19.8, and it is this result that connects function smoothing with curve smoothing.

18.4 The principal curvatures of a level surface

We saw in Exercise 18.5 that curv(u) was contrast invariant. This idea will be generalized to ℝ^{N} by introducing other differential operators that are contrast invariant. These operators will be functions of the principal curvatures of the level surfaces of u. For z ∈ ℝ^{N}, z^{⊥} will denote the hyperplane {y | z · y = 0} “JMMBookOct04” 30/10/2009 page 243

[figure] [figure]

18.4. THE PRINCIPAL CURVATURES OF A LEVEL SURFACE 243

that is orthogonal to z. (There should be no confusion with this notation and the same notation for z ∈ ℝ^{2}. In ℝ^{2}, z^{⊥} is a vector orthogonal to z, and the corresponding “hyperplane” is the line {tz^{⊥} | t ∈ ℝ}.)

Proposition 18.16. Assume that u : ℝ^{N} → ℝ is C^{2} in a neighborhood of a point x_{0} and assume that Du(x_{0}) ≠ 0. Let g : ℝ → ℝ be a C^{2} contrast change such that g′(s) > 0 for all s ∈ ℝ. Then Dg(u(x_{0})) = g′(u(x_{0}))Du(x_{0}), and D̃^{2}g(u(x_{0})) = g′(u(x_{0}))D̃^{2}u(x_{0}), where D̃^{2}u(x_{0}) denotes the restriction of the quadratic form D^{2}u(x_{0}) to the hyperplane Du(x_{0})^{⊥}. This means, in particular, that (1/|Du(x_{0})|)D̃^{2}u(x_{0}) is invariant under such a contrast change.

Proof. To simplify the notation, we will suppress the argument x_{0}; thus, we write Du for Du(x_{0}), and so on. We use the notation y ⊗ y, y ∈ ℝ^{N}, to denote the linear mapping y ⊗ y : ℝ^{N} → ℝ^{N} defined by (y ⊗ y)(x) = (x · y)y. The range of y ⊗ y is the one-dimensional space ℝy.

An application of the chain rule shows that Dg(u) = g′(u)Du. This implies that Du^{⊥} = Dg(u)^{⊥}. (Recall that g′(s) > 0 for all s ∈ ℝ.) A second differentiation shows that

D^{2}g(u) = g′′(u)Du ⊗ Du + g′(u)D^{2}u.

If y ∈ Du^{⊥}, then (Du ⊗ Du)(y) = 0 and D^{2}g(u)(y, y) = g′(u)D^{2}u(y, y). This means that D^{2}g(u) = g′(u)D^{2}u on Du^{⊥} = Dg(u)^{⊥}, which proves the result. □

Exercise 18.6. Taking euclidian coordinates, give the matrix of y ⊗ y. Check the above differentiations.

We are now going to define locally the level surface of a smooth function u, and for this we quote one more version of the implicit function theorem, in arbitrary dimension N.

Theorem 18.17 (Implicit function theorem). Assume that u : ℝ^{N} → ℝ is C^{m} in the neighborhood of x_{0} and assume that Du(x_{0}) ≠ 0. Write x = x_{0} + y + zi, where i = Du(x_{0})/|Du(x_{0})| and y ∈ Du(x_{0})^{⊥}. Then there exists a ball B(x_{0}, ρ) and a unique real-valued C^{m} function φ defined on B(x_{0}, ρ) ∩ {x | x = x_{0} + y, i · y = 0} such that for every x ∈ B(x_{0}, ρ)

u(x) = u(x_{0}) ⟺ φ(y) = z.

In other words, the equation φ(y) = z describes the set {x | u(x) = u(x_{0})} near x_{0} as the graph of a C^{m} function φ. Thus, locally we have a surface passing through x_{0} that we call the level surface of u around x_{0}.

We are going to use Proposition 18.16 and Theorem 18.17, first, to give a simple intrinsic representation for the level surface of a function u around a point x_{0} and, second, to relate the eigenvalues of the quadratic form introduced in Proposition 18.16 to the curvatures of lines drawn on the level surface of u.

Proposition 18.18. Assume that u : ℝ^{N} → ℝ is C^{2} in a neighborhood of x_{0} ∈ ℝ^{N} and that p = Du(x_{0}) ≠ 0. Denote the eigenvalues of the restriction of the quadratic form D^{2}u(x_{0}) to the hyperplane Du(x_{0})^{⊥} by μ_{1}, …, μ_{N−1}. Let i_{N} = Du(x_{0})/|Du(x_{0})| and select i_{1}, …, i_{N−1} so they form an orthonormal “JMMBookOct04” 30/10/2009 page 244

[figure] [figure]

244 CHAPTER 18. CURVES AND CURVATURES

basis of eigenvectors of the restriction of D^{2}u(x_{0}) to Du(x_{0})^{⊥}. Write x = x_{0}+z, where z = x_{1}i_{1} + ⋯ + x_{N}i_{N} = y + x_{N}i_{N}. Then if |z| is sufficiently small, the function φ(y) = x_{N} that solves the equation u(y, φ(y)) = u(x_{0}) can be expressed locally as

x_{N} = \frac{−1}{2p} ∑_{i=1}^{N−1} μ_{i}x_{i}^{2} + o(|y|^{2}).

Proof. Assume, without loss of generality, that x_{0} = 0 and that u(0) = 0. Using the notation of Theorem 18.17, for x ∈ B(0, ρ), u(y, x_{N}) = 0 if and only if φ(y) = x_{N}, and φ is C^{2} in B(0, ρ). Furthermore, by differentiating the expression u(y, φ(y)) = 0, we see that u_{x_{i}} + u_{x_{N}} φ_{x_{i}} = 0, i = 1, …, N − 1 for |x| < ρ. In particular, u_{x_{i}}(0)+u_{x_{N}} (0)φ_{x_{i}}(0) = 0. In the local coordinate system we have chosen, |Du(0)| = |u_{x_{N}} (0)|, and since Du(0) ≠ 0, we conclude that u_{x_{i}}(0) = 0 for i = 1, …, N − 1 and hence that φ_{x_{i}}(0) = 0 for i = 1, …, N − 1. This means that the local expansion of φ has the form

φ(y) = \frac{1}{2}D^{2}φ(0)(y, y) + o(|y|^{2}).

Now differentiate the relation u_{x_{i}} + u_{x_{N}} φ_{x_{i}} = 0 again to obtain

u_{x_{i}x_{j}} + u_{x_{i}x_{N}} φ_{x_{j}} + (u_{x_{N} x_{j}} + u_{x_{N} x_{N}} φ_{x_{j}})φ_{x_{i}} + u_{x_{N}} φ_{x_{i}x_{j}} = 0.

Since we have just shown that φ_{x_{i}}(0) = 0 for i = 1, …, N − 1, we see from this last expression that D̃^{2}u(0) + pD̃^{2}φ(0) = 0, where p = u_{x_{N}} (0) and D̃^{2}u(0), D̃^{2}φ(0) are the restrictions of the quadratic forms D^{2}u(0) and D^{2}φ(0) to the hyperplane Du(0)^{⊥}. Thus we have

x_{N} = \frac{−1}{2p}D^{2}u(0)(y, y) + o(|y|^{2}).  (18.10)

Recall that y ∈ Du(0)^{⊥} and that y = x_{1}i_{1} + ⋯ + x_{N−1}i_{N−1}, where the i_{i} are an orthonormal basis of eigenvectors of D^{2}(0) restricted to Du(0)^{⊥}. Thus,

x_{N} = \frac{−1}{2p} ∑_{i=1}^{N−1} μ_{i}x_{i}^{2} + o(|y|^{2}),

which is what we wished to prove. □

This formula reads

x_{2} = \frac{−1}{2p}μ_{1}x_{1}^{2} + o(|x_{1}|^{2})

if N = 2, which is just equation (18.5) with different notation. Thus, μ_{1} = |Du|curv(u), confirming that μ_{1} = ∂^{2}u/∂x_{1}^{2}. We are now going to use our twodimensional analysis to give a further interpretation of the eigenvalues μ_{i} for N > 2. We begin by considering the curve Γ_{ν} defined by the two equations x = x_{0} + tν + x_{N}i_{N} and φ(tν) = x_{N}, where ν is a unit vector in Du(x_{0})^{⊥}. Their solution in the local coordinates is φ(tν) = x_{N}, whenever t ∈ ℝ is small. Thus, Γ_{ν} is a curve passing by x_{0}, drawn on the level surface of u and projecting into a straight line of Du^{⊥}. By (18.10) its equation is

x_{N} = φ(tν) = \frac{−1}{2|Du(x_{0})|}D^{2}u(x_{0})(ν, ν)t^{2} + o(t^{2}),

“JMMBookOct04” 30/10/2009 page 245

[figure] [figure]

18.4. THE PRINCIPAL CURVATURES OF A LEVEL SURFACE 245

and its normal at x_{0} is \frac{Du(x_{0})}{|Du(x_{0})|}. Thus the curvature vector of Γ_{ν} at x_{0} is

κ(Γ_{ν})(x_{0}) = \frac{−1}{|Du(x_{0})|}D^{2}u(x_{0})(ν, ν)\frac{Du(x_{0})}{|Du(x_{0})|}.

By defining κ_{ν} = |Du(x_{0})|^{−1}D^{2}u(x_{0})(ν, ν), we have

κ(Γ_{ν})(x_{0}) = −κ_{ν} \frac{Du(x_{0})}{|Du(x_{0})|},

which has the same form as equation (18.8). So the modulus of κ_{ν} is equal to the modulus of the curvature of Γ_{ν} at x_{0}. This leads us to call principal curvatures of the level surface of u at x_{0} the numbers κ_{ν} obtained by letting ν = i_{j}, j = 1, …, N − 1, where the unit vectors i_{j} are an orthonormal system of eigenvectors of D^{2}u(x_{0}) restricted to Du(x_{0})^{⊥}.

Definition 18.19. Let u : ℝ^{2} → ℝ be C^{2} at x_{0}, with Du(x_{0}) ≠ 0. The principal curvatures of u at x_{0} are the real numbers

κ_{j} = \frac{μ_{j}}{|Du(x_{0})|},

where μ_{j} are the eigenvalues of D^{2}u(x_{0}) restricted to Du(x_{0})^{⊥}.

It follows from Proposition 18.16 that the principal curvatures are invariant under a C^{2} contrast change g such that g′(s) > 0 for all s ∈ ℝ.

Definition 18.20. The mean curvature of a C^{2} function u : ℝ^{N} → ℝ at x_{0} ∈ ℝ^{N} is the sum of the principal curvatures at x_{0}. It is denoted by curv(u)(x_{0}).

Note that this definition agrees with Definition 18.2 when N = 2. The next result provides another representation for curv(u).

Proposition 18.21. The mean curvature of u is given by

curv(u) = div(\frac{Du}{|Du|}).

Proof. Represent the matrix D^{2}u in the coordinate system i_{j}, j = 1, …, N −1, and i_{N} = Du(x_{0})/|Du(x_{0})|, where the i_{j}, j = 1, …, N −1, form a complete set of eigenvectors of the linear mapping D^{2}u(x_{0}) restricted to Du^{⊥}(x_{0}). Then in this coordinate system, D^{2}u(x_{0}) has the following form (illustrated for N = 5):

D^{2}u(x_{0}) = [\begin{matrix}u_{11} & 0 & 0 & 0 & u_{15} \\ 0 & u_{22} & 0 & 0 & u_{25} \\ 0 & 0 & u_{33} & 0 & u_{35} \\ 0 & 0 & 0 & u_{44} & u_{45} \\ u_{51} & u_{52} & u_{53} & u_{54} & u_{55} \end{matrix}] ,

where u_{jk} = u_{x_{j}x_{k}}(x_{0}), and u_{jj} = κ_{j} is the eigenvalue associated with i_{j}. Thus, by definition, we see that

curv(u) = \frac{Δu}{|Du|} − \frac{1}{|Du|}D^{2}u(\frac{Du}{|Du|}, \frac{Du}{|Du|}).

“JMMBookOct04” 30/10/2009 page 246

[figure] [figure]

246 CHAPTER 18. CURVES AND CURVATURES

We also have

div(\frac{Du}{|Du|}) = ∑_{j=1}^{N} \frac{∂}{∂x_{j}} (\frac{u_{x_{j}}}{|Du|})

= \frac{1}{|Du|} ∑_{j=1}^{N} u_{x_{j}x_{j}} − \frac{1}{|Du|^{3}} ∑_{j,k=1}^{N} u_{x_{j}x_{k}}u_{x_{j}}u_{x_{k}}

= \frac{Δu}{|Du|} − \frac{1}{|Du|}D^{2}u(\frac{Du}{|Du|}, \frac{Du}{|Du|}).

□

With this representation, it is clear that the mean curvature has the same invariance properties as the curvature of a C^{2} function defined on ℝ^{2}. (See Exercise 18.5.)

18.5 Exercises

Exercise 18.7. Let Γ by a Jordan arc parameterized by x : [0, 1] → Γ and by y : [0, 1] → Γ. Show that x = y ∘ f or x = y ∘ (1 − f), where f is a continuous, strictly increasing function that maps [0, 1] onto [0, 1]. Hint: x and y are one-to-one, and since [0, 1] is compact, they are homeomorphisms. Thus, y^{−1}(x) = f is a one-to-one continuous mapping of [0, 1] onto itself. As an application, give a proof of Proposition 18.4.

Exercise 18.8. State and prove an adaptation of Propositions 18.4 and 18.5 to a Jordan arc.

The curvature vector has been defined in terms of the arc length. Curves, however, are often naturally defined in terms of other parameters. The next two exercises develop the differential relations between an arc-length parameterization and another parameterization.

Exercise 18.9. Assume that Γ is a C^{2} Jordan arc or curve. Let s ↦ x(s) be an arc-length parameterization and let t ↦ y(t) be any other parameterization with the property that y′(t) ≠ 0. Since x and y are one-to-one, we can consider the function y^{−1}(x) = φ. Then x(s) = y(φ(s)), where φ(s) = t. The inverse function φ^{−1} is given by

s = φ^{−1}(t) = ∫_{t_{0}}^{t} \sqrt{y′(r) · y′(r)} dr,

so we know immediately that φ^{−1} is absolutely continuous with continuous derivative equal to \sqrt{y′(t) · y′(t)}. Thus, we also know that φ′(s) = |y′(φ(s))|^{−1}. Note that we made a choice above by taking \sqrt{y′(r) · y′(r)} to be positive. This is equivalent to assuming that x′(s) and y′(φ(s)) point in the same direction or that φ′(s) > 0.

(i) Show that κ(s) = x′′(s) = y′′(φ(s))[φ′(s)]^{2} + y′(φ(s))φ′′(s) and deduce that

φ′′(s) = − \frac{y′′(φ(s))φ′(s) · y′(φ(s))}{|y′(φ(s))|^{3}} = − \frac{y′′(φ(s)) · y′(φ(s))}{|y′(φ(s))|^{4}} .

(ii) Use the results of (i) to show that

κ(s) = x′′(s) = \frac{1}{|y′(t)|^{2}} [y′′(t) − (y′′(t) · \frac{y′(t)}{|y′(t)|} ) \frac{y′(t)}{|y′(t)|} ] ,  (18.11)

“JMMBookOct04” 30/10/2009 page 247

[figure] [figure]

18.5. EXERCISES 247

where φ(s) = t. Show that we get the same expression for the right-hand side of (18.11) with the assumption that φ′(s) < 0. This shows that the curvature vector κ does not depend on the choice of parameter.

(iii) Consider the scalar function κ(y) defined by κ(y)(s) = κ(s)·x′(s)^{⊥}. Use equation (18.11) to show that

κ(y)(t) = \frac{y′′(t) · [y′(t)]^{⊥}}{|y′(t)|^{3}}

Note that κ(y) is determined up to a sign that depends on the sign of φ′(s); however, |κ(y)| = |κ| is uniquely determined.

Exercise 18.10. Assume that Γ is a Jordan arc or curve that is represented by a C^{1} function t ↦ x(t) with the property that x′(t) ≠ 0. Prove that Γ is C^{1}. Exercise 18.11.

(i) Consider the arc-length parameterization of the circle with radius r centered at the origin given by x(s) = (r cos(s/r), r sin(s/r)). Show that the length of the curvature vector is 1/r.

(ii) Compute the scalar curvature of the graph of y = (a/2)x^{2} at x = 0. Exercise 18.12. Complete the proof of Corollary 18.9. Exercise 18.13. The kinds of techniques used in this exercise are important for work in later chapters. The exercise demonstrates that it is possible to bracket a C^{2} function locally with two functions that are radial and either increasing or decreasing. We say that a function f is radial and increasing if there exists an increasing function g : ℝ^{+} → ℝ such that f(x) = g(|x_{c} − x|^{2}), x_{c} ∈ ℝ^{2}. We say that f is radial and decreasing if g is decreasing. Let u : ℝ^{2} → ℝ be C^{2} and assume that Du(x_{0}) ≠ 0. We wish to show that for every ε > 0 there exist two C^{2} radial functions f_{ε}^{−} and f_{ε}^{+} (increasing or decreasing, depending on the situation) that satisfy the following four conditions:

f_{ε}^{−}(x_{0}) = u(x_{0}) = f_{ε}^{+}(x_{0}),  (18.12)
Df_{ε}^{−}(x_{0}) = Du(x_{0}) = Df_{ε}^{+}(x_{0}),  (18.13)
curv(f_{ε}^{−})(x_{0}) + \frac{2ε}{p} = curv(u)(x_{0}) = curvf_{ε}^{+}(x_{0}) − \frac{2ε}{p} ,  (18.14)
f_{ε}^{−}(x) + o(|x_{0} − x|^{2}) ≤ u(x) ≤ f_{ε}^{+}(x) + o(|x_{0} − x|^{2}).  (18.15)

1. Without loss of generality, take x_{0} = (0, 0), u(0, 0) = 0, and Du(x_{0}) = (p, 0), p > 0. Then we have the Taylor expansion

u(x) = px + ax^{2} + by^{2} + cxy + o(x^{2} + y^{2}),

where a, b, and c are given in (18.4). Show that for every ε > 0,

px+(− \frac{c^{2}}{ε} +a)x^{2}+(b−ε)y^{2}+o(x^{2}+y^{2}) ≤ u(x, y) ≤ px+( \frac{c^{2}}{ε} +a)x^{2}+(b+ε)y^{2}+o(x^{2}+y^{2}).

2. Let f be a radial function defined by f(x, y) = g((x − x_{c})^{2} + y^{2}), where g : ℝ^{+} → ℝ is C^{2} and either increasing or decreasing. Show by expanding f at (0, 0) that

f(x, y) = g(x_{c}^{2}) − 2x_{c}g′(x_{c}^{2})x + (2x_{c}^{2}g′′(x_{c}^{2}) + g′(x_{c}^{2}))x^{2} + g′(x_{c}^{2})y^{2} + o(x^{2} + y^{2}).

3. The idea is to construct f_{ε}^{+} and f_{ε}^{−} by matching the coefficients of the expansion of f with the coefficients of the functions px + (±(c^{2}/ε) + a)x^{2} + (b ± ε)y^{2}. There are three cases to consider: b < 0, b = 0, and b > 0. Show that in each case it is possible to find values of x_{c} and functions g so the functions f_{ε}^{+} and f_{ε}^{−} satisfy the four condition. Note that both x_{c} and g depend on ε. Discuss the geometry for each case. Exercise 18.14. By computing explicitly the terms ∂g(u)/∂x_{i}, verify that Dg(u) = g′(u)Du. Similarly, verify that D^{2}(g(u)) = g′′(u)Du ⊗ Du + g′(u)D^{2}u by computing the second-order terms ∂^{2}g(u)/∂x_{i}∂x_{j}.

“JMMBookOct04” 30/10/2009 page 248

[figure] [figure]

248 CHAPTER 18. CURVES AND CURVATURES

18.6 Comments and references

Calculus and differential geometry. The differential calculus of curves and surfaces used in this chapter can be found in many books, and no doubt most readers are familiar with this material. Nevertheless, a few references to specific results may be useful. As a general reference on calculus, and as a specific reference for the implicit function theorem, we suggest the text by Courant and John [57]. (The implicit function theorem can be found on page 221 of volume II.) Elementary results about classical differential geometry can be found in [237]. A statement and proof of Sard’s theorem can be found in [138].

Level lines. An introduction to the use of level lines in computer vision can be found in [45]. A complete discussion of the definition of level lines for BV functions can be found in [11]. One can decompose an image into into its level lines at quantized levels and conversely reconstruct the image from this topographic map. A fast algorithm, the Fast Level Set Transform (FLST) performing these algorithms is described in [148]. Its principle is very simple: a) perform the bilinear interpolation, b) rule out all singular levels where saddle point occur c) quantize the other levels, in which the level lines are finite unions of parametric Jordan curves. The image is then parsed into a set of parametric Jordan curves. This set is easily ordered in a tree structure, since two Jordan level curves do not meet. Thus either one surrounds the other one or conversely. The level lines tree is a shape parser for the image, many level lines surrounding perceptual shapes or parts of perceptual shapes.

Curvature. It is a well-known mathematical technique to define a set implicitly as the zero set of its distance function. In case the set is a curve, one can compute its curvature at a point x by computing the curvature curv(u)(x), where u is a signed distance function of the curve. This yields an intrinsic formula for the curvature that is not dependent on a parameterization of the curve. The same technique has been applied in recent years as a useful numerical tool. This started with Barles report on flame propagation [24] and was extended by Sethian [231] and by Osher and Sethian [199] in a series of papers on the numerical simulation of the motion of a surface by its mean curvature.

“JMMBookOct04” 30/10/2009 page 249

[figure] [figure]

Chapter 19

The Main Curvature Equations

The purpose of this chapter is to introduce the curvature motion PDE’s for Jordan curves and images. Our main task is to establish a formal link between curve evolution and image evolution. This link will be established through the PDE formulation. The basic differential geometry used in this chapter was thoroughly developed in Chapter 18, which must therefore be read first.

19.1 The definition of a shape and how it is recognized

Relevant information in images has been reduced to the image level sets in Chapter 12. By Corollary 18.9, if the image is C^{1}, the boundary of its level sets is a finite set Jordan curves at almost every level. Thus, shape analysis can be led back to the study of these curves which we shall call “elementary shapes”.

Definition 19.1. We call elementary shape any C^{1} planar Jordan curve.

The many experiments where we display level lines of digital images make clear enough why a smoothing is necessary to restore their structure. These experiments also show that we can in no way assimilate these level lines with our common notion of shape as the silhouette of a physical object in full view. Indeed, in images of a natural environment, most observed objects are partially hidden (occluded) by other objects and often deformed by perspective. When we observe a level line we cannot be sure that it belongs to a single object; it may be composed of pieces of the boundaries of several objects that are occluding each other. Shape recognition technology has therefore focused on local methods, that is, methods that work even if a shape is not in full view or if the visible part is distorted. As a consequence, image analysis adopts the following principle: Shape recognition must be based on local features of the shape’s boundary, in this case local features of the Jordan curve, and not on its global features. If the boundary has some degree of smoothness, then these local features are based on the derivatives of the curve, namely the tangent vector, the curvature, and

249 “JMMBookOct04” 30/10/2009 page 250

[figure] [figure]

250 CHAPTER 19. THE MAIN CURVATURE EQUATIONS

so on. Many local recognition methods involve the “salient” points of a shape, which are the points where the curvature is zero (inflection points) and points where the curvature has a maximum or minimum (the “corners” of the shape).

19.2 Multiscale features and scale space

Computational shape recognition methods often make the following two basic assumptions, neither of which is true in practice for the rough shape data:

• The shape is a smooth Jordan curve.

• The boundary has a small number of inflexion points and curvature extrema. This number can be made as small as desired by smoothing.

The fact that these conditions can be obtained by properly smoothing a C^{1} Jordan curve was proven in 1986-87 by Gage and Hamilton [90] and Grayson [98]. They showed that it is possible to transform a C^{1} Jordan curve into a C^{∞} Jordan curve by using the so-called intrinsic heat equation.

For convenience, and unless it would cause ambiguity, we will not make a distinction between a Jordan curve Γ as a subset of the plane and a function s ↦ x(s) such that Γ = {x(s)}. As we have already done, we will speak of the Jordan curve x. Since we will be speaking of families of Jordan curves dependent on a parameter t > 0, we will most often denote these families by x(t, s), where the second variable is a parameterization of the Jordan curve. Thus, x(t, s) has three meanings: a family of Jordan curves, a family of functions that represent these curves, and a particular point on one of these curves. The notation s will be usually reserved to an arc-length parameter.

Definition 19.2. Let x(t), t > 0, be a family of C^{2} Jordan curves. We say that x(t) satisfies the intrinsic heat equation if

\frac{∂x}{∂t} = κ(x(t)).  (19.1)

Theorem 19.3 (Grayson). Let x_{0} be a C^{1} Jordan curve. By using the intrinsic heat equation, it is possible to evolve x_{0} into a family of Jordan curves x(t, s) such that x(0, s) = x_{0}(s) and such that for every t > 0, x(t, s) is C^{∞} (actually analytical) and satisfies the equation (19.1). Furthermore, for every t > 0, x(t, s) has only a finite number of inflection points and curvature extrema, and the number of these points does not increase with t. For every initial curve, there is a scale t_{0} such that the curve x(t, s) is convex for t ≥ t_{0} and there is a scale t_{1} such that the curve x(t, s) is a single point for t ≥ t_{1}.

It is time to say what we mean by “curve scale space”, or “shape scale space.” We will refer to any process that smooths a Jordan curve and that depends on a real parameter t. Thus a shape scale space associates with an initial Jordan curve x(0, s) = x_{0}(s) a family of smooth curves x(t, s). For example, the intrinsic heat equation eliminates spurious details of the initial shape and retains simpler, more reliable versions of the shape, and these smoothed shapes have finite codes. A scale space is causal in the terminology of vision theory if it does not introduce new features. Grayson’s theorem therefore defines a causal scale space.

“JMMBookOct04” 30/10/2009 page 251

[figure] [figure]

19.3. FROM IMAGE MOTION TO CURVE MOTION 251

19.3 From image motion to curve motion

The intrinsic heat equation is only one example from a large family of nonlinear equations that move curves with a curvature-dependent speed, that is, ∂x/∂t is a function of the curvature of the curve x. There are two conditions on the curvature dependence. The velocity vector must always point towards the concavity of the curve. Its norm must be a nondecreasing function of the magnitude of the curvature |κ(x)|. The first condition ensures that the equation is a smoothing which reduces asperities. The second condition intuitively preserves the inclusion between curves. This can be appreciated by considering two circles C and C′ such that C′ surround C and C′ and C are tangent at some point x. Then the first condition implies that both circles shrink, but the second condition implies that the smaller circle C shrinks faster, so that the inclusion between circles is preserved by the evolution.

Definition 19.4. We say that a C^{2} function u : ℝ^{+} × ℝ^{2} → ℝ satisfies a curvature equation if for some real-valued function g(κ, t), which is nondecreasing in κ and satisfies g(0, t) = 0,

\frac{∂u}{∂t}(t, x) = g(curv(u)(t, x), t)|Du|(t, x).  (19.2)

Definition 19.5. Let x(t) be a family of C^{2} Jordan curves. We say that the functions x(t) satisfy a curvature equation if for some real-valued function g(κ, t) nondecreasing in κ with g(0, t) = 0, they satisfy

\frac{∂x}{∂t} = g(|κ(x)|, t)n(t),  (19.3)

where n is a unit vector in the direction of κ(x).

In the preceding definition, the equation makes sense if κ(x) = 0 since then the second member is zero. As we shall see, these equations are the only candidates to be curve or image scale spaces, and one of the objectives of this book is to identify which forms for g are particularly relevant for image analysis. The above definitions are quite restrictive because they require the curves or images to be C^{2}. A more generally applicable definition of solutions for these equations will be given in Chapter ?? with the introduction of viscosity solutions. Our immediate objective is to establish the link between the motion of an image and the motion of its level lines. This will establish the relation between equations (19.2) and (19.3).

19.3.1 A link between image and curve evolution

Lemma 19.6. (Definition of the “normal flow”). Suppose that (t, x) ↦ u(t, x) is C^{2} in a neighborhood T ×U of the point (t_{0}, x_{0}) ∈ ℝ×ℝ^{2}, and assume that Du(t_{0}, x_{0}) ≠ 0. Then there exists an open interval J centered at t_{0}, an open disk V centered at x_{0}, and a unique C^{1} function x : J × V → ℝ^{2} that satisfy the following properties:

(i) u(t, x(t, y)) = u(t_{0}, y) and x(t_{0}, y) = y for all (t, y) ∈ J × V.

“JMMBookOct04” 30/10/2009 page 252

[figure] [figure]

252 CHAPTER 19. THE MAIN CURVATURE EQUATIONS

(ii) The vectors (∂x/∂t)(t, y) and Du(t, x(t, y)) are collinear.

In addition, the function x satisfies the following differential equation:

\frac{∂x}{∂t}(t, y) = −(\frac{Du}{|Du|^{2}} \frac{∂u}{∂t})(t, x(t, y)).  (19.4)

The trajectory t ↦ x(t, y) is called the normal flow starting from (t_{0}, y).

Proof. Differentiating the relation u(t, x(t)) = 0 with respect to t yields \frac{∂u}{∂t} + Du.\frac{∂x}{∂t} = 0. By multiplying this equation by the vector Du we see that \frac{∂x}{∂t} is collinear to Du if and only if (19.4) holds. Now, this relation defines x(t) as the solution of an ordinary differential equation, with initial condition x(t_{0}) = y. Since u is C^{2}, the second member of (19.4) appears to be a Lipschitz function of (t, x) provided Du(t, x) ≠ 0, which is ensured for (t, x) close enough to (t_{0}, x_{0}). Thus, by Cauchy-Lipschitz Theorem, there exists an open interval J such that the O.D.E. (19.4) has a unique solution x(t, y) for all y in a neighborhood of x_{0} and t ∈ J. □

Proposition 19.7. Assume that the function (t, x) ↦ u(t, x) is C^{2} in a neighborhood of (t_{0}, x_{0}) and that Du(t_{0}, x_{0}) ≠ 0. Then u satisfies the curvature motion equation

\frac{∂u}{∂t}(t, x) = curv(u)(t, x)|Du|(t, x)  (19.5)

in a neighborhood of (t_{0}, x_{0}) if and only if the normal flow x(t, y) of u in this neighborhood satisfies the intrinsic heat equation

\frac{∂x}{∂t}(t, y) = κ(x(t, y)),  (19.6)

where κ(x(t, y)) denotes the curvature vector of the level line of u(t) passing by x(t, y).

Proof. Assume first that x(t, y) satisfies (19.6). Applying (18.8) for all t in a neighborhood of t_{0} to each image u(t) : x → u(t, x) yields

κ(x(t, y)) = −curv(u)\frac{Du}{|Du|}(t, x(t, y)).

Substituting (19.6) in this last relation we obtain

\frac{∂x}{∂t}(t, y) = −curv(u)\frac{Du}{|Du|}(t, x(t, y))

and by the normal flow equation (19.4),

(\frac{∂u}{∂t} \frac{Du}{|Du|^{2}} )(t, x(t, y)) = curv(u)\frac{Du}{|Du|}(t, x(t, y)).

Multiplying this equation by Du(t, x(t, y)) yields the curvature motion equation (19.5).

The converse statement follows exactly the same lines backwards. □

“JMMBookOct04” 30/10/2009 page 253

[figure] [figure]

19.3. FROM IMAGE MOTION TO CURVE MOTION 253

Exercise 19.1. Write the proof of the converse statement of Proposition 19.7.

The preceding proof is immediately adaptable to all curvature equations :

Proposition 19.8. Assume that the function (t, x) ↦ u(t, x) is C^{2} in a neighborhood of (t_{0}, x_{0}) and that Du(t_{0}, x_{0}) ≠ 0. Let g : ℝ × ℝ^{+} → ℝ be continuous and nondecreasing with respect to κ and such that g(−κ, t) = −g(κ, t). Then u satisfies the curvature motion equation

\frac{∂u}{∂t}(t, x) = g(curv(u)(t, x), t)|Du|(t, x)  (19.7)

in a neighborhood of (t_{0}, x_{0}) if and only if the normal flow t ↦ x(t, ·) satisfies the curvature equation

\frac{∂x}{∂t}(t, y) = g(|κ(x(t, y))|)\frac{κ(x(t, y))}{|κ(x(t, y))|}.  (19.8)

19.3.2 Introduction to the affine curve and function equations

There are two curvature equations that are affine invariant and are therefore particularly well suited for use in shape recognition. In their definition, for x ∈ ℝ, x^{1/3} stands for sign(x)|x|^{1/3}.

Definition 19.9. The image evolution equation

\frac{∂u}{∂t}(t, x) = (curv(u)(t, x))^{1/3}|Du(t, x)|  (19.9)

is called affine morphological scale space (AMSS). The curve evolution equation

\frac{∂x}{∂t}(t, s) = |κ(x(t, s))|^{1/3}n(t, s) ( = \frac{κ(x(t, s))}{|κ(x(t, s))|^{2/3}} )  (19.10)

is called affine scale space (ASS).

It is clear that AMSS and ASS are equivalent in the sense of Proposition 19.8. As one would expect from the names of these equations, they both have some sort of affine invariance. This is the subject of the next definition, Exercises 19.3 and 19.4 and the next section.

Definition 19.10. We say that a curvature equation (E) (image evolution equation) is affine invariant, if for every linear map A with positive determinant, there is a positive constant c = c(A) such that (t, x) ↦ u(t, x) is a solution of (E) if and only if (ct, Ax) ↦ u(ct, Ax) is a solution of (E).

19.3.3 The affine scale space as an intrinsic heat equation

Suppose that for each scale t, σ ↦ x(t, σ) is a Jordan arc (or curve) parameterized by σ, which is not in general an arc length. As in Chapter 18, we will “JMMBookOct04” 30/10/2009 page 254

[figure] [figure]

254 CHAPTER 19. THE MAIN CURVATURE EQUATIONS

denote the curvature of x by κ. We wish to demonstrate a formal equivalence between the affine scale space,

\frac{∂x}{∂t} = |κ|^{1/3}n(x),  (19.11)

and an “intrinsic heat equation”

\frac{∂x}{∂t} = \frac{∂^{2}x}{∂σ^{2}},  (19.12)

where σ is a special parameterization called affine length. We define an affine length parameter of a Jordan curve (or arc) to be any parameterization σ ↦ x(σ) such that

[x_{σ}, x_{σσ}] = 1,  (19.13)

where [x, y] = x^{⊥} · y. If s is an arc-length parameterization, then we have (Definition 18.3)

τ = x_{s} n = |κ|^{−1}x_{ss} ( = \frac{κ(x)}{|κ(x)|}).  (19.14)

We also have

x_{σ} = x_{s} \frac{∂s}{∂σ} and x_{σσ} = x_{ss}(\frac{∂s}{∂σ} )^{2} + x_{s} \frac{∂^{2}s}{∂σ^{2}} .  (19.15)

Thus,

[x_{σ}, x_{σσ}] = [x_{s}, x_{ss}](\frac{∂s}{∂σ} )^{3},

and if (19.13) holds, then

[x_{s}, x_{ss}](\frac{∂s}{∂σ} )^{3} = 1.

Since by (19.14) [x_{s}, x_{ss}] = sign([x_{s}, x_{ss}])|κ|, we conclude that

\frac{∂s}{∂σ} = (sign([x_{s}, x_{ss}])|κ|)^{−1/3}.  (19.16)

Substituting this result in the expression for x_{σσ} shown in (19.15) and writing x_{s} = τ, we see that

x_{σσ} = |κ|^{1/3}n + (\frac{∂^{2}s}{∂σ^{2}} )τ .

This tells us that equation (19.12) is equivalent to the following equation:

\frac{∂x}{∂t} = |κ|^{1/3}n + (\frac{∂^{2}s}{∂σ^{2}} )τ .  (19.17)

Now it turns out that the graphs of the functions x that you get from one time to another do not depend on the term involving τ; you could drop this term and get the same graphs. More precisely, Epstein and Gage [72] have shown that the tangential component of an equation like (19.17) does not matter as far as the geometric evolution of the curve is concerned. In fact, the tangential term just moves points along the curve itself, and the total curve evolution is determined by the normal term. As a consequence, equation (19.11) is equivalent to equation (19.12) in any neighborhood that avoids an inflection point, that is, in any neighborhood where n(x) ≠ 0. At an inflection point, κ = 0, and the two equations give the same result.

“JMMBookOct04” 30/10/2009 page 255

[figure] [figure]

19.4. CURVATURE MOTION IN N DIMENSIONS 255

19.4 Curvature motion in N dimensions

We consider an evolution (t, x) ↦ u(t, x), where x ∈ ℝ^{N} and u(0, ·) = u_{0} is an initial N–dimensional image. Let κ_{i}(u)(t, x), i = 1, …, N − 1, denote the i^{th} principal curvature at the point (t, x). By definition 18.20 the mean curvature is curv(u) = ∑_{i=1}^{N−1} κ_{i}. We will now define three curvature motion flow equations in N dimensions.

Mean curvature motion. This equation is a direct translation of equation (19.5) in N dimensions:

\frac{∂u}{∂t} = |Du|curv(u).

This says that the motion of a level hypersurface of u in the normal direction is proportional to its mean curvature.

Gaussian curvature motion for convex functions. We say that a function is convex if all of its principal curvatures have the same sign. An example of such a function is the signed distance function to a regular convex shape. The equation is

\frac{∂u}{∂t} = |Du| ∏_{i=1}^{N−1} κ_{i}.

The motion of a level hypersurface is proportional to the product of its principal curvatures, which is the Gaussian curvature. As we will see in Chapter ??, this must be modified before it can be applied to a nonconvex function.

Affine-invariant curvature motion. The equation is

\frac{∂u}{∂t} = |Du|| ∏_{i=1}^{N−1} κ_{i}|^{1/(N+1)}H( ∑_{i=1}^{N−1} sign(κ_{i})),

where H(N − 1) = 1, H(−N + 1) = −1, and H(n) = 0 otherwise. The motion is similar to Gaussian curvature motion, but the affine invariance requires that the Gaussian curvature be raised to the power 1/(N + 1). There is no motion at a point where the principal curvatures have mixed signs. This means that only concave or convex parts of level surfaces get move by such an equation.

19.5 Exercises

Exercise 19.2. Check that all of the curvature equations (19.2) are contrast invariant. That is, assuming that h is a real-valued C^{2} increasing function defined on ℝ and u is C^{2}, show that the function v defined by v(t, x) = h(u(t, x)) satisfies one of these equations if and only if u satisfies the same equation. Exercise 19.3. Assume that (t, x) ↦ u(t, x) is a C^{2} function and that A is a 2 × 2 matrix with positive determinant, which we denote by |A|. Define the function v by v(t, x) = u(ct, Ax), where c = |A|^{−2/3}.

(i) Prove that for each point x such that Du(x) ≠ 0 one has the relation

curv(v)(x)|Dv(x)|^{3} = |A|^{2}curv(u)(Ax)|Du(Ax)|^{3}.

“JMMBookOct04” 30/10/2009 page 256

[figure] [figure]

256 CHAPTER 19. THE MAIN CURVATURE EQUATIONS

(ii) Use (i) to deduce that the AMSS equation (19.9) is affine invariant, that is,

(t, x) ↦ u(t, x) is a solution of AMSS if and only (t, x) ↦ v(t, x) does. Exercise 19.4. This exercise is to show that the affine scale space (equation (19.10)) is affine invariant. It relies directly on results from Exercise 18.9. Let σ ↦ c(σ) be a C^{2} curve, and assume that |c′(σ)| > 0. Then we know from Exercise 18.9 that

κ(c)(σ) = \frac{1}{|c′(σ)|^{2}} [c′′(σ) − (c′′(σ) · \frac{c′(σ)}{|c′(σ)|} ) \frac{c′(σ)}{|c′(σ)|} ] .  (19.18)

Now assume that we have a family of C^{2} Jordan arcs (t, σ) ↦ c(t, σ). By projecting both sides of the intrinsic heat equation onto the unit vector c^{′⊥}/|c′| and by using (19.18), we have the following equation:

\frac{∂c}{∂t} · \frac{c^{′⊥}}{|c′|} = \frac{c′′ · c^{′⊥}}{|c′|^{3}}  (19.19)

We say that c satisfies a parametric curvature equation if it satisfies equation (19.19). In the same spirit, we say that c satisfies a parametric affine equation if for some constant γ > 0

\frac{∂c}{∂t} · c^{′⊥} = γ(c′′ · c^{′⊥})^{1/3}.  (19.20)

(i) Suppose that σ = s, an arc-length parameterization of c. Show that equation

(19.19) can be written as

\frac{∂c}{∂t} = κ(c) + λτ,

where λ is a real-valued function and τ is the unit tangent vector ∂c/∂s. (See the remark following equation (19.17).)

(ii) Let A be a 2 × 2 matrix with positive determinant, and define the curve y by y(t, σ) = Ac(t, σ). We wish to show that if c satisfies a parametric affine motion, then so does y. As a first step, show that Ax · (Ay)^{⊥} = |A|x · y and hence that A(x^{⊥}) · (Ax)^{⊥} = |A||x|^{2} for any x, y ∈ ℝ^{2}.

(iii) Show that if c satisfies equation (19.20), then y satisfies

\frac{∂y}{∂t} · y^{′⊥} = γ|A|^{2/3}(y′′ · y^{′⊥})^{1/3}.

19.6 Comments and references

Our definition of shape. The Italian mathematician Renato Caccioppoli proposed a theory of sets whose boundaries have finite length (finite Hausdorff measure). From his theory, it can be deduced that the boundary of a Caccioppoli set is composed of a countable number of Jordan curves, up to a set with zero length. This decomposition can even be made unambiguous. In other words, the set of Jordan curves associated with a given Caccioppoli set is unique and gives enough information to reconstruct the set [10]. This result justifies our focus on Jordan curves as the representatives of shapes.

The role of curvature in shape analysis. After Attneave’s founding paper [20], let us mention the thesis by G. J. Agin [4] as being one of the first references dealing with the use of curvature for the representation and recognition of objects in computer vision. The now-classic paper by Asada and Brady [17] entitled “The curvature primal sketch” introduced the notion of computing a “JMMBookOct04” 30/10/2009 page 257

[figure] [figure]

19.6. COMMENTS AND REFERENCES 257

“multiscale curvature” as a tool for object recognition. (The title is an allusion to David Marr’s famous “raw primal sketch,” which is a set of geometric primitives extracted from and representing an image.) The Asada–Brady paper led to a long series of increasingly sophisticated attempts to represent shape from curvature [70, 71] and to compute curvature correctly [186]. The shape recognition programme we sketched in the beginning of this chapter was anticipated in a visionary paper by Attneave [20] and has been very recently fully developed in the works of José Luis Lisani, Pablo Musé, Frédéric Sur, Yann Gousseau and Frédéric Cao [188], [190], [39], [40].

Curve shortening. The mathematical study of the intrinsic heat equation (or curvature motion in two dimensions) was done is a series of brilliant papers in differential geometry between 1983 and 1987. We repeat a few of the titles, which indicate the progress: There was Gage [88] and Gage [89]: “Curve shortening makes convex curves circular.” Then there was Gage and Hamilton [90]: “The heat equation shrinking convex plane curves.” In this paper the authors showed that a plane convex curve became asymptotically close to a shrinking circle. In 1987 there was the paper by Epstein and Gage [72], and, in the same year, Grayson removed the convexity condition and finished the job [98]: “The heat equation shrinks embedded plane curves to round points.” As the reviewer, U. Pinkall, wrote, “This paper contains the final solution of the long-standing curve-shortening problem for plane curves.”

The first papers that brought curve shortening (and some variations) to image analysis were by Kimia, Tannenbaum, and Zucker [130] and by Mackworth and Mokhtarian [154]. Curve shortening was introduced as a way to do a multiscale analysis of curves, which were considered as shapes extracted from an image. In the latter paper, curve shortening was proposed as an efficient numerical tool for multiscale shape analysis.

Affine-invariant curve shortening. Affine-invariant geometry seems to have been founded by W. Blaschke. His three-volume work “Vorlesungen über Differentialgeometrie” (1921–1929) contains definitions of affine length and affine curvature. Curves with constant affine curvature are discussed in [155]. The term “affine shortening” and the corresponding curve evolution equation were introduced by Sapiro and Tannenbaum in [221]. Several mathematical properties were developed by the same authors in [222] and [223]. Angenent, Sapiro, and Tannenbaum gave the first existence and uniqueness proof of affine shortening in [14] and prove a theorem comparable to Grayson’s theorem : they prove that a shape eventually becomes convex and thereafter evolves towards an ellipse before collapsing.

Mean curvature motion. In his famous paper entitled “Shapes of worn stones,” Firey proposed a model for the natural erosion of stones on a beach [81]. He suggested that the rate of erosion of the surface of a stone was proportional to the Gaussian curvature of the surface, so that areas with high Gaussian curvature eroded faster than areas with lower curvature, and he conjectured that the final shape was a sphere. The first attempt at a mathematical definition of the mean curvature motion is found in Brakke [31]. Later in the book, we will discuss the Sethian’s clever numerical implementation of the same equation “JMMBookOct04” 30/10/2009 page 258

[figure] [figure]

258 CHAPTER 19. THE MAIN CURVATURE EQUATIONS

[233]. Almgren, Taylor, Wang proposed a more general formulation of mean curvature motion that is applicable to crystal growth and, in general, to the evolution of anisotropic solids [5].

“JMMBookOct04” 30/10/2009 page 259

[figure] [figure]

Chapter 20

Finite Difference Schemes for Curvature Motions

We shall consider the classical discrete representation of an image u on a grid u_{i,j} = u(i, j), with 1 ≤ i ≤ N, 1 ≤ j ≤ N . The image is the union of the squares centered at the points (i, j).

20.1 Case of Mean curvature motion.

We start with the “Mean curvature motion” equation (M.C.M.) given by

\frac{∂u}{∂t} = |Du|curv(u) = \frac{u_{y}^{2}u_{xx} − 2u_{x}u_{y}u_{xy} + u_{x}^{2}u_{yy}}{u_{x}^{2} + u_{y}^{2}}

In order to discretize this equation by finite differences we shall introduce an explicit scheme which uses a fixed 3 × 3 stencil to discretize the differential operators. We denote by Δx = Δy the pixel width. From the PDE viewpoint Δx is considered as an infinitesimal length with respect to the image scale. Thus we shall write formulas containing o(Δx). Numerically Δx is equal to 1, and the image scale ranges from 512 to 4096 and more. By the order 1 Taylor formula one can give the following discrete versions of the first derivatives u_{x} and u_{y} at a point (i, j) of the grid:

(u_{x})_{i,j} = \frac{2(u_{i+1,j} − u_{i−1,j}) + u_{i+1,j+1} − u_{i−1,j+1} + u_{i+1,j−1} − u_{i−1,j−1}}{8Δx} +O(Δx^{2});

(u_{y})_{i,j} = \frac{2(u_{i,j+1} − u_{i,j−1}) + u_{i+1,j+1} − u_{i+1,j−1} + u_{i−1,j+1} − u_{i−1,j−1}}{8Δx} +O(Δx^{2});

|Du_{i,j}| = ((u_{x})_{i,j}^{2} + (u_{y})_{i,j}^{2})^{\frac{1}{2}} .

Definition 20.1. A discrete scheme approximating a differential operator is said to be consistent if, when the grid mesh Δx tends to zero, the discrete scheme tends to the differential operator.

259 “JMMBookOct04” 30/10/2009 page 260

[figure] [figure]

260 CHAPTER 20. FINITE DIFFERENCE SCHEMES

Figure 20.1: A 3 × 3 stencil

Clearly the above discrete versions of the partial derivatives and of the gradient of u are consistent. When |Du| ≠ 0, we can denote by ξ the direction orthogonal to the gradient of u. It is easily deduced from Definition 18.14 that

|Du|curv(u) = u_{ξξ}.

Exercise 20.1. Show this formula.

Defining θ as the angle between the x direction and the gradient, we have

ξ = (− sin θ, cos θ) = (\frac{−u_{y}}{\sqrt{u_{x}^{2} + u_{y}^{2}}} , \frac{u_{x}}{\sqrt{u_{x}^{2} + u_{y}^{2}}} ), and

u_{ξξ} = sin^{2}(θ)u_{xx} − 2 sin(θ) cos(θ)u_{xy} + cos^{2}(θ)u_{yy}.  (20.1)

We would like to write u_{ξξ} as a linear combination of the values of u on the fixed 3×3 stencil. Of course, the coefficients of the linear combination will depend on ξ. Since the direction of ξ is defined modulo π, we must assume by symmetry that the coefficients of points symmetrical with respect to the central point of the stencil are equal (see Figure 20.1.)

In order to ensure consistency with the differential operator u_{ξξ}, we must find λ_{0}, λ_{1}, λ_{2}, λ_{3}, λ_{4}, such that

(u_{ξξ})_{i,j} = \frac{1}{Δx^{2}} (−4λ_{0}u_{i,j} + λ_{1}(u_{i+1,j} + u_{i−1,j}) + λ_{2}(u_{i,j+1} + u_{i,j−1})

+λ_{3}(u_{i−1,j−1} + u_{i+1,j+1}) + λ_{4}(u_{i−1,j+1} + u_{i+1,j−1})) + ε(Δx).  (20.2)

We write

u_{i+1,j} = u_{i,j} + Δx(u_{x})_{i,j} + \frac{Δx^{2}}{2} (u_{xx})_{i,j} + o((Δx)^{3}),

and the corresponding relations for the other points of the stencil. By substituting these relations into (20.2) and by using (20.1) one obtains four links between the five coefficients, namely

\{ \begin{matrix}λ_{1}(θ) = 2λ_{0}(θ) − sin^{2} θ \\ λ_{2}(θ) = 2λ_{0}(θ) − cos^{2} θ \\ λ_{3}(θ) = −λ_{0}(θ) + 0.5(sin θ cos θ + 1) \\ λ_{4}(θ) = −λ_{0}(θ) + 0.5(− sin θ cos θ + 1) \end{matrix}  (20.3)

“JMMBookOct04” 30/10/2009 page 261

[figure] [figure]

20.1. CASE OF MEAN CURVATURE MOTION. 261

Exercise 20.2. Prove these four relations.

Thus, one degree of freedom is left for our coefficients : we can for example choose λ_{0}(θ) as we wish. This choice will be driven by stability and geometric invariance requirements. Denoting by u_{i,j}^{n} an approximation of u(iΔx, jΔx, nΔt) we can write our explicit scheme as

u_{i,j}^{n+1} = u_{i,j}^{n} + Δt(u_{ξξ}^{n})_{i,j}  (20.4)

Notice that this scheme can be rewritten as u_{i,j}^{n+1} = ∑_{k,l=−1}^{1} α_{k,l}u_{i+k,j+l}^{n} where the α_{k,l} satisfy ∑_{k,l=−1}^{1} α_{k,l} = 1. The following obvious lemma shows a general condition to have L^{∞} stability in this kind of scheme.

Lemma 20.2. Let a finite difference scheme given by

T (u)_{i,j} = ∑_{k,l=−1}^{1} α_{k,l}u_{i+k,j+l}

where α_{k,l} satisfy ∑_{k,l=−1}^{1} α_{k,l} = 1. We say that the scheme is L^{∞}-stable if for all i, j,

min_{i,j} u(i, j) ≤ T (u)_{i,j} ≤ max_{i,j} u(i, j).

Then the scheme is L^{∞} stable if and only if α_{k,l} ≥ 0 for any k, l.

Proof. If α_{k,l} ≥ 0 for any k, l, set min = inf_{i,j}{u_{i,j}}, max = sup_{i,j}{u_{i,j}} and take a point (i, j). Then the L^{∞} stability follows from the inequality:

min = ∑_{k,l=−1}^{1} α_{k,l}min ≤ ∑_{k,l=−1}^{1} α_{k,l}u_{i+k,j+l} = (T u)_{i,j} ≤ ∑_{k,l=−1}^{1} α_{k,l}max = max

On the other hand, if there exists α_{k_{0},l_{0}} < 0 then choosing u and (i, j) such that u_{i+k_{0},j+l_{0}} = min and u_{i+k,j+l} = max for any other k, l, we obtain

(T u)_{i,j} = ∑_{k≠k_{0},l≠l_{0}}^{1} α_{k,l}max + α_{k_{0},l_{0}}min = max + α_{k_{0},l_{0}}(min − max) > max,

which means that the L^{∞} stability is violated.

□

Following this lemma, in order to guarantee the L^{∞} stability in the scheme (20.4) we should look for λ_{0} such that λ_{1}, λ_{2}, λ_{3}, λ_{4} ≥ 0 and (1 − \frac{4λ_{0}}{Δx^{2}} ) ≥ 0. Unfortunately the links between these coefficients make it impossible to obtain these relations, except for the particular values of θ = (0, \frac{π}{4} , \frac{π}{2} , ...). Indeed, for θ in [0, \frac{π}{4} ],

λ_{1} ≥ λ_{2}  and  λ_{3} ≥ λ_{4}

But

λ_{2}(θ) ≥ 0 ⇒ λ_{0}(θ) ≥ \frac{cos^{2}(θ)}{2}

λ_{4}(θ) ≥ 0 ⇒ λ_{0}(θ) ≤ \frac{1 − sin(θ) cos(θ)}{2}

“JMMBookOct04” 30/10/2009 page 262

[figure] [figure]

262 CHAPTER 20. FINITE DIFFERENCE SCHEMES

0.25 0.3 0.35 0.4 0.45 0.5 0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8 λ_{0}(θ)  θ Search of the optimal λ_{0} 0.5 − cos^{2}(θ) + cos^{4}(θ) cos^{2}(θ)/2 (1 − sin(θ) cos(θ))/2

Figure 20.2: The middle curve represents the choice of the function λ_{0} of Formula 20.6. The upper function represents the smallest possibility for λ_{0}(θ) securing λ_{2} ≥ 0 for all angles and the lower one represents the largest values of λ_{0}(θ) securing λ_{4}(θ) ≥ 0. Thus, it is not possible to satisfy simultaneously both conditions. The intermediate curve is the simplest trigonometric function which lies between these two bounds.

We cannot find λ_{0}(θ) satisfying both inequalities, since

\frac{cos^{2}(θ)}{2} ≥ \frac{1 − sin(θ) cos(θ)}{2}

If we chose λ_{0}(θ) ≥ \frac{cos^{2}(θ)}{2}, λ_{4}(θ) would be significantly below zero. If we took λ_{0}(θ) ≤ \frac{1−sin(θ) cos(θ)}{2}, λ_{2}(θ) would be significantly below zero. Thus we shall choose λ_{0} somewhere between both functions, so that λ_{2} and λ_{4} become only slightly negative. (see Figure 20.2.)

In addition, we can try to impose on λ_{0} the following geometrical requirements

(i). Invariance by rotation of angle \frac{π}{2}

λ_{0}(θ + \frac{π}{2} ) = λ_{0}(θ)

(ii). Purely one-dimensional diffusion in the case θ = 0, \frac{π}{2} , ...

λ_{0}(0) = 0.5
This condition implies that λ_{2}(0) = λ_{3}(0) = λ_{4}(0) = 0

“JMMBookOct04” 30/10/2009 page 263

[figure] [figure]

20.1. CASE OF MEAN CURVATURE MOTION. 263

(iii). Pure one-dimensional diffusion in the case θ = \frac{π}{4} , \frac{3π}{4} , ...

λ_{0}(\frac{π}{4} ) = 0.25

This condition implies that λ_{1}(\frac{π}{4} ) = λ_{2}(\frac{π}{4} ) = λ_{4}(\frac{π}{4} ) = 0

(iv). Symmetry with respect to the axes i+j and i-j,

λ_{0}(\frac{π}{2} − θ) = λ_{0}(θ)

We remark that by the above conditions it is enough to define the function λ_{0}(θ) in the interval [0, \frac{π}{4} ] because it can be extended by periodicity elsewhere.

Two choices for the function λ_{0}(θ) using as basis the trigonometric polynomials were tested. The first one corresponds to an average of the boundary functions:

λ_{0}(θ) = \frac{cos^{2}(θ) + 1 − sin(θ)cos(θ)}{4}  (20.5)

As we shall see this choice is well-adapted to the “affine curvature motion” equation. However, if we extend this function by periodicity, the extended function is not smooth at \frac{π}{4} . If we seek for a smooth function for λ_{0}(θ), we must impose λ′_{0}(0) = λ′_{0}(\frac{π}{4} ) = 0. The trigonometric polynomial with least degree satisfying the above conditions and lying between both boundary functions is

λ_{0}(θ)) = 0.5 − cos^{2}(θ) sin^{2}(θ)  (20.6)

The formulas of the other λ_{i}’s are deduced using (20.3). For instance with the above choice of λ_{0}(θ) we have

λ_{1}(θ) = cos^{2}(θ)(cos^{2}(θ) − sin^{2}(θ));
λ_{2}(θ) = sin^{2}(θ)(sin^{2}(θ) − cos^{2}(θ));
λ_{3}(θ) = cos^{2}(θ) sin^{2}(θ) + 0.5 sin(θ) cos(θ);
λ_{4}(θ) = cos^{2}(θ) sin^{2}(θ) − 0.5 sin(θ) cos(θ).

When |Du| = 0, the direction of the gradient is unknown. Therefore the diffusion term u_{ξξ} is not defined. We chose to replace this term by half the Laplacian. (The Laplacian is equal to the sum of the two second derivatives in orthogonal directions, whereas the diffusion term u_{ξξ} is the second derivative in just one). However, other possibilities will be considered in Section 20.6. Summarizing, a consistent, almost L^{∞} stable finite difference scheme for the mean curvature motion is (iterations start with u^{0} as initial function)

1. If |Du| ≥ T_{g}

u^{n+1} = u^{n} + \frac{Δt}{Δx^{2}} (−4λ_{0}u_{i,j} + λ_{1}(u_{i+1,j} + u_{i−1,j}) + λ_{2}(u_{i,j+1} + u_{i,j−1})

+λ_{3}(u_{i−1,j−1} + u_{i+1,j+1}) + λ_{4}(u_{i−1,j+1} + u_{i+1,j−1})).

“JMMBookOct04” 30/10/2009 page 264

[figure] [figure]

264 CHAPTER 20. FINITE DIFFERENCE SCHEMES

2. Otherwise,

u^{n+1} = u^{n} + \frac{1}{2} \frac{Δt}{Δx^{2}} (−4λ_{0}u_{i,j} + u_{i+1,j} + u_{i−1,j} + u_{i,j+1} + u_{i,j−1})

Two parameters have to be fixed in the previous algorithm:

• The iteration step scale s := \frac{Δt}{Δx^{2}} has to be chosen as large as possible in order to reduce the number of iterations. However, 1/2 is a natural upper bound for s. Indeed, consider the discrete image defined by u_{i,j}^{0} = 0 for all i, j, except for i = j = 0 where u_{0,0}^{0} = 1. Then the second formula yields u_{0,0}^{1} = 1 − 2 ∗ s. If we want L^{∞} stability to be ensured we must have u^{1}(0, 0) ≥ 0, which yields s ≤ 1/2. Imposing this condition

\frac{Δt}{Δx^{2}} ≤ \frac{1}{2}  (20.7)

it is an experimental observation that there is a (small with respect to 255) ϵ > 0 such that for any n ∈ IN and (i, j),

−ϵ + inf_{i,j}{u_{i,j}^{0}} ≤ u_{i,j}^{n} ≤ sup_{i,j}{u_{i,j}^{0}} + ϵ.

• The threshold on the spatial gradient norm : T_{g} has been fixed experimentally to 6 for 0 to 255 images.

Figure 20.3: Curvature motion finite difference scheme and scale calibration. Image filtered by curvature motion at scales 1, 2, 3, 4, 5. In order to give a sound numerical meaning to the scale, a calibration of the numerical scales (number of iterations) is made in such a way that a disk with radius t shrinks to a point at scale t.

20.2 FDS for AMSS

We will use the ideas developed in the above section. We rewrite the AMSS equation as “JMMBookOct04” 30/10/2009 page 265

[figure] [figure]

20.2. FDS FOR AMSS 265

Figure 20.4: Curvature motion finite difference scheme applied on each level set separately, at scales 1, 2, 3, 4, 5. The processed image is then reconstructed by the threshold superposition principle. In contrast with the same scheme directly applied on the image, this scheme yields a fully contrast invariant smoothing. However, a comparison with Figure 20.3 shows that the resulting images are very close to each other. This shows that the contrast invariance is almost achieved when applying the finite difference scheme directly on a good quality image. As we shall see in Figure 20.6, if the initial image is noisy, the difference between both methods can be huge.

Figure 20.5: Iterated median filter with normalized scales 1, 2, 3, 4, 5. The scale normalization permits to compare very different schemes on the same images. Compare with Figure 20.4. The striking similarity of the results anticipates Theorem ??, according to which the application of the median filter is equivalent to a mean curvature motion.

\frac{∂u}{∂t} = (|Du|^{3}curv(u))^{\frac{1}{3}} = (u_{y}^{2}u_{xx} − 2u_{x}u_{y}u_{xy} + u_{x}^{2}u_{yy})^{\frac{1}{3}}  (20.8)

“JMMBookOct04” 30/10/2009 page 266

[figure] [figure]

266 CHAPTER 20. FINITE DIFFERENCE SCHEMES

We remark that |Du|^{3}curv(u) = |Du|^{2}u_{ξξ} where ξ corresponds to the direction orthogonal to the gradient. Therefore, in order to discretize this operator, it is enough to multiply the discretization of u_{ξξ} presented in the above section by |Du|^{2}. We choose λ_{0}(θ) given by (20.5) because it corresponds to a trigonometric polynomial of degree two and then multiplying it by |Du|^{2} the coefficients η_{i} = |Du|^{2}λ_{i}, i = 0, 1, 2, 3, 4, are polynomials of degree two with respect to u_{x} and u_{y}. Indeed, we obtain for θ ∈ [0, \frac{π}{4} ]

(|Du|^{2}u_{ξξ})_{i,j} = \frac{1}{Δx^{2}} (−4η_{0}u_{i,j} + η_{1}(u_{i+1,j} + u_{i−1,j}) + η_{2}(u_{i,j+1} + u_{i,j−1})

+η_{3}(u_{i−1,j−1} + u_{i+1,j+1}) + η_{4}(u_{i−1,j+1} + u_{i+1,j−1})) + O(Δx^{2})

where η_{0}, η_{1}, η_{2}, η_{3}, η_{4} are given by

\{ \begin{matrix}η_{0} = 0.25(2u_{x}^{2} + u_{y}^{2} − u_{x}u_{y}) \\ η_{1} = 0.5(2u_{x}^{2} − u_{y}^{2} − u_{x}u_{y}) \\ η_{2} = 0.5(u_{y}^{2} − u_{x}u_{y}) \\ η_{3} = 0.25(u_{y}^{2} + 3u_{x}u_{y}) \\ η_{4} = 0.25(u_{y}^{2} − u_{x}u_{y}) \end{matrix}

Finally, the finite difference scheme for the A.M.S.S. equation is

u_{i,j}^{n+1} = u_{i,j}^{n} + Δt(|Du^{n}|^{2}u_{ξξ}^{n})i,j^{\frac{1}{3}}  (20.9)

We have tested this algorithm and we have noticed that in this case the condition for the experimental stability (in the sense presented in the above subsection) is

\frac{Δt}{Δx^{2}} ≤ \frac{1}{10}.

Remark. The finite difference schemes presented above are consistent. Contrast invariance can only be obtained asymptotically by taking a very small time step Δt. The experimental results presented in Figures 20.3 and ?? have been obtained by using these schemes with Δt = 0.1 in the case of mean curvature motion and Δt = 0.01 in the case of affine curvature motion. Indeed, while experimental stability is achieved with Δt ≤ 0.1, the experimental affine invariance needs Δt < 0.05 (see Figure ??.)

20.3 IL MANQUE UNE EXPERIENCE AMSS SUR L’INVARIANCE AFFINE!

20.4 Numerical normalization of scale.

(or Relation between scale and the number of iterations).

“JMMBookOct04” 30/10/2009 page 267

[figure] [figure]

20.5. CONTRAST INVARIANCE AND THE LEVEL SET EXTENSION 267

The case of the curvature motion. Setting the distance between pixels Δx to 1, the scale achieved with N iterations is simply N × Δt. Now, the scale t associated with the PDE is somewhat arbitrary : It has no geometric meaning. In order to get it, we need a rescaling T → t(T) which we will call scale normalization.

A good way to perform this scale normalization is to define the correspondence t(T) as the time for which a circle with initial radius T vanishes under curvature motion. Such a circle moves at a speed equal to its curvature, which is the inverse of its radius. Thus have for a disk with radius R(t)

\frac{dR(t)}{dt} = −\frac{1}{R(t)}

which yields

\frac{1}{2}(R^{2}(0) − R^{2}(t)) = t.

Exercise 20.3. Check this relation!

The disk disappears when R(t) = 0, that is, at scale T = R^{2}(0)/2. This last relation gives a scale normalization: In order to arrive at the normalized scale T (at which any disk with radius less or equal to T vanishes), we have to evolve the PDE at t = NΔt = T^{2}/2. This fixes the number of needed iterations as

N = T^{2}/2Δt.

The case of AMSS We can perform similar calculations. The radius of an evolving disk satisfies

\frac{dR(t)}{dt} = −\frac{1}{R(t)^{\frac{1}{3}}}

which yields

\frac{3}{4}(R^{\frac{4}{3}} (0) − R^{\frac{4}{3}} (t)) = t

The disappearance time is therefore t = \frac{3}{4}R^{\frac{4}{3}} . As for the curvature motion, we define the normalized scale T as the one at which a disk with radius T vanishes. In order to achieve this scale T, the needed number of iterations is

N = \frac{3}{4Δt}T^{\frac{4}{3}} .

Exercise 20.4. Check the last two formulas!

20.5 Contrast invariance and the level set extension

Both schemes (M.C.M and A.M.S.S) presented above are not numerically contrast invariant. We have seen that a contrast operator cannot create new gray levels (Exercise 14.22.) Now, starting with a binary image u^{0} and applying a scheme defined by such a formula as

u^{n+1} = u^{n} + Δt(...)

“JMMBookOct04” 30/10/2009 page 268

[figure] [figure]

268 CHAPTER 20. FINITE DIFFERENCE SCHEMES

does not ensure that u^{n+1} will be also a binary image.

A natural idea to overcome this problem is the following. Starting with a binary image (with values 0 and 1): apply the scheme until the expected scale is achieved, then threshold the obtained image at λ = \frac{1}{2}. This of course works only for binary images. However, the level set extension (see Section 14.3) gives us the key to extend this to general images.

The contrast invariance can be fully obtained by first applying the finite difference scheme on each level set (considered as a binary image) separately. Then by the superposition principle the evolved image is computed from the evolved level sets. The procedure is the following :

Algorithm starting with an image u_{0} and evolving it to u(t, x) by curvature motion

For each λ ∈ [0, 255], in increasing order:

• Let v_{λ}(x) be the characteristic function of X_{λ}u_{0}. (This function is equal to 1 inside the level set and to 0 outside.)

• Apply to v_{λ} the MCM or AMSS FDS-scheme until scale t. This yields the images w_{λ}(t, .).

• Set u(t, x) = λ at each point (t, x) where w_{λ}(t, x) ≥ 0.5.

20.6 Problems at extrema

For MCM and AMSS we raised the question of performing numerically the equation when |Du| = 0. For MCM the right hand part of the equation is simply not defined. For AMSS one can set by continuity as Du → 0, (|Du^{n}|^{2}u_{ξξ}^{n})i,j^{\frac{1}{3}} = 0. Now, numerically, this would imply that isolated black or white extrema will not evolve by the equation. We know that this is simply wrong, since small sets collapse by curvature motion.

In short, FDS for MCM and AMSS are not consistent with the equation at extrema. In Figure 20.6, we added to an image a strong “salt and pepper” noise. More than one fourth of the pixels have been given a uniform random value in [0, 255] and most of them have become local extrema. Not only these values do not evolve but they contaminate their neighboring pixels. There are easy ways to avoid this spurious effect :

• One can first zoom by 2 the image by duplicating pixels. This, however, multiplies by 16 the number of computations.

• One can first remove pixels extrema with diameter k since they must anyway disappear by the equation at normalized scale \frac{k}{2} .

• One can use the level set method. This multiplies the number of computations by the initial number of gray-levels.

All of these solutions are efficient, as shown in Figure 20.6.

“JMMBookOct04” 30/10/2009 page 269

[figure] [figure]

20.6. PROBLEMS AT EXTREMA 269

Figure 20.6: Various implementations of curvature motion on a noisy image. Top left : image with 40% pixels replaced by a uniform random value in [0, 255]. Top right: application of the finite difference scheme (FDS) at normalized scale 3. On the lines 2 to 4, we see various solutions to the disastrous diffusion of extrema. On the left the image is processed at normalized scale 1 and on the right at normalized scale 3. Second line: FDS applied on the image previously zoomed by a factor 2; third line: FDS applied on the image after its extrema have been ”killed” (the reference area is given by the area of the disk vanishing at the desired scale). Fourth line: FDS applied separately on each level set and application of the threshold superposition principle. The third scheme offers a good speed-quality compromise.

“JMMBookOct04” 30/10/2009 page 270

[figure] [figure]

270 CHAPTER 20. FINITE DIFFERENCE SCHEMES

20.7 Conclusion

We have seen that standard finite difference schemes are easy to implement but cannot handle properly the invariance properties satisfied by the equations.

1. There is no finite difference scheme that insures the monotonicity. This leads to slightly oscilatory solutions.

2. No full contrast invariance. For instance FDS create new grey levels and blur edges. Also, a spurious diffusion occurs around the image extrema. However this last problem was dealt with efficiently in the previous section. The full contrast invariance has been restored by the level set extension of the numerical schemes.

3. The worst drawback of FDS is the lack of Euclidean or affine invariance which can be only approximately obtained by grid local schemes. A much more clever strategy to achieve full invariance is to evolve all level curves of the image and the reconstruct it. This is the aim of Section ??, but we have already seen in Chapter 8 how to evolve curves by curvature.

20.8 Comments and references

Difference schemes for the curvature motion and the AMSS The presented difference scheme follows mainly [99], improved in Alvarez et al. [9]. This scheme is somehow optimal among the rotationally invariant numerical schemes for curvature motion and the AMSS. Now, this presentation is specific of those two motions, while other many authors have analysed more general nonlinear anisotropic diffusions in image processing, namely Acton [2], Kacur and Mikula [122, 123]. Weickert and the Ütrecht school [195, 255, 1, 258] address many aspects of implementation of nonlinear scale spaces, namely speed, parallelism and robustness. Crandall and Lions [61] also proposed a finite difference scheme for mean curvature motion, valid in any dimension. Sethian’s book [232] explains how to implement fast the motion of a curve or surface by the so called ”level set method”, where a distance function to the curve or surface is evolved. Dynamic programming allows a fast implementation (the ”fast marching method”).

“JMMBookOct04” 30/10/2009 page 271

[figure] [figure]

Bibliography

[1] Parallel implementations of AOS schemes: A fast way of nonlinear diffusion filtering.

[2] S. Acton, Multigrid anisotropic diffusion, Image Processing, IEEE Transactions on, 7 (1998), pp. 280–291.

[3] A. Agarwala, M. Agrawala, M. Cohen, D. Salesin, and R. Szeliski, Photographing long scenes with multi-viewpoint panoramas, International Conference on Computer Graphics and Interactive Techniques, (2006), pp. 853–861.

[4] G. J. Agin, Representation and Description of Curved Objects, PhD thesis, Stanford U., October 1972.

[5] F. Almgren, J. E. Taylor, and L. Wang, Curvature-driven flows: A variational approach, SIAM J. Control Optimization, 31 (1993), pp. 387– 438.

[6] L. Alvarez, Y. Gousseau, and J.-M. Morel, The size of objects in natural and artificial images, in Advances in Imaging and Electron Physics, vol. 111, Academic Press, 1999, pp. 167–242.

[7] L. Alvarez, F. Guichard, P.-L. Lions, and J.-M. Morel, Axioms and fundamental equations of image processing, Arch. Rational Mech. Anal., 16 (1993), pp. 200–257.

[8] L. Alvarez, P.-L. Lions, and J.-M. Morel, Image selective smoothing and edge detection by nonlinear diffusion (ii), SIAM J. Numer. Anal., 29 (1992), pp. 845–866.

[9] L. Alvarez and J.-M. Morel, Formalization and computational aspects of image analysis, Acta Numerica, (1999), pp. 1–59.

[10] L. Ambrosio, V. Caselles, S. Masnou, and J.-M. Morel, The connected components of caccioppoli sets and applications to image processing, J. Eur. Math. Soc., 3 (2001), pp. 213–266.

[11] L. Ambrosio, V. Caselles, S. Masnou, and J.-M. Morel, Connected components of sets of finite perimeter and applications to image processing, J. Eur. Math. Soc., 3 (2001), pp. 39–92.

271 “JMMBookOct04” 30/10/2009 page 272

[figure] [figure]

272 BIBLIOGRAPHY

[12] F. Andreu, C. Ballester, V. Caselles, and J. M. Mazon, Minimizing total variation flow, C. R. Acad. Sci. Paris Sér. I, 331 (2000), pp. 867–872.

[13] F. Andreu, C. Ballester, V. Caselles, and J. M. Mazon, Minimizing total variation flow, Differential and Integral Equations, 4 (2001), pp. 321–360.

[14] S. B. Angenent, G. Sapiro, and A. Tannenbaum, On the affine heat flow for nonconvex curves, J. Amer. Math. Soc., 11 (1998), pp. 601–634.

[15] A. Anjulan and N. Canagarajah, Affine Invariant Feature Extraction Using Symmetry, Advanced Concepts for Intelligent Vision Systems: 7th International Conference, ACIVS 2005, Antwerp, Belgium, September 20-23, 2005: Proceedings, (2005).

[16] G. R. Arce and M. P. McLoughlin, Theoretical analysis of the max/median filter, 35 (1987), pp. 60–69.

[17] H. Asada and M. Brady, The curvature primal sketch, IEEE Trans. Patt. Anal. Mach. Intell., 8 (1986), pp. 2–14.

[18] A. Ashbrook, N. Thacker, P. Rockett, and C. Brown, Robust recognition of scaled shapes using pairwise geometric histograms, Proc. BMVC, (1995), pp. 503–512.

[19] J. Astola, P. Heinonen, and Y. Neuvo, On root structures of median and median-type filters, IEEE Trans. Acoust. Speech Signal Process., 35 (1987), pp. 1199–1201.

[20] F. ATTNEAVE, Some informational aspects of visual perception., Psychol Rev, 61 (1954), pp. 183–93.

[21] C. Ballester, V. Caselles, and P. Monasse, The tree of shapes of an image, tech. rep., CMLA, Ecole Normale Supérieure-Cachan, France, 2001. Available at http://www.cmla.ens-

cachan.fr/Cmla/index.html.

[22] C. Ballester, E. Cubero-Castan, M. González, and J.-M. Morel, Fast contrast invariant image intersection. Preprint CERE-MADE, 1998.

[23] J. A. Bangham, P. Ling, and R. Young, Multiscale recursive medians, scale-space, and transforms with applications to image processing, IEEE Trans. Image Process., 5 (1996).

[24] G. Barles, Remarks on a Flame Propagation Model, Institut national de recherche en informatique et en automatique, 1985.

[25] A. Baumberg, Reliable feature matching across widely separated views, Proc. IEEE CVPR, 1 (2000), pp. 774–781.

[26] H. Bay, T. Tuytelaars, and L. Van Gool, Surf: Speeded up robust features, European Conference on Computer Vision, 1 (2006), pp. 404– 417.

“JMMBookOct04” 30/10/2009 page 273

[figure] [figure]

BIBLIOGRAPHY 273

[27] S. Belongie, J. Malik, and J. Puzicha, Shape Matching and Object Recognition Using Shape Contexts, IEEE Trans. Pattern Anal. Mach. Intell., (2002).

[28] M. Bennewitz, C. Stachniss, W. Burgard, and S. Behnke, Metric Localization with Scale-Invariant Visual Features Using a Single Perspective Camera, European Robotics Symposium 2006, (2006).

[29] M. J. Black, G. Sapiro, D. H. Marimont, and D. Heeger, Robust anisotropic diffusion, IEEE Trans. Image Process., 7 (1998), pp. 421–432.

[30] J. Blom, B. T. te Haar Romeny, A. Bel, and J. J. Koenderink, Spatial derivatives and the propagation of noise in Gaussian scale-space, Tech. Rep. 3DCV 91-03, U. of Utrecht, Dept. of Medical and Physiological Physics, 1991.

[31] K. A. Brakke, The motion of a surface by its mean curvature, Mathematical Notes, 20, Princeton University Press, Princeton, NJ, 1978.

[32] L. Breiman, Probability, Addison-Wesley, 1968.

[33] H. Brezis, Analyse Fonctionnelle, Théorie et Applications, Masson, Paris, 1983.

[34] M. Brown and D. Lowe, Recognising panoramas, Proc. ICCV, 1 (2003), p. 3.

[35] D. R. K. Brownrigg, The weighted median filter, Communications of the ACM, 27 (1984), pp. 807–818.

[36] P. J. Burt and E. H. Adelson, The laplacian pyramid as a compact image code, IEEE Trans. Comm., 31 (1983), pp. 532–540.

[37] A. R. Butz, A class of rank order smoothers, IEEE Trans. Acoust. Speech Signal Process., 34 (1986), pp. 157–165.

[38] J. F. Canny, A computational approach to edge detection, IEEE Trans. Patt. Anal. Mach. Intell., 8 (1986), pp. 679–698.

[39] F. Cao, J. Delon, A. Desolneux, P. Musé, and F. Sur, A unified framework for detecting groups and application to shape recognition, Rapport interne, 1746, pp. 1166–8687.

[40] F. Cao, Y. Gousseau, P. Muse, F. Sur, and J. Morel, Accurate estimates of false alarm number in shape recognition, tech. rep., Technical report, Cachan, France (http://www. cmla. enscachan. fr/Cmla/), 2004. 4, 5.

[41] F. Cao, J. Lisani, J. Morel, P. Muse, and F. Sur, A Theory of Shape Identification, Springer Verlag, 2008.

[42] L. J.-L. M. J.-M. M. P. S. F. Cao, F., A Theory of Shape Identification, Springer Verlag, 2008.

[43] R. A. Carmona and S. F. Zhong, Adaptive smoothing respecting feature directions, IEEE Trans. Image Process., 7 (1998), pp. 353–358.

“JMMBookOct04” 30/10/2009 page 274

[figure] [figure]

274 BIBLIOGRAPHY

[44] V. Caselles, F. Catté, T. Coll, and F. Dibos, A geometric model for active contours, Numerische Mathematik, 66 (1993), pp. 1–31.

[45] V. Caselles, B. Coll, and J.-M. Morel, Topographic maps and local contrast changes in natural images, International Journal of Computer Vision, volume=.

[46] V. Caselles, B. Coll, and J.-M. Morel, Topographic maps and local contrast changes in natural images, Int. J. Comput. Vision, 33 (1999), pp. 5–27.

[47] V. Caselles, R. Kimmel, and G. Sapiro, Geodesic active contours, Int. J. Comput. Vision, 1 (1997), pp. 61–79.

[48] V. Caselles, J. L. Lisani, J.-M. Morel, and G. Sapiro, Shape preserving local histogram modification, 1999.

[49] V. Caselles, J.-M. Morel, and C. Sbert, An axiomatic approach to image interpolation, IEEE Trans. Image Process., 7 (1998), pp. 376–386.

[50] V. Caselles, G. Sapiro, and D. H. Chung, Vector median filters, vector morphology, and coupled PDE’s: Theoretical connections, J. Math. Imaging Vision, 12 (2000), pp. 109–120.

[51] E. Chang, EXTENT: fusing context, content, and semantic ontology for photo annotation, Proceedings of the 2nd international workshop on Computer vision meets databases, (2005), pp. 5–11.

[52] R. C. Chen and P. T. Yu, On the optimal design of rational rank selection filters for image restoration, in Proceedings International Conference on Image Processing, 1996, p. 16A9.

[53] Y. G. Chen, Y. Giga, and S. Goto, Uniqueness and existence of viscosity solutions of generalized mean curvature flow equations, Proc. Japan Acad. Ser. A Math. Sci., 65 (1989), pp. 207–210.

[54] Y. G. Chen, Y. Giga, and S. Goto, Uniqueness and existence of viscosity solutions of generalized mean curvature flow equations, J. Differential Geom., 33 (1991), pp. 749–786.

[55] F. Cheng and A. N. Venetsanopoulos, An adaptive morphological filter for image processing, IEEE Trans. Image Process., 1 (1992), pp. 533– 539.

[56] H. Cornelius and G. Loy, Detecting Bilateral Symmetry in Perspective, 5th Workshop on Perceptual Organization in Computer Vision, (2006), pp. 191–198.

[57] R. Courant and F. John, Introduction to Calculus and Analysis, Springer, 2000. Reprint of the 1989 edition.

[58] I. Cox, S. Roy, and S. Hingorani, Dynamic histogram warping of image pairs for constant image brightness, Image Processing, 1995. Proceedings, International Conference on, 2 (1995).

“JMMBookOct04” 30/10/2009 page 275

[figure] [figure]

BIBLIOGRAPHY 275

[59] E. Coyle, N. Gallagher, S. Bass, J. Fitch, and R. Harber, Digital filtering by threshold decomposition, 1987. U.S. Patent 4,868,773.

[60] E. J. Coyle, Rank order operators and the mean absolute error criterion, IEEE Trans. Acoust. Speech Signal Process., 36 (1988), pp. 63–76.

[61] M. Crandall and P. Lions, Convergent difference schemes for nonlinear parabolic equations and mean curvature motion, Numerische Mathematik, 75 (1996), pp. 17–41.

[62] M. G. Crandall, H. Ishii, and P.-L. Lions, User’s guide to viscosity solutions of second order partial differential equations, Bull. Amer. Math. Soc. (N.S.), 27 (1992), pp. 1–67.

[63] J. Damon, Generic structure of two dimensional images under Gaussian blurring, SIAM J. Appl. Math., 59 (1998), pp. 97–138.

[64] P. E. Danielsson, Getting the median faster, Computer Graphics Image Processing, 17 (1981), pp. 71–78.

[65] J. G. Daugman, Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by 2D visual cortical filters, Journal of the Optical Society of America, 2 (1985), pp. 1160–1169.

[66] E. R. Davies, On the noise suppression and image enhancement characteristics of the median, truncated median and mode filters, Pattern Recognition Letters, 7 (1988), pp. 87–97.

[67] J. Delon, Midway Image Equalization, Journal of Mathematical Imaging and Vision, 21 (2004), pp. 119–134.

[68] J. Delon, Movie and video scale-time equalization application to flicker reduction, Image Processing, IEEE Transactions on, 15 (2006), pp. 241– 248.

[69] R. Deriche, Using canny’s criteria to derive a recursively implemented optimal edge detector, Int. J. Comput. Vision, 1 (1987), pp. 167–187.

[70] G. Dudek, Shape Representation from Curvature, PhD thesis, U. Toronto, 1990.

[71] G. Dudek and J. K. Tsotsos, Robustly recognizing curves using curvature-tuned smoothing, in IEEE Conference on Computer Vision and Pattern Recognition, 1991, pp. 35–41.

[72] C. L. Epstein and M. Gage, The curve shortening flow, in Wave motion: theory, modelling, and computation, Springer, NY, 1987, pp. 15–59. Proc. Conf. Hon. 60th Birthday P. D. Lax (Berkeley, CA, 1986) Math. Sci. Res. Inst. Publ. 7.

[73] L. C. Evans, Partial Differential Equations, Graduate Studies in Mathematics, 19, American Mathematical Society, Providence, RI, 1998.

[74] L. C. Evans and J. Spruck, Motion of level sets by mean curvature. I., J. Differential Geom., 33 (1991), pp. 635–681.

“JMMBookOct04” 30/10/2009 page 276

[figure] [figure]

276 BIBLIOGRAPHY

[75] L. C. Evans and J. Spruck, Motion of level sets by mean curvature. II., Trans. Am. Math. Soc., 330 (1992), pp. 321–332.

[76] L. C. Evans and J. Spruck, Motion of level sets by mean curvature. III., J. Geom. Anal., 2 (1992), pp. 121–150.

[77] L. C. Evans and J. Spruck, Motion of level sets by mean curvature. IV., J. Geom. Anal., 5 (1995), pp. 79–116.

[78] F. Falzon and G. Giraudon, Singularity analysis and derivative scalespace, in IEEE Computer Vision and Pattern Recognition, 1994, pp. 245– 250.

[79] Q. Fan, K. Barnard, A. Amir, A. Efrat, and M. Lin, Matching slides to presentation videos using SIFT and scene background matching, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, (2006), pp. 239–248.

[80] L. Fevrier, A wide-baseline matching library for Zeno, Technical report, (2007).

[81] W. J. Firey, Shapes of worn stones, Mathematika, 21 (1974), pp. 1–11.

[82] J. P. Fitch, E. J. Coyle, and N. C. Gallagher, Jr., Median filtering by threshold decomposition, IEEE Trans. Acoust. Speech Signal Process., 32 (1984), pp. 1183–1188.

[83] L. M. J. Florack, Image Structure, vol. 10 of Computational Imaging and Vision, Kluwer Academic Publishers, Dordrecht, NL, 1997.

[84] L. M. J. Florack, B. M. ter Haar Romeny, J. J. Koenderink, and M. A. Viergever, Linear scale-space, J. Math. Imaging Vision, 4 (1994), pp. 325–351.

[85] J. Foo and R. Sinha, Pruning SIFT for scalable near-duplicate image matching, Proceedings of the eighteenth conference on Australasian database-Volume 63, (2007), pp. 63–71.

[86] G. Fritz, C. Seifert, M. Kumar, and L. Paletta, Building detection from mobile imagery using informative SIFT descriptors, Lecture notes in computer science, pp. 629–638.

[87] D. Gabor, Information theory in electron microscopy, Journal of Laboratory Investigation, 14 (1965), pp. 801–807. The Journal of Laboratory Investigation was published by the International Academy of Pathology.

[88] M. Gage, An isometric inequality with applications to curve shortening, Duke Math. J., 50 (1983), pp. 1225–1229.

[89] M. Gage, Curve shortening makes convex curves circular, Invent. Math., 76 (1984), pp. 357–364.

[90] M. Gage and R. S. Hamilton, The heat equation shrinking convex plane curves, J. Differential Geom., 23 (1986), pp. 69–96.

“JMMBookOct04” 30/10/2009 page 277

[figure] [figure]

BIBLIOGRAPHY 277

[91] C. Gasquet and P. Witomski, Fourier Analysis and Applications: Filtering, Numerical Computation, Wavelets, Springer Verlag, 1999.

[92] E. D. Giorgi and L. Ambrosio, Un nuovo tipo di funzionale del calcolo delle variazioni, Atti. Accad. Naz. Lincei, s.8 (1988), pp. 199–210.

[93] M. J. E. Golay, Smoothing data by least squares procedures and by filtering, IEEE Trans. Computer, 21 (1972), pp. 299–301.

[94] G. Golub and C. Van Loan, Matrix computations, 1996.

[95] R. Gonzalez and R. Woods, Digital image processing, Addison-Wesley Reading, Mass, 1987.

[96] I. Gordon and D. Lowe, What and Where: 3D Object Recognition with Accurate Pose, Lecture Notes in Computer Science, 4170 (2006), p. 67.

[97] M. Grabner, H. Grabner, and H. Bischof, Fast Approximated SIFT, in Asian Conference on Computer Vision, Hyderabad, India, 2006, pp. 918–927.

[98] M. A. Grayson, The heat equation shrinks embedded plane curves to round points, J. Differential Geom., 26 (1987), pp. 285–314.

[99] F. Guichard, Axiomatisation des analyses multi-échelles d’images et de films - Axiomatization of images and movies scale-space., PhD thesis, Univ. Paris IX-Dauphine, 1994.

[100] F. Guichard and J. Morel, Mathematical morphology almost everywhere, Proceedings of ISMM, pp. 293–303.

[101] R. M. Haralick, Digital step edges from zero-crossing of second directional derivatives, IEEE Trans. Patt. Anal. Mach. Intell., 6 (1984), pp. 58– 68.

[102] R. M. Haralick, S. R. Sternberg, and X. Zhuang, Image analysis using mathematical morphology, IEEE Trans. Patt. Anal. Mach. Intell., 9 (1987), pp. 532–550.

[103] R. C. Hardie and K. E. Barner, Rank conditioned rank selection filters for signal restoration, IEEE Trans. Image Process., 3 (1994), pp. 192–206.

[104] J. Hare and P. Lewis, Salient regions for query by image content, Image and Video Retrieval: Third International Conference, CIVR, (2004), pp. 317–325.

[105] C. Harris and M. Stephens, A combined corner and edge detector, Alvey Vision Conference, 15 (1988), p. 50.

[106] H. J. A. M. Heijmans and C. Ronse, The algebraic basis of mathematical morphology: I. dilations and erosions, Computer Vision, Graphics, and Image Processing, 50 (1990), pp. 245–295.

[107] C. W. Helstrom, Image restoration by the method of least squares, J. Opt. Soc. Amer., 57 (1967), pp. 297–303.

“JMMBookOct04” 30/10/2009 page 278

[figure] [figure]

278 BIBLIOGRAPHY

[108] J. M. Hereford and W. T. Rhodes, Nonlinear optical image filtering by time-sequential threshold decomposition, Opt. Eng., 27 (1988), pp. 274– 279.

[109] B. K. P. Horn and E. J. Weldon, Jr., Filtering closed curves, IEEE Trans. Patt. Anal. Mach. Intell., 8 (1986), pp. 665–668.

[110] C. T. Huang and O. R. Mitchell, A Euclidean distance transform using grayscale morphology decomposition, IEEE Trans. Patt. Anal. Mach. Intell., 16 (1994), pp. 443–448.

[111] T. S. Huang, G. J. Yang, and G. Y. Tang, A fast two-dimensional median filtering algorithm, IEEE Trans. Acoust. Speech Signal Process., 27 (1979), pp. 13–18.

[112] D. H. Hubel and T. N. Wiesel, Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex, J. Physiol., 160 (1962), pp. 106–154.

[113] R. A. Hummel, Representations based on zero-crossing in scale-space, in IEEE Computer Vision and Pattern Recognition, 1986, pp. 204–209.

[114] R. A. Hummel, B. Kimia, and S. W. Zucker, Gaussian blur and the heat equation: Forward and inverse solutions, in IEEE Computer Vision and Pattern Recognition, 1983, pp. 416–421.

[115] R. A. Hummel, B. Kimia, and S. W. Zucker, Deblurring gaussian blur, Computer Vision, Graphics, and Image Processing, 38 (1987), pp. 66–80.

[116] R. A. Hummel and R. Moniot, Reconstructions from zero-crossings in scale-space, IEEE Trans. Acoust. Speech Signal Process., 37 (1989), pp. 2111–2130.

[117] H. H. S. Ip, D. J. Potter, and D. S. Lebedev, Impulse noise cleaning by iterative threshold median filtering, Pattern Recognition Letters, 2 (1983), pp. 89–93.

[118] H. Ishii and P. Souganidis, Generalized motion of noncompact hypersurfaces with velocity having arbitrary growth on the curvature tensor, Tohoku Math. J. II, 47 (1995), pp. 227–250.

[119] S. Jaffard, Y. Meyer, and R. D. Ryan, Wavelets: Tools for Science and Technology, Society for Industrial and Applied Mathematics, Philadelphia, 2001.

[120] A. M. Joseph, Building local safety maps for a wheelchair robot using vision and lasers.

[121] B. Julesz, A method of coding TV signals based on edge detection, Bell System Tech. J. 38 (1959), pp. 1001–1020.

[122] J. Kacur and K. Mikula, Solution of nonlinear diffusion appearing in image smoothing and edge detection, Applied Numerical Mathematics, 17 (1995), pp. 47–59.

“JMMBookOct04” 30/10/2009 page 279

[figure] [figure]

BIBLIOGRAPHY 279

[123] J. Kacur and K. Mikula, Slowed Anisotropic Diffusion, Scale-Space Theory in Computer Vision: First International Conference, Scale-Space’97, Utrecht, The Netherlands, July 2-4, 1997: Proceedings, (1997).

[124] T. Kadir, A. Zisserman, and M. Brady, An Affine Invariant Salient Region Detector, in European Conference on Computer Vision, 2004, pp. 228–241.

[125] M. Kass, A. Witkin, and D. Terzopoulos, Snakes: Active contour models, Int. J. Comput. Vision, 1 (1998), pp. 321–331.

[126] Y. Ke and R. Sukthankar, PCA-SIFT: A more distinctive representation for local image descriptors, Proc. CVPR, 2 (2004), pp. 506–513.

[127] S. Kichenassamy, The Perona–Malik paradox, SIAM J. Appl. Math., 57 (1997), pp. 1328–1342.

[128] J. Kim, S. Seitz, and M. Agrawala, Video-based document tracking: unifying your physical and electronic desktops, Symposium on User Interface Software and Technology: Proceedings of the 17 th annual ACM symposium on User interface software and technology, 24 (2004), pp. 99– 107.

[129] V. Kim and L. Yaroslavskii, Rank algorithms for picture processing, Computer Vision, Graphics, and Image Processing, 35 (1986), pp. 234– 258.

[130] B. Kimia, A. Tannenbaum, and S. W. Zucker, On the evolution of curves via a function of curvature. I. The classical case, J. Math. Anal. Appl., 163 (1992), pp. 438–458.

[131] B. Kimia, A. R. Tannenbaum, and S. W. Zucker, Shapes, shocks, and deformations I: The components of 2-dimensional shape and the reactiondiffusion space, Int. J. Comput. Vision, 15 (1995), pp. 189–224.

[132] R. Kimmel and A. M. Bruckstein, Shape offsets via level sets., Computer Aided Design, 5 (1993), pp. 154–162.

[133] R. Kimmel, D. Shaked, N. Kiryati, and A. M. Bruckstein, Skeletonization via distance maps and level sets, Computer Vision and Image Understanding, 3 (1995), pp. 382–391.

[134] J. J. Koenderink, The structure of images, Biological Cybernetics, 50 (1984), pp. 363–370.

[135] J. J. Koenderink and A. J. van Doorn, Dynamic shape, Biological Cybernetics, 53 (1986), pp. 383–396.

[136] L. S. G. Kovasznay and H. M. Joseph, Image processing, Proc. IRE, 43 (1955), p. 560.

[137] H. P. Kramer and J. B. Bruckner, Iterations of a non-linear transformation for enhancement of digital images, Pattern Recognition, 7 (1975).

“JMMBookOct04” 30/10/2009 page 280

[figure] [figure]

280 BIBLIOGRAPHY

[138] S. G. Krantz and H. R. Parks, The Geometry of Domains in Space, Birkäuser, 1999.

[139] A. S. Kronrod, On functions of two variables, Uspehi Matem. Nauk (N. S.), 5 (1950), pp. 24–134.

[140] P. D. Lax, Numerical solution of partial differential equations, Amer. Math. Monthly, 72 (1965), pp. 74–85.

[141] B. Lee, W. Chen, and E. Chang, Fotofiti: web service for photo management, Proceedings of the 14th annual ACM international conference on Multimedia, (2006), pp. 485–486.

[142] Y. H. Lee and S. A. Kassam, Generalized median filtering and related nonlinear filtering techniques, IEEE Trans. Acoust. Speech Signal Process., 33 (1985), pp. 672–683.

[143] H. Lejsek, F. Ásmundsson, B. Jónsson, and L. Amsaleg, Scalability of local image descriptors: a comparative study, Proceedings of the 14th annual ACM international conference on Multimedia, (2006), pp. 589–598.

[144] T. Lindeberg, Scale-Space Theory in Computer Vision, vol. 256 of Kluwer International Series in Engineering and Computer Science, Kluwer Academic Publishers, Dordrecht, NL, 1993.

[145] T. Lindeberg, Scale-space theory: a basic tool for analyzing structures at different scales, Journal of Applied Statistics, 21 (1994), pp. 225–270.

[146] T. Lindeberg and J. Garding, Shape-adapted smoothing in estimation of 3-d depth cues from affine distortions of local 2-d brightness structure, Proc. ECCV, (1994), pp. 389–400.

[147] M. Lindenbaum, M. Fischer, and A. M. Bruckstein, On gabor’s contribution to image enhancement, Pattern Recognition, 27 (1994), pp. 1–8.

[148] J. Lisani, P. Monasse, and L. Rudin, Fast Shape Extraction and Applications, to appear, prepublication of CMLA ENS Cachan, 16 (2001).

[149] J.-L. Lisani, L. Moisan, P. Monasse, and J.-M. Morel, Affine invariant mathematical morphology applied to a generic shape recognition algorithm, in Proceedings of the International Symposium of Mathematical Morphology, Palo Alto, CA, 2000. Available at

http://pascal.monasse.free.fr/index.html.

[150] D. Lowe,  SIFT Keypoint Detector:  online demo

http://www.cs.ubc.ca/∼lowe/keypoints/.

[151] D. Lowe, Distinctive image features from scale-invariant key points, International Journal of Computer Vision, 60 (2004), pp. 91–110.

[152] G. Loy and J. Eklundh, Detecting symmetry and symmetric constellations of features, Proceedings of ECCV, 2 (2006), pp. 508–521.

“JMMBookOct04” 30/10/2009 page 281

[figure] [figure]

BIBLIOGRAPHY 281

[153] A. Mackworth and F. Mokhtarian, Scale based description and recognition of planar curves and two-dimensional shapes, IEEE Trans. Patt. Anal. Mach. Intell., 8 (1986), pp. 34–43.

[154] A. K. Mackworth and F. Mokhtarian.

[155] J. Maeda, A characteristic property of space curves of constant first affine curvature, Tohoku Math. J., 48 (1941), pp. 148–151.

[156] R. Malladi, J. A. Sethian, and B. C. Vemuri, Shape modeling with front propagation: A level set approach, IEEE Trans. Patt. Anal. Mach. Intell., 17 (1995), pp. 158–175.

[157] P. Maragos, A representation theory for morphological image and signal processing, IEEE Trans. Patt. Anal. Mach. Intell., 11 (1989), pp. 586–599.

[158] P. Maragos, Differential morphology and image-processing, IEEE Trans. Image Process., 5 (1996), pp. 922–937.

[159] P. Maragos, Partial differential equations in image analysis: Continuous modeling, discrete processing, in Eusipco, 1998.

[160] P. Maragos and R. W. Schafer, Morphological filters. Part I: Their set-theoretic analysis and relations to linear shift-invariant filters, IEEE Trans. Acoust. Speech Signal Process., 35 (1987), pp. 1153–1169.

[161] P. Maragos and R. W. Schafer, Morphological filters. Part II: Their relations to median, order-statistic, and stack filters, IEEE Trans. Acoust. Speech Signal Process., 35 (1987), pp. 1170–1184.

[162] P. Maragos and R. D. Ziff, Threshold superposition in morphological image analysis systems, IEEE Trans. Patt. Anal. Mach. Intell., 12 (1990), pp. 498–504.

[163] D. Marr, Vision, W. H. Freeman and Company, New York, NY, 1982.

[164] D. Marr, Vision, Freeman Publishers, 1982.

[165] D. Marr and E. Hildreth, Theory of edge detection, Proc. Roy. Soc. London Ser. B, 207 (1980), pp. 187–217.

[166] S. Masnou, Image restoration involving connectedness, in Proceedings of the 6^{th} International Workshop on Digital Image Processing and Computer Graphics, 1998.

[167] J. Matas, O. Chum, M. Urban, and T. Pajdla, Wbs image matcher: online demo http://cmp.felk.cvut.cz/∼wbsdemo/demo/.

[168] J. Matas, O. Chum, M. Urban, and T. Pajdla, Robust wide-baseline stereo from maximally stable extremal regions, Image and Vision Computing, 22 (2004), pp. 761–767.

[169] G. Matheron, Random Sets and Integral Geometry, John Wiley & Sons, New York, NY, 1975.

“JMMBookOct04” 30/10/2009 page 282

[figure] [figure]

282 BIBLIOGRAPHY

[170] B. Merriman, J. Bence, and S. Osher, Diffusion generated motion by mean curvature, in Computational Crystal Growers Workshop. Proceedings of the Geometry Center Workshop held in Minneapolis, Minnesota, February 22–28, 1992., J. E. Taylor, ed., Selected Lectures in Mathematics, American Mathematical Society, 1992, pp. 73–83.

[171] K. Mikolajczyk, http://www.robots.ox.ac.uk/∼vgg/research/affine/.

[172] K. Mikolajczyk and C. Schmid, Indexing based on scale invariant interest points, Proc. ICCV, 1 (2001), pp. 525–531.

[173] K. Mikolajczyk and C. Schmid, An affine invariant interest point detector, Proc. ECCV, 1 (2002), pp. 128–142.

[174] K. Mikolajczyk and C. Schmid, A Performance Evaluation of Local Descriptors, in International Conference on Computer Vision and Pattern Recognition, vol. 2, June 2003, pp. 257–263.

[175] K. Mikolajczyk and C. Schmid, Scale and Affine Invariant Interest Point Detectors, International Journal of Computer Vision, 60 (2004), pp. 63–86.

[176] K. Mikolajczyk and C. Schmid, A Performance Evaluation of Local Descriptors, IEEE Trans. PAMI, (2005), pp. 1615–1630.

[177] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and L. Gool, A Comparison of Affine Region Detectors, International Journal of Computer Vision, 65 (2005), pp. 43–72.

[178] L. Moisan, Affine plane curve evolution: A fully consistent scheme, IEEE Trans. Image Process., 7 (1998), pp. 411–420.

[179] P. Monasse, Contrast invariant image registration, Proc. of the International Conf. on Acoustics, Speech and Signal Processing, Phoenix, Arizona, 6 (1999), pp. 3221–3224.

[180] P. Monasse, Représentation morphologique d’images numériques et application au recalage, PhD thesis, Ecole Normale Supérieure-Cachan, France, 2000. French and English versions available at

http://pascal.monasse.free.fr.

[181] P. Monasse and F. Guichard, Scale-space from a level lines tree, Journal of Visual Communication and Image Representation, 11 (2000),

pp. 224–236. Available at http://pascal.monasse.free.fr.

[182] P. Moreels and P. Perona, Common-frame model for object recognition, Proc. NIPS, (2004).

[183] P. Moreels and P. Perona, Evaluation of Features Detectors and Descriptors based on 3D Objects, International Journal of Computer Vision, 73 (2007), pp. 263–284.

[184] J. Morel and G. Yu, On the consistency of the SIFT method, Tech. Rep. Prepublication, CMLA, ENS Cachan, 2008.

“JMMBookOct04” 30/10/2009 page 283

[figure] [figure]

BIBLIOGRAPHY 283

[185] D. Mukherjee, A. Zisserman, and J. Brady, Shape from symmetry— detecting and exploiting symmetry in affine images, Philosophical Transactions: Physical Sciences and Engineering, 351 (1995), pp. 77–106.

[186] J. L. Mundy, Object recognition: The search for representation, in ORCV95, 1995, pp. 19–50.

[187] P. Musé, F. Sur, F. Cao, and Y. Gousseau, Unsupervised thresholds for shape matching, Image Processing, 2003. Proceedings. 2003 International Conference on, 2 (2003).

[188] P. Muse, F. Sur, F. Cao, and Y. Gousseau, Unsupervised thresholds for shape matching, Image Processing, 2003. Proceedings. 2003 International Conference on, 2 (2003).

[189] P. Musé, F. Sur, F. Cao, Y. Gousseau, and J. Morel, An A Contrario Decision Method for Shape Element Recognition, International Journal of Computer Vision, 69 (2006), pp. 295–315.

[190] P. Musé, F. Sur, F. Cao, Y. Gousseau, and J. Morel, An A Contrario Decision Method for Shape Element Recognition, International Journal of Computer Vision, 69 (2006), pp. 295–315.

[191] P. Musé, F. Sur, F. Cao, J. Lisani, and J. Morel, Three-Dimensional Computer Vision: A Geometric Viewpoint, Mit Press, 2007.

[192] Y. Nakagawa and A. Rosenfeld, A note on the use of local min and max operations in digital image processing, IEEE Trans. Systems, Man and Cybernetics, 8 (1978), pp. 632–635.

[193] P. M. Narendra, Noise reduction by median filtering, in IEEE Conf. Pattern Recogn. Image Processing, 1978.

[194] A. Negre, H. Tran, N. Gourier, D. Hall, A. Lux, and J. Crowley, Comparative study of People Detection in Surveillance Scenes, Structural, Syntactic and Statistical Pattern Recognition, Proceedings Lecture Notes in Computer Science, 4109 (2006), pp. 100–108.

[195] W. Niessen, B. ter Haar Romeny, and M. Viergever, Numerical Analysis of Geometry-Driven Diffusion Equations, Geometry Driven Diffusion in Computer Vision, pp. 393–410.

[196] D. Nister and H. Stewenius, Scalable recognition with a vocabulary tree, Proc. CVPR, (2006), pp. 2161–2168.

[197] E. Ochoa, J. P. Allebach, and D. W. Sweeney, Optical median filtering by threshold decomposition, Appl. Opt., 26 (1987), pp. 252–260.

[198] S. Osher and L. I. Rudin, Feature-oriented image enhancement using shock filters, SIAM J. Numer. Anal., 27 (1990), pp. 919–940.

[199] S. Osher and J. Sethian, Fronts propagating with curvature-dependent speed: Algorithms based on the Hamilton–Jacobi formulation, J. Comput. Phys., 79 (1988), pp. 12–49.

“JMMBookOct04” 30/10/2009 page 284

[figure] [figure]

284 BIBLIOGRAPHY

[200] H. C. Park and R. T. Chin, Decomposition of arbitrarily-shaped morphological structuring elements, IEEE Trans. Patt. Anal. Mach. Intell., 17 (1995), pp. 2–15.

[201] D. Pasquignon, Computation of skeleton by PDE, in International Conference in Image Processing, IEEE, ed., 1995.

[202] S. S. Perlman, S. Eisenhandler, P. W. Lyons, and M. J. Shumila, Adaptive median filtering for impulse noise elimination in real-time TV signals, IEEE Trans. Communications, 35 (1987), pp. 646–652.

[203] P. Perona and J. Malik, Scale space and edge detection using anisotropic diffusion, in Proc. Workshop on Computer Vision, Representation and Control, IEEE Computer Society, IEEE Press, Piscataway, NJ, 1987, pp. 16–22.

[204] P. Perona and J. Malik, Scale space and edge detection using anisotropic diffusion, IEEE Trans. Patt. Anal. Mach. Intell., 12 (1990), pp. 629–639.

[205] D. Pritchard and W. Heidrich, Cloth Motion Capture, Computer Graphics Forum, 22 (2003), pp. 263–271.

[206] J. Rabin, Y. Gousseau, and J. Delon, A contrario matching of local descriptors, Tech. Rep. hal-00168285, Ecole Nationale Supérieure des Télécommunications, Paris, France, 2007.

[207] L. Rawtani, Image smoothing by median filtering – an efficient algorithm, J.M.A.C.T., 28 (1995), pp. 93–101.

[208] C. S. Regazzoni and A. Teschioni, A new approach to vector median filtering based on space-filling curves, IEEE Trans. Image Process., 6 (1997), pp. 1025–1037.

[209] C. H. Richardson and R. W. Schafer, A lower bound for structuring element decompositions, IEEE Trans. Patt. Anal. Mach. Intell., 13 (1991), pp. 365–369.

[210] F. Riesz and B. Sz.-Nagy, Functional Analysis, Frederick Ungar Publishing Co., New York, 1955.

[211] F. Riggi, M. Toews, and T. Arbel, Fundamental Matrix Estimation via TIP-Transfer of Invariant Parameters, Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06)-Volume 02, (2006), pp. 21–24.

[212] C. Ronse and H. J. A. M. Heijmans, The algebraic basis of mathematical morphology: II. openings and closings, Computer Vision, Graphics, and Image Processing, 54 (1991), pp. 74–97.

[213] E. Rouy and A. Tourin, A Viscosity Solutions Approach to Shape-From-Shading, SIAM Journal on Numerical Analysis, 29 (1992), pp. 867– 884.

“JMMBookOct04” 30/10/2009 page 285

[figure] [figure]

BIBLIOGRAPHY 285

[214] L. I. Rudin, Images, Numerical Analysis of Singularities and Shock Filters, PhD thesis, California Institute of Technology, Pasadena, CA, 1987.

[215] L. I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), pp. 259–268.

[216] W. Rudin, Real and Complex Analysis. Third edition, McGraw-Hill Book Co., New York, 1987.

[217] J. Ruiz-del Solar, P. Loncomilla, and C. Devia, A New Approach for Fingerprint Verification Based on Wide Baseline Matching Using Local Interest Points and Descriptors, Lecture Notes in Computer Science, 4872 (2007), p. 586.

[218] P. Salembier, Structuring element adaptation for morphological filters, Journal of Visual Communication and Image Representation, 3 (1992), pp. 115–136.

[219] P. Salembier and J. Serra, Flat zones filtering, connected operators and filters by reconstruction, IEEE Trans. Image Process., 8 (1995).

[220] G. Sapiro, R. Kimmel, D. Shaked, B. Kimia, and A. M. Bruckstein, Implementing continuous-scale morphology via curve evolution, Pattern Recognition, 9 (1993), pp. 1363–1372.

[221] G. Sapiro and A. Tannenbaum, Affine invariant scale space, International Journal of Computer Vision, 11 (1993), pp. 25–44.

[222] G. Sapiro and A. Tannenbaum, On invariant curve evolution and image analysis, Indiana Univ. Math. J., 42 (1993), pp. 985–1009.

[223] G. Sapiro and A. Tannenbaum, On affine plane curve evolution, J. Funct. Anal., 119 (1994), pp. 79–120.

[224] F. Schaffalitzky and A. Zisserman, Multi-view matching for unordered image sets, or How do I organize my holiday snaps?, Proc. ECCV, 1 (2002), pp. 414–431.

[225] D. Schonfeld, Optimal structuring elements for the morphological pattern restoration of binary images, IEEE Trans. Patt. Anal. Mach. Intell., 16 (1994), pp. 589–601.

[226] P. Scovanner, S. Ali, and M. Shah, A 3-dimensional SIFT descriptor and its application to action recognition, Proceedings of the 15th international conference on Multimedia, (2007), pp. 357–360.

[227] S. Se, D. Lowe, and J. Little, Vision-based mobile robot localization and mapping using scale-invariant features, Robotics and Automation, 2001. Proceedings 2001 ICRA. IEEE International Conference on, 2 (2001).

[228] J. Serra, Image Analysis and Mathematical Morphology, Academic Press, San Diego, CA, 1982.

“JMMBookOct04” 30/10/2009 page 286

[figure] [figure]

286 BIBLIOGRAPHY

[229] J. Serra, Introduction to mathematical morphology, Computer Vision, Graphics, and Image Processing, 35 (1986), pp. 283–305.

[230] J. Serra, Image Analysis and Mathematical Morphology. Volume2: Theoretical Advances, Academic Press, 1988.

[231] J. Sethian, An analysis of flow propagation, PhD thesis, UCLA, 1982.

[232] J. Sethian, Level Set Methods: Evolving Interfaces in Geometry, Fluid Mechanics, Computer Vision, and Materials Science, Cambridge University Press, 1996.

[233] J. A. Sethian, Curvature and the evolution of fronts, Comm. Math. Phys., 101 (1985).

[234] F. Y. Shih and O. R. Mitchell, A mathematical morphology approach to Euclidean distance transformation, IEEE Trans. Image Process., 1 (1992), pp. 197–204.

[235] N. Snavely, S. Seitz, and R. Szeliski, Photo tourism: exploring photo collections in 3D, ACM Transactions on Graphics (TOG), 25 (2006), pp. 835–846.

[236] N. Sochen, R. Kimmel, and R. Malladi, A general framework for low-level vision, IEEE Trans. Image Process., 7 (1998), pp. 310–318.

[237] D. Struik, Differential Geometry, Addison-Wesley, 1950.

[238] M. P. Sudipta N Sinha, Jan-Michael Frahm and Y. Genc, Gpubased video feature tracking and matching, EDGE 2006, workshop on Edge Computing Using New Commodity Architectures, Chapel Hill, (2006).

[239] T. Sun and Y. Neuvo, Detail-preserving median based filters in imageprocessing, Pattern Recognition Letters, 15.

[240] I. Svalbe and R. Jones, The design of morphological filters using multiple structuring elements, part I: Openings and closings, Pattern Recognition Letters, 13 (1992), pp. 123–129.

[241] H. D. Tagare and R. J. P. deFigueiredo, Order filters, Proceedings of IEEE, 73 (1985), pp. 163–165.

[242] M. E. Taylor, Partial Differential Equations: Basic Theory, Springer, New York, 1996.

[243] B. M. ter Haar Romeny, Front-End Vision and Multiscale Image Analysis: Introduction to Scale-Space Theory, Kluwer Academic Publishers, Dordrecht, NL, In preparation.

[244] T. Tuytelaars and L. Van Gool, Wide baseline stereo matching based on local, affinely invariant regions, British Machine Vision Conference, (2000), pp. 412–425.

[245] T. Tuytelaars and L. Van Gool, Matching Widely Separated Views Based on Affine Invariant Regions, International Journal of Computer Vision, 59 (2004), pp. 61–85.

“JMMBookOct04” 30/10/2009 page 287

[figure] [figure]

BIBLIOGRAPHY 287

[246] T. Tuytelaars, L. Van Gool, et al., Content-based image retrieval based on local affinely invariant regions, Int. Conf. on Visual Information Systems, (1999), pp. 493–500.

[247] L. Vacchetti, V. Lepetit, and P. Fua, Stable Real-Time 3D Tracking Using Online and Offline Information, IEEE Trans PAMI, (2004), pp. 1385–1391.

[248] R. van den Boomgaard, Numerical Solution Schemes for Continuous-Scale Morphology, Proceedings of the Second International Conference on Scale-Space Theories in Computer Vision, (1999), pp. 199–210.

[249] R. van den Boomgaard and A. Smeulders, The morphological structure of images: The differential equations of morphological scale-space, IEEE Trans. Patt. Anal. Mach. Intell., 16 (1994), pp. 1101–1113.

[250] L. Van Gool, T. Moons, and D. Ungureanu, Affine/Photometric Invariants for Planar Intensity Patterns, Proceedings of the 4th European Conference on Computer Vision-Volume I-Volume I, (1996), pp. 642–651.

[251] M. Veloso, F. von Hundelshausen, and P. Rybski, Learning visual object definitions by observing human activities, Humanoid Robots, 2005 5th IEEE-RAS International Conference on, (2005), pp. 148–153.

[252] M. Vergauwen and L. Van Gool, Web-based 3D Reconstruction Service, Machine Vision and Applications, 17 (2005), pp. 411–426.

[253] L. Vincent, Grayscale area openings and closings, their efficient implementation and applications, in Proceedings of the 1^{st} Workshop on Mathematical Morphology and its Applications to Signal Processing, J. Serra and P. Salembier, eds., Barcelona, Spain, 1993, pp. 22–27.

[254] J. von Neumann and O. Morgenstern, Theory of Games and Economic Behavior, Princeton U. Press, Princeton, NJ, 1947.

[255] J. Weickert, Nonlinear diffusion scale-spaces: from the continuous to the discrete setting, Proc. ICAOS96, pp. 111–118.

[256] J. Weickert, Anisotropic Diffusion in Image Processing, ECMI Series, Teubner-Verlag, Stuttgart, 1998.

[257] J. Weickert and B. Benhamouda, A semidiscrete nonlinear scalespace theory and its relation to the Perona–Malik paradox, in Advances in Computer Vision, F. Solina, W. G. Kropatsch, R. Klette, and R. Bajcsy, eds., Springer, 1997, pp. 1–10.

[258] J. Weickert, B. Romeny, and M. Viergever, Efficient and reliable schemes for nonlinear diffusion filtering, Image Processing, IEEE Transactions on, 7 (1998), pp. 398–410.

[259] P. D. Wendt, E. J. Coyle, and N. C. Gallagher, Jr., Stack filters, IEEE Trans. Acoust. Speech Signal Process., 34 (1986), pp. 898–911.

[260] M. Wertheimer, Untersuchungen zur Lehre der Gestalt, II, Psychologische Forschung, 4 (1923), pp. 301–350.

“JMMBookOct04” 30/10/2009 page 288

[figure] [figure]

288 BIBLIOGRAPHY

[261] J. Wilburn, Developments in generalized ranked-order filters, Journal of the Optical Society of America, 15 (1998), pp. 1084–1099.

[262] A. P. Witkin, Scale-space filtering, in Proceedings of the International Joint Conference on Artificial Intelligence, Karlsruhe, Germany, 1983, Morgan Kaufmann Publishers, San Francisco, pp. 1019–1021.

[263] K. Yanai, Image collector III: a web image-gathering system with bagof-keypoints, Proceedings of the 16th international conference on World Wide Web, (2007), pp. 1295–1296.

[264] G. Yang, C. Stewart, M. Sofka, and C. Tsai, Alignment of challenging image pairs: Refinement and region growing starting from a single keypoint correspondence, IEEE Trans. Pattern Anal. Machine Intell, (2007).

[265] J. Yao and W. Cham, Robust multi-view feature matching from multiple unordered views, Pattern Recognition, 40 (2007), pp. 3081–3099.

[266] O. Yli-Harja, J. Astola, and Y. Neuvo, Analysis of the properties of median and weighted median filters using threshold logic and stack filter representation, IEEE Trans. Signal Process., 39 (1991), pp. 395–410.

[267] A. L. Yuille, The creation of structure in dynamic shape, in Proceedings Second International Conference on Computer Vision, 1988, pp. 685–689.

[268] A. L. Yuille and T. A. Poggio, Fingerprint theorems for zerocrossings, J. Opt. Soc. Amer., 2 (1985), pp. 683–692.

[269] A. L. Yuille and T. A. Poggio, Scaling theorems for zero-crossings, IEEE Trans. Patt. Anal. Mach. Intell., 8 (1986), pp. 15–25.

[270] X. Zhuang, Decomposition of morphological structuring elements, J. Math. Imaging Vision, 4 (1994), pp. 5–18.

[271] X. Zhuang and R. M. Haralick, Morphological structural element decomposition, Computer Vision, Graphics, and Image Processing, 35 (1986), pp. 370–382.