Affine shape adaptation

The interest points obtained from the scale-adapted Laplacian blob detector or the multi-scale Harris corner detector with automatic scale selection are invariant to translations, rotations and uniform rescalings in the spatial domain. The images that constitute the input to a computer vision system are, however, also subject to perspective distortions. To obtain interest points that are more robust to perspective transformations, a natural approach is to devise a feature detector that is invariant to affine transformations.

Affine invariance can be accomplished from measurements of the same multi-scale windowed second moment matrix $\backslash mu$ as is used in the multi-scale Harris operator provided that we extend the regular scale space concept obtained by convolution with rotationally symmetric Gaussian kernels to an affine Gaussian scale-space obtained by shape-adapted Gaussian kernels ({{harvnb|Lindeberg|1994|loc=section 15.3}}; {{harvnb|Lindeberg|Garding|1997}}). For a two-dimensional image $I$, let $\backslash bar\{x\}\; =\; (x,\; y)^T$ and let $\backslash Sigma\_t$ be a positive definite 2×2 matrix. Then, a non-uniform Gaussian kernel can be defined as

:$g(\backslash bar\{x\};\; \backslash Sigma)\; =\; \backslash frac\{1\}\{2\; \backslash pi\; \backslash sqrt\{\backslash operatorname\{det\}\; \backslash Sigma\_t\}\}\; e^\{-\backslash bar\{x\}\; \backslash Sigma\_t^\{-1\}\; \backslash bar\{x\}/2\}$

and given any input image $I\_L$ the affine Gaussian scale-space is the three-parameter scale-space defined as

:$L(\backslash bar\{x\};\; \backslash Sigma\_t)\; =\; \backslash int\_\{\backslash bar\{xi\}\}\; I\_L(x-\backslash xi)\; \backslash ,\; g(\backslash bar\{\backslash xi\};\; \backslash Sigma\_t)\; \backslash ,\; d\backslash bar\{\backslash xi\}.$

Next, introduce an affine transformation $\backslash eta\; =\; B\; \backslash xi$ where $B$ is a 2×2-matrix, and define a transformed image $I\_R$ as

:$I\_L(\backslash bar\{\backslash xi\})\; =\; I\_R(\backslash bar\{\backslash eta\})$.

Then, the affine scale-space representations $L$ and $R$ of $I\_L$ and $I\_R$, respectively, are related according to

:$L(\backslash bar\{\backslash xi\},\; \backslash Sigma\_L)\; =\; R(\backslash bar\{\backslash eta\},\; \backslash Sigma\_R)$

provided that the affine shape matrices $\backslash Sigma\_L$ and $\backslash Sigma\_R$ are related according to

:$\backslash Sigma\_R\; =\; B\; \backslash Sigma\_L\; B^T$.

Disregarding mathematical details, which unfortunately become somewhat technical if one aims at a precise description of what is going on, the important message is that the affine Gaussian scale-space is closed under affine transformations.

If we, given the notation $\backslash nabla\; L\; =\; (L\_x,\; L\_y)^T$ as well as local shape matrix $\backslash Sigma\_t$ and an integration shape matrix $\backslash Sigma\_s$, introduce an affine-adapted multi-scale second-moment matrix according to

:$\backslash mu\_L(\backslash bar\{x\};\; \backslash Sigma\_t,\; \backslash Sigma\_s)\; =\; g(\backslash bar\{x\}\; -\; \backslash bar\{\backslash xi\};\; \backslash Sigma\_s)\; \backslash ,\; \backslash left(\; \backslash nabla\_L(\backslash bar\{\backslash xi\};\; \backslash Sigma\_t)\; \backslash nabla\_L^T(\backslash bar\{\backslash xi\};\; \backslash Sigma\_t)\; \backslash right)$

it can be shown that under any affine transformation $\backslash bar\{q\}\; =\; B\; \backslash bar\{p\}$ the affine-adapted multi-scale second-moment matrix transforms according to

:$\backslash mu\_L(\backslash bar\{p\};\; \backslash Sigma\_t,\; \backslash Sigma\_s)\; =\; B^T\; \backslash mu\_R(\backslash bar\{q\};\; B\; \backslash Sigma\_t\; B^T,\; B\; \backslash Sigma\_s\; B^T)\; B$.

Again, disregarding somewhat messy technical details, the important message here is that given a correspondence between the image points $\backslash bar\{p\}$ and $\backslash bar\{q\}$, the affine transformation $B$ can be estimated from measurements of the multi-scale second-moment matrices $\backslash mu\_L$ and $\backslash mu\_R$ in the two domains.

An important consequence of this study is that if we can find an affine transformation $B$ such that $\backslash mu\_R$ is a constant times the unit matrix, then we obtain a fixed-point that is invariant to affine transformations ({{harvnb|Lindeberg|1994|loc=section 15.4}}; {{harvnb|Lindeberg|Garding|1997}}). For the purpose of practical implementation, this property can often be reached by in either of two main ways. The first approach is based on transformations of the smoothing filters and consists of:

• estimating the second-moment matrix $\backslash mu$ in the image domain,
• determining a new adapted smoothing kernel with covariance matrix proportional to $\backslash mu^\{-1\}$,
• smoothing the original image by the shape-adapted smoothing kernel, and
• repeating this operation until the difference between two successive second-moment matrices is sufficiently small.

The second approach is based on warpings in the image domain and implies:

• estimating $\backslash mu$ in the image domain,
• estimating a local affine transformation proportional to $\backslash hat\{B\}\; =\; \backslash mu^\{1/2\}$ where $\backslash mu^\{1/2\}$ denotes the square root matrix of $\backslash mu$,
• warping the input image by the affine transformation $\backslash hat\{B\}^\{-1\}$ and
• repeating this operation until $\backslash mu$ is sufficiently close to a constant times the unit matrix.

This overall process is referred to as affine shape adaptation ({{harvnb|Lindeberg|Garding|1997}}; {{harvnb|Baumberg|2000}}; {{harvnb|Mikolajczyk|Schmid|2004}}; {{harvnb|Tuytelaars|van Gool|2004}}; {{harvnb|Ravela|2004}}; {{harvnb|Lindeberg|2008}}). In the ideal continuous case, the two approaches are mathematically equivalent. In practical implementations, however, the first filter-based approach is usually more accurate in the presence of noise while the second warping-based approach is usually faster.

In practice, the affine shape adaptation process described here is often combined with interest point detection automatic scale selection as described in the articles on blob detection and corner detection, to obtain interest points that are invariant to the full affine group, including scale changes. Besides the commonly used multi-scale Harris operator, this affine shape adaptation can also be applied to other types of interest point operators such as the Laplacian/Difference of Gaussian blob operator and the determinant of the Hessian ({{harvnb|Lindeberg|2008}}). Affine shape adaptation can also be used for affine invariant texture recognition and affine invariant texture segmentation.

Closely related to the notion of affine shape adaptation is the notion of affine normalization, which defines an affine invariant reference frame as further described in Lindeberg ({{citeref|Lindeberg|2013a|2013a|style=plain}},{{citeref|Lindeberg|2013b|b|style=plain}}, {{citeref|Lindeberg|2021|2021|style=plain}}:Appendix I.3), such that any image measurement performed in the affine invariant reference frame is affine invariant.

• Blob detection
• Corner detection
• Gaussian function
• Harris affine region detector
• Hessian affine region detector
• Scale space

• {{cite conference

| first=A.

| last=Baumberg

| chapter=Reliable feature matching across widely separated views

| title=Proceedings of IEEE Conference on Computer Vision and Pattern Recognition

|pages=I:1774–1781

| year=2000

| doi=10.1109/CVPR.2000.855899

}}

• {{cite book |

first=T. |

last=Lindeberg |

title= Scale-Space Theory in Computer Vision |

url = http://www.csc.kth.se/~tony/book.html |

publisher= Springer |

year=1994 |

isbn=0-7923-9418-6}}

• {{cite journal

| first1=T.

| last1=Lindeberg

| first2=J.

| last2=Garding

| title=Shape-adapted smoothing in estimation of 3-D depth cues from affine distortions of local 2-D structure

| journal=Image and Vision Computing

| year=1997

| volume=15

| issue=6

|pages=415–434

| url=http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A472972&dswid=5025

| doi=10.1016/S0262-8856(97)01144-X

}}

• {{cite encyclopedia

| first=T.

| last=Lindeberg

| title=Scale-space

| encyclopedia=Encyclopedia of Computer Science and Engineering (Benjamin Wah, ed), John Wiley and Sons

| volume = IV

| pages = 2495–2504

| year = 2008

| doi=10.1002/9780470050118.ecse609

| isbn=978-0470050118

| url = http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A441147&dswid=9229

}}

• {{cite journal

| first=T.

| last=Lindeberg

| title=Invariance of visual operations at the level of receptive fields

| year=2013a

| journal=PLOS ONE

| volume=8

| issue=7

| pages=e66990:1–33

| doi=10.1371/journal.pone.0066990

| pmid=23894283

| pmc=3716821

| arxiv=1210.0754

| bibcode=2013PLoSO...866990L

| doi-access=free

}}

• {{cite journal

| first=T.

| last=Lindeberg

| title=Generalized axiomatic scale-space theory

| year=2013b

| journal=Advances in Imaging and Electron Physics

| volume=178

| issue=7

| pages=1–96

| url=http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A607456&dswid=-5433

| doi=10.1016/B978-0-12-407701-0.00001-7

| isbn=9780124077010

}}

• {{cite journal

| first=T.

| last=Lindeberg

| title=Normative theory of visual receptive fields

| year=2021

| journal=Heliyon

| volume=7

| issue=1

| pages=e05897

| doi=10.1016/j.heliyon.2021.e05897

| pmid=33521348

| pmc=7820928

| doi-access=free

}}

• {{cite journal

| first1=K.

| last1=Mikolajczyk

| first2=C.

| last2=Schmid

| title=Scale and affine invariant interest point detectors

| year=2004

| journal=International Journal of Computer Vision

| volume=60

| issue=1

|pages=63–86

| url=http://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolajczyk_ijcv2004.pdf

| quote=Integration of the multi-scale Harris operator with the methodology for automatic scale selection as well as with affine shape adaptation.

| doi=10.1023/B:VISI.0000027790.02288.f2

| s2cid=1704741

}}

• {{cite journal

|first1=T.

|last1=Tuytelaars

|first2=L.

|last2=van Gool

|title=Matching Widely Separated Views Based on Affine Invariant Regions

|year=2004

|journal=International Journal of Computer Vision

|volume=59

|issue=1

|pages=63–86

|url=http://www.vis.uky.edu/~dnister/Teaching/CS684Fall2005/tuytelaars_ijcv2004.pdf

|doi=10.1023/B:VISI.0000020671.28016.e8

|s2cid=5107897

|url-status=dead

|archive-url=https://web.archive.org/web/20100612233617/http://vis.uky.edu/~dnister/Teaching/CS684Fall2005/tuytelaars_ijcv2004.pdf

|archive-date=2010-06-12

}}

• {{cite conference

|chapter=Shaping receptive fields for affine invariance

|title=Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004

|year=2004

|last1=Ravela

|first1=S.

|volume=2

|pages=725–730

|isbn=0-7695-2158-4

|doi=10.1109/CVPR.2004.1315236

}}

{{Reflist}}

{{DEFAULTSORT:Affine Shape Adaptation}}

Category:Feature detection (computer vision)