Description

Case: Build a Panorama. . M. Chestnut and D. G. Lowe. Perceiving Panoramas. ICCV 2003. . A persuading application Building a display. We have to match/adjust/register pictures. Building a scene. 1) Detect highlight focuses in both pictures. Building a scene. Recognize highlight focuses in both imagesFind relating sets.

Transcripts

Highlight focuses extraction A low level building obstruct in numerous applications: Structure from movement Object ID: Video Google Objects acknowledgment . Many slides are politeness of Darya Frolova, Denis Simakov

A persuading application Building a display We have to coordinate/adjust/enlist pictures

Building a scene 1) Detect include focuses in both pictures

Building a scene Detect highlight focuses in both pictures Find relating sets

Building a scene Detect include focuses in both pictures Find comparing sets Find a parametric change (e.g. homography) Warp (right picture to left picture)

2 (n) sees geometry Today\'s discussion Matching with Features Detect include focuses in both pictures Find relating sets Find a parametric change

Criteria for good Features Repeatable locator Distinctive descriptor Accurate 2D position

Repeatable identifier Property 1: Detect a similar point freely in both pictures no possibility to coordinate!

Distinctive descriptor Property 2: Reliable coordinating of a comparing point ?

Accurate 2D position Property 3: Localization Where precisely is the point ? Sub-pixel precise 2D position

Examples of regularly utilized components Harris, Corner Detector (1988) KLT Kanade-Lucas-Tomasi (80\'s 90\'s) Lowe, SIFT (Scale Invariant Features Transform) Mikolajczyk &Schmid, "Harris Laplacian" (2000) Tuytelaars &V.Gool. Affinely Invariant Regions Matas et.al. "Recognized Regions" Bay et.al. "SURF" (Speeded Up Robust Features) (2006)

Detection : focuses with high "Cornerness" (next slide) Descriptor : a little window around it (i.e., coordinating by SSD, SAD) Corner identifiers Harris & KLT Localization : pinnacle of a fitted parabola that approximates the "cornerness" surface C.Harris, M.Stephens. "A Combined Corner and Edge Detector". 1988 Lucas Kanade. An Iterative Image Registration Technique 1981. Tomasi Kanade. Location and Tracking of Point Features. 1991. Shi Tomasi. Great Features to Track 1994.

"Cornerness" R (x 0 , y 0 ) of a point is characterized as: Cornerness (formally) where M is a 2 2 "structure network" processed from picture subsidiaries: And k – is a scale consistent, and w(x,y) is a weight work

Descriptors & Matching Descriptors ROI around the point (rectangle/Gaussian ) run of the mill sizes 8X8 up to 16X16. Coordinating : (agent choices) Sum Absolute Difference Sum Square Difference Correlation (Normalized Correlation)

Localization Fit a surface/parabola P(x,y) (utilizing 3x3 R values) Compute its maxima Yields a non whole number position.

Harris corner finder is roused by exact restriction Find focuses with the end goal that: little move high force change Hidden presumption: Good confinement in one picture great limitation in another picture

Window work Shifted power Intensity E Window work w(x,y) = or 1 in window, 0 outside Gaussian u v Cornerness ≈ High change of force for each move Harris Detector Cont. Change of force for the move [ u,v ]:

Harris Detector: Basic Idea "level" area: "edge" : "corner" :

Measuring the "properties" of E() M relies on upon picture properties

"properties" of E() ↔ "properties" of M Harris Detector Cont. For little moves [ u,v ] we have a bilinear estimation: where M is a 2 2 lattice processed from picture subordinates:

Bilinear frame and its eigenvalue 1 , 2 – eigenvalues of M Ellipse E(u,v) = const bearing of the slowest alter course of the quickest change ( min ) - 1/2 ( max ) - 1/2

"Cornerness" of a point R(x 0 , y 0 ) is characterized as: KLT And k – is a scale consistent, and w(x,y) is a weight work

Classification of picture focuses utilizing eigenvalues of M : 2 "Edge" 2 >> 1 "Corner" 1 and 2 are expansive, 1 ~ 2 ; E increments every which way 1 and 2 are little; E is practically steady every which way "Edge" 1 >> 2 "Level" area 1

"Cornerness" of a point R(x 0 , y 0 ) > limit >0: And 0< k<0.25 (~0.05) is a scale steady, Harris corner locator Computed utilizing 2 traps: C.Harris, M.Stephens. "A Combined Corner and Edge Detector". 1988

Harris Detector 2 "Edge" "Corner" R depends just on eigenvalues of M R is extensive for a corner R is negative with vast greatness for an edge | R | is little for a level area R < 0 R > 0 "Level" "Edge" |R| little R < 0 1

Harris Detector (outline) The Algorithm: Detection : Find focuses with substantial corner reaction work R ( R > edge) Localization : Approximate (parabola) neighborhood maxima of R Descriptors ROI around (rectangle) the point. Coordinating : SSD, SAD, NC.

Harris Detector: Workflow

Harris Detector: Workflow Compute corner reaction R

Harris Detector: Workflow Find focuses with substantial corner reaction: R> edge

Harris Detector: Workflow Take just the purposes of neighborhood maxima of R

Harris Detector: Workflow

If I distinguished this point Will I identify this point If I identified this point Will I recognize this point If I distinguished this point Will I identify this direct Detector Properties toward be "Invariant" to 2D revolutions Illumination Scale Surface introduction Viewpoint (benchmark between 2 cameras)

Harris Detector: Properties Rotation invariance Ellipse turns however its shape (i.e. eigenvalues) continues as before Corner reaction R is invariant to picture pivot

R limit x (picture arrange) x (picture organize) Harris Detector: Properties Partial invariance to force change Only subordinates are utilized to manufacture M => invariance to power move I I + b

Harris Detector: Properties Non-invariant to picture scale ! focuses "ordered" as edges Corner !

Harris Detector: Properties Non-invariant for scale changes Repeatability rate is: # correspondences # conceivable "Correspondences" in controlled setting (i.e., take a picture and scale it) is insignificant C.Schmid et.al. "Assessment of Interest Point Detectors". IJCV 2000

Rotation Invariant Detection Harris Corner Detector C.Schmid et.al. "Assessment of Interest Point Detectors". IJCV 2000

Examples of regularly utilized elements Harris, Corner Detector (1988) KTL Kanade-Lucas-Tomasi Lowe, SIFT (Scale Invariant Features Transform) Mikolajczyk &Schmid, "Harris Laplacian" (2000) Tuytelaars &V.Gool. Affinely Invariant Regions Matas et.al. "Recognized Regions" Bay et.al. "SURF" (Speeded Up Robust Features) (2006)

Scale Invariant issue outline Consider locales (e.g. circles) of various sizes around a point

Scale invariance approach Find a "local" scale. A similar local scale ought to redetected (at pictures of various scale).

scale Laplacian y x Harris Scale Invariant Detectors Harris-Laplacian Find nearby most extreme of: Harris corner indicator for set of Laplacian pictures 1 K.Mikolajczyk, C.Schmid . "Ordering Based on Scale Invariant Interest Points". ICCV 2001

SIFT (Lowe) Find neighborhood greatest of Difference of Gaussians scale DoG y x DoG D.Lowe . "Unmistakable Image Features from Scale-Invariant Keypoints". IJCV 2004

Difference of Gaussians pictures Kernels: Functions for deciding scale (Difference of Gaussians) where Gaussian Note: both pieces are invariant to scale and turn

SIFT Localization Fit a 3D quadric D(x,y,s) (utilizing 3x3X3 DoG values) Compute its maxima Yields a non whole number position (in x,y) . Chestnut and Lowe, 2002

D(x,y,s) is likewise utilized for pruning non-stable maxima D.Lowe . "Unmistakable Image Features from Scale-Invariant Keypoints". IJCV 2004

Scale Invariant Detectors Experimental assessment of finders w.r.t. scale change Repeatability rate: # correspondences # conceivable correspondences K.Mikolajczyk, C.Schmid. "Ordering Based on Scale Invariant Interest Points". ICCV 2001

SIFT Descriptors

SIFT – Descriptor A vector of 128 values each between [0 - 1] We additionally processed area scale "local" introduction D.Lowe. "Particular Image Features from Scale-Invariant Keypoints". IJCV 2004

"local" introduction Peaks in an angle introduction histogram Gradient is processed at the chosen scale 36 canisters (determination of 10 degrees). Ordinarily (15%) more than 1 crest !? The "powerless chain" in SIFT descriptor. D.Lowe. "Unmistakable Image Features from Scale-Invariant Keypoints". IJCV 2004

Computing a SIFT descriptor Determine scale (by boosting DoG in scale and in space), Determine neighborhood introduction (bearing prevailing inclination). characterize a local facilitate framework. Register slope introduction histograms (of a 16x16 window) 16 windows 128 qualities for every point/(4x4 histograms of 8 receptacles) Normalize the descriptor to roll out it invariant to power improvement D.Lowe. "Particular Image Features from Scale-Invariant Keypoints". IJCV 2004

Matching SIFT Descriptors vectors of 128 qualities Using L2 standard. A scan for NN (or KNN) can\'t be driven insignificantly , and is implemente