Lightfield and refocusing algorithms

View this page on my blog for moving GIFs.

1. Plenoptic image

The plenoptic image is restructured as a matrix of size \((U, V, S, T, C) = (16, 16, 700, 400, 3)\).

• \(U, V\) are the coordinates on the aperture or coordinates of the pinhole, viewpoint, or the origin of a light ray.
• \(S, T\) are the the coordinates on the lenslet array or the coordinates of the coordinates of the world object for a fixed pinhole, or the endpoint of a light ray.

Sub-aperture views

This data has 16x16 “viewpoints” or sub-apertures. Each viewpoint is depicted here, in a grid of 16x16 images.

Refocusing / Focal stack simulation

\(\iint_{(u,v) \in A} L(u, v, s, t, c) \text{d}v \text{d}u\)

Depth from focus

Using \(\sigma_1=1, \sigma_2=4\), we recover the following depth image and all-in-focus image.

Focus-Aperture stack

The aperture size varies horizontally and focal distance varies vertically along the grid in the following image.

Aperture-focus images

Depth from Focus-Aperture stack

2. Unstructured lightfield

Capturing unstructured lightfield

Calculating shifts

1. I employ a coarse-to-fine search to save computation. I slide the template by step_size across the search window and find the position where NCC is highest.

2. When such initial position is found, I search within a smaller search window within step_size distance from the initial position by sliding the template by a single pixel at a time.

3. To calculate NCC (Normalized Cross Correlation),
   • center both template and image patch at zero mean.
   • apply box filter on image patch
   • divide by l2 norm
   • take sum of correlation scipy.signal.correlate2d

4. The position found in step 2 offset by shifting the same amount in the negative direction, in order to align all the frames on the same patch.

5. Average all aligned frames and crop to attain the refocused image.