Rolling Shutter Stereo

Olivier Saurer, Kevin Köser, Jean-Yves Bouguet, Marc Pollefeys

ETH ZürichSwitzerland

GEOMAR KielGermany

Google Inc,Mountain View, CA

ETH ZürichSwitzerland

2

Motivation

Resolution Rolling Shutter Time [ms]

iPhone 4S2

1920 x 1080 26.04

Galaxy S32

1920 x 1080 32.67

• Most CMOS chips have an electronic rolling shutter

Sequential exposure of scanline (rolling shutter)Cheap & Compact Light sensitive

2Oth et al. 2013

Ro

lling Sh

utter D

irection

3

When to Consider Rolling Shutter (RS)?

• RS distortion depends on:• Motion & Depth

• Moving car:

• Human motion:

• Velocity = 25 km / h• RS distortion is measurable

up to: 125m

• Velocity = 5 km / h• RS distortion is measurable

up to: 25m

4

Baker et al. 2010

Related Work

Rolling Shutter Calibration

Oth et al. 2013

Removing Rolling Shutter Wobble

Rolling Shutter Structure from Motion

Klingner et al. 2013

Forssén et al. 2010

Geyer et al. 2005

Hedborg et al. 2010

Dense 3D Reconstruction

?

5

Outline

Rolling Shutter 2-View Geometry

Rolling Shutter Stereo (Proposed Method)

Evaluation

6

Case 1: Valid Epipolar Geometry

RS direction Camera motion direction

Left RScamera

Right RS camera

7

Case 1: Valid Epipolar Geometry

RS direction Camera motion direction

Inter baseline

Intra baselines

Left RScamera

Right RS camera

Important special case: “Streetview”• Intra & inter baseline coincide

8

Case 1: Valid Epipolar Geometry

RS direction Camera motion direction

Inter baseline

Intra baselines

Epipolar line

P

Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry

Left RScamera

Right RS camera

9

Case 1: Valid Epipolar Geometry

RS direction Camera motion direction

Inter baseline

Intra baselines

Epipolar line

P

Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry

Left RScamera

Right RS camera

10

Case 1: Valid Epipolar Geometry

RS direction Camera motion direction

Inter baseline

Intra baselines

Epipolar line

P

Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry

Correct pixel correspondence

Left RScamera

Right RS camera

11

Case 1: Valid Epipolar Geometry

RS direction Camera motion direction

Inter baseline

Intra baselines

Epipolar line

GS

P

Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry

Correct pixel correspondence• Global Shutter triangulated 3D point

Left RScamera

Right RS camera

12

Case 1: Valid Epipolar Geometry

Inter baseline

Intra baselines

Epipolar line

GSRS

P

RS direction Camera motion direction

Important special case: “Streetview”• Intra & inter baseline coincide• Valid epipolar geometry

Correct pixel correspondenceGS triangulated 3D points havewrong depthRS triangulated 3D points havecorrect depth, considering correct pose at time of exposure

Left RScamera

Right RS camera

13

Standard Stereo vs. RS Stereo

Input Images Standard Stereo

Proposed Method(RS Stereo)

Ground Truth

RS & Camera Motion

RS direction Camera motion direction

14

• Intra & inter baseline do not coincide

Case 2: In General no Epipolar Geometry!1

Proposed MethodStandard Stereo:Only random matches

Left RS camera Right RS camera

1Besides very special configurations [Seitz 2001, Pajdla 2001]

15

Outline

Rolling Shutter 2-View Geometry

Rolling Shutter Stereo (Proposed Method)

Evaluation

16

Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure

Left RS camera

RS direction Camera motion direction

P P’

17

Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure

Left RS camera

RS direction Camera motion direction

P P’

1. Sample 3D planes1

1Plane Sweep: [Collins 1996], [Yang et al. 2003]

18

Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure

Left RS camera

RS direction Camera motion direction

P P’

1. Sample 3D planes1

2. Given:• 3D Point• Camera trajectorySolve for time of exposureusing RS projection model

1Plane Sweep: [Collins 1996], [Yang et al. 2003]

19

Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure

Left RS camera

RS direction Camera motion direction

P P’

1. Sample 3D planes1

2. Given:• 3D Point• Camera trajectorySolve for time of exposureusing RS projection model

3. Texture lookup using time of exposure

1Plane Sweep: [Collins 1996], [Yang et al. 2003]

20

Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure

Left RS camera

RS direction Camera motion direction

P P’P’’

Correlate

1Plane Sweep: [Collins 1996], [Yang et al. 2003]

1. Sample 3D planes1

2. Given:• 3D Point• Camera trajectorySolve for time of exposureusing RS projection model

3. Texture lookup using time of exposure

4. Find best correlation1

21

Rolling Shutter Stereo - Idea• Solve simultaneously for depth & time of exposure

Left RS camera

RS direction Camera motion direction

P P’P’’

Correlate

1Plane Sweep: [Collins 1996], [Yang et al. 2003]

1. Sample 3D planes1

2. Given:• 3D Point• Camera trajectorySolve for time of exposureusing RS projection model

3. Texture lookup using time of exposure

4. Find best correlation1

22

Rolling Shutter Projection

• RS projection matrix changes with time :

RS camera

RS direction Camera motion direction

?

23

Rolling Shutter Projection

• RS projection matrix changes with time :

• Find time of exposure when is seen. Solve fix-point function: RS camera

RS direction Camera motion direction

?

RS image sensor

Projection of:Quadratic polynomial1

1Assuming Linear motion [Geyer et al. 2005]

24

Rolling Shutter Projection

• RS projection matrix changes with time :

• Find time of exposure when is seen. Solve fix-point function:

• Valid solution:

RS camera

RS image sensor

Projection of:

RS direction Camera motion direction

?

Quadratic polynomial1

1Assuming Linear motion [Geyer et al. 2005]

25

RS Projection (With Lens Distortion)

• Find time of exposure when is seen. Solve fix-point function:

RS direction Camera motion direction

Polynomial of degree 8

Lens distortion

RS image sensor

Projection of:

26

RS Projection (With Lens Distortion)

• Find time of exposure when is seen. Solve fix-point function:

• Global undistortion does not help

RS direction Camera motion direction

Polynomial of degree 8

Lens distortion

RS image sensor

Projection of:

Polynomial of degree 8

27

Motion Models

• Further motion models are discussed in the paper

Translation Orientation Distortion (# coefficient)

Polynomial Degree

Linear / Orbital / Spiral Const / Linear / Linear 0 2

Linear / Orbital / Spiral Const / Linear / Linear 1 4

Linear Linear 1 5

Linear Const 5 8

Linear Linear 5 9

28

Outline

Rolling Shutter 2 View Geometry

Rolling Shutter Stereo (Proposed Method)

Evaluation

29

RS Warp1.

2. Solve 8 degree polynomial

Rolling Shutter WarpFast Approximation (FA)1. RS warp at grid vertices

2. Interpolate texture coordinates within grid cell

Image size: 976 x 732Speed: 27.7ms / warp (CUDA)

Grid size: 1/10 of image sizeSpeed: 2.2 ms / warp (CUDA)

Left image Right image Interpolate

30

RS Warp1.

2. Solve 8 degree polynomial

Rolling Shutter WarpFast Approximation (FA)1. RS warp at grid vertices

2. Interpolate texture coordinates within grid cell

Image size: 976 x 732Speed: 27.7ms / warp (CUDA)

Grid size: 1/10 of image sizeSpeed: 2.2 ms / warp (CUDA)

Left image Right image Interpolate

~6 Hz with FA, with 50 planes

31

Results – Rendered Ground Truth

Ground Truth(LiDAR)

FARS StereoStandard Stereo

Intra baseline

Intra baseline

RS & Motion

RS & Motion

Castle

Old Town

32

RS

& M

oti

on

Results Standard Stereo vs. RS Stereo

Driving Speed: 25km/hInter baseline: 2mIntra baseline: 0.5m

RS reconstruction & fusion

Standard Stereo reconstruction & fusion

Bird’s eye views

33

Conclusion

• RS hurts 3D reconstruction if ignored!

• Radial distortion is depth dependentand can’t be undone without depth.

• With little additional computational cost, similar quality RS 3D reconstructionis possible as with GS images.

Thank you!

Supported by

Olivier Saurer, Kevin Köser, Jean-Yves Bouguet, Marc Pollefeys

35

Reconstruction from RS Streetview Images