UHD als Format-agnostisches Produktionsverfahren und als ... · UHD als Format-agnostisches...

Fraunhofer

Heinrich Hertz Institute

©

Fraunhofer


Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587 Berlin, www.hhi.fraunhofer.de

UHD als Format-agnostisches

Produktionsverfahren und als eine Plattform für

interaktives Video

Ralf Schäfer

1

Fraunhofer


© ©

Immersive Telepresence Systems

2 Ralf Schäfer

Immersive sound

Immersive displays & surrounding (panorama) video

Users feel like „being there“ – standing in the middle of a scene or

sitting at the best seat

Users not having access to an „immersive screen“ can navigate with

their smaller screen in the Ultra High Definition panorama

Fraunhofer


© ©

Outline

3

Introduction - Beyond HD

Ultra High Definition (UHD) @ HHI – TiME Lab

FASCINATE project: Format agnostic video production and

distribution

Shooting of UHD panoramas – OMNICAM

Production examples

Combining panorama and broadcast camera views

Streaming for interactive navigation in UHD panoramas

Further developments

Conclusions

Ralf Schäfer

Fraunhofer


© ©

Video Beyond HD

4

Video formats evolve beyond HD

New standards like 4k (3840x2160) and 8k (7680x4320) are

available

4k equipment is already available today, 8k is targeted for 2020

In the future flexible displays will be available which can be used

like wall paper

The European project FASCINATE is working on a completely new

production approach: Format Agnostic Video

Format Agnostic Video means: Combination of any number of

video sensors to create Ultra High Resolution Video of arbitrary

spatial and temporal resolution

Ralf Schäfer

Fraunhofer


© ©

Real-time acquisition and projection of high-

resolution video panoramas

Ultra high resolution video @ HHI

5

Omnidirectional

6k camera

system

(OMNICAM)

CineBox for

stitching and

segmentation

Transmission

CineBox for

playout,

decoding,

warping and

blending

7k multiprojection system

Ralf Schäfer

Fraunhofer


© © 6

Immersive Viewing at HHI‘s TiME Lab

Surround view by 180°panorama projection in 7k in 2D and 3D and

surround sound by IOSONO WFS system

mirrors

for HD projectors

(7x2)

Row of 120

loudspeakers Curved screen (3m x 12m)

acoustically transparent

TiME: Tomorrow‘s immersive Media Experience

Ralf Schäfer

Fraunhofer


© © 7

TiME Lab –

a Laboratory for the Creative Community

TiME Lab is a place, where creative people can experiment with a 180°video

panorama and a WFS sound system.

Partners are

• HFF Konrad Wolf

• BTK Hochschule

• Berliner Philharmoniker

• Berlinale Talent Campus

• Lichtmond

• Neue Roadmovies/Wim Wenders

• Red Bull Media

• KUK - Filmproduktion

• Moro Images

• Arri

• ........

Ralf Schäfer

Fraunhofer


© ©

FASCINATE - Overall System Architecture

8 Ralf Schäfer

Fraunhofer


© ©

Format Agnostic Video Production

9

A layered scene representation is used to represent the camera clusters

used to record a scene

A powerfull set of metadata is used to describe the single sensor data,

their positions, viewing directions and their geometrical relations

Video streams and metadata can be transmitted and used to create

appropriate formats for further processing and/or for available displays

Such a format agnostic video representation is especially suitable for

interactive viewing of large panoramas

Ralf Schäfer

CC: camera cluster

Cam: camera

Video

Scene CC 1

CC 2

CC N

Cam 1.1

Cam 1.n

Fraunhofer


© ©

Test shoot at Chelsea

FascinatE test shoot at Premier League Match, Oct. 2010

Real live event

Usage of professional equipment by outside broadcaster

Visual sensors

OMNICAM by HHI

ALEXA by ARRI

4 HD broadcast cameras by BBC

Audio sensors

12 Stadium shot gun mics around

the pitch

Stereo mics

2 Soundfield® mics

1 Eigenmike®

Ralf Schäfer 10

Fraunhofer


© ©

Test shoot arrangement

OMNICAM Broadcast cam Alexa

Soundfield Stereo pair

Eigenmike

Ralf Schäfer 11

Fraunhofer


© © 12

Capturing from six cameras: 6 x HD

Ralf Schäfer

Fraunhofer


© © 19.2.2010 13

Postprocessing: Stitching and

geometric correction

Ralf Schäfer

Fraunhofer


© © 14

Postprocessing: Photometric &

colorimetric correction

Ralf Schäfer

Fraunhofer


© © 15

Final Video Panorama: 6000 x 1920 Pixels

Ralf Schäfer

Fraunhofer


© ©

Real-Time Stitching Engine

16

Realtime

stitching of 6

HD cameras on

a single PC

platform -

demonstrated

@ IBC 2011

Ralf Schäfer

Fraunhofer


© ©

Mindelheim Shooting

17 Ralf Schäfer

Fraunhofer


© ©

Collaboration with HFF

18

Production of feature film „Orlac Reloaded“ presented at Talent

Campus of Berlinale 2010

OMNICAM

Ralf Schäfer

Fraunhofer


© ©

Shooting a concert of BPO*

19

OMNICAM

*Berlin Philharmonic Orchestra

Ralf Schäfer

Fraunhofer


© ©

Improving resolution of OMNICAM

First shootings have been performed with an OMNICAM using 1st

generation micro HD cameras, which have strong limitations in spatial

resolution, temporal resolution (25 fps) and dynamic range.

Especially the limitation to 25 frames caused severe motion blur and

motion judder.

Meanwhile a new OMNICAM using 6 ALEXA-m cameras is available

The ALEXA-m has a high dynamic range (15 stops), a 3k sensor and can

shoot at 50 fps or higher

Two first shootings with the new OMNICAM have already been made:

• Berliner Philharmoniker & Sasha Waltz @ ARENA in Berlin

• Night of the Proms @ Royal Albert Hall in London

Ralf Schäfer 20

Fraunhofer


© ©

OMNICAM with ALEXA-M

New rig with 6 high end electronic film cameras using 6 segments for 180° panoramas:

• 6 x ALEXA-M

• vertical FOV: 60° (portrait format)

• horizontal FOV 30° per segment => 180°

• resolution: 6k x 2k

• upgradable up to 360°

Ralf Schäfer

Fraunhofer


© © Ralf Schäfer

Shooting BPO & Sasha Waltz @ ARENA

Fraunhofer


© © 23 Ralf Schäfer

Live

Preview

Director‘s room

Panorama Shooting @ Arena / Berlin

Fraunhofer


©

Fraunhofer


24

Panorama Shooting – BBC Proms / London

Ralf Schäfer

Royal Albert Hall

Fraunhofer


© © 25

Format agnostic production scenario

Ralf Schäfer

OMNICAM

Fraunhofer


© ©

Comparison of resolutions

Full broadcast

camera image

6k x 2k panorama

Detail from

panoramic image

Merging of Broadcast Camera(s) and

Panorama

Ralf Schäfer 26

Fraunhofer


© ©

Registration between cameras

Region of OMNICAM image

Ralf Schäfer 27

Fraunhofer


© ©

Registration: Colour Matching Problems

Simple overlay shows problems with photometric matching and parallax

Ralf Schäfer 28

Fraunhofer


© ©

Histogram matching can largely solve photometric issues

Registration with Colour Correction

Ralf Schäfer 29

Fraunhofer


© ©

Parallax problems are more fundamental: depth variation makes global

registration impossible

Clearly visible when trying to take low frequencies from OMNICAM and

high frequences (detail) from broadcast camera

Registration:Parallax Problems

Ralf Schäfer 30

Movies/Omnicam_Depth_Wrong_1.wmv

Fraunhofer


© ©

Registration with Parallax Correction

Correction can be applied for each depth plane:

Estimate depth of the scene part in the panorama, at which the broadcast

camera points

Use Depth Image Based Rendering (DIBR) to compensate the parallax

Demo: Blending from broadcast camera into OMNICAM panorama

Ralf Schäfer 31

Fraunhofer


© © 34

Shooting of UHD Panoramas for

Interactive Usage

production

phone/tablet

Interactive TV

distribution

network and/or

Internet

HD

UHD

Ralf Schäfer

OMNICAM

http://www.fug-verlag.de/on1577&bildanzeige=4501

Fraunhofer


© ©

Interactive Tiled Streaming of ROIs

Control API to

Combiner

Monitor combiner output:

Stitched video of the ROI

Panorama Sequence

total of 385 spatially segmented and

compressed video files of resolution

480x270 from 10 resolution layers

Ralf Schäfer 35

Fraunhofer


© ©

Server-side Panning and Zooming

High resolution panoramic content

Real-Time Reframing and Coding

Frame width / bitrate / quality

2D navigation / zoom commands

Remote A/V Proxy Thin Clients

- Set-up per client

- Interactive View streaming per client

- Decoded once for all clients

Ralf Schäfer 36

Fraunhofer


©

Fraunhofer


38

3D CGI Production – Lichtmond 2

"Precious Life" – joint project with

Alan Parsons

Cooperation with Lichtmond for the

production of a 180°animation in

3D

WFS production of sound

Ralf Schäfer

Fraunhofer


© © 39 Ralf Schäfer

• First prototype of OMNICAM 3D

with 3 camera pairs has been

developed

• Presentation of first 3D

panoramas took place during

Berlinale on February 17th, 2011

(PRIME Symposium:

http://ip.hhi.de/PRIME_Workshop

/index_de.html

• 180°3D OMNICAM will be

available by mid 2013

OMNICAM 3D

http://ip.hhi.de/PRIME_Workshop/index_de.html

http://ip.hhi.de/PRIME_Workshop/index_de.html

Fraunhofer


© © 40

Mirror Surface

concentric

arrangement

of stereo

sub-systems

The 3D OmniCam

3D OMNICAM

Ralf Schäfer

left view right view

stereo content from crossed mirror segments

Stitching of 3D panoramas

Fraunhofer


©

Fraunhofer

Heinrich Hertz Institute Compact OMNICAM

41

• Compact 360°version

• 10 HD cameras

• Vertical FOV: 60° (portrait format)

• Horizontal FOV: 36° per segment

• Resolution: 10k x 2k

• Diameter: 50 cm

• Height: 40 cm

<<< size comparison

Fraunhofer


©

Fraunhofer


42

Mobile TiME Lab

180°panorama

14 HD projectors

3D projection using Infitec

3D audio (32 loudspeakers) with

IOSONO-WFS

Premiere at IFA 2012

Part of the program of

Medienwoche

Ralf Schäfer

Fraunhofer


© © 43

Conclusions

Video technology is evolving beyond HD

New video formats such as 4k and 8k have been standardized

In a long term wall displays will be available

For HHI video systems beyond HD are a strategic R&&D topic

The European project FASCINATE is working on a new concept, which is called

“Format Agnostic Video” and which is especially suitable for sport events such

as soccer

The complete chain is being developed in FASCINATE

Several panorama productions using the OMNICAM have been made

Technologies for combining video from panorama and broadcast cameras have

been developed

A test bed for adaptive streaming using interactive navigation has been

developed

Further developments (Mobile TiME Lab, 3D OMNICAM, compact OMNICAM)

are on its way

Ralf Schäfer

Fraunhofer


© © 44

Contact:

Dr. Ralf Schäfer

Fraunhofer HHI

Einsteinufer 37

10587 Berlin, Germany

Phone: +49 30 31002-560

[email protected]

www.hhi.fraunhofer.de

Thank you!

Ralf Schäfer

Acknowledgement: Parts of this presentation are based on work within the FP7

project FASCINATE funded by the European Commission

mailto:[email protected]

http://www.hhi.fraunhofer.de/

UHD als Format-agnostisches Produktionsverfahren und als ... · UHD als Format-agnostisches...

Documents

Transcript of UHD als Format-agnostisches Produktionsverfahren und als ... · UHD als Format-agnostisches...