2. Index
• Introduction
• 3DTV Broadcast
• 3DTV Over IP Networks
• Discussion and Conclusion
2
3. Introduction
• Ultimate goal
• dynamic holography
• Most systems available today
• via stereoscopy
• Actually, 3DTV systems can be designed to support
• fixed-view stereoscopy: only two views
• free-view stereoscopy: multiple views
3
4. Introduction
• History of 3-D movie
• 1903, first stereoscopic 3-D movie was created
• 1922, the first full length stereoscopic movie was shown
• in the 1950s, Hollywood started 3-D movie production in big
numbers
• Consensus: a lasting success
• backwards compatible
• supports different numbers of users
• with affordable 3-D display technologies
• requires low additional transport/transmission overhead
• perceived quality and viewing comfort is better 4
5. 3DTV Broadcast
• Analog Transmission
• US
• April 29th, 1953: a trial live broadcast of the series SPACE
PATROL was run in Los Angeles
• viewers with a pair of special polarization lenses
• December 19th, 1980: The first “nonexperimental” 3DTV “Miss
Sadie Thompson,” and Three Stooges
• 3-D Video Corporation developed a system: anaglyph format
• April 10th, 1981: musical classic, “Kiss Me Kate.”
• 3-D Video Corporation: perfect in color
5
6. 3DTV Broadcast
• Analog Transmission
• European
• 1982: Netherlands two popular-scientific 3-D series
• a simple red/green anaglyph format
• H.-J. Herbst (Hamburg, Germany) and Philips Research Lab
• More than 40 million red/green viewing spectacles were sold
• “the TV of the future” was disillusioned
• 1983 at the International Audio and Video Fair in Berlin
• based on a standard PAL channel chain in two-channel mode
• For display, two projectors with orthogonal polarization filters were
used
• so successful that were continued at IAVF in 1985 and 1987
• Unfortunately, transmission system required custom receiver.
6
LIMITED
7. 3DTV Broadcast
• Digital Transmission
• Background: ongoing transition from analogue to digital TV
services
• MPEG developed a new compression technology as part of
MPEG-2
• The MPEG-2 multiview profile (MVP)
• Left-eye view --- MPEG-2 main profile --- backwards-
compatibility
• Right-eye enhancement layer using the scalable coding tools
• MVP, unfortunately, has not found use in commercially
services 7
8. 3DTV Broadcast
• Digital Transmission
• Some promising attempts
• 1998 Nagano Winter Games in Japan
• right-eye and left-eye HDTV images @ 45Mbps
• projected onto a large screen. Impressing and Powerful
• 2002 FIFA World Cup in Korea/Japan
• the right-eye and left-eye HDTV images were compressed in
side-by-side format using the MPEG-2 Main Profile
8
9. 3DTV Broadcast
• Digital Transmission
• Fixed-view -> flexible 3-D visual data representation
formats
• Australian DDD company : “video-plus-depth”
representation
• combination of monoscopic color video and associated per-
pixel depth maps
• encodes the depth data low bit rate format
• transmitted in the “user data” of an MPEG-2 Transport Stream
• receiver : rendered by using depth-image-based rendering
(DIBR)
9
10. 3DTV Broadcast
• Digital Transmission
• European IST project ATTEST
• “video-plus-depth” representation
• Standard MPEG technologies: H.264/AVC
• depth data: 200–300 kbps
• overhead for the 3-D visual information is only 10% CMP 2-D
10
11. 3DTV Broadcast
• Digital Transmission
• European IST project ATTEST
• First demonstration based on ATTEST
• Diagram as follows
1st demo of a 3DTV service 3-D programs,the “video-plus-depth” 3-D
TS Contained two
on each contains
basedvideo stream
• an MPEG-2 coded color
data representation formatcoded depth-image DVB-T transmission.
• an H.264/AVC using a real sequence.
DTV-Recorder-Generator PC with a PCI DVB-T card
11
real-time replay of an offline- Received MPEG-2 TS was demultiplexed in software
generated MPEG-2 TS video bit streams were decoded in real-time
12. 3DTV Broadcast
• Digital Transmission
• “video-plus-depth” representation has been standardized
within MPEG as a result of work initiated by Philips and
Fraunhofer HHI.
• The new standard has been published in two parts:
• Specification of the depth format itself is called ISO/IEC
23002-3 (MPEG-C)
• a method for transmitting “video-plus-depth” within a
conventional MPEG-2 TS has become an amendment (Amd. 2)
to ISO/IEC 13818-1 (MPEG-2 Systems).
12
13. 3DTV Over IP Networks
• Background
• IP is proving to be flexible in accommodating
communication services
• Classical telephone -> VOIP
• Transmission of video over Internet is active in R & D
• VoD
• 2.5G and 3G offer wireless video service
• The IP itself leaves many aspects of the transmission to be
defined by other layers of the protocol stack and,
• thus, offers flexibility in designing the optimal
communications system for various 3-D data 13
representations and encoding schemes.
14. 3DTV Over IP Networks
• General Outline
• 3DTV streaming architectures
• Server Unicasting
• Server Multicasting
• P2P Unicasting
• P2P Multicasting
• Protocol
• Current state of the art: RTP/UDP/IP 14
• Next generation: RTP/DCCP/IP
15. 3DTV Over IP Networks
• Streaming Protocols
• Most widely used : RTP over UDP
• does not contain any congestion control mechanism
• lead to congestion collapse when large volumes of video are
delivered
• Datagram congestion control protocol (DCCP) is designed as
a replacement for UDP for media delivery
• TCP minus reliability and in-order packet delivery
• UDP plus congestion control, connection setup, and
acknowledgements 15
16. 3DTV Over IP Networks
• Streaming Protocols
• DCCP is a transport protocol that implements bi-directional
unicast connections of congestion-controlled, unreliable
datagrams.
• Despite of the unreliable datagram flow
• Reliable handshakes for connection setup/teardown and
reliable negotiation of options
16
17. 3DTV Over IP Networks
• Streaming Protocols
• DCCP also accommodates two congestion control
mechanisms.
• TCP-like Congestion Control
• TCP-Friendly Rate Control(TFRC)
• TCP-like Congestion Control
• identified by CongestionCCID2 in DCCP
• behaves similar to TCP’s AIMD congestion control
• halving the congestion window in response to a packet drop
• respond quickly to changes in available bandwidth 17
• must tolerate the abrupt changes in the congestion window size
18. 3DTV Over IP Networks
• Streaming Protocols
• TCP-Friendly Rate Control(TFRC)
• identified by CCID3
• a form of equation-based flow control that minimizes abrupt
changes in the sending rate while maintaining longer-term
fairness with TCP
• Appropriate for applications that would prefer a rather smooth
sending-rate with a small or moderate receiver buffer
• streaming media applications
18
19. 3DTV Over IP Networks
• Streaming Protocols
• TCP-Friendly Rate Control(TFRC)
• Operation: CCID3/TFRC calculates TFRC rate
• using the TCP throughput equation
• Request gives feedback to sender application
• Sender may use this rate information to adjust rate to get better
results
19
20. 3DTV Over IP Networks
• Streaming Protocols
• (exp)RFC for TCP-Friendly Multicast Congestion Control
(TFMC)
• compute the TFRC rate in a multicast scenario
• each receiver computes own TFRC rate as a function of RTT
loss rate
• server then selects the minimum of these rates
• (limited number clients to prevent feedback explosion)
• DCCP is the same way doing this.
• Hence, future 3DTV over IP services is expected to employ
the DCCP protocol with effective video rate adaptation to 20
match the TFRC rate.
21. 3DTV Over IP Networks
• Multiview Video Encoding and Rate
Allocation/Adaptation
• Multiview 3-D video can be represented and encoded
• Implicitly: “video-plus-depth” representation (discussed)
• Explicitly: in raw form
• a trade-off between
• random access
• ease of rate adaptation
• compression efficiency
• simulcast coding
• scalable simulcast coding
• multiview coding
• scalable multiview coding 21
22. 3DTV Over IP Networks
• Multiview Video Encoding and Rate Allocation/Adaptation
• The rate adaptation differs, since rate allocation between views
offers new flexibilities.
• According to the suppression theory of human visual
perception
• if the right and left views are transmitted and displayed with
unequal spatial, temporal and/or quality resolutions, the overall 3-
D video quality is determined by the view with the better resolution
• Therefore, rate adaptation may be achieved by
• adaptation of the spatial, temporal and/or signal-to-noise (SNR)
resolution of one of the views
• while encoding/transmitting the other view at full rate. 22
23. 3DTV Over IP Networks
• Multiview Video Encoding and Rate
Allocation/Adaptation
• Several open loop and closed loop rate adaptation strategies
• closed loop strategies
• client estimates some function of the received signal and feeds
it back to the transmitter
• The transmitter determines an optimized rate
• open loop strategies
• transmitter does not use feedback from the receiver
23
24. 3DTV Over IP Networks
• Multiview Video Encoding and Rate
Allocation/Adaptation
• open-loop rate adaptation strategies
• First paper: content-adaptive video scaling
• Rate adaptation has been achieved by
• 1) spatial subsampling;
• 2) temporal subsampling;
• 3) scaling the quantization step-size;
• 4) content-adaptive scaling
• content-adaptive video scaling approach
• Four categories: high/low temporal spatial detail.
• Scaling their resolutions
24
• Experiments show that better compression with better quality
25. 3DTV Over IP Networks
• Multiview Video Encoding and Rate Allocation/Adaptation
• open-loop rate adaptation strategies
• Second paper: adaptive selection of temporal levels and quality
layers
• video is encoded offline with a predetermined number of spatial,
temporal and SNR scalability layers.
• Content-aware bit allocation among the views is performed during
bit stream extraction by adaptive selection scalability layers
• The required bit rate reduction is only applied to one of the views.
• Experiments shows that better rate-distortion performance
compared to static cases.
25
26. 3DTV Over IP Networks
• Multiview Video Encoding and Rate Allocation/Adaptation
• closed-loop rate adaptation strategies
• rate adaptation is done at the server side by feedback from the user.
• First paper:
• The user’s head position is tracked and predicted
• The system allocates more bandwidth to the selected views in order
to render the current viewing angle.
• Make use of MVC and SVC
• Second paper:
• Each view is streamed to a different IP-multicast address
• A viewer’s client joins appropriate multicast groups to only receive
the 3-D information relevant to its current viewpoint 26
27. 3DTV Over IP Networks
• Error Correction and Concealment
• Sources: packet losses in the wired or wireless IP links
• Wired Internet: Congestion -> packet losses
• Wireless Internet: capacity limited by bandwidth of radio
spectrum
• Noise, interference and fading, error bursts(from mobility)
• Joint source and channel coding techniques
• Error concealment methods (at the decoder) to limit temporal
error propagation
27
28. 3DTV Over IP Networks
• Error Correction and Concealment
• Common error correction approaches for reliable
transmission
• ARQ
• ACK
• Delay, not desirable
• FEC
• In broadcast and multicast services, channel coding
techniques have been widely applied
28
29. 3DTV Over IP Networks
• Error Correction and Concealment
• First paper:
• Macroblock classification into unequally important slice groups
• Using FMO tool of H.264/AVC
• LT codes are used for error protection for low complexity and
advanced performace
29
30. 3DTV Over IP Networks
• Error Correction and Concealment
• Second paper:
• Stereoscopic video streaming using FEC techniques
• Frames are classified according to their contribution to overall
quality
• three layers used for UEP
• I-frame
• P-frame
• Left
• Right
• To find optimum packetization and UEP strategies
• Comparative analysis and simulation of Reed–Solomon (RS) and 30
systematic Luby transform (LT) codes
31. 3DTV Over IP Networks
• Error Correction and Concealment
• Error concealment algorithm for monoscopic not applicable
for stereoscopic.
• Based on interpolation -> is not sufficient for not depth info is
preserved.
• Human perception of errors in 3-D video is different
• A small degradation -> significant perceptual distortion
• Third paper: an error concealment algorithm
• Make full use of characteristic of stereoscopic video
• Based on the relativity of prediction mode of right frames ->
prediction mode of macroblock
• restore the lost macroblock according to the estimated motion 31
vector or disparity vector.
32. 3DTV Over IP Networks
• Error Correction and Concealment
• capabilities of error concealment
• To increase the quality of the reconstructed block
• a stereoscopic movie: the two views are highly
correlated(why)
• information about the corresponding region is highly useful
for the reconstruction of the lost block.
• corresponding pixel pairs identified using feature matching and
principles of epipolar geometry
• robust estimation of the transformation parameters is used to 32
educe the negative effect of outliers
33. 3DTV Over IP Networks
• 3D Video Streaming Demonstrations
• end-to-end prototype system for point-to-point streaming of
stereoscopic video over UDP
supports the
1.over a LAN autostereoscopic
Sharp 3-D laptop
with no
packet losses
supports a monocular
display to demonstrate
2.employs the backwards
protocol stack compatibility
RTP/UDP/IP
supports an in-house
polarized 3-D
projection display
system
that uses a pair
projector
33
34. Discussion and conclusion
• A comprehensive survey of the state-of-the art in
transport techniques
• While the transport solutions must address backwards
compatibility issues with the existing digital TV
standards and infrastructure
• 3DTV flexible
• Current and future research issues for 3-D TV
transmission
• joint transport and coding
• Why
• determination of the best rate adaptation method
• error resilient video encoding and streaming strategies 34