The expanding demand for imaging- and vision-based systems in mobile, IoT and automotive products is making the need for multi camera and sensor fusion systems look for novel ways to gather and process multiple data streams while still fitting into the mobile interface. The CSI-2 protocol allows camera sensor and processed image data to be combined into a single data stream using interleaving, allowing the application processor to extract the image data using the virtual channel or data type information. In this presentation, Richard Sproul of Cadence Design Systems will highlight some of the key details and requirements for a system with image processing of multi camera/sensor systems.

1. MIPI CSI-2℠
CSI-2℠ Application for Vision
and Sensor Fusion Systems
Richard Sproul – Cadence
Design Systems, IP Architect

2. Overview
• The expanding demand for imaging and vision based
systems in mobile, IoT and automotive products is
creating the need for multi-camera and sensor fusion
systems to look for novel ways to gather and process
multiple camera/sensor data streams whilst still fitting
into the mobile interface.
• The presentation will highlight some of the key details
and requirements for a system with image processing
of a multi-camera/sensor system.
2

3. CSI-2 Application for Multi-Sensor Systems
Multi-Camera Applications
• Imaging applications are everywhere
• Mobile Phone
– Selfie Picture in Picture
– Gesture Recognition
• Video Games
– Gesture Recognition
• Autonomous Driving
– Pedestrian Detection
– Signage Recognition
– Night Vision
– Parallel Parking!
• In-Car Control
– Gesture Recognition

4. 4
More than the
naked eye…
CSI-2 Application for Multi-Sensor Systems
Machine Vision

5. 5
CSI-2 Application for Multi-Sensor Systems
Camera Applications
Op0mal pathway for mul0ple forward-looking advancements in
imaging
– Key Drivers: Health, Convenience, Security, Lifestyle, Efficiency
– High-perf pixel conduit needs met with C/D-PHY advancements
– Broad defini0ons and fuzzy range: (i.e. Wearable: Near Body, On
Body, In Body)
• Explore possibili0es of overlap between Imaging and low-speed
sensor requirements and solu0ons
Camera Controller Interface (CCI/CCS)
advancement considera0ons:
- Point-to-Point and Mul0-Drop configura0ons
- Energy consumed / Gb transfer
- Limit latency for VB & HB
- Precision Timing & Sync
- Independent Transport: Pixel Data & Control
- Channel Integrity (Error Detec0on)
- FW Upload (ISP, Neural)

6. • Options for physical interface
• Pins, legacy, bandwidth
6
CSI-2 Application for Multi-Sensor Systems
MIPI CSI-2 Interfaces

7. • Evolution of the CSI-2
7
CSI-2 Application for Multi-Sensor Systems
CSI-2 Generations

8. 8
CSI-2 Application for Multi-Sensor Systems
CSI-2 Performance

9. • CSI-2 packets V1.x to V2.0
9
LP
LP
LP
LP
Transi0on between packets to LP state for PHY data lane (100ns)
Transi0on between packets by using filler pa_erns
CSI-2 Application for Multi-Sensor Systems
CSI-2 Packet Structure

10. • Improve the effective bandwidth
10
0
20
40
60
80
100
120
1000 1250 1500 1800 2000 2250 2500
Frame Rate fps
Bit Rate (Mbps)
CSI-2 Frame Rate Improvement V1.x to V2.x
1920x1080 RAW12
FPS (V1.x) FPF (V2.x)
CSI-2 Application for Multi-Sensor Systems
CSI-2 Packet Transmission

11. • Automotive application for driver assistance
- External systems and for in-car control
• Objects
• High
resolu0on
• Night image
and IR
• In-car
gesture
• People
detec0on
• Medium
resolu0on
• Road signage
• Medium
resolu0on
Parking
assistance
CSI2 Application for Multi-Sensor Systems
Advanced Driver-Assistance System

12. CSI-2 Application for Multi-Sensor Systems
• Application in a Multi-Camera Platform
12
Automotive AV
Reference Subsystem
MIPI
DPHY
Audio DSP
$I $D
System Interconnect
Image/Vision
DSP
DMA I-RAM D-RAM
AXI2
AHB
UART I2C
32b APB
Timer I2S GPIO
AHB2
APB
32b AHB
QSP
I
Soun
dWir
e
Audio
USB
2/3
devic
e
Ethe
rnet
MAC
On-Chip
System SRAM
1300MT/s
DDR3
Controller
DDR-PHY
64b DDR3 SODIMM
SD
SDIO
eMM
C
Displa
y
Inm.
BR
PHY
USB
PHY
Pixel
2AXI
Color
Conver
t
Video
Scalar
HDMI
PHY
Image/Vision
DSP
DMA I-RAM D-RAM
MIPI
CSI-2
Rx
MIPI
CSI-2
RX
MIPI
CSI-2
Rx
MIPI
DPHY
MIPI
DPHY
Sensor DSP
$I

13. 13
CSI-2 Application for Multi-Sensor Systems
Sensors Everywhere

14. • The data does not have to be images…
• LiDAR
• The resolution is low (IR RAW data, typically 64 pixels high, though much more
horizontally)
• Range is limited. Typical LiDARs see well to about 70 metres.
• Refresh rates tend to be slower, at around 10fps.
• RADAR
• Long range – cruise control, brake assist
• Ultrasonic
• Short-range parking assist
• Self parking ☺
• Protocol support with user-defined data to transfer the bytes
14
CSI-2 Application for Multi-Sensor Systems
CSI-2 for Sensors

15. CSI-2 Application for Multi-Sensor Systems
CSI-2 Example Video Frame
• Bandwidth on CSI-2 V1.1 – 4 Lanes 6Gbps
• So with our 30fps, we have 200M bit to use
• 3 HD camera RGB888 1920x1200x24=165.888M
• Also adding 100ns gaps (150 bit clocks)
• 3 x(1920x24) +3x150 = 138240
• Embedded data line with image processed data (clusters, edges)

16. 16
CSI-2 TX Controller
Video Buffer
Centralized ECU for
Infotainment or ADAS
Pixel
Pixel
Data
Data
Sensor DSP
Sensor DSP
PCIe
Sensor
DSP
I2S
I2S
I2S
SoundWire
eMMC
Apps
USB
DSI
DDR
DPHY
PCIe
PHY
USB
PHY
Sensor
Sensor
Video EP
Video EP
PPI
PPI
DPHY
DPHY
Pixel
Pixel
PHY
Data
Data
CSI-2 domain
Video Buffer
Data Buffer
Data Buffer
I2S
SoundWire
Sensor
DSP
CSI-2 RX
CTRL
Vision
DSP
CSI-2
Vision
DSP
CSI-2
CSI-2 Application for Multi-Sensor Systems
CSI-2 Sensor Fusion Example
• Sensor Fusion ADAS System Topology
• Merge the data from image and other sensors
• Pre-processing the inline data for the application

17. • Image Processing and the Application
• Application processing will need to perform the ADAS system and
sensor analysis
17
CSI2 Application for Multi-Sensor Systems
CSI-2 Sensor Fusion Example

18. 18
CSI-2 Application for Multi-Sensor Systems
Filling the Channels
D PHY
(MCNN)
D-PHY
(MFEN)
Pixel
Processor/
Application
DP
DP
DN
DN
D PHY
(SCNN)
D-PHY
(SFEN)
Pixel Stream 0
PPI
PPI
D-PHY
(MFEN)
DP
DN
D-PHY
(SFEN)
PPI
Pixel Stream 1
Pixel Stream 2
Pixel Stream 3
Sensor
Processor/
Application
Pixel Stream 4
Pixel Stream 5
Pixel Stream 6
Pixel Stream 7
CSI-2 Host
Controller
PPI
PPI
PPI
D-PHY
(MFEN)
D-PHY
(SFEN)
DP
DN
CSI-2 Slave
Controller
PPI PPI
Sensor
Control
Sensor
Control
Sensor
Sensor
Sensor
Sensor
Sensor
DATA
DATA
DATA
DATA
DATA
DATA DATA
CSI-2 Slave
Controller
CSI-2 Host
Controller

19. 19
CSI-2 Application for Multi-Sensor Systems
Physical Interface

20. CSI-2 Application for Multi-Sensor Systems
CSI-2 Example Video Frame
• Bandwidth on CSI-2 V1.1 – 4 Lanes 6Gbps
– Using LS/LE to keep synchronisation and sequence
– Use the virtual channel to identify the sensor
– Use the data types (RAW, RGB, YUV and user defined)
– Use the short packet sync events

21. • Functional safety considerations
• DPHY BER, RX error detection
• Packet header ECC
• Payload CRC
• SP sync sequences, counting values
21
CSI2 Application for Multi-Sensor Systems
Functional Safety In CSI2 ADAS

22. • Pitfalls of interleaved streams
22
CSI-2 Application for Multi-Sensor Systems
CSI-2 Interleaving Data

23. 23
CSI-2 Application for Multi-Sensor Systems
CSI-2 Interleaving

24. CSI-2 Beyond Mobile
24

25. CSI-2 Beyond Mobile
• System architecture considerations for CNN
applications:
• Assist
• Co-pilot
• Automated
• Optimal platform arch for the CNN engines
• Central processing (+SW dev, lacks scalability / modularity, cost
may not align w/ entry-level cars)
• Distributed processing (plug-and-play, scalable, each camera unit
enhances capability, complex system)
• AlgoEngine: CPU / GPU / DSP / FPGA
• Overall risks and uncertainty:
• Market, product, execution, timing, regulators, infrastructure
25

26. CSI-2 Beyond Mobile
• What can technology do for us?
• Imaging: digital photography vs. vision
• Scene capture, object capture & track, modeling & measurement
• Perception and decision-making using real-time
streaming image data:
• Camera, RADAR, LiDAR, sonar (varying detection capabilities vs.
cost)
• Performance vs robustness – consequence of error
• Camera position, lighting, environmental factors, required accuracy
for object detection
26