Self-driving cars, voice assistants, autonomous robots, smart devices… Autonomous systems are reaching to and changing every part of our lives, and Deep Learning is the technology behind that change. Advanced levels of perception, enabled by Deep Learning, are key to the success of automated driving, from advanced driver assistance systems (ADAS) to fully autonomous driving.
Designing and deploying deep learning applications to embedded CPU and GPU platforms (as it is the case with cars) is challenging because of resource constraints inherent in embedded devices. In this session, you will be exposed to some of the most relevant and stimulating real-world problems in ADAS, focusing on the role played by Deep Neural Networks (DNN): image classification, object detection and localization, semantic segmentation and deployment. You will get a walkthrough of a complete workflow including data preparation (properly ground-truth labeling of datasets and driving scenes) and visualization, creating or fine-tuning DNNs, training such networks leveraging NVIDIA GPUs to build automated driving capabilities and generating portable and optimized CUDA code that can be deployed on boards (NVIDIA Jetson TX2 and NVIDIA DRIVE PX) leveraging TensorRT for very fast inference.
Video: https://youtu.be/Oal-ac7QbkE
2. 2
A brief history of the automobile
Attribution: DaimlerChrysler AG (CC-BY-SA-3.0),via Wikimedia Commons
1885
FIRST COMMERCIAL GAS CAR
Benz Patent-Motorwagen
3. 3
1885
FIRST COMMERCIAL GAS CAR
Benz Patent-Motorwagen
A brief history of the automobile
1908
FIRST MASS PRODUCED CAR
Ford Model T
Attribution: Harry Shipler (Public domain), via Wikimedia Commons
4. 4
1885
FIRST COMMERCIAL GAS CAR
Benz Patent-Motorwagen
A brief history of the automobile
1908
FIRST MASS PRODUCED CAR
Ford Model T
1911
ELECTRIC SELF-STARTER
C.F. Kettering (DELCO) Attribution: Charles F. Kettering, U.S. Patent 1,150,523
5. 5
1911
ELECTRIC SELF-STARTER
C.F. Kettering (DELCO)
A brief history of the automobile
1939
FIRST AUTOMATIC TRANSMISSION
Hydra-Matic Drive - Cadillac &
Oldsmobile
Attribution: Michael Barera (CC BY-SA 4.0),via Wikimedia Commons
6. 6
1911
ELECTRIC SELF-STARTER
C.F. Kettering (DELCO)
A brief history of the automobile
1939
FIRST AUTOMATIC TRANSMISSION
Hydra-Matic Drive - Cadillac &
Oldsmobile
1958
MODERN CRUISE CONTROL
Chrysler Imperial Convertible
Attribution: Lars-Göran Lindgren, Sweden (CC BY-SA 3.0), via Wikimedia Commons
7. 7
1911
ELECTRIC SELF-STARTER
C.F. Kettering (DELCO)
A brief history of the automobile
1939
FIRST AUTOMATIC TRANSMISSION
Hydra-Matic Drive - Cadillac &
Oldsmobile
1958
MODERN CRUISE CONTROL
Chrysler Imperial Convertible
1971
ANTI-LOCK BRAKING SYSTEM
“Sure-Brake”, First Computerized ABS
Attribution: Chris828 (Public domain),
via Wikimedia Commons
8. 8
A brief history of the automobile
1971
ANTI-LOCK BRAKING SYSTEM
“Sure-Brake”, First Computerized ABS
1981
FIRST ECU
General Motors, Motorola
9. 9
A brief history of the automobile
1971
ANTI-LOCK BRAKING SYSTEM
“Sure-Brake”, First Computerized ABS
1981
FIRST ECU
General Motors, Motorola
1996
FIRST CONNECTED CAR
OnStar
Attribution: Tyler from Riverside, USA (CC BY 2.0), via Wikimedia Commons
10. 10
A brief history of the automobile
1971
ANTI-LOCK BRAKING SYSTEM
“Sure-Brake”, First Computerized ABS
1981
FIRST ECU
General Motors, Motorola
1996
FIRST CONNECTED CAR
OnStar
2000
LANE DEPARTURE WARNING SYSTEM
Mercedes-Benz Actros
11. 11
A brief history of the automobile
1971
ANTI-LOCK BRAKING SYSTEM
“Sure-Brake”, First Computerized ABS
1981
FIRST ECU
General Motors, Motorola
1996
FIRST CONNECTED CAR
OnStar
2000
LANE DEPARTURE WARNING SYSTEM
Mercedes-Benz Actros
13. 13
Localization and PlanningLocalization and Planning
Perception Control
Most prominent areas in automated driving
ControlPerception
Deep learning
Path planning
Sensor models &
model predictive control
Sensor fusion
14. 14
Perception Control
Localization and Planning
Focus of today’s presentation
Perception
Deep learning
Sensor fusion
Sensor models &
model predictive control
Path planning
15. 15
Shallow Machine Learning vs. Deep Learning
Shallow Machine Learning
Deep Learning
Deep Learning learns
both features and tasks
directly from data
Machine Learning learns
tasks using features
extracted manually from data
End-to-End Learning
16. 16
▪ Train “deep” neural networks on structured data (e.g. images, signals, text)
▪ Implements Feature Learning: Eliminates need for “hand crafted” features
▪ Trained using GPUs for performance
Convolutional Neural Networks
Convolution +
ReLu PoolingInput
Convolution +
ReLu Pooling
…
…
Flatten Fully
Connected
Softmax
car
truck
bicycle
…
van
…
…
Feature Learning Classification
17. 17
Deep Learning Workflow
Select Network
Architecture
Build from scratch
Interoperability
Use/tune pretrained
networks
3
Images
Signals
Text
Access and Explore Data
1
Share and Deploy
Share and export
Enterprise Scale
Systems
Embedded Devices
and Hardware
5
Perform Training
CPU vs. GPU
Hyperparameter
tuning
Scale training
4
Label and
Preprocess Data
Label training data
Data augmentation
Synthetic Data
2
24. 24
Semantic Segmentation
CamVid Dataset
1. Segmentation and Recognition Using Structure from Motion Point Clouds, ECCV 2008
2. Semantic Object Classes in Video: A High-Definition Ground Truth Database, Pattern Recognition Letters
25. 25
Load and plot training images
% Create datastore for images
imds = imageDatastore(imgDir);
I = readimage(imds, 1);
I = histeq(I);
imshow(I)
imageDatastore
manages large collections
of images
26. 26
Load and overlay pixel labels
% Load pixel labels
classes = ["Sky"; "Building";...
"Pole"; "Road"; "Pavement"; "Tree";...
"SignSymbol"; "Fence"; "Car";...
"Pedestrian"; "Bicyclist"];
pxds = pixelLabelDatastore(...
labelDir,classes,labelIDs);
% Display labeled image
C = readimage(pxds, 1);
cmap = camvidColorMap;
B = labeloverlay(I,C,'ColorMap',cmap);
imshow(B)
pixelLabelDatastore
manages large collections
of pixel labels
27. 27
Visualize distribution of labeled pixels
% Visualize label count by class
tbl = countEachLabel(pxds)
frequency = tbl.PixelCount / ...
sum(tbl.PixelCount);
bar(1:numel(classes),frequency)
xticks(1:numel(classes))
xticklabels(tbl.Name)
xtickangle(45)
ylabel('Frequency')
Likely to
detect roads
Unlikely to
detect
bicyclist
Labeled pixels in this set are
imbalanced
32. 32
options = trainingOptions('sgdm', ...
'Momentum', 0.9, ...
'InitialLearnRate', 1e-2, ...
'L2Regularization', 0.0005, ...
'MaxEpochs', 120, ...
'MiniBatchSize', 4, ...
'Shuffle', 'every-epoch', ...
'Verbose', false, ...
'ExecutionEnvironment', 'auto', ...
'Plots','training-progress');
Deep learning on CPU, GPU, multi-GPU and clusters
Single CPU Single CPU
Single GPU
Single CPU
Multiple GPUs
On-prem server with
GPUs
Cloud GPUs
(AWS, Azure, etc.)
34. 34
Evaluate trained network on image
% Plot actual results
I = read(imdsTest);
actual = semanticseg(I, net);
B = labeloverlay(I, ...
actual,...
'Colormap', cmap,...
'Transparency',0.4);
imshow(B)
pixelLabelColorbar(cmap, classes);
title('Actual')
35. 35
Visually compare actual with original labeled results
% Plot expected results
% using original labels
expected = read(pxdsTest);
E = labeloverlay(I,...
expected,...
'Colormap', cmap,...
'Transparency', 0.4);
imshow(E)
title('Expected');
36. 36
Visually compare actual with original labeled results
% Plot differences
imshowpair(...
uint8(actual),...
uint8(expected));
title('Difference');
39. 39
Distribution of labels in data affects intersection-over-union (IoU)
Underrepresented classes such as Pedestrian and Bicyclist are
not segmented as well as classes such as Sky and Road
Distribution of labels in original data set Evaluation metrics of network
44. 44
Data preparation and labeling of LiDAR is a challenge
Trained
DNN
DNN
design + training
Accessing
LiDAR data
TrainingLiDAR pre-
processing
Labeling
LiDAR data
45. 45
Access and Visualize LiDAR Data
Access Stored Lidar Data
▪ Velodyne file I/O (pcap)
▪ Individual point clouds (.pcd,ply)
▪ Custom binary formats
Visualize Lidar Data
▪ Streaming LiDAR player
▪ Static point cloud display
▪ Point cloud differences
46. 46
Lidar Preprocessing
Remove Ground
• Fit plane using RANSAC
• segmentGroundFromLidarData
Cluster
• Segment clusters using
Euclidean distance
• segmentLidarData
61. 61
Learn more about perception applications for Deep Learning and
Automated Driving
Deep Learning Toolbox | Automated Driving System Toolbox | GPU Coder
Deep Learning
https://www.mathworks.com/deeplearning
Automated Driving
https://www.mathworks.com/adas
Visit our booth!
