Smart phones are equipped with many sensors which provide detailed and continuous information of the device's location and movement. The use of such signals for vehicle movement inference presents many challenges due to signal noise, unknown phone orientation, varying device sensor quality and so on. Signal processing and feature engineering are generally difficult and require deep domain knowledge and manual pattern recognition. We discuss how deep learning can be leveraged in this context for automatic signal processing and feature engineering. We present several applications of deep learning in vehicle telematics as well as the deep learning architecture designed for learning sensor embeddings for vehicle movement events. One challenge we face is that model training requires huge volumes of sensor data, which must be processed efficiently. We present a solution using Spark for model development and batch deployment.

1. Wayne Zhang, Uber
Applications of Deep
Learning in Telematics
#UnifiedAnalytics #SparkAISummit

2. We want Uber to be the
safest transportation
platform on the planet.
Safety should be our
number-one priority. We
have to, as a company,
stand for safety.”
Dara Khosrowshahi (2018)
Stand for Safety

3. Driving Hours Limit Speed Limit Alert Ride Check
Example Driving Safety Products

4. Telematics
4
Source: Smartphone-based Vehicle Telematics - A Ten-Year Anniversary
- Wide availability
- Cheap
- Short upgrade cycle
- Lower quality
- Measure phone motion

5. Sensor Data (Driver Device)
● GPS
○ Absolute location, velocity and time
○ Low frequency (~0.5Hz)
● IMU
○ Relative motion of phone
○ Accelerometer: 3D linear acceleration
○ Gyroscope: 3D angular velocity
○ High frequency (~25Hz)

6. Motivation
● High-frequency signals
○ Intricate and diverse patterns
○ Dynamic over time
Driving WalkingDrivingHandling
Phone
On Train

7. Motivation

8. Sequence Classification
Classify whole sequence to certain events:
● Crash
● Driving events (brake, turn, speeding)
● Phone handling
● Rider complaint
● ...

9. Sequence-Sequence Prediction
Input sequence
(Phone sensor data)
Output sequence
Align to other sensor
phone => vehicle
Pinpoint telematics
events
(turn, activity)
Vehicle sensor
Turn event (binary encoding)

10. Pre-filtering
● High-frequency data result in huge # time steps
○ Pre-filtering: identify specific time window of interest
○ Window segmentation: divide input sequence into small windows
Design Choice
Window Segmentation
window0
window1
window2 windowT

11. Raw data
LSTM
Embedding
Window Prediction
[optional]

12. Feature Extraction
window 1 window 2 window t window T
- Time domain stats
(min, max, mean, sd)
- Frequency domain feature
(FFT)
1-D CNNSample Summary
Raw data
New Feature
Vector
LSTM LSTM

13. Data Augmentation
● Sensor readings depend on phone orientation
● Create augmented data by artificially rotating phone
○ New sensor readings
○ Label stays the same

14. Model
- SparkML Transformer
- XgBoost
- xM trips in training
- xM trips in validation
- Saved model pipeline
Data
- Sensor (Driver)
Score
- Sensor embedding
Model Dev Pipeline
Data
- Sensor (Driver)
- Map
- Trip
- Other
Label/Feature
- Telematics
- Trip
Label/Feature
- Event definition
- Feature
Model
- Multi-layer LSTM
- xM trips in training
- Saved protocol buffer
Non-DL
DL
Score
- Score and classify

15. Horovod
● Open source library developed at Uber
● Distributed training for TensorFlow, Keras & PyTorch
● Uses bandwidth-optimal communication protocols &
makes use of advanced networking
● Seamlessly installs via pip install horovod

16. ● Open source library developed at Uber ATG
● Enables deep learning directly from Parquet
● Supports Tensorflow, PyTorch, and PySpark
Petastorm
Apache Parquet as a dataframe with
tensors
nd-arrays,
scalars
(e.g. images,
lidar point
clouds)
Apache Parquet
store
Fog
Horse
Hedgehog

17. Performance
DL model DL model

18. Proprietary and confidential © 2019 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any
form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without
permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains
information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified
that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate,
or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent
necessary for consultations with authorized personnel of Uber.

19. Thank You!
Wayne Zhang
actuaryzhang@uber.com