This project was developed for a Digital Control class. It consists of a system that is able to identify and track lane marks in a video acquired by webcam. It's interesting how the Kalman filter is used in such a context in order to make the lane detection computationally feasible in the small amount of time between two subsequent video frames

1. A VISION-BASED LANE TRACKING
SYSTEM FOR AUTONOMOUS DRIVING
USING KALMAN FILTER
Department of Information Engineering
University of Pisa
A. Biondi – .
F. Corucci – .
2011
Project for Digital Control class
Prof. A. Balestrino, Prof. A. Landi

2. Aim of the project
 Build a vision-based lane tracking system using
Kalman filter
 Test Kalman filter effectiveness in a practical noisy
scenario
 Experience with some computer vision algorithms
and techniques
Project for Digital Control class - University of Pisa 2

3. Why lane tracking is useful?
Lane
tracking
Steering
control to
follow the
road
Camera
view
 Vision-based lane
tracking is commonly
used to build
autonomous cars
capable of following a
road without human
intervention
Project for Digital Control class - University of Pisa 3

4. Video acquisition
 In order to collect a
dataset to work with, we
placed a netbook on an
hand-cart and recorded
a video from the
webcam while following
a realistic circuit as best
as possible
Project for Digital Control class - University of Pisa 4

5. Video acquisition
Project for Digital Control class - University of Pisa 5
RGB video frame grabbed from the webcam

6. Image preprocessing - Overview
Project for Digital Control class - University of Pisa 6
 Some preprocessing is needed on the RGB video frame
in order to perform feature discrimination:
RGB frame
Grayscale
Equalization Filtering
Binarization
Edge detection

7. Grayscale conversion
Project for Digital Control class - University of Pisa 7
Grayscale video frame
Color distribution
Floor
Lane
markers

8. Enhancing color separation
Project for Digital Control class - University of Pisa 8
Enhanced color
separation
Resulting color distribution
Floor
Lane
markers
This was achieved using the
decorrelation stretch method,
commonly used for feature
discrimination in images.
(Correlated images does not
usually have sharp contours)

9. Filtering floor components
Project for Digital Control class - University of Pisa 9
Image after filtering
Resulting color distribution
Floor
Lane
markers
FILTER
This was achieved enhancing the
image contrast through color
saturation (with appropriate
thresholds).
Floor is saturated to black

10. Image binarization
Project for Digital Control class - University of Pisa 10
Binarized image
 The image is now a matrix of 0 and 1

11. Edge detection
 Once the interesting features (i.e. lane marks) are
enhanced, we detect edges in the video frame, in
order to identify lane-marks contours to be
followed
 The Canny algorithm (a derivative filter) was used
for this purpose
Project for Digital Control class - University of Pisa 11

12. Edge detection - Canny
Project for Digital Control class - University of Pisa 12
Frame after Canny application
The linear contours are now clearly identifiable

13. Line detection
 We are now able to extrapolate the analytic
equations of the lane-mark borders
 At the first frame, we have no idea of where lane
marks could be, so we need to perform a global
scan
 We used the Hough Transformation in order to
detect lines in this situation
Project for Digital Control class - University of Pisa 13

14. Outline on Hough transformation
Project for Digital Control class - University of Pisa 14
θ
 Allows to discriminate features
described by an analytical
equation
 It has been generalized in order
to detect arbitrarily complex
shapes
 The problem of searching for a
feature is translated in a max
problem
Example of Hough
detection on a complex
shape

15. More on Hough transformation
Project for Digital Control class - University of Pisa 15
θ
 The equation that ties the curve parameters 𝑎𝑖 to the
coordinates 𝑥, 𝑦 looks like:
𝑓 𝑥, 𝑦 , 𝑎1, … , 𝑎 𝑛 = 0
 Every point 𝑥𝑖, 𝑦𝑖 of the image space generates a
hyper-curve in the parametric space
 A point in the parametric space identifies uniquely a
curve in the image space
 N points of the image space generate N curves in the
parametric space with a common intersection that is the
curve on which they possibly lie on

16. More on Hough transformation
Project for Digital Control class - University of Pisa 16
θ
 In our case the curves in the parametric space are still
lines
𝑦 = 𝑚𝑥 + 𝑞 → 𝑓 𝑥, 𝑦 , 𝑚, 𝑞 = 0
 Given a point 𝑥𝑖, 𝑦𝑖 , the parametric equation is
𝑞 = 𝑦𝑖 − 𝑚 ∗ 𝑥𝑖
 Every intersection in the parameter space is interpreted
as a vote for the corresponding curve
 A curve that has lot of votes identifies (with a certain
confidence) a relevant feature

17. Line detection
Project for Digital Control class - University of Pisa 17
Hough transform parameters space
Peaks identifiyng lines
Detected lines,
superimposed on the
original RGB frame
θ

18. Inverse perspective transform
 Work in perspective space is not convenient
 A common way to avoid it is performing an inverse
perspective transformation (→ Bird’s Eye view)
 The achieved «virtual» top view is much more
convenient in order to measure the distances and
the angles we need
 Various auxiliary operation in our application are
performed in this space
Project for Digital Control class - University of Pisa 18

19. Inverse perspective transform
Project for Digital Control class - University of Pisa 19
I → W map

20. Inverse perspective transform
Project for Digital Control class - University of Pisa 20
Bird’s eye view («virtual» top view)
I space
W space
Perspective view

21. So, where’s Kalman?
 The problem with this approach is that performing a
HoughTransformation on the whole image is
computationally heavy: with a reasonable video frame
rate for a driving application, there is no time to
perform Hough between two subsequent frames
We can exploit Kalman to drastically reduce the search
area in every frame and subsequently detect lane-markers
in a computationally efficient way
Project for Digital Control class - University of Pisa 21

22. Kalman – the intuition
 Once we have an idea of where the lane marks are at the
first frame, a Kalman filter is used in order to predict where
the lane marks will be in the next video frame
 The system then evolves step-by-step, frame after frame,
through Kalman predictions, until this is possible (i.e.
Kalman is able to lock-on the lane mark)
 Every Kalman prediction identifies a very thin region of the
next frame, in which we can perform a local search in a very
efficient way (preprocessing, binarization, pixels fit)
 If something goes wrong and we are not able to identify the
lane mark in the predicted band, a global Hough scan is
performed
Project for Digital Control class - University of Pisa 22

23. Algorithm overview
Project for Digital Control class - University of Pisa 23
1-st frame
• Hough detection
• Kalman initialization
• Kalman prediction for frame no. 2
k-th frame
• Fit inside Kalman predicted band
• If fit fails -> Perform Hough Detection and init. Kalman
• Kalman prediction for frame no. K+1
…
…
…
…

24. Model details
𝐴 𝑘+1= 𝐹𝐴 𝑘 + 𝐺𝑢 𝑘+ 𝑤 𝑘
𝑦 𝑘 = 𝐻𝐴 𝑘+ 𝑣 𝑘
Project for Digital Control class - University of Pisa 24
 𝑨 = state vector =
𝒎
𝒒
Where 𝑚 and 𝑞 are coefficients of a line expressed as: 𝑦 = 𝑚𝑥 + 𝑞
A linear model was sufficient for our short-range view
 𝑭 = autonomous dynamic
Models the evolution of the state when no input is applied. In order to simplify the
model, considered the low velocity and the high frame rate, we have taken 𝐹 = 𝐼
 𝒖 = input vector, models the vehicle steering (here simulated)
 𝑮 = maps the input on the state
 𝒘 𝒌 = process noise ~ 𝑵(𝟎, 𝑷)
 𝒗 𝒌 = measure noise (fit error due to pixel discretization) ~ 𝑵(𝟎, 𝑸)
 𝒚 𝒌 = output (for us, output = state)
 𝑯 = maps state on the output (for us, 𝐻 = 𝐼)

25. Kalman algorithm
Project for Digital Control class - University of Pisa 25
 𝑃𝑘 = covariance matrix of the process noise 𝒘
 𝑄 𝑘 = covariance matrix of the measurement noise 𝒗
 𝐾𝑘 = Kalman gain
Predicted state Estimated state

26. Kalman-based detection
Project for Digital Control class - University of Pisa 26
Left lane-mark Right lane-mark Legend:
- Green region:
Kalman predicted
band
- Blue line:
Fitted line
- White line:
Lane-mark
contour (real
pixels)

27. Experimental results
Project for Digital Control class - University of Pisa 27
<video simulation>
 As shown in the simulation, Kalman is able to track
lane-marks without problems, even:
 in presence of sudden anomalous car movements
 with a simplified linear model (the curve is well tracked!)
 with a simplified model that does not consider the autonomous
dynamic of the system
 Hough recalculation is triggered only when the left lane-
mark disappears from the camera-field: a polling mode
is triggered in this situation, and the lane-mark is
locked-on again when it returns in the camera-field

28. Future work
Project for Digital Control class - University of Pisa 28
 Use the tracking information in order to make a
vehicle autonomously follow the circuit (a simple
PID can be used to control the steering)
 Simulation
 Implementation
 Mounting a netbook running MATLAB on a toy car equipped with a
camera
 DSP + μcontroller based implementation

29. Bibliography
Project for Digital Control class - University of Pisa 29
 «A massively parallel approach to real-time
vision-based road markings detection»
Alberto Broggi – University of Parma
http://www.ce.unipr.it/people/broggi/publications/detroit.pdf
 «Lane detection and Kalman-based
linear-parabolic lane tracking»
Lim, Seng,Ang, Chin -The University of Nottingham Malaysia campus
Published at IHMSC’09 - Intelligent Human-Machine Systems and Cybernetics
 «La trasformata di Hough»
Padova University, Computer vision course 07/08
http://vision.unipv.it/corsi/VisioneArtificiale-ls/lucidi/VA-06.pdf

30. Thank you!
Project for Digital Control class - University of Pisa 30