Point Cloud Processing: Estimating Normal Vectors and Curvature Indicators using Eigenvectors
1. 1Challenge the future
Basic Point Cloud Processing
Estimating Normal Vectors and Curvature Indicators
Ir. Pirouz Nourian
PhD candidate & Instructor, chair of Design Informatics, since 2010
MSc in Architecture 2009
BSc in Control Engineering 2005
Geo1004, Geomatics Master Track
Directed by Dr. ir. Sisi Zlatannova
2. 2Challenge the future
How do we make a computer ‘see’ what we understand from this?
What do you make of this?
What is this all about?
3. 3Challenge the future
Data to Information: LIDAR to LOD2
How do we make sense out of such big data?
Image courtesy of GEOCONNEXION
?
4. 4Challenge the future
SOFTWARE APPLICATIONS
• FME, Safe Software
• LASTools, rapidlasso
• Point Cloud Library (PCL)
• Cloud Compare
• MeshLab
• LP360 QCoherent
A NEW VERSION OF TOIDAR! YES!
http://opentopo.sdsc.eduInformation from:
6. 6Challenge the future
LASTools, rapidlasso
filtering, clipping, reprojecting, compression, classification, DSM, DTM, TIN, contours bare-earth
de facto standards for
point cloud data:
.las
.laz
8. 8Challenge the future
Point Cloud Library PCL
point clouds, visualization, processing, segmentation, filtering,
feature estimation, registration
Using this library in Rhino?
9. 9Challenge the future
Cloud Compare
Implements PCL and more methods, handy to use for point
cloud processing
Image from software.informer.com
10. 10Challenge the future
MeshLab
Has some surface reconstruction methods, handy to have when
working with point clouds.
Image from https://danieleferdani.wordpress.com/
12. 12Challenge the future
Part 1 finished
We will continue with one specific way of dealing with point
clouds and estimating normal vectors and curvatures…
14. 14Challenge the future
A Curvature-Based Approach to Point Cloud
Segmentation & Feature Detection
Removing
Outliers
Elevation
Classification
Slope
Classification
Aspect
Classification
Region-Growing
Segmentation
Point Cloud
of a Building
Noise
reduction
Forming
Neighbourhoods:
e.g. KNN or Range
Forming
Covariance
Matrices for
Neighbourhoods
Using PCA, Estimating:
Normal Vectors &
Curvature Indicators
Normal
Vectors
Curvature
Indicators
Eigen
System
Triangulation
Mesh
Surface
15. 15Challenge the future
Different Approaches for Finding Fitting Planes/Edges
1. Using curvature values computed (estimated) by eigenvalues of
covariance matrices, to run a segmentation algorithm based on region
growing.
2. Using Octree Voxels, to detect edge voxels, remove them and create
segments.
3. Using Hough transform on a 2.5D point cloud, converting to the
parameter space and using DBScan clustering method to find
clusters of parameters, each of which correspond to a segment in a
point cloud.
4. And a few more in the literature…
16. 16Challenge the future
Different Approaches for Finding Fitting Planes/Edges
Region growing based on
normals and curvature Segmentation usingOctree voxels Hough transformation
17. 17Challenge the future
Different Approaches for Finding Fitting Planes/Edges:
our experience!
Curvature based region
growing
Voxel based region
growing
Hough transform
Noise management Fair Good Good
Density variation
management
Fair Fair Good
Efficiency Good Good Poor
Ambiguity of vantage
point management
Fair Good Good
Avoiding priori
knowledge
Good Good Poor
Ease of further
processing
Good Poor Poor
18. 18Challenge the future
Neighbourhoods of Points
Fixed Distance Neighbors (FDN) and K-Nearest Neighbors (KNN)
• Rabbani, T., van den Heuvel, F., & Vosselmann, G. (2006). Segmentation of point clouds using smoothness
constraint. International Archives of Photogrammetry, Remote Sensing and Spatial Information
Sciences, 36(5), 248-253.
• Pauling, Frederick, Michael Bosse, and Robert Zlot. "Automatic segmentation of 3d laser point clouds by
ellipsoidal region growing." Australasian Conference on Robotics and Automation. 2009.
The idea is to form neighborhoods based on [squared] distance
These two options are usually provided. Apart from normal vector
estimation KNN can be useful also for removal of outliers.
19. 19Challenge the future
Fitting Planes to Neighbourhoods
The idea is that the underlying surface is a 2-manifold; therefore it
resembles a 2D plane locally
Least Square Plane Fitting Estimation Problem
Principal Component Analysis Problem
Intuition: Defining plane as the locus of lines that have
direction vectors perpendicular to a normal vector;
considering an origin for the plane in questions, we can
consider the plane as the locus of points A such that A-
O is perpendicular to N.
20. 20Challenge the future
Fitting Planes to Neighbourhoods
The idea is that the underlying surface is a 2-manifold; therefore it
resembles a 2D plane locally, we look at ellipsoids showing local
dispersions
Images courtesy of Olga Sorkine
21. 21Challenge the future
How to estimate normals using PCA
The idea is that the underlying surface is a 2-manifold; therefore it
resembles a 2D plane locally
• Pauly, Mark, Markus Gross, and Leif P. Kobbelt. "Efficient simplification of
point-sampled surfaces." Proceedings of the conference on
Visualization'02. IEEE Computer Society, 2002.
• Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W. Surface
reconstruction from unorganized points. SIGGRAPH 92, 1992
• Shaffer, E., Garland, M. Efficient Adaptive Simplification of Massive
Meshes. IEEE Visualization 01, 2001
22. 22Challenge the future
How to estimate normals using PCA
We form a covariance matrix for each neighborhood, showing how
neighbors are dispersed around their average (centroid).
This will be a 3
by 3 symmetric
matrix!
23. 23Challenge the future
How to estimate normals using PCA
We form a covariance matrix for each neighborhood, showing how
neighbors are dispersed around their average (centroid).
This will be a 3
by 3 symmetric
matrix!
Form an Eigen system for this matrix using a
linear algebra library, and the first eigenvector
corresponding to least eigenvalue will be the
normal vector at each neighbourhood.
Explanation follows…
PCL implementation:
http://pointclouds.org/documentation/tutorials/normal_estimation.php
24. 24Challenge the future
How to estimate curvature using PCA
The idea is to use an indication of change along the normal vector
Jolliffe, I. Principle Component Analysis. Springer-Verlag, 1986
25. 25Challenge the future
How to do all this in code?
• We try not to reinvent the wheel; the idea is to use free open
source libraries like Math.NET and Accord.NET
• We are not the first people dealing with such issues, these are
generally matters of data mining and machine learning
• We can find KNN neighbourhoods using
Accord.NET http://accord-framework.net/
• We can find eigenvalues and eigenvectors
using MetaNumerics.dll, MathNet.dll or
Accord.NET
• You will receive example code implementing
MetaNumerics
26. 26Challenge the future
Fitting Planes to Neighbourhoods
How do we minimize the error in fitting a plane to the
mentioned neighbourhood? First we define it…
Images courtesy of Olga Sorkine
27. 27Challenge the future
An explanation after Olga Sorkine:
• Input points:
• Centroid:
• Vectors from the centroid:
m
29. 29Challenge the future
Continued…
m
m minimizes SSD, formally it is the
“arg min” of the following function
(to be minimized), i.e. the argument
that makes it reach its minimum. and
it can be shown that m is the
centroid C
Solving this turns out to be equal to
solving an Eigen system for the
covariance matrix; you can see how…
31. 31Challenge the future
Long story short…
We find eigenvalues and eigenvectors of the covariance matrices…
What was this all about? Segmenting the point cloud taking into
account [underlying] surface variations, in search of smooth patches,
made disjoint by edges (where we find high curvature/variation).
Therefore we can use the above estimated vectors and values for
such a segmentation.
35. 35Challenge the future
•Constrained minimization – Lagrange multipliers
Continued from lecture notes of Olga Sorkine…
Given this equality in
differentiation of vector
valued functions
36. 36Challenge the future
•Constrained minimization – Lagrange multipliers
Continued from lecture notes of Olga Sorkine…
maximize f(x, y) subject to g(x, y) = c.
Lagrangian:
The prblem is transformed to finding a ‘’staionary
point for the Langrangian at which the partial
derivatives of Lagrangian function are zero.
37. 37Challenge the future
•Constrained minimization – Lagrange multipliers
Continued from lecture notes of Olga Sorkine…
This one means that the
normal vector is
normalized, meaning its
length is equal to 1; why,
because the dot product
of a vector by itself gives
the squared length
38. 38Challenge the future
How to form covariance matrices in code?
VB.NET (confirm and debug for yourself)
If(Pts.Count > 3) Then
Dim CovMatrices As New list(Of Meta.Numerics.Matrices.SymmetricMatrix)
For j As Integer=0 To Pts.Count - 1
Dim CovM As New Meta.Numerics.Matrices.SymmetricMatrix(3)
Dim Neighbors As List(Of Integer) = DirectCast(Neigh(j), List(Of Integer))
Dim Centroid As New Point3d
Dim NPts As New List(Of Point3d)
For Each neighbor As Integer In Neighbors
Centroid = Centroid + Pts(neighbor)
NPts.Add(Pts(neighbor))
Next
Centroid = Centroid / Neighbors.Count
Dim CiCBar As New RectangularMatrix(3, Neighbors.Count)
For k As Integer=0 To Neighbors.count - 1
Dim Diff As point3d = Pts(Neighbors(k)) - Centroid
CiCBar(0, k) = Diff.X
CiCBar(1, k) = Diff.y
CiCBar(2, k) = Diff.z
Next
CovM = CiCBar.MultiplySelfByTranspose()
CovM = (1 / (Neighbors.count - 1)) * CovM
CovMatrices.Add(CovM)
Next
A = CovMatrices(0).ToArray()
C = CovMatrices
End If
39. 39Challenge the future
How to form covariance matrices in code?
C#.NET (confirm and debug for yourself)
if ((Pts.Count > 3)) {
List<Meta.Numerics.Matrices.SymmetricMatrix> CovMatrices = new List<Meta.Numerics.Matrices.SymmetricMatrix>();
for (int j = 0; j <= Pts.Count - 1; j++) {
Meta.Numerics.Matrices.SymmetricMatrix CovM = new Meta.Numerics.Matrices.SymmetricMatrix(3);
List<int> Neighbors = (List<int>)Neigh(j);
Point3d Centroid = new Point3d();
List<Point3d> NPts = new List<Point3d>();
foreach (int neighbor in Neighbors) {
Centroid = Centroid + Pts(neighbor);
NPts.Add(Pts(neighbor));
}
Centroid = Centroid / Neighbors.Count;
RectangularMatrix CiCBar = new RectangularMatrix(3, Neighbors.Count);
for (int k = 0; k <= Neighbors.count - 1; k++) {
point3d Diff = Pts(Neighbors(k)) - Centroid;
CiCBar(0, k) = Diff.X;
CiCBar(1, k) = Diff.y;
CiCBar(2, k) = Diff.z;
}
CovM = CiCBar.MultiplySelfByTranspose();
CovM = (1 / (Neighbors.count - 1)) * CovM;
CovMatrices.Add(CovM);
}
A = CovMatrices(0).ToArray();
C = CovMatrices;
}
41. 41Challenge the future
Grid Surface Reconstruction
Pseudo-Code
Define a new Mesh with a list of Vertices and a list of Faces
Mesh.Vertices=Points
For j as Integer=0 to V-2//choose only bottom-left corners as pivot points
For i As Integer=0 To U - 2
define n0,n1,n2,n3 As Integer
n0 = j * U + I //bottom-left
n1 = n0 + 1 //bottom-right
n2 = n0 + U //top-right
n3 = n1 + U //top-left
Define face As new MeshFace(n0, n1, n3, n2)
Mesh.Faces.AddFace(face)
Next
Next
42. 42Challenge the future
Grid Surface Reconstruction
VB.NET
Dim M As New Mesh
M.Vertices.AddVertices(P)
If Not C Is Nothing Then
M.VertexColors.AppendColors(C.toArray)
For j As Integer=0 To V - 2
For i As Integer=0 To U - 2
Dim n0,n1,n2,n3 As Integer
n0 = j * U + i
n1 = n0 + 1
n2 = n0 + U
n3 = n1 + U
Dim face As MeshFace = New MeshFace(n0, n1, n3, n2)
M.Faces.AddFace(face)
Next
Next
with two nested loops
43. 43Challenge the future
Grid Surface Reconstruction
VB.NET
Dim M As New Mesh
M.Vertices.AddVertices(P)
If Not C Is Nothing Then
M.VertexColors.AppendColors(C.toArray)
For i As Integer=0 To U * (V - 1) - 1
If (i Mod u) < u - 1 Then
Dim n0,n1,n2,n3 As Integer
n0 = i
n1 = n0 + 1
n2 = n0 + U
n3 = n1 + U
Dim face As MeshFace = New MeshFace(n0, n1, n3, n2)
M.Faces.AddFace(face)
End If
Next
with one loop
44. 44Challenge the future
Grid Surface Reconstruction
C#.NET
Mesh M = new Mesh();
M.Vertices.AddVertices(P);
if ((C != null))
M.VertexColors.AppendColors(C.toArray);
for (int i = 0; i <= U * (V - 1) - 1; i++) {
if ((i % u) < u - 1) {
int n0 = 0;
int n1 = 0;
int n2 = 0;
int n3 = 0;
n0 = i;
n1 = n0 + 1;
n2 = n0 + U;
n3 = n1 + U;
MeshFace face = new MeshFace(n0,
n1, n3, n2);
M.Faces.AddFace(face);
}
}
with one loop
45. 45Challenge the future
Simple Estimation of Normal
Vectors Pseudo-Code
Form an empty list of normal vectors
Define deviation as a double
For each point as Point3d in the point cloud
find neighbors
fit a plane to neighbors
Get the normal of this plane and put it out as the normal of the point
form a vector from the vantage point VP to point=VP-point and call it dir
if this normal.dir>0 then
Add the normal to the list of normals
Else
Add –normal to the list of normals
End
Next
46. 46Challenge the future
Estimation Normal Vectors
C#.NET
List<Vector3d> Normals = new List<Vector3d>();
Point3dList PCList = new Point3dList();
PCList.AddRange(x);
double Dev = MD;
foreach (Point3d point in PCList) {
dynamic Neighbors = PCList.FindAll(V => V.DistanceTo(point) < D);
plane NP = default(Plane);
Plane.FitPlaneToPoints(Neighbors, NP, Dev);
if (NP.Normal * (VP - point) > 0) {
Normals.Add(NP.Normal);
} else {
Normals.Add(-NP.Normal);
}
}
A = Normals;
B = PCList.FindAll(VT => VT.DistanceTo(x(654)) < D);
47. 47Challenge the future
Estimation Normal Vectors
VB.NET
Dim Normals As New list(Of Vector3d)
Dim PCList As New point3dlist
PClist.AddRange(x)
Dim Dev As Double = MD
For Each point As point3d In PClist
Dim Neighbors = PClist.FindAll(Function(V) V.distanceto(point) < D)
Dim NP As plane
Plane.FitPlaneToPoints(Neighbors, NP, Dev)
If NP.Normal * (VP - point) > 0 Then
Normals.Add(NP.Normal)
Else
Normals.Add(-NP.Normal)
End if
Next
A = Normals
B = PClist.FindAll(Function(VT) VT.DistanceTo(x(654)) < D)
48. 48Challenge the future
Basics of Scripting in Rhino + GH
• VB.NET (Sub[routine], Function, ByVal, ByRef)
• C#.NET (Void, [Functions], [val],ref)
• Python (defs)
Basic concepts of systems and modules, Inputs & Outputs
Private Sub RunScript(ByVal x As Object, ByVal y As Object, ByRef A As
Object)
End Sub
'<Custom additional code>
'</Custom additional code>
Function plus(A As Integer, B As Integer) As
Integer
Return A + B
End Function
private void RunScript(object x, object y, ref object A)
{
}
// <Custom additional code>
// </Custom additional code>
public int plus(int a, int b)
{
return a + b;
}
def plus(a,b):
return a+b
Nothing visible.
49. 49Challenge the future
Basics of Scripting in Rhino + GH
• Rhinocommon
• Rhinoscript
• Grasshopper Kernel
Rhinocommon
Library of
Geometry
Operations
VB.NET. C#.NET &
Python
Grasshopper Kernel
Library of Some
Special Geometry
Operations
VB.NET. C#.NET &
Python
Rhionscript (with [Iron]Python)
Operations as in Rhino Command Line
Python
http://4.rhino3d.com/5/rhinocommon/Rhinocommon SDK:
Grasshopper SDK: [Rhino]Command: GrasshopperGetSDKDocumentation
Rhinoscript Syntax SDK: [GH Python]Help: Rhinoscript syntax help
50. 50Challenge the future
Questions? p.nourian@tudelft.nl
• I will give you the tools we have developed so far as open
source;
• You will make them better using the following libraries;
• Using either Math.NET or MetaNumerics is a must;
• Using Accord.NET is a big plus!
• If you are using Python, use similar libraries such as SciPy
• If you want to do something else let us discuss it now!