1. Mobile Terrestrial LiDAR Datasets
in a Spatial Database Framework
Dr. Conor Mc Elhinney
Postdoctoral Researcher
Mobile Mapping Group
7th MMT 16th June 2011

2. Mobile Mapping Systems Group
Develop automated processing
algorithms for MMS data.

3. Mobile Mapping Systems Group
Develop automated processing
algorithms for MMS data.

4. Mobile Mapping Systems Group
Develop automated processing
algorithms for MMS data.

5. Mobile Mapping Systems Group
Develop automated processing
algorithms for MMS data.

6. Survey based
LiDAR folder

7. Survey based
LiDAR folder
Survey 10 Apr
Block 1
Block 2
Block 3
.
.
Block N
.
MetaData: Geo Bounds, date,
processing done

8. Survey based
LiDAR folder
Survey 10 Apr Survey 5 Dec
Block 1 Block 1
Block 2 Block 2
Block 3 Block 3
. .
. .
Block N
. Block N
.
MetaData: Geo Bounds, date,
MetaData: Geo Bounds, date,
processing done
processing done

9. Survey based
LiDAR folder
Survey 10 Apr Survey 5 Dec Survey 2 May
Block 1 Block 1 Block 1
Block 2 Block 2 ....... Block 2
Block 3 Block 3 Block 3
. . .
. . .
Block N
. Block N
. Block N
.
MetaData: Geo Bounds, date,
MetaData: Geo Bounds, date, MetaData: Geo Bounds, date,
processing done
processing done processing done

10. Survey based
LiDAR folder

11. Survey based
LiDAR folder

12. Survey based
Give me the data from Dublin city
LiDAR folder

13. Survey based
Do we have data in a given area?
LiDAR folder

14. Survey based
Give me 10mx5m cross sections at
5m intervals
LiDAR folder

15. Storage

16. Storage

17. Storage
Database

18. Storage
Database

19. Storage
Database

20. Storage
PostgreSQL
PostGIS
Database

21. Storage
PostGIS PostgreSQL
Database

22. Storage
Spatial Database

23. Storage
DB Index: 3D point
Spatial Database

24. Data Handling and Upload

25. Why is upload an issue
50Km – 1 way

26. Why is upload an issue
50GB
50Km – 1 way

27. Why is upload an issue
50GB
50Km – 1 way
500
Million points

28. Why is upload an issue
80Km – 1 way

29. Why is upload an issue
130GB
80Km – 1 way

30. Why is upload an issue
130GB
80Km – 1 way
1,300
million points

31. First tests
DB on powerful desktop

32. First tests
DB on powerful desktop
> 60m records

33. First tests
DB on powerful desktop
> 60m records
> 4hrs to upload

34. First tests
DB on powerful desktop
Small survey >40hrs
> 60m records
> 4hrs to upload time
upload

35. Our hardware
1 Processing Server
8 Intel Xeons, 2.8 GHz
32 GBs RAM
1 Storage Server
7TBs Raided Drives

36. Experiments

37. Compared
PostgreSQL Copy
with
pg_bulkload

38. Test files
LiDAR data over 66m rows
2 files:
10 columns -> 4.4Gbs
14 columns -> 6.8Gbs

39. Postgresql upload
1. Create Table
2. Load data
3. Create Geometry Column
4. Update Geometry Column
5. Create Index
6. Vacuum Table

40. Our upload process
1. Pre-process - Python
2. Create Table
3. Create Geometry Column
4. Create Index
5. Load data

41. Time per row
PostgreSQL
Exp 1
Exp 2
Exp 3
Copy 10
pg_bulkload columns
0.025 0.05 0.075 0.1
Time per row (ms)
Copy
0.25 0.5 0.75 1 1.25
14
pg_bulkload columns
0.025 0.05 0.075 0.1
Time per row (ms)

42. Time per row Exp 1
PostgreSQL
Exp 1
2
New
Exp 2
3
Exp 3
Copy 10
pg_bulkload columns
0.025 0.05 0.075 0.1
Time per row (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.025 0.05 0.075 0.1
Time per row (ms)

43. Row size impacts time Exp 1
PostgreSQL
1
Exp 2
New2
Exp 3
Exp 3
Copy 10
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)

44. Row size impacts time Exp 1
PostgreSQL
1
Exp 2
New2
Exp 3
Exp 3
Copy 10
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)

45. Row size impacts time Exp 1
PostgreSQL
1
Exp 2
New2
Exp 3
Exp 3
Copy 10
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)

46. Row size impacts time Exp 1
PostgreSQL
1
Exp 2
New2
Exp 3
Exp 3
Copy 10
pg_bulkload
As row size
0.25 0.5 0.75 1 1.25
columns
Time per kb
Time per KB (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)

47. Time difference
1. Our Method
>40 %
2. pg_bulkload
>12 %

48. Benefit?

49. 60
1.5 1.5
Copy Orig
Copy
Direct
50 Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)

50. 60
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)

51. 60
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)

52. 60
>24hrs
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)

53. 60
>24hrs
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 PG_bulkload
Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)

54. 60
4hrs
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 PG_bulkload
Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)

55. Back to the big picture

56. Access “Mobile Mapping System LiDAR Data
Framework “
3D GeoInfo 2010
Spatial Database

57. Access “Mobile Mapping System LiDAR Data
Framework “
3D GeoInfo 2010
Spatial Database

58. Access “Mobile Mapping System LiDAR Data
Framework “
3D GeoInfo 2010
Spatial Database

59. Access “Mobile Mapping System LiDAR Data
Framework “
3D GeoInfo 2010
Spatial Database

60. Access
Spatial Database

61. Access
Spatial Database

62. Access
Spatial Database

63. Process
Spatial Database

64. Process
Spatial Database

65. Process
> 140km
Spatial Database

66. Visualising

67. Conclusions
Store
Access
Automatically
Process
Visualise

68. Future work
Finalise DB schema
Formalise DB Upload / Access
Release Free Mobile Mapping
Spatial DB
Automated algorithms

69. Questions
Conor Mc Elhinney, Paul Lewis, Tim McCarthy
conormce@cs.nuim.ie