Ilke Demir, Postdoctoral Research Scientist, Facebook

1. Geospatial Machine Learning
for Urban Development
Ilke Demir
Facebook
MLConf – The Machine Learning Conference

2. Understanding the World

3. Exploring the World

4. Understanding the World
urban safety
socioeconomic data &
voting patterns
poverty
mapping
disaster mapping

5. Open Datasets
o DeepGlobe: https://deepglobe.org
o Road extraction
o Building detection
o Land cover classification
o SpaceNet: https://spacenetchallenge.github.io/
o Road network extraction
o Building detection
o Earth observation challenge: http://eochallenge.bemyapp.com/
o Water resource extraction
o Change detection
o Data fusion contest: http://www.grss-ieee.org/data-fusion-contest/
o Land cover land use classification from various sensor data
o Functional map of the world: https://www.iarpa.gov/challenges/fmow.html
o Labeling the world into land use categories

6. Case Study: DeepGlobe
o Public datasets and benchmarks for scalable and reliable approaches
o Satellite imagery is powerful as it is more structured than everyday images
DeepGlobe focuses on machine learning and computer vision approaches on
satellite images and brings together researchers with different perspectives by;
o Publishing public datasets and baselines
o Creating public challenges to benchmark different approaches
o Organizing a workshop to sparkle new collaborations and ideas

7. DeepGlobe Tracks
Road Extraction Challenge:
- Maps, accessibility, and connectivity
- Economic and developmental inclusion
- Crisis response

8. DeepGlobe Tracks
Building Detection Challenge:
- Population dynamics and demographics
- Disaster recovery and damage coordination
- Urban development

9. DeepGlobe Tracks
Land Cover Classification Challenge:
- Sustainable development
- Automation in agriculture
- Urban planning and growth

10. DeepGlobe Challenges
1. Road Extraction Challenge
- DigitalGlobe Vivid+
- 50 cm/pixel
- Pixel-wise manual annotation
- 2 classes
- Thailand, Indonesia, India
- 8570 images of 2220km2
- 70%/15%/15% split
- ~4% positive pixels
- Diverse road networks
- 345 participants
- 2150 submissions
- 84 results in the leaderboard

11. DeepGlobe Challenges
2. Building Detection Challenge
- SpaceNet Buildings v2
- 31cm single-band panchromatic
- 1.24m 8 band multi-spectral
- Manual annotation of polygons
- 2 classes
- Las Vegas, Paris, Shanghai, Khartoum
- 24586 images of 9623 km2
- 60%/20%/20% split
- 302701 buildings
- 296 participants
- 576 submissions
- 25 results in the leaderboard

12. DeepGlobe Challenges
3. Land Cover Classification Challenge
- DigitalGlobe Vivid+
- 50 cm/pixel
- Pixel-wise manual annotation
- 7 classes
- Thailand, India, Indonesia
- 1146 images of 1717 km2
- 70%/15%/15% split
- 20m minimum granularity area
- 311 participants
- 1155 submissions
- 28 results in the leaderboard

13. DeepGlobe Results and Baselines
Roads
DeepLab variation
Only data augmentation by rotation
IoU score 0.545

14. DeepGlobe Results and Baselines
Roads
DeepLab variation
Only data augmentation by rotation
IoU score 0.545
Buildings
Ensemble 3 U-Net models
Boost by OpenStreetMap data
F1 score 0.693

15. DeepGlobe Results and Baselines
Roads
DeepLab variation
Only data augmentation by rotation
IoU score 0.545
Buildings
Ensemble 3 U-Net models
Boost by OpenStreetMap data
F1 score 0.693
Lands
DeepLab variation
Data augmentation by rotations and class weights
IoU score of 0.433

16. DeepGlobe Winners
Roads Buildings Land
D-LinkNet: LinkNet with
Pretrained Encoder and Dilated
Convolution for High Resolution
Satellite Imagery Road Extraction
Lichen
Zhou,
BUPT
Building Detection
from Satellite Imagery
using Ensemble of
Size-specific Detectors
Ryuhei
Hamaguc
hi,
Pasco
Dense Fusion
Classmate Network
for Land Cover
Classification
Chao Tian,
Harbin
Institute of
Technology
1. Semantic Binary Segmentation using Convolutional Networks
without Decoders
Shubhra Aich*; William van der Kamp; Ian Stavness, University of
Saskatchewan
2. Stacked U-Nets with Multi-Output for Road Extraction
Tao Sun*; Zehui Chen; Wenxiang Yang; Yin Wang, Tongji University
3. D-LinkNet: LinkNet with Pretrained Encoder and Dilated
Convolution for High Resolution Satellite Imagery Road Extraction
Lichen Zhou*; Chuang Zhang; Ming Wu, Beijing University of Posts
and Telecommunications
4. Fully Convolutional Network for Automatic Road Extraction from
Satellite Imagery
Alexander Buslaev*, Mapbox; Selim Seferbekov, Veeva Systems;
Vladimir Iglovikov, Lyft Inc; Alexey Shvets Massachusetts Institute of
Technology
5. Road Detection with EOSResUNet and Post Vectorizing Algorithm
Oleksandr Filin*; Serhii Panchenko; Anton Zapara, EOS Data Analytics
6. Residual Inception Skip Network for Binary Segmentation
Jigar Doshi*, CrowdAI
7. Roadmap Generation using a Multi-Stage Ensemble of Neural
Networks with Smoothing-Based Optimization
Dragos Costea*; Alina Marcu; Emil Slusanschi; Marius Leordeanu,
University Politehnica of Bucharest
8. Rotated Rectangles for Symbolized Building Footprint Extraction
Matthew Dickenson*; Lionel Gueguen, Uber
9. Building Detection from Satellite Imagery Using Composite Loss
Function
Sergey Golovanov*; Rauf Kurbanov; Aleksey Artamonov; Alex
Davydow; Sergey Nikolenko, Neuromation
10. Building Detection from Satellite Imagery using Ensemble of Size-
specific Detectors
Ryuhei Hamaguchi*; Shuhei Hikosaka, Pasco Corporation
11. TernausNetV2: Fully Convolutional Network for Instance
Segmentation
Vladimir Iglovikov*, Lyft Inc; Selim Seferbekov, Veeva Systems;
Alexander Buslaev, Mapbox; Alexey Shvets Massachusetts Institute of
Technology
12. Semantic Segmentation based Building Extraction Method using
Multi-source GIS Map Datasets and Satellite Imagery
Weijia Li*; Conghui He; Jiarui Fang; Haohuan Fu, Tsinghua University
13. CNNs Fusion for Building Detection in Aerial Images for the
Building Detection Challenge
Remi Delassus*, Qucit - LaBRI; Romain Giot, Univ. Bordeaux
14. Building Extraction from Satellite Images Using Mask R-CNN with
Building Boundary Regularization
Kang Zhao*; Jungwon Kang; Jaewook Jung; Gunho Sohn, York
University
15. Deep Aggregation Net for Land Cover Classification
Tzu-Sheng Kuo*; Keng-Sen Tseng; Jia-Wei Yan; Yen-Cheng Liu; Yu-
Chiang Frank Wang, National Taiwan University
16. Stacked U-Nets for Ground Material Segmentation in Remote
Sensing Imagery
Arthita Ghosh*; Max Ehrlich; Sohil Shah; Larry Davis; Rama
Chellappa, University of Maryland
17. Land Cover Classification from Satellite Imagery With U-Net and
Lovasz-Softmax Loss
Alexander Rakhlin*; Alex Davydow; Sergey Nikolenko, Neuromation
18. Dense Fusion Classmate Network for Land Cover Classification
Chao Tian*, Harbin Institute of Technology; Cong Li; Jianping Shi,
Sensetime 19. NU-Net: Deep Residual Wide Field of View
Convolutional Neural Network for Semantic Segmentation
Mohamed Samy; Karim Amer*; Kareem Eissa; Mahmoud Shaker;
Mohamed ElHelw, Nile University;
20. Feature Pyramid Network for Multi-Class Land Segmentation
Selim Seferbekov*, Veeva Systems; Vladimir Iglovikov, Lyft Inc;
Alexander Buslaev, Mapbox; Alexey Shvets Massachusetts Institute of
Technology
21. Uncertainty Gated Network for Land Cover Segmentation
Guillem Pascual*; Santi Seguí; Jordi Vitria, Universitat de Barcelona
22. Land Cover Classification With Superpixels and Jaccard Index
Post-Optimization
Alex Davydow*; Sergey Nikolenko, Neuromation

17. Changing the World
urban planning simulations generative models

18. Proceduralization
Generalized Proceduralization

19. Shape Understanding
Point clouds:
Arch
Physical objects:
Urban spaces: Satellite Images: Architectural models:

20. Point clouds:
Satellite images:
Urban spaces: Architecture:
Generative Modeling

21. Case Study: Street Addresses
• 75% of the world lives
without adequate addressing.
What3Words
• 4 billion people are ‘invisible’.
United Nations
• Haiti earthquake: 48 hours
reaction time, 6 months
complete road vectors.
OpenStreetMap

22. Geocoding Solutions
“issuer.lollipop.ripe”
“ItsADream”
“37.482825, -122.145661”
“75682SB3084”
“parrot.casino.failed”
“SweetPotato”
“102.22556, -12.166981”
“7098HGT3083”
• f(lat, lon) = “address”
• Hashes, random words, manual tags
• No spatial relation cue
• No city/state/country
• No accessibility
• No geometry
A B Humans need streets!

23. Traditional Addressing Systems
London postal code system:
Radial regions based on orientation and distance
South Korea streets:
Meter markers
Japan block system:
Hard to decipher
Dubai addressing:
Uses districts
Berlin numbering:
Zigzag house pattern

24. Our Generative Scheme
• 5 alphanumeric fields
• Hierarchical and linear descriptors
• To close the gap between physical
addresses and automated geocoding
Road naming scheme:
- distance from the center
- orientation in odd parity
Region naming scheme:
- orientation wrt downtown
- distance from downtown
House numbering scheme:
- meter markers on the road
- block letters from the road
“I7 Hacker Way, Menlo Park, CA, US”

25. Pipeline: Satellite Images
ç
• Irregular urban structure
• Illumination/weather/country
• Different road types

26. Pipeline: Road Predictions
• Binary road masks
• 19K*19K, 0.5m/pixel
• SegNet

27. Pipeline: Road Network
• Orientation based median filtering
• Road segments by orientation
bucketing

28. NF
NH
NE
Pipeline: Regions
• Road graph: Node=intersection,
edge=road, weight=length
• Partition for max inter, min intra
connectivity, using normalized min-cut.

29. Pipeline: Naming
• Orientation bucketing into N, S, W, E
• Trace regions based on distance to CA
• Orientation bucketing into major axes
• Trace roads based on order

30. Pipeline: Address Cells
• 5 meter marker along the road
• Odd/even based on RHR
• Distance field of roads: block offset

31. Results: Unmapped Developing Country
• Improve coverage up to 80%
• Processed more than 200 districts (and increasing!)
• Regions follow natural boundaries
• Road network is being discovered in non-urban settings

32. Results: Unmapped Developing Country
• Improve coverage up to 80%
• Processed more than 200 districts (and increasing!)
• Regions follow natural boundaries
• Road network is being discovered in non-urban settings

33. Results
• Improve coverage up to 80%
• Processed more than 200 districts (and increasing!)
• Regions follow natural boundaries
• Road network is being discovered in non-urban settings
• Changing the world!

34. News & Ads!
o Geospatial Modeling and Visualization, Special Issue in Big Earth Data Journal
http://bit.ly/BigEarthData
o SUMO Challenge: Understanding indoor scenes from 360 RGBD data
https://sumochallenge.org/
o Challenges and opportunities for deep learning in remote sensing,
Special session in Living Planet Symposium 2019
https://lps19.esa.int/
o EarthVision 2019! (coming soon…)
o DeepGlobe v2! (coming some day…)

35. Thanks… and your turn!
Generative Street Addresses
Code: https://github.com/facebookresearch/street-addresses
Paper: https://research.fb.com/publications/robocodes
DeepGlobe Benchmark
Papers: http://bit.ly/deepglobe_papers
Website: http://deepglobe.org
Dataset: http://bit.ly/deepglobe
Ilke Demir
e-mail: idemir@fb.com
Twitter: @ilkedemir

36. Additional Slides 36

37. Evaluation Metrics
Roads
Pixel-wise mean IoU
Buildings
Average F1 score
Lands
Pixel-wise mean IoU

38. Design Choices
Linear: similar addresses stored in a linear fashion
Hierarchical: top-down structure for spatial encapsulation
Compressible: 5x4 max (chars x words)
Universal: independent of local language
Inquirable: useful for geometric, proximity-based, and type-ahead queries
Extendible: dynamically modifiable for new places
Robust: flexible for overestimation and noise
StructuralDesignParameters
forefficientcomputerimplementation
Linear: closer addresses are given related names
Hierarchical: top-down subdivision of the world
Memorable: short and alphanumeric, easily convertible
Intuitive: with a sense of direction and distance
Topological: consistent with road topology
Inclusive: with local names (city, state)
Physical: consistent with natural boundaries
SemanticDesignParameters
foruserfriendliness
Machine
Needs
Human
Needs

39. 39Geometric Shape Processing: Satellite Images
[*] I. Demir et al., 2018. “Generative Street Addresses from Satellite Imagery”.
International Journal on Geo-Information (IJGI).

40. Output Maps and Tools
• .osm maps with roads (meter marking and offsetting on the fly)
• ID-tool of MapBox for on-demand inverse/forward geocoding
• rtree extension for efficient spatial querying
• Experimental mobile app for self navigation
• 21.7% decrease in arrival time using Robocodes

41. Results: Evaluation with Ground Truth
• System learns 90.51% of roads
• Approximately 80% on average
• Better in urban environments
• Ground truth prepared as if
training data

42. Results: Mapped US City
• More than 95% of the roads are found (compared to OSM).
• Traditional addresses are more established, however
• Robocodes are contextually and spatially easier to remember.

43. Results: Comparison
Automated geocoding:
A: parrot.casino.failed
B: issuer.lollipop.ripe
- Have irrelevant words
based on lat/lon.
Robocodes:
715D.NE127.Dhule.MhIn
716C.NE127.Dhule.MhIn
- Have hierarchical and
linear addresses.
Landmark based:
Green Park
Green Park
- Have roads but no
addresses or labels.
OSM:
lat/lon
lat/lon
- Have neither road
geometry, nor labels.

44. Limitations & Future Work
• Robotic meter marking and offsetting:
• (i) use smart parcel subdivision,
• (ii) adapt to population density.
• Imperfect training data: sample more countries.
• Metric to evaluate regions: supervised learning of
land annotations.

45. Inaccessible Areas
• To extend our format to cover areas that are not accessible by
streets, we explored different implementations to cover such
areas, which are 26*5 m away from any street.
• Geocoding as a function (excluding the version field):
f (info, lat, lon) = x.y.z.t
• For places with roads, info={road network, city, country}
f (R, C) = x.y.city.country
• Extreme case: only reliable information is latitude/longitude!
45

46. f(C,lat,lon) = hash(round(lat,3)) + dir(lat) .
hash(round(lon,3)) +dir(lon) . C
L-A-T-dir.L-O-N-dir.name.area
Inaccessible Areas: Blackholes!
• Linear hashing:
• 26 letters + 10 digits
• 100m x 100 m granularity
• Last letter is the hemisphere
• Range: 359.999, longitude: 7PRZ W
• Hierarchical hashing:
• Enlarge the grid from to 1 km x 1 km
• Using two floating points = three letters
• Within each cell, re-hash it to a 36 x 36 grid = one letter
• New resolution: 30m, represented by five letters
46
f(C,lat,lon) = hash(round(lat,2)) + hash(lat - round(lat,2)) + dir(lat) .
hash(round(lon,2)) + hash(lon - round(lon,2)) + dir(lon) . C
LlatLlatHlatDlat .LlonLlonHlonDlon . name . Ocean /Continent /etc

47. Completion & Reconstruction 47
• Voxelize building proxy from
footprint
• Find roofs with photo-
consistency in aerial images
• Apply graph-cuts:
•Building
•Building-ground
•Ground
[*] I. Garcia-Dorado I. Demir, D. Aliaga.
2013. “Automatic Urban Modeling Using
Volumetric Reconstruction with Surface
Graph-cuts”. Computers & Graphics.

48. Website: http://deepglobe.org
Papers: http://bit.ly/deepglobe_papers
Dataset: http://bit.ly/deepglobe
Hashtags: #DeepGlobe