Spatial Networks geospatial data assets fit the gambit of Human Geography domains and Geospatial Intelligence, with a contextually relevant and intimate understanding from local professional experts collecting data on the ground. We are transforming this data into insightful analytics and rich, robust data sets of 21 unique data types to help our clients solve challenging geospatial problems. FME has helped us to streamline our workflows and automate several processes along our data pipeline. We look to scale our operations significantly in 2018 and FME will ease many of our challenges as we move forward.

1. Spatial Networks Inc.
Data Management and Transformation
Todd Pollard

2. Agenda
• Background
• Challenges
• Successes
• Future

3. Business decisions powered by ground truth.

4. 86,000+
3M+
63M+
users in 172 countries
field surveys recorded per month
survey records
Mobile app platform enabling businesses to build
customized workflows for field data collection.

5. • Legacy datasets in PostGIS of varying schemas and data types
• Ongoing mobile survey field collection ETL to a standard data model(s)
• Maintain data integrity and provenance
• Data validation and QA/QC
• Standardization of data storage and workflows
• Automation of workflows with minimal programming
• Metadata, temporal accuracy and update records for attribute changes -
versioning
• Basically, a decade plus worth of data assets requiring intense curation,
management and governance for a rapidly growing company.
Challenges

6. FME - Data Pipeline Backbone
• Project Management
• Data Acquisition
• Data Exploitation
• Data Conditioning
• Data Enrichment
• Data Integration
• Product Development
• Data Representation
• Metadata Management
• Product/Data Delivery

7. Build Foundational Datasets

8. Baseline Data Model
Field Survey Data Types
Standard Mobile Data Collection
 Field Survey Apps
E T L
Human Geography Themes
Standard Theme Domains
 World Wide Human Geography
Data Model(s)
Commodities
Urban Survey Business
Production Facilities
Economy

9. Baseline Foundational Dataset
• Read Survey Schema
• Import from Dataset (write to
pre-structured PostGIS tables
 Conditioning
 Validation
 Enrichment
 Integration
 Representation

10. Data Conditioning

11. Conditioning Foundational Datasets
Distinct Values
11 or 7?

12. Conditioning Foundational Datasets
• BulkAttributeMapper
• StringReplacer
• StringCaseChanger
• QA/QC Standards into workflows
• AttributeManager
• SchemaMapper
• AttributeRenamer
• Normalizing certain attributes

13. Validating Features and Attributes
• SQL Statements on Read
• TestFilters & Conditional Statements
• Consistency in Updates
• QA/QC and Error Logging into
Workflows

14. Database Updating

15. Updating Features and Attributes
Update Validation
• FME Cloud Server runs scheduled
routines
• SQL Statements on Read in ETL Log
file
• ChangeDetector / UpdateDetector
Database Updates
• fme_db_operation
• SQL Statement on Write to ETL Log file
• DatabaseUpdater
• AttributeCreator for INSERT, UPDATE,
DELETE

16. Simple to Complex…and Evolving

17. • FME can probably do it! And in more than one way!
• FME Knowledge Center
○ On-line Training Courses, Videos(YouTube) and Tutorials are excellent learning tools!
○ Q&A Forum and Knowledge Base are an excellent resource.
• Best Practices
○ Use Bookmarks!
○ Use consistent naming, numbering and versioning standards
• FME Quick Translator
○ Integral to packaging AOI datasets quickly
What We’ve learned…

18. • Transform legacy datasets in PostGIS of varying schemas and data types
into a data model standard
• Automate ETL processes of near-real time mobile survey field collection to a
standard data model
• Ensure the preservation of data integrity and provenance
• Maintain consistent data validation and QA/QC standards
• Standardization of data storage and workflows
• Automation of workflows with minimal programming
• Metadata, temporal accuracy and update records for attribute changes -
versioning
• We still have much more to explore and learn about FME’s capability to help
us streamline our workflows and automate processes.
What FME has Helped Us Achieve…

19. THANK YOU!
Todd Pollard
todd@spatialnetworks.com
www.spatialnetworks.com
1.727.403.3058