"Attend this session to learn how the City of Vancouver uses FME to:
• Provide data for both their Open Data and Road Ahead initiatives.
• Improve the City's Road Ahead map with more current information and open formats. (Learn how FME is being used to retrieve the source data from SQL Server and Oracle Spatial, and to generate SDF3/SDF2 files and a GeoRSS/RSS feed integrated with Google Street View, which is accessible from the Road Ahead home page. Plus, see how the Road Ahead FME workspace is also published to FME Server where the relevant data format files can be generated on the fly via data streaming services.)
• Prepare datasets for the City's OpenData website on a weekly basis. "
1. Vancouver Uses FME to Open
Data to the World
Meng Li, GIS Systems Analyst, City of Vancouver
Jonathan Mark, GIS Manager, City of Vancouver
March 29, 2010
2. We’ll Talk About
Overview
The Open Data Initiative and ICIS support
The Road Ahead Website Upgrade
Safe software in use at the City
Open Data Infrastructure Platform
Workflow, ETL scripts, and deployment
The Road Ahead Website Upgrade Project
ETL, Atom, GeoRSS, data streaming experience
3. The Open Data Initiative
The Open3 Council motion of May 2009
Open data
Open standards
Open source software
More sharing with ICIS
Directive to make City’s data freely accessible
subject to
Security concerns
Privacy concerns
Terms of use
4. Open Data Website
September 2009 launch
About 50 layers, many hard to find
Mostly static datasets, limited formats
FME used for most of initial creation
January 2010 launch
About 100 layers, all easy to find
Most in DWG, KML, SHP with VanMap groupings
Weekly updating (thanks to FME software)
Can display in Google Maps or Bing Map easily
ICIS-specific data groups for GIS & AddressBC
Demo
5. The Road Ahead Website
Map in AJAX Viewer of MGE using SDF3 files
Atom and GeoRSS feed data generated
Source data: SQL Server and Oracle Spatial
FME ETL process to update SDF3, SDF2 &
feed data hourly 7am-7pm Mon to Fri
The Road Ahead feed data being widely used
Feed data with Street View via data
streaming
6. Safe Software in Use
FME Desktop
The City has gradually increased its pool of
licenses to 10
Being used for IMS, Open Data, Road Ahead,
and a variety of day-to-day data ETL needs
FME Server
Acquired in November 2009
Important foundation component of our GIS
software infrastructure
Moving Open Data and Road Ahead to Server
7. Open Data Infrastructure
Requirements of source and target data:
Public VanMap consistency
layers/groups specifications (filtering, grouping, naming)
source formats: MDB, DWG, SDF, SQL Server, Oracle
Target Data Catalogue in OpenData Website
formats: DWG, KML/KMZ, SHAPE, CSV, XLS
naming convention: low casing and no spaces
file structure: data folders for FTP and Web servers
flexible for data packaging: via published parameters
ICIS specific data specifications
Automated Execution of the Data Extraction
Scripts via Windows scheduler tasks
8. Open Data—Processes/workflow
outputs
VM source Windows
Scheduler
Task
ETL Workspace and scripts
loaded with Python pre-
processing, filtering, routing
& post-processing logic
11. Open Data - Deployment
All files in KML/KMZ format uploaded weekly
to the Open Data web server
All files in other formats uploaded weekly to
the public FTP server
FME ETL Scripts (now Desktop) Updates for
Flexible Deployment
12. Open Data- Challenges
Data Catalogue Design: groupings and classification
Required Data Groupings and File Structure
Naming Conventions: machine and user friendly
KML Specification Compliance/Validation Issue
Data fitness of use: attribute requests & selection
14. The Road Ahead—New Features
Hourly update replacing nightly update:
FME ETL job runs hourly from 7am to 7pm
weekdays
update SDF3/SDF2 (for VanMap) and feeds
data
MapGuide Enterprise AJAX Viewer uses SDF3
Atom and Geo RSS feed data generated
The Road Ahead feed data replaced an email
list and is being widely used (including EOC)
Feed data with Street View via data
streaming
15. The Road Ahead—Data Extract
Source Data
SQL Server--point & street-based projects
Oracle Spatial--street network reference data
Output files:
SDF3 file of point & line features with 3 Road Ahead categories
Atom xml file, and a GeoRSS xml file
SDF2 files for existing VanMap Road Ahead layers
Feed xml file with Google Street View (not in production)
FME script data extract job automation
17. The Road Ahead-FME Server Deployment
The Road Ahead FME script workspace
published to our FME Server (2009)
The same ETL script job is run hourly against
the FME Server:
"D:FMEServerClientsFMEServerConsolefmeserverconsole.exe"
RUN_WORKSPACE Roadahead_dataextract/
uat_roadahead_sdf3_georss_sdf2_lp.fmw
Feed data (Atom and GeoRSS) integrated with
Google Street View is accessible live via data
streaming services from FME Server
18. The Road Ahead-FME Server
Interface
http://gisweb9/fmedatastreaming/
Roadahead_dataextract/
prd_roadahead_georss_lp_sv.fmw
Feed data:
19. The Road Ahead- Challenges
FME Date Formatting: customized transformer
Feed format (RSS or Atom) and specification
issues:
unique ID for feed data,
duplicate feed entries on disjoin hundred block streets
FME Server user account: ID required cross
domain access
Authentication limitation: use FME 2010!
20. Summary
FME is very good solution for both Open Data
and Road Ahead (and lots of other things!)
FME Desktop and Server have greatly
improved the openness and interoperability of
City’s GIS infrastructure platform
FME Server 2010 will expand our abilities in
supporting GIS, Open3, Road Ahead and
other initiatives going forward
21. Thank You!
Questions?
For more information:
meng.li@vancouver.ca
jonathan.mark@vancouver.ca
IT Department
City of Vancouver