SlideShare uma empresa Scribd logo
1 de 21
Baixar para ler offline
Vancouver Uses FME to Open
Data to the World




Meng Li, GIS Systems Analyst, City of Vancouver

Jonathan Mark, GIS Manager, City of Vancouver

March 29, 2010
We’ll Talk About


  Overview
    The Open Data Initiative and ICIS support
    The Road Ahead Website Upgrade
    Safe software in use at the City


  Open Data Infrastructure Platform
    Workflow, ETL scripts, and deployment


  The Road Ahead Website Upgrade Project
    ETL, Atom, GeoRSS, data streaming experience
The Open Data Initiative


  The Open3 Council motion of May 2009
      Open data
      Open standards
      Open source software
      More sharing with ICIS
  Directive to make City’s data freely accessible
   subject to
    Security concerns
    Privacy concerns
    Terms of use
Open Data Website

  September 2009 launch
    About 50 layers, many hard to find
    Mostly static datasets, limited formats
    FME used for most of initial creation
  January 2010 launch
      About 100 layers, all easy to find
      Most in DWG, KML, SHP with VanMap groupings
      Weekly updating (thanks to FME software)
      Can display in Google Maps or Bing Map easily
      ICIS-specific data groups for GIS & AddressBC
  Demo
The Road Ahead Website


    Map in AJAX Viewer of MGE using SDF3 files
    Atom and GeoRSS feed data generated
    Source data: SQL Server and Oracle Spatial
    FME ETL process to update SDF3, SDF2 &
     feed data hourly 7am-7pm Mon to Fri

  The Road Ahead feed data being widely used
  Feed data with Street View via data
   streaming
Safe Software in Use


  FME Desktop
    The City has gradually increased its pool of
     licenses to 10
    Being used for IMS, Open Data, Road Ahead,
     and a variety of day-to-day data ETL needs


  FME Server
    Acquired in November 2009
    Important foundation component of our GIS
     software infrastructure
    Moving Open Data and Road Ahead to Server
Open Data Infrastructure


  Requirements of source and target data:
    Public VanMap consistency
       layers/groups specifications (filtering, grouping, naming)
       source formats: MDB, DWG, SDF, SQL Server, Oracle


    Target Data Catalogue in OpenData Website
         formats: DWG, KML/KMZ, SHAPE, CSV, XLS
         naming convention: low casing and no spaces
         file structure: data folders for FTP and Web servers
         flexible for data packaging: via published parameters
         ICIS specific data specifications


    Automated Execution of the Data Extraction
     Scripts via Windows scheduler tasks
Open Data—Processes/workflow




outputs




          VM source                          Windows
                                             Scheduler
                                             Task
                        ETL Workspace and scripts
                          loaded with Python pre-
                        processing, filtering, routing
                           & post-processing logic
Open Data: one Workbench script




                         shape




                                       KML




                                 DWG
Open Data—Automation


  Scheduler Tasks for ETL Scripts (run weekly)
Open Data - Deployment


  All files in KML/KMZ format uploaded weekly
   to the Open Data web server
  All files in other formats uploaded weekly to
   the public FTP server
  FME ETL Scripts (now Desktop) Updates for
   Flexible Deployment
Open Data- Challenges


  Data Catalogue Design:       groupings and classification



  Required Data Groupings and File Structure

  Naming Conventions:      machine and user friendly



  KML Specification Compliance/Validation Issue

  Data fitness of use:   attribute requests & selection
The Road Ahead Website
The Road Ahead—New Features


   Hourly update replacing nightly update:
     FME ETL job runs hourly from 7am to 7pm
      weekdays
     update SDF3/SDF2 (for VanMap) and feeds
      data
   MapGuide Enterprise AJAX Viewer uses SDF3
   Atom and Geo RSS feed data generated
   The Road Ahead feed data replaced an email
    list and is being widely used (including EOC)
   Feed data with Street View via data
    streaming
The Road Ahead—Data Extract


  Source Data
     SQL Server--point & street-based projects
     Oracle Spatial--street network reference data


  Output files:
       SDF3 file of point & line features with 3 Road Ahead categories
       Atom xml file, and a GeoRSS xml file
       SDF2 files for existing VanMap Road Ahead layers
       Feed xml file with Google Street View (not in production)


  FME script data extract job automation
Road Ahead— Workbench file
The Road Ahead-FME Server Deployment



    The Road Ahead FME script workspace
     published to our FME Server (2009)

    The same ETL script job is run hourly against
     the FME Server:
       "D:FMEServerClientsFMEServerConsolefmeserverconsole.exe"
        RUN_WORKSPACE Roadahead_dataextract/
        uat_roadahead_sdf3_georss_sdf2_lp.fmw


    Feed data (Atom and GeoRSS) integrated with
     Google Street View is accessible live via data
     streaming services from FME Server
The Road Ahead-FME Server
Interface




   http://gisweb9/fmedatastreaming/
    Roadahead_dataextract/
    prd_roadahead_georss_lp_sv.fmw
 Feed data:
The Road Ahead- Challenges


  FME Date Formatting: customized transformer

  Feed format (RSS or Atom) and specification
   issues:
    unique ID for feed data,
    duplicate feed entries on disjoin hundred block streets


  FME Server user account: ID required cross
   domain access

  Authentication limitation: use FME 2010!
Summary


  FME is very good solution for both Open Data
   and Road Ahead (and lots of other things!)

  FME Desktop and Server have greatly
   improved the openness and interoperability of
   City’s GIS infrastructure platform

  FME Server 2010 will expand our abilities in
   supporting GIS, Open3, Road Ahead and
   other initiatives going forward
Thank You!


  Questions?

  For more information:
      meng.li@vancouver.ca
      jonathan.mark@vancouver.ca
      IT Department
      City of Vancouver

Mais conteúdo relacionado

Mais procurados

C-SCAN Engine and Language
C-SCAN Engine and LanguageC-SCAN Engine and Language
C-SCAN Engine and LanguageHaim Ben Zagmi
 
ISPF Recent and Coming Enhancements
ISPF Recent and Coming EnhancementsISPF Recent and Coming Enhancements
ISPF Recent and Coming Enhancementsibmsamr
 
Apache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAseApache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAseenissoz
 
Spatialware_2_Sql08
Spatialware_2_Sql08Spatialware_2_Sql08
Spatialware_2_Sql08Mike Osbourn
 
Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020Databricks
 
HPCC Systems Engineering Summit Presentation - Improving Thor Data Loading us...
HPCC Systems Engineering Summit Presentation - Improving Thor Data Loading us...HPCC Systems Engineering Summit Presentation - Improving Thor Data Loading us...
HPCC Systems Engineering Summit Presentation - Improving Thor Data Loading us...HPCC Systems
 
Dev Summit 2011 - Talk
Dev Summit 2011 - TalkDev Summit 2011 - Talk
Dev Summit 2011 - TalkArc2Earth
 
Local Secondary Indexes in Apache Phoenix
Local Secondary Indexes in Apache PhoenixLocal Secondary Indexes in Apache Phoenix
Local Secondary Indexes in Apache PhoenixRajeshbabu Chintaguntla
 
A TPC Benchmark of Hive LLAP and Comparison with Presto
A TPC Benchmark of Hive LLAP and Comparison with PrestoA TPC Benchmark of Hive LLAP and Comparison with Presto
A TPC Benchmark of Hive LLAP and Comparison with PrestoYu Liu
 
The Evolution of a Relational Database Layer over HBase
The Evolution of a Relational Database Layer over HBaseThe Evolution of a Relational Database Layer over HBase
The Evolution of a Relational Database Layer over HBaseDataWorks Summit
 
Apache Phoenix: Transforming HBase into a SQL Database
Apache Phoenix: Transforming HBase into a SQL DatabaseApache Phoenix: Transforming HBase into a SQL Database
Apache Phoenix: Transforming HBase into a SQL DatabaseDataWorks Summit
 
Exceptions are the Norm: Dealing with Bad Actors in ETL
Exceptions are the Norm: Dealing with Bad Actors in ETLExceptions are the Norm: Dealing with Bad Actors in ETL
Exceptions are the Norm: Dealing with Bad Actors in ETLDatabricks
 
Major advancements in Apache Hive towards full support of SQL compliance
Major advancements in Apache Hive towards full support of SQL complianceMajor advancements in Apache Hive towards full support of SQL compliance
Major advancements in Apache Hive towards full support of SQL complianceDataWorks Summit/Hadoop Summit
 

Mais procurados (17)

C-SCAN Engine and Language
C-SCAN Engine and LanguageC-SCAN Engine and Language
C-SCAN Engine and Language
 
ISPF Recent and Coming Enhancements
ISPF Recent and Coming EnhancementsISPF Recent and Coming Enhancements
ISPF Recent and Coming Enhancements
 
Apache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAseApache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAse
 
Spatialware_2_Sql08
Spatialware_2_Sql08Spatialware_2_Sql08
Spatialware_2_Sql08
 
Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
 
HPCC Systems Engineering Summit Presentation - Improving Thor Data Loading us...
HPCC Systems Engineering Summit Presentation - Improving Thor Data Loading us...HPCC Systems Engineering Summit Presentation - Improving Thor Data Loading us...
HPCC Systems Engineering Summit Presentation - Improving Thor Data Loading us...
 
Dev Summit 2011 - Talk
Dev Summit 2011 - TalkDev Summit 2011 - Talk
Dev Summit 2011 - Talk
 
Ecsr tutorial
Ecsr tutorialEcsr tutorial
Ecsr tutorial
 
Harmonizing Alpine traffic data
Harmonizing Alpine traffic data Harmonizing Alpine traffic data
Harmonizing Alpine traffic data
 
Data provenance in Hopsworks
Data provenance in HopsworksData provenance in Hopsworks
Data provenance in Hopsworks
 
Local Secondary Indexes in Apache Phoenix
Local Secondary Indexes in Apache PhoenixLocal Secondary Indexes in Apache Phoenix
Local Secondary Indexes in Apache Phoenix
 
A TPC Benchmark of Hive LLAP and Comparison with Presto
A TPC Benchmark of Hive LLAP and Comparison with PrestoA TPC Benchmark of Hive LLAP and Comparison with Presto
A TPC Benchmark of Hive LLAP and Comparison with Presto
 
The Evolution of a Relational Database Layer over HBase
The Evolution of a Relational Database Layer over HBaseThe Evolution of a Relational Database Layer over HBase
The Evolution of a Relational Database Layer over HBase
 
Apache Phoenix: Transforming HBase into a SQL Database
Apache Phoenix: Transforming HBase into a SQL DatabaseApache Phoenix: Transforming HBase into a SQL Database
Apache Phoenix: Transforming HBase into a SQL Database
 
Exceptions are the Norm: Dealing with Bad Actors in ETL
Exceptions are the Norm: Dealing with Bad Actors in ETLExceptions are the Norm: Dealing with Bad Actors in ETL
Exceptions are the Norm: Dealing with Bad Actors in ETL
 
Major advancements in Apache Hive towards full support of SQL compliance
Major advancements in Apache Hive towards full support of SQL complianceMajor advancements in Apache Hive towards full support of SQL compliance
Major advancements in Apache Hive towards full support of SQL compliance
 
Apache phoenix
Apache phoenixApache phoenix
Apache phoenix
 

Semelhante a Vancouver Uses FME to Open Data to the World

FME User Stories from Around the World
FME User Stories from Around the WorldFME User Stories from Around the World
FME User Stories from Around the WorldSafe Software
 
Managing data interoperability with FME
Managing data interoperability with FMEManaging data interoperability with FME
Managing data interoperability with FMEIMGS
 
Migrating legacy applications (Cobol, PL/I) to Java on z/OS
Migrating legacy applications (Cobol, PL/I) to Java on z/OSMigrating legacy applications (Cobol, PL/I) to Java on z/OS
Migrating legacy applications (Cobol, PL/I) to Java on z/OSEranea
 
IMGeospatial: FME World Tour 2018 Nottingham: FME at the Heart of Everything ...
IMGeospatial: FME World Tour 2018 Nottingham: FME at the Heart of Everything ...IMGeospatial: FME World Tour 2018 Nottingham: FME at the Heart of Everything ...
IMGeospatial: FME World Tour 2018 Nottingham: FME at the Heart of Everything ...IMGeospatial
 
Generating Pipeline Alignment Sheets Using FME
Generating Pipeline Alignment Sheets Using FMEGenerating Pipeline Alignment Sheets Using FME
Generating Pipeline Alignment Sheets Using FMESafe Software
 
Automating Enterprise Workflows with FME Server
 Automating Enterprise Workflows with FME Server Automating Enterprise Workflows with FME Server
Automating Enterprise Workflows with FME ServerSafe Software
 
Reporting Summary Information of Spatial Datasets and Non-Compliance Issues U...
Reporting Summary Information of Spatial Datasets and Non-Compliance Issues U...Reporting Summary Information of Spatial Datasets and Non-Compliance Issues U...
Reporting Summary Information of Spatial Datasets and Non-Compliance Issues U...Safe Software
 
Extracting Data from GE Smallworld into Common Information Model (CIM XML)
Extracting Data from GE Smallworld into Common Information Model (CIM XML)Extracting Data from GE Smallworld into Common Information Model (CIM XML)
Extracting Data from GE Smallworld into Common Information Model (CIM XML)Safe Software
 
WMS Performance Shootout 2009
WMS Performance Shootout 2009WMS Performance Shootout 2009
WMS Performance Shootout 2009Jeff McKenna
 
Mainframe Architecture & Product Overview
Mainframe Architecture & Product OverviewMainframe Architecture & Product Overview
Mainframe Architecture & Product Overviewabhi1112
 
204810 xer and xml
204810 xer and xml204810 xer and xml
204810 xer and xmlp6academy
 
Finns Using FME Like Crazy
Finns Using FME Like CrazyFinns Using FME Like Crazy
Finns Using FME Like CrazySafe Software
 
Using FME to support open data initiatives and INSPIRE
Using FME to support open data initiatives and INSPIREUsing FME to support open data initiatives and INSPIRE
Using FME to support open data initiatives and INSPIREIMGS
 

Semelhante a Vancouver Uses FME to Open Data to the World (20)

FME User Stories from Around the World
FME User Stories from Around the WorldFME User Stories from Around the World
FME User Stories from Around the World
 
Managing data interoperability with FME
Managing data interoperability with FMEManaging data interoperability with FME
Managing data interoperability with FME
 
HDF-EOS Maintenance, Current Development and Tools
HDF-EOS Maintenance, Current Development and ToolsHDF-EOS Maintenance, Current Development and Tools
HDF-EOS Maintenance, Current Development and Tools
 
Anton_Vysotskiy_resume
Anton_Vysotskiy_resumeAnton_Vysotskiy_resume
Anton_Vysotskiy_resume
 
Migrating legacy applications (Cobol, PL/I) to Java on z/OS
Migrating legacy applications (Cobol, PL/I) to Java on z/OSMigrating legacy applications (Cobol, PL/I) to Java on z/OS
Migrating legacy applications (Cobol, PL/I) to Java on z/OS
 
IMGeospatial: FME World Tour 2018 Nottingham: FME at the Heart of Everything ...
IMGeospatial: FME World Tour 2018 Nottingham: FME at the Heart of Everything ...IMGeospatial: FME World Tour 2018 Nottingham: FME at the Heart of Everything ...
IMGeospatial: FME World Tour 2018 Nottingham: FME at the Heart of Everything ...
 
Generating Pipeline Alignment Sheets Using FME
Generating Pipeline Alignment Sheets Using FMEGenerating Pipeline Alignment Sheets Using FME
Generating Pipeline Alignment Sheets Using FME
 
Automating Enterprise Workflows with FME Server
 Automating Enterprise Workflows with FME Server Automating Enterprise Workflows with FME Server
Automating Enterprise Workflows with FME Server
 
Status of HDF-EOS, Related Software, and Tools
Status of HDF-EOS, Related Software, and ToolsStatus of HDF-EOS, Related Software, and Tools
Status of HDF-EOS, Related Software, and Tools
 
Reporting Summary Information of Spatial Datasets and Non-Compliance Issues U...
Reporting Summary Information of Spatial Datasets and Non-Compliance Issues U...Reporting Summary Information of Spatial Datasets and Non-Compliance Issues U...
Reporting Summary Information of Spatial Datasets and Non-Compliance Issues U...
 
DhevendranResume
DhevendranResumeDhevendranResume
DhevendranResume
 
Extracting Data from GE Smallworld into Common Information Model (CIM XML)
Extracting Data from GE Smallworld into Common Information Model (CIM XML)Extracting Data from GE Smallworld into Common Information Model (CIM XML)
Extracting Data from GE Smallworld into Common Information Model (CIM XML)
 
WMS Performance Shootout 2009
WMS Performance Shootout 2009WMS Performance Shootout 2009
WMS Performance Shootout 2009
 
CDF Embraces XML and SOAP
CDF Embraces XML and SOAPCDF Embraces XML and SOAP
CDF Embraces XML and SOAP
 
Mainframe Architecture & Product Overview
Mainframe Architecture & Product OverviewMainframe Architecture & Product Overview
Mainframe Architecture & Product Overview
 
204810 xer and xml
204810 xer and xml204810 xer and xml
204810 xer and xml
 
Finns Using FME Like Crazy
Finns Using FME Like CrazyFinns Using FME Like Crazy
Finns Using FME Like Crazy
 
Hdf eos status-workshp_xi_nov_2007
Hdf eos status-workshp_xi_nov_2007Hdf eos status-workshp_xi_nov_2007
Hdf eos status-workshp_xi_nov_2007
 
Phnom penh mapping meetup #15
Phnom penh mapping meetup #15Phnom penh mapping meetup #15
Phnom penh mapping meetup #15
 
Using FME to support open data initiatives and INSPIRE
Using FME to support open data initiatives and INSPIREUsing FME to support open data initiatives and INSPIRE
Using FME to support open data initiatives and INSPIRE
 

Mais de Safe Software

Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISSafe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersSafe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsSafe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategySafe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Safe Software
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Safe Software
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Safe Software
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Safe Software
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleSafe Software
 
Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Safe Software
 
Datendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentDatendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentSafe Software
 
Zugriff auf alle raumbezogenen Daten - direkt in Ihrem WebGIS durch FME
Zugriff auf alle raumbezogenen Daten - direkt in Ihrem WebGIS durch FMEZugriff auf alle raumbezogenen Daten - direkt in Ihrem WebGIS durch FME
Zugriff auf alle raumbezogenen Daten - direkt in Ihrem WebGIS durch FMESafe Software
 
Let’s Talk About EX – WFS mit Filter EXpressions gezielt abfragen
Let’s Talk About EX – WFS mit Filter EXpressions gezielt abfragenLet’s Talk About EX – WFS mit Filter EXpressions gezielt abfragen
Let’s Talk About EX – WFS mit Filter EXpressions gezielt abfragenSafe Software
 

Mais de Safe Software (20)

Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
Geospatial Synergy: Amplifying Efficiency with FME & Esri ft. Peak Guest Spea...
 
Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...Identification of landscape features eligible for EU subsidy - Analysis of la...
Identification of landscape features eligible for EU subsidy - Analysis of la...
 
Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework Utilizing FME as an API Test Framework
Utilizing FME as an API Test Framework
 
Bringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) PeopleBringing Open Data Integration to the (SWECO) People
Bringing Open Data Integration to the (SWECO) People
 
Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...Working Forward: From Data to Location Intelligence for Renewables Siting and...
Working Forward: From Data to Location Intelligence for Renewables Siting and...
 
Datendrehscheibe im eGovernment
Datendrehscheibe im eGovernmentDatendrehscheibe im eGovernment
Datendrehscheibe im eGovernment
 
Zugriff auf alle raumbezogenen Daten - direkt in Ihrem WebGIS durch FME
Zugriff auf alle raumbezogenen Daten - direkt in Ihrem WebGIS durch FMEZugriff auf alle raumbezogenen Daten - direkt in Ihrem WebGIS durch FME
Zugriff auf alle raumbezogenen Daten - direkt in Ihrem WebGIS durch FME
 
Let’s Talk About EX – WFS mit Filter EXpressions gezielt abfragen
Let’s Talk About EX – WFS mit Filter EXpressions gezielt abfragenLet’s Talk About EX – WFS mit Filter EXpressions gezielt abfragen
Let’s Talk About EX – WFS mit Filter EXpressions gezielt abfragen
 

Último

Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 

Último (20)

Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 

Vancouver Uses FME to Open Data to the World

  • 1. Vancouver Uses FME to Open Data to the World Meng Li, GIS Systems Analyst, City of Vancouver Jonathan Mark, GIS Manager, City of Vancouver March 29, 2010
  • 2. We’ll Talk About   Overview   The Open Data Initiative and ICIS support   The Road Ahead Website Upgrade   Safe software in use at the City   Open Data Infrastructure Platform   Workflow, ETL scripts, and deployment   The Road Ahead Website Upgrade Project   ETL, Atom, GeoRSS, data streaming experience
  • 3. The Open Data Initiative   The Open3 Council motion of May 2009   Open data   Open standards   Open source software   More sharing with ICIS   Directive to make City’s data freely accessible subject to   Security concerns   Privacy concerns   Terms of use
  • 4. Open Data Website   September 2009 launch   About 50 layers, many hard to find   Mostly static datasets, limited formats   FME used for most of initial creation   January 2010 launch   About 100 layers, all easy to find   Most in DWG, KML, SHP with VanMap groupings   Weekly updating (thanks to FME software)   Can display in Google Maps or Bing Map easily   ICIS-specific data groups for GIS & AddressBC   Demo
  • 5. The Road Ahead Website   Map in AJAX Viewer of MGE using SDF3 files   Atom and GeoRSS feed data generated   Source data: SQL Server and Oracle Spatial   FME ETL process to update SDF3, SDF2 & feed data hourly 7am-7pm Mon to Fri   The Road Ahead feed data being widely used   Feed data with Street View via data streaming
  • 6. Safe Software in Use   FME Desktop   The City has gradually increased its pool of licenses to 10   Being used for IMS, Open Data, Road Ahead, and a variety of day-to-day data ETL needs   FME Server   Acquired in November 2009   Important foundation component of our GIS software infrastructure   Moving Open Data and Road Ahead to Server
  • 7. Open Data Infrastructure   Requirements of source and target data:   Public VanMap consistency   layers/groups specifications (filtering, grouping, naming)   source formats: MDB, DWG, SDF, SQL Server, Oracle   Target Data Catalogue in OpenData Website   formats: DWG, KML/KMZ, SHAPE, CSV, XLS   naming convention: low casing and no spaces   file structure: data folders for FTP and Web servers   flexible for data packaging: via published parameters   ICIS specific data specifications   Automated Execution of the Data Extraction Scripts via Windows scheduler tasks
  • 8. Open Data—Processes/workflow outputs VM source Windows Scheduler Task ETL Workspace and scripts loaded with Python pre- processing, filtering, routing & post-processing logic
  • 9. Open Data: one Workbench script shape KML DWG
  • 10. Open Data—Automation   Scheduler Tasks for ETL Scripts (run weekly)
  • 11. Open Data - Deployment   All files in KML/KMZ format uploaded weekly to the Open Data web server   All files in other formats uploaded weekly to the public FTP server   FME ETL Scripts (now Desktop) Updates for Flexible Deployment
  • 12. Open Data- Challenges   Data Catalogue Design: groupings and classification   Required Data Groupings and File Structure   Naming Conventions: machine and user friendly   KML Specification Compliance/Validation Issue   Data fitness of use: attribute requests & selection
  • 13. The Road Ahead Website
  • 14. The Road Ahead—New Features   Hourly update replacing nightly update:   FME ETL job runs hourly from 7am to 7pm weekdays   update SDF3/SDF2 (for VanMap) and feeds data   MapGuide Enterprise AJAX Viewer uses SDF3   Atom and Geo RSS feed data generated   The Road Ahead feed data replaced an email list and is being widely used (including EOC)   Feed data with Street View via data streaming
  • 15. The Road Ahead—Data Extract   Source Data   SQL Server--point & street-based projects   Oracle Spatial--street network reference data   Output files:   SDF3 file of point & line features with 3 Road Ahead categories   Atom xml file, and a GeoRSS xml file   SDF2 files for existing VanMap Road Ahead layers   Feed xml file with Google Street View (not in production)   FME script data extract job automation
  • 17. The Road Ahead-FME Server Deployment   The Road Ahead FME script workspace published to our FME Server (2009)   The same ETL script job is run hourly against the FME Server:   "D:FMEServerClientsFMEServerConsolefmeserverconsole.exe" RUN_WORKSPACE Roadahead_dataextract/ uat_roadahead_sdf3_georss_sdf2_lp.fmw   Feed data (Atom and GeoRSS) integrated with Google Street View is accessible live via data streaming services from FME Server
  • 18. The Road Ahead-FME Server Interface   http://gisweb9/fmedatastreaming/ Roadahead_dataextract/ prd_roadahead_georss_lp_sv.fmw Feed data:
  • 19. The Road Ahead- Challenges   FME Date Formatting: customized transformer   Feed format (RSS or Atom) and specification issues:   unique ID for feed data,   duplicate feed entries on disjoin hundred block streets   FME Server user account: ID required cross domain access   Authentication limitation: use FME 2010!
  • 20. Summary   FME is very good solution for both Open Data and Road Ahead (and lots of other things!)   FME Desktop and Server have greatly improved the openness and interoperability of City’s GIS infrastructure platform   FME Server 2010 will expand our abilities in supporting GIS, Open3, Road Ahead and other initiatives going forward
  • 21. Thank You!   Questions?   For more information:   meng.li@vancouver.ca   jonathan.mark@vancouver.ca   IT Department   City of Vancouver