Mais conteúdo relacionado

Similar a Leveraging Data Integration for Strategic GIS Governance(20)

Mais de Safe Software(20)


Leveraging Data Integration for Strategic GIS Governance

  1. Leveraging Data Integration for Strategic GIS Governance
  2. Meet the Presenters Lesley MacKenzie Director of Innovation, Consortech Dean Hintz Senior Analyst, Safe Software
  3. Welcome to Livestorm! A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps. *Weʼll also bring up a poll on your screen during the webinar.
  4. How to download slides 1. Hover over the slide deck in the webinar room 2. Click this button How to download slides
  5. Agenda ● Intro: Data & GIS governance ● FME and its capabilities ● Using FME for Data & GIS governance ● Governance Demo ● Best practices: Integrating FME into governance frameworks ● Real-world FME examples: Data and GIS governance in practice ● Governance trends ● Conclusion and next steps ● Q&A
  6. Time for a Poll…
  7. Governance? Why Bother?
  8. “Governance is one of the most important factors determining the long-term success of a GIS” - Matthew Lewin, Esri
  9. Flash Demo
  10. Dallas Fort Worth Airport: Digital Twin Source : DFW & Willow, Inc
  11. GIS Governance & Data Governance There is room in the sandbox for all the castles, but we have to agree on how to share the sand According to this article, data governance is the process of managing the availability, usability, integrity and security of the data in enterprise systems, based on internal data standards and policies that also control data usage. GIS governance is defined as the processes and actions required to manage the planning, design, implementation, and on-going innovative use and maintenance of Geographic Information Systems (GIS) technology.
  12. GIS Governance & Data Governance How does it help us? Building something significant in our multi-system and platform world takes collaboration and purpose = Governance
  13. Benefits of a GIS Governance Program
  14. GIS Governance Framework Source: Matthew Lewin, Esri
  15. FME implemented as a platform - a versatile tool that can turn governance policy into action
  16. FME® Enterprise Integration Platform FME Desktop Build & Run Data Workflows FME Server Automate Data Workflows FME Mobile Deliver the Power of FME to Mobile The only Enterprise Integration Solution with comprehensive support for Spatial Data Powering your dataflows on premises, in the cloud, and on mobile FME Cloud FME Server in the cloud
  17. Unrivaled Data Support
  18. Using FME for Data & GIS governance
  19. Demo
  20. Metadata Change Detection and Update The Goal Update Feature Service Metadata based on local updates The Obstacles Metadata is stored in a complex nested XML structure - difficult to manage The Solution XSD XML Reader and XMLUpdater enable FME to detect changes and make updates as needed The Result Feature service metadata is automatically updated according to latest metadata
  21. Governance Demo: Detect Metadata Changes
  22. Governance Demo: Detect Metadata Changes
  23. Governance Demo: Detect Metadata Changes
  24. Metadata Harvest The Goal Automate extraction of metadata directly from database tables The Obstacles Database metadata is often out of date or incomplete The Solution FME workflow that dynamically scans records and geometry for metadata properties The Result Automatically generated table level metadata XML
  25. Governance Demo: Metadata Harvest Auto-extract table name, mod time, ID, category, fields and extents per table
  26. Governance Demo: Metadata Harvest
  27. Best practices: Integrating FME into governance frameworks
  28. Data Governance Plan: Components ● Standardized data structure ● Policy ○ Standards enforcement and procedures ○ Storage, backup, destruction ○ Security, distribution, handling ● Roles and responsibilities ● Data maintenance and procedures, metrics ● Training and Education ● Governing organization
  29. FME and Data Governance: Data Management ● Upload & Update ● Processing ● Lineage ● Sharing & Distribution ● Standards ● Validation ● Reporting
  30. FME and Data Governance: FME Engine ● Data Fusion: span silos ● Updates: validate then load ● Standards: ISO, OGC ● Validation: geometry and attributes ● Reporting: HTML, PDF, Logs
  31. FME and Data Governance: FME Server ● Automation: reliability & repeatability ● Enterprise workflow integration: control workflows that span systems ● Uploads, update & process: validation before update, change tracking ● Sharing & Distribution: optimized for data sharing ● Version control: workspaces, data ● Access control: roles and permissions ● Security: standards - OAuth2 ● Notifications: warning messages
  32. How Has FME Embraced Open Standards ● Support for > 30 OGC open standards: (GML, WFS, CityGML, IMDF, CSW) ● OGC Certification for GML, Geopackage, GeoTIFF, OGC KML ● Improvements to existing formats: XSD XML, Cesium, I3S, GML 3.3 ● Continual functionality and performance improvements to open standards infrastructure: HTTPCaller, XML / GML / JSON core, security updates ● New formats: CSW, CityJSON, WMTS, OGC API Features, CityGML3 ● Cloud Native formats: COG, COPC, ZARR, STAC ● New wrapper formats coming for XML, GML, JSON ( InfraGML, GeoSciML etc) ● Participation in OGC Pilots: Disaster, Climate, Maritime, Indoor
  33. Standards: Metadata • ISO 19115, 19139 • OGC CSW • STAC • Dublin core • Esri shp.xml • FGDC / NIEM • EU INSPIRE Supports data management, quality and automation
  34. FME and Data Governance: Validation Validation of: ● Metadata ● Schema ● Content ● Standards conformance Key Transformers: ● AttributeValidator, AttributeRangeMapper, DuplicateFilter, GeometryValidator, NullAttributeMapper, Tester, XMLUpdater, XMLValidator ● Sufficient to capture gaps in data structure as well as missing fields, bad ids, geometry etc
  35. FME and Data Governance: IMDF Validation
  36. FME and Data Governance: IMDF Validation
  37. FME and Data Governance: IMDF Validation
  38. FME and Data Governance: IMDF Validation
  39. Real-world examples: FME in data and GIS governance
  40. Publishing to Open Standards for Compliance: SOLO Matrice for Cadastral Management in Québec The Goal Deliver the updated assessorʼs data in the provincial GML format The Obstacles Constant updating of the cadastre City needs to maintain their own business structure The Solution FME Workspaces automate the import from outside sources and automate the export to GML The Result Data gets delivered successfully to the province
  41. SOLO Workflow
  42. City of Sudbury: Optimizing Road Patrol Activities The Goal Monitor road patrol activities to provide a solid data basis for resident claims Leverage in house tools The Obstacles Multiple audiences in different departments require tools and access to data for their business needs The Solution Full solution mix of ESRI and FME FME helped in both change management and in the final solution The Result FME Automations allow all stakeholders to get their data in the required format
  43. Validation for NextGen 9-1-1 The Goal Maintain data in the Next Gen 9-1-1 schema Quality is important The Obstacles Address data is constantly changing Merging multiple jurisdiction Volume of data The Solution FME Workspaces automate data validations: Schema Topological Geometry The Result Emergency agencies are able to consolidate large amounts of address data into NENA
  44. Project 33 Countries continuously contribute vast volumes of data in a diverse array of formats and structures: Excel, Shape, Access, GML, CSV, OGC geopackage, geodatabase Solution Build streamlined workflows to transform data, harvest reports and databases, catalogue metadata, perform QA/QC, and deliver dashboards and datasets Results ● Office 365 integration, calendar and tasks for Administrators ● Deliver complex dataset to Tableau for BI user consumption ● Improved operational efficiency with automated data validation. ● Improved authoritative data sharing among federal departments and agencies and increased government transparency. European Environment Agency Customer Story “FME is an amazing platform and has really changed our life.” - Jan Bliki, EEA
  45. European Environment Agency: Data Management The Task Manage INSPIRE related data submitted from across EU member states. Validate and process it to support INSPIRE SDI and data services The Problem Vast volumes of data continuously arrives in a diverse array of formats and structures: Excel, Shape, Access, GML, CSV, OGC geopackage, geodatabase The Solution Comprehensive automated, ETL based data processing platform Extract -> QA/QC -> Clean -> Conform -> Deliver The Result FME platform with 16 engines processing about 11,000 jobs a day High degree of automation providing a rich array of timely data services Next step: ReportNet3 - moving from files to APIs Data exchange and harvesting - e-Reporting and thematic portals
  46. CUSTOMER STORY PROJECT Integrate disparate datasets from internal departments and external sources, both globally and locally. SOLUTION Implemented FME within existing GIS architecture to enable data integration, management, and automation. RESULTS ● Automate and simplify complex workflows. ● Feed data to Tableau & ArcGIS. ● Publish data to web applications via MapBox, CartoDB & PDF reports. INTERNATIONAL COMMITTEE OF THE RED CROSS “FME Server plays an important role in scheduling processes, publishing data, and process monitoring.” - Yann Rebois, ICRC
  47. #4 Data Cleaning Mixed data needs to be harmonized and cleaned. Types of cleaning done by FME: • ArcSDEGridSapper • Self-intersection • Minimum area • OGC tests • ArcPy repair
  48. Geospatial Environment
  49. Powerlink Project Create a fully automated, data-driven Emergency Management system for natural disasters. Solution Use FME to monitor for events and distribute reports with maps when infrastructure is at risk. Results ● Notifications are sent to stakeholders when emergency criteria are met. ● Reports contain informative, layered maps in accessible formats like PDF. ● Time spent by staff on emergency event handling has been reduced from days to hours. Customer Story
  50. Mapping Products
  51. Emergency Spatial Journey 2019 2018 2017 2015 2011 2013 Pre 2010 Fully Automated, Data driven approach to Emergency awareness Optimised data & information management Expanded data access to increase reliability Full automation of data downloads FME Server Introduced manual reactive mapping & data retrieval & creation No internal emergency data or Information Support Development of FME mapping automation Patent Pending
  52. Best Practices & Trends
  53. How might your work be impacted by ungoverned data and systems?
  54. Lesley’s Best Practices ● Understand the different roles the platform plays ● Implement with purpose ● Integrate FME into all facets of GIS governance ● Continual improvement mindset
  55. Governance: Trends Good governance enables: ● Automation: standards powers data flow between systems ● Collaboration: data sharing, integrate new tools ● ROI: data management, maintenance and search efficiencies ● Risk management: data quality, legal, privacy, security ● Innovation: BIM, 3D, Digital Twins, Real time, AI, ML
  56. Digital Twin: San Francisco Airport ● CAD to GIS BIM to GIS workflows ● QA/QC workflows including Exterior Data Model, Interior Data Model & Subsurface Utilities model ● Real-time device tracking of assets ● Real-time tracking of Wag Brigade animal locations ● Email notifications to stakeholders ● Tracking of state wildfire activity and evacuation orders ● Convert data to IMDF & other indoor formats support public and 3rd party routing ● 911 Computer Aided Dispatch ● Many are not spatial workflows
  57. Other Data sources ACI AGIS Schema IMDF KML Indoor GML Airport GIS Data Airport Map Model Read & Transform Read & Write Validate . . . HERE SFO: Indoor Map Data Flow Using ACI AGIS Schema Staging Database
  58. Customer Story Project Schiphol is Europeʼs busiest airport. Theyʼve won almost 200 awards for their efficiency, innovation, and passenger experience. Their goal was to become the #1 preferred airport in Europe. The vision for this was to offer data to the public via open APIs; enable indoor mapping; and create a digital twin of the airport. Solution Integrated asset and infrastructure data, enhanced it, validated it, and distributed it across teams. Unified data across departments into a central GIS database. Results ● Synchronized GIS database ● Open APIs available to developers ● Foundation for building digital twin (IFC) ● Indoor mapping in Apple Maps and a custom augmented reality app Improving operations and passenger experience to become a top airport in Europe. Digital Twin: Amsterdam Airport Schiphol
  59. Data Integration ESB Platform at Schiphol GIS Core-database with Assets and Infrastructure information Edit Platform Schiphol Internal Schiphol Optimized Viewing data Viewing Platform Schiphol Internal Viewing Platform Schiphol Internal&External Optimized Geo Data Realtime External Sources API Platform Public Geo API Wayfinding POI’s Maps Routing IFC XML via FME is the “Glue” between systems to keep data up to date and in Sync.
  60. Schiphol Airport
  61. Governance Trends: Lessons Learned ● Automation: standards powers data flow -> BIM ● Collaboration: data sharing, integrate new tools -> Indoor Mapping ● ROI: data management, maintenance efficiencies -> auto updates via ESB ● Risk management: data quality, legal, lineage,privacy, security -> validation detects problems ● Organizational Risk: staff turnover / retirement, institutional knowledge, access ● Innovation: enabled by governance -> BIM, Digital Twins, Real-time, AI
  62. Conclusion & Next Steps
  63. Governance: Key Take-Aways ● Standardized data structure: internal standards based on external standards smooths integration (OpenAPI, JSON, IMDF, IFC, ACI) ● Careful Policy planning to ensure: ○ Standards are followed ○ Management of full data life cycle (create, mod, destroy) ○ Data authorship lineage, security and access is controlled ○ Data Resilience ● Human aspects: roles, responsibilities, training ● Governing organizations (internal and external) ● FME helps you implement your governance policies by giving you control over data management
  64. You can achieve GIS Governance in your organization.
  65. Next Steps Download FME 2022.2 & the webinar workspaces Apply the workspaces to your own data to manage changes Free Trial | Upgrade Contact us. We would love to chat with you about anything data
  66. Free Resources Upcoming & on- demand webinars eBook: Spatial Data for the Enterprise For Dummies Free 90-min Workshop: FME Accelerator
  67. GIS Governance Resources ● Blogs ○ GIS Governance 101 with FME by Consortech ○ Data Governance: The Basic Information ○ Practical Guidance Series ● Webinars & Presentations ○ Metadata Matters! What it is and How to Manage it ○ FME Governance: How to Structure the Use and Growth of FME ● Knowledge articles: FME and Metadata Creating and Validating IMDF ● Kit: FME GOVERNANCE KIT: Harness the Full Power of FME ● Study: Surveying Data Governance Policy Models
  68. Claim Your Community Badge Get community badges for watching webinars! Todayʼs Code: GWFCM
  69. Registration is open! The Peak of Data Integration 2023 Europe UC September 5-7, 2023 | Bonn, Germany
  70. Q&A
  71. Thank you! Download FME Free Trial | Upgrade Chat with us Connect with us in the Community Please share your feedback with us through the webinar survey!
  72. Bonus slides!
  73. #1 Simplifying Geometry A single SDE database is behind: ● large scale paper maps ● cross-border maps ● web applications ● Importing to Tableau
  74. #2 Site Reports Essential for safety and security of ICRC teams ● Premises Management web app ● PDF creation for site reports ● Raster backgrounds from Google or Esri services, where available
  75. #3 Supplying Data to Tableau Tableau used for real time analysis of operations and programs. •Preparing spatial data for Tableau •Joining business data to spatial data
  76. #5 Integrating Source Data Integrating source data from 3rd-party web services etc. Examples: • Free & open healthcare location data. •ACLED – armed conflict location & event data.
  77. CUSTOMER STORY “We run a lot of Jobs.” - Lance Bohy, USFS OCIO PROJECT Culminate roads data from multiple sources for validation. SOLUTION Automate and scale change detection processes. RESULTS ● Clean, High quality, uptodate Data ● Hands off synchronization ● Facilitate the sharing/publishing of geospatial data between partners and business units. US Forest Service
  78. #6 Population Kiosk Reference site for population data Helps answers questions like: •What populations are negatively affected by disease or displacement? •In what villages have abuses been committed or outbreaks occurred?