SlideShare uma empresa Scribd logo
1 de 18
Baixar para ler offline
CONNECT. TRANSFORM. AUTOMATE.
Managing Data
Synchronization
Jerrod Stutzman
GIS Platform Engineer – Devon Energy
Project Goal
CONNECT. TRANSFORM. AUTOMATE.
!  Create a centralized spatial system for storing
and displaying SCADA (supervisory control and
data acquisition) data
!  View data on map via desktop and mobile
!  Google-like search and performance
Challenges
!  Need to see consistent data on desktop web app
and custom iPhone app
!  Location information from SCADA database is
inconsistent and unreliable – need to join with
reliable data for mapping
!  Need to show on map along with other Devon
assets (wells, pipelines, leases, etc)
!  ArcSDE is authoritative spatial database
!  Won’t be used if it isn’t fast!
Solution
!  Spatial Reasoning System (SRS)
!  OpenGeo Suite (POSTGIS data served via
GeoServer using GeoWebCache)
!  Apache SOLR search platform
!  Internally developed web and mobile apps
!  FME & FME Server for data creation and
synchronization
!  Custom code for certain integrations
Why OpenSource
!  PERFORMANCE
!  Biggest pain point historically is our well dataset.
Over 4 million features + 3 geometries (surface
location, bottom location, directional path)
!  POSTGIS + GeoServer render and cache this
dataset quite fast
!  POSTGIS can store all 3 geometries on one row!
!  Reduced database complexity
!  Low cost
!  FME is used within the SRS
to create and sync data for
wells, pipelines, leases,
seismic, facilities, etc
!  FME Server handles
scheduling and failure
reporting (email)
SRS Integrations Diagram
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
!  SCADA Radio Tower data is maintained in Excel
!  Updated quarterly. Full dataset update
!  GPS Location and Range need to be shown
!  Need to store 1 to 3 geometries for each tower
Selected Workbench Example 1
EXCEL Spreadsheet to POSTGIS
Selected Workbench Example 2
Non-spatial Oracle to POSTGIS
Selected Workbench Example 2
Non-Spatial Oracle to POSTGIS
!  Non-spatial well production data from Oracle DB
is synchronized with POSTGIS
!  Workflow:
Retrieve last
run time from
POSTGIS
Query
ORACLE for
changes
Query
POSTGIS for
matches
Compare to
determine
INSERT/UPDATE
Determine
DELETES
Update
POSTGIS
DB triggers
update
SOLR
Selected Workbench Example 2
Non-Spatial Oracle to POSTGIS
Selected Workbench Example 3
Seismic Data from Oracle DB
Selected Workbench Example 3
Seismic Data from Oracle DB
!  Seismic data can be represented spatially as lines
or polygons
!  This is stored in the seismic DB as a coordinate list
of vertices (numbers only, non-spatial DB)
!  This data needs to be ‘spatialized’ and stored in a
spatial database – synced nightly (incremental
updates)
!  Complexity Reduction – store all data (2 different
geometry types) in same table
Selected Workbench Example 3
Seismic Data from Oracle DB
Result
Future
!  Ability to do an “UPSERT” DB transaction would
greatly simplify some of the logic required to
determine DB transaction!
!  ARCSDE30 Writer actually had this capability, if it
did not find the record to update, it would insert
!  Need to revise old workbenches to use best
practices and gain efficiencies
!  Scaling SRS to handle more data and users
Thank You!
!  Questions?
!  For more information:
!  Jerrod Stutzman
!  Jerrod.Stutzman@dvn.com
!  Devon Energy Corporation
CONNECT. TRANSFORM. AUTOMATE.

Mais conteúdo relacionado

Mais procurados

WHODIS_kearns_presentation.v0a
WHODIS_kearns_presentation.v0aWHODIS_kearns_presentation.v0a
WHODIS_kearns_presentation.v0a
Edward Kearns
 
06 how to write a map reduce version of k-means clustering
06 how to write a map reduce version of k-means clustering06 how to write a map reduce version of k-means clustering
06 how to write a map reduce version of k-means clustering
Subhas Kumar Ghosh
 
Hw09 Making Hadoop Easy On Amazon Web Services
Hw09   Making Hadoop Easy On Amazon Web ServicesHw09   Making Hadoop Easy On Amazon Web Services
Hw09 Making Hadoop Easy On Amazon Web Services
Cloudera, Inc.
 
CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
CloudClustering: Toward an Iterative Data Processing Pattern on the CloudCloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
Ankur Dave
 

Mais procurados (20)

FME 2013 Based Spatial Export As A Service
FME 2013 Based Spatial Export As A ServiceFME 2013 Based Spatial Export As A Service
FME 2013 Based Spatial Export As A Service
 
Plantpredict: Solar Performance Modeling Made Simple
Plantpredict: Solar Performance Modeling Made SimplePlantpredict: Solar Performance Modeling Made Simple
Plantpredict: Solar Performance Modeling Made Simple
 
A Study on New York City Taxi Rides
A Study on New York City Taxi RidesA Study on New York City Taxi Rides
A Study on New York City Taxi Rides
 
WHODIS_kearns_presentation.v0a
WHODIS_kearns_presentation.v0aWHODIS_kearns_presentation.v0a
WHODIS_kearns_presentation.v0a
 
NDGISUC2017 - Mobile Data Collection & Reporting System
NDGISUC2017 - Mobile Data Collection & Reporting SystemNDGISUC2017 - Mobile Data Collection & Reporting System
NDGISUC2017 - Mobile Data Collection & Reporting System
 
AWS EMR (Elastic Map Reduce) explained
AWS EMR (Elastic Map Reduce) explainedAWS EMR (Elastic Map Reduce) explained
AWS EMR (Elastic Map Reduce) explained
 
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonough
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonoughFME World Tour 2015 Belfast - Donegal County Council - Daragh McDonough
FME World Tour 2015 Belfast - Donegal County Council - Daragh McDonough
 
Streaming API, Spark and Ruby
Streaming API, Spark and RubyStreaming API, Spark and Ruby
Streaming API, Spark and Ruby
 
Apache spark vs hadoop
Apache spark vs hadoopApache spark vs hadoop
Apache spark vs hadoop
 
06 how to write a map reduce version of k-means clustering
06 how to write a map reduce version of k-means clustering06 how to write a map reduce version of k-means clustering
06 how to write a map reduce version of k-means clustering
 
Low cost solutions for Lidar and GIS analysis
Low cost solutions for Lidar and GIS analysisLow cost solutions for Lidar and GIS analysis
Low cost solutions for Lidar and GIS analysis
 
Llnl talk
Llnl talkLlnl talk
Llnl talk
 
Mapreduce introduction
Mapreduce introductionMapreduce introduction
Mapreduce introduction
 
FME World Tour 2016: Your Data in Motion (Safe Software)
FME World Tour 2016: Your Data in Motion (Safe Software)FME World Tour 2016: Your Data in Motion (Safe Software)
FME World Tour 2016: Your Data in Motion (Safe Software)
 
Scalable Data Analytics and Visualization with Cloud Optimized Services
Scalable Data Analytics and Visualization with Cloud Optimized ServicesScalable Data Analytics and Visualization with Cloud Optimized Services
Scalable Data Analytics and Visualization with Cloud Optimized Services
 
developmentSEED Presentation for Earth Observation in the Cloud Demo Day
developmentSEED Presentation for Earth Observation in the Cloud Demo DaydevelopmentSEED Presentation for Earth Observation in the Cloud Demo Day
developmentSEED Presentation for Earth Observation in the Cloud Demo Day
 
AAPG Geoscience Technology Workshop 2019
AAPG Geoscience Technology Workshop 2019AAPG Geoscience Technology Workshop 2019
AAPG Geoscience Technology Workshop 2019
 
ESIP 2018 - The Case for Archives of Convenience
ESIP 2018 - The Case for Archives of ConvenienceESIP 2018 - The Case for Archives of Convenience
ESIP 2018 - The Case for Archives of Convenience
 
Hw09 Making Hadoop Easy On Amazon Web Services
Hw09   Making Hadoop Easy On Amazon Web ServicesHw09   Making Hadoop Easy On Amazon Web Services
Hw09 Making Hadoop Easy On Amazon Web Services
 
CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
CloudClustering: Toward an Iterative Data Processing Pattern on the CloudCloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
CloudClustering: Toward an Iterative Data Processing Pattern on the Cloud
 

Semelhante a Managing Data Synchronization Between ArcSDE and POSTGIS using FME

Hot-Spot analysis Using Apache Spark framework
Hot-Spot analysis Using Apache Spark frameworkHot-Spot analysis Using Apache Spark framework
Hot-Spot analysis Using Apache Spark framework
Supriya .
 
2004-11-13 Supersite Relational Database Project: (Data Portal?)
2004-11-13 Supersite Relational Database Project: (Data Portal?)2004-11-13 Supersite Relational Database Project: (Data Portal?)
2004-11-13 Supersite Relational Database Project: (Data Portal?)
Rudolf Husar
 
Leicester 2010 notes
Leicester 2010 notesLeicester 2010 notes
Leicester 2010 notes
Joanne Cook
 
EEDC - Apache Pig
EEDC - Apache PigEEDC - Apache Pig
EEDC - Apache Pig
javicid
 

Semelhante a Managing Data Synchronization Between ArcSDE and POSTGIS using FME (20)

Open Source Databases And Gis
Open Source Databases And GisOpen Source Databases And Gis
Open Source Databases And Gis
 
Hot-Spot analysis Using Apache Spark framework
Hot-Spot analysis Using Apache Spark frameworkHot-Spot analysis Using Apache Spark framework
Hot-Spot analysis Using Apache Spark framework
 
Dev Summit 2011 - Talk
Dev Summit 2011 - TalkDev Summit 2011 - Talk
Dev Summit 2011 - Talk
 
2004-11-13 Supersite Relational Database Project: (Data Portal?)
2004-11-13 Supersite Relational Database Project: (Data Portal?)2004-11-13 Supersite Relational Database Project: (Data Portal?)
2004-11-13 Supersite Relational Database Project: (Data Portal?)
 
Srds Pres011120
Srds Pres011120Srds Pres011120
Srds Pres011120
 
Stsg17 speaker yousunjeong
Stsg17 speaker yousunjeongStsg17 speaker yousunjeong
Stsg17 speaker yousunjeong
 
GeoKettle: A powerful open source spatial ETL tool
GeoKettle: A powerful open source spatial ETL toolGeoKettle: A powerful open source spatial ETL tool
GeoKettle: A powerful open source spatial ETL tool
 
Building a Spatial Database in PostgreSQL
Building a Spatial Database in PostgreSQLBuilding a Spatial Database in PostgreSQL
Building a Spatial Database in PostgreSQL
 
SPL_ALL_EN.pptx
SPL_ALL_EN.pptxSPL_ALL_EN.pptx
SPL_ALL_EN.pptx
 
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
IRJET - Evaluating and Comparing the Two Variation with Current Scheduling Al...
 
Azure Cosmos DB - Technical Deep Dive
Azure Cosmos DB - Technical Deep DiveAzure Cosmos DB - Technical Deep Dive
Azure Cosmos DB - Technical Deep Dive
 
ADAPTER
ADAPTERADAPTER
ADAPTER
 
Eedc.apache.pig last
Eedc.apache.pig lastEedc.apache.pig last
Eedc.apache.pig last
 
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
Azure Cosmos DB - NoSQL Strikes Back (An introduction to the dark side of you...
 
226 team project-report-manjula kollipara
226 team project-report-manjula kollipara226 team project-report-manjula kollipara
226 team project-report-manjula kollipara
 
Leicester 2010 notes
Leicester 2010 notesLeicester 2010 notes
Leicester 2010 notes
 
NoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQLNoSQL and Spatial Database Capabilities using PostgreSQL
NoSQL and Spatial Database Capabilities using PostgreSQL
 
Data Warehouse Offload
Data Warehouse OffloadData Warehouse Offload
Data Warehouse Offload
 
EEDC - Apache Pig
EEDC - Apache PigEEDC - Apache Pig
EEDC - Apache Pig
 
Real-Time Analytics with Spark and MemSQL
Real-Time Analytics with Spark and MemSQLReal-Time Analytics with Spark and MemSQL
Real-Time Analytics with Spark and MemSQL
 

Mais de Safe Software

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Safe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Safe Software
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Safe Software
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
Safe Software
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
Safe Software
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software
 

Mais de Safe Software (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action:  Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action:  Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Powering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data StreamsPowering Real-Time Decisions with Continuous Data Streams
Powering Real-Time Decisions with Continuous Data Streams
 
The Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data EcosystemThe Critical Role of Spatial Data in Today's Data Ecosystem
The Critical Role of Spatial Data in Today's Data Ecosystem
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Mastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GISMastering MicroStation DGN: How to Integrate CAD and GIS
Mastering MicroStation DGN: How to Integrate CAD and GIS
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...Best Practices to Navigating Data and Application Integration for the Enterpr...
Best Practices to Navigating Data and Application Integration for the Enterpr...
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
New Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s FoundersNew Year's Fireside Chat with Safe Software’s Founders
New Year's Fireside Chat with Safe Software’s Founders
 
Taking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New HeightsTaking Off with FME: Elevating Airport Operations to New Heights
Taking Off with FME: Elevating Airport Operations to New Heights
 
Initiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 

Último

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Último (20)

Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 

Managing Data Synchronization Between ArcSDE and POSTGIS using FME

  • 1. CONNECT. TRANSFORM. AUTOMATE. Managing Data Synchronization Jerrod Stutzman GIS Platform Engineer – Devon Energy
  • 2. Project Goal CONNECT. TRANSFORM. AUTOMATE. !  Create a centralized spatial system for storing and displaying SCADA (supervisory control and data acquisition) data !  View data on map via desktop and mobile !  Google-like search and performance
  • 3. Challenges !  Need to see consistent data on desktop web app and custom iPhone app !  Location information from SCADA database is inconsistent and unreliable – need to join with reliable data for mapping !  Need to show on map along with other Devon assets (wells, pipelines, leases, etc) !  ArcSDE is authoritative spatial database !  Won’t be used if it isn’t fast!
  • 4. Solution !  Spatial Reasoning System (SRS) !  OpenGeo Suite (POSTGIS data served via GeoServer using GeoWebCache) !  Apache SOLR search platform !  Internally developed web and mobile apps !  FME & FME Server for data creation and synchronization !  Custom code for certain integrations
  • 5. Why OpenSource !  PERFORMANCE !  Biggest pain point historically is our well dataset. Over 4 million features + 3 geometries (surface location, bottom location, directional path) !  POSTGIS + GeoServer render and cache this dataset quite fast !  POSTGIS can store all 3 geometries on one row! !  Reduced database complexity !  Low cost
  • 6. !  FME is used within the SRS to create and sync data for wells, pipelines, leases, seismic, facilities, etc !  FME Server handles scheduling and failure reporting (email) SRS Integrations Diagram
  • 7. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
  • 8. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS !  SCADA Radio Tower data is maintained in Excel !  Updated quarterly. Full dataset update !  GPS Location and Range need to be shown !  Need to store 1 to 3 geometries for each tower
  • 9. Selected Workbench Example 1 EXCEL Spreadsheet to POSTGIS
  • 10. Selected Workbench Example 2 Non-spatial Oracle to POSTGIS
  • 11. Selected Workbench Example 2 Non-Spatial Oracle to POSTGIS !  Non-spatial well production data from Oracle DB is synchronized with POSTGIS !  Workflow: Retrieve last run time from POSTGIS Query ORACLE for changes Query POSTGIS for matches Compare to determine INSERT/UPDATE Determine DELETES Update POSTGIS DB triggers update SOLR
  • 12. Selected Workbench Example 2 Non-Spatial Oracle to POSTGIS
  • 13. Selected Workbench Example 3 Seismic Data from Oracle DB
  • 14. Selected Workbench Example 3 Seismic Data from Oracle DB !  Seismic data can be represented spatially as lines or polygons !  This is stored in the seismic DB as a coordinate list of vertices (numbers only, non-spatial DB) !  This data needs to be ‘spatialized’ and stored in a spatial database – synced nightly (incremental updates) !  Complexity Reduction – store all data (2 different geometry types) in same table
  • 15. Selected Workbench Example 3 Seismic Data from Oracle DB
  • 17. Future !  Ability to do an “UPSERT” DB transaction would greatly simplify some of the logic required to determine DB transaction! !  ARCSDE30 Writer actually had this capability, if it did not find the record to update, it would insert !  Need to revise old workbenches to use best practices and gain efficiencies !  Scaling SRS to handle more data and users
  • 18. Thank You! !  Questions? !  For more information: !  Jerrod Stutzman !  Jerrod.Stutzman@dvn.com !  Devon Energy Corporation CONNECT. TRANSFORM. AUTOMATE.