Geospatial ETL with Stetl - GeoPython 2016
•
3 gostaram•2,746 visualizações
Explains basics of Stetl, an Open Source spatial ETL framework. More on http://stetl.org. Talk given on GeoPython conference, June 24, 2016, Basel, Switserland.
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (8)
Semelhante a Geospatial ETL with Stetl - GeoPython 2016
Semelhante a Geospatial ETL with Stetl - GeoPython 2016 (20)
Mais de Just van den Broecke
Mais de Just van den Broecke (20)
Último
Último (20)
Geospatial ETL with Stetl - GeoPython 2016
- 1. Geospatial Data Processing with Stetl Just van den Broecke GeoPython 2016 Muttenz - Switserland June 24, 2016 www.justobjects.nl www.stetl.org
- 2. About Me Independent Open Source Geospatial Professional + Secretary OSGeo Dutch Local Chapter + Member of the Dutch OpenGeoGroep Just van den Broecke just@justobjects.nl www.justobjects.nl
- 3. Agenda • Spatial ETL • Stetl • Concepts • Cases • Status • Q & A
- 4. Spatial ETL
- 6. ETL - Extract Transform Load
- 8. GML PostGIS Shapefile WFS CSV SQLite XML Spatial ETL Example non-standard source
- 11. Plenty of Tools… Each tool is powerful by itself but cannot do the entire ETL ogr2ogr Spatial ETL
- 12. FOSS ETL - How to Combine Components? = + + ?+ ..
- 13. Example - 2011 INSPIRE-FOSS http://inspire.kademo.nl/doc/design-etl.html Nice ideas but hard to scale, deploy and reuse. Need Framework
- 14. Solution: Add Python to the Equation =+ + ?( )+ ..
- 15. Stetl Solution: Add Python to the Equation =+ +( )+ ..
- 17. Stetl Concepts
- 18. Process Chain Input Filter OutputFilter Stetl concepts Source Target
- 19. Process Chain Input Filter Output gml Filter Stetl concepts
- 20. Example: GML to PostGIS GML Reader PG Output gml Stetl concepts
- 21. Example: Data Model Transform OGR Reader XSLT GML Writer gml Stetl concepts Simple Features Complex Features or Jinja2 !
- 22. Process Chain - How? Input Filters Output Stetl concepts Stetl Config File Instantiate
- 23. Example: XML to Shape XML Input XSLT Filter OGR Output
- 24. Example: XML to Shape The Source File
- 25. Example: XML to Shape XML Input
- 26. Example: XML to Shape XML Input XSLT Filter
- 27. Example: XML to Shape Prepare XSLT Script
- 28. Example: XML to Shape XSLT GML Output
- 29. Example: XML to Shape XML Input XSLT Filter OGR Output OGC Simple Features XML DOM
- 30. Example: XML to Shape The Stetl Config File Process Chain XML InputXSLT Filter ogr2ogr Output
- 31. Running Stetl stetl -c etl.cfg stetl -c etl.cfg [-a <properties>]
- 32. Installing Stetl - PyPi Deps •GDAL+Python bindings •lxml (xml proc) •psycopg2 (Postgres) sudo pip install stetl
- 33. Installing Stetl - new: Docker https://hub.docker.com/r/ justb4/stetl
- 34. Speed: Streaming Input Filter Output gml Stetl concepts
- 35. Speed: Going Native Input Filter Output gml ogr2ogr StetlStetl Native C Libs/Progs Calls Stetl concepts
- 36. Example Components Input Filters Output Stetl concepts XMLFile XSLT GML ogr2ogr XMLAssembler GDAL/OGR LineStream XMLValidator WFS-T Postgres/PostGIS Jinja2 Postgres/PostGIS deegree* FeatureExtractor deegree* YourInput YourFilter YourOutput
- 37. Example: XsltFilter Pythonfrom util import Util, etree from filter import Filter from packet import FORMAT log = Util.get_log("xsltfilter") class XsltFilter(Filter): # Constructor def __init__(self, configdict, section): Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc) self.xslt_file_path = self.cfg.get('script') self.xslt_file = open(self.xslt_file_path, 'r') # Parse XSLT file only once self.xslt_doc = etree.parse(self.xslt_file) self.xslt_obj = etree.XSLT(self.xslt_doc) self.xslt_file.close() def invoke(self, packet): if packet.data is None: return packet return self.transform(packet) def transform(self, packet): packet.data = self.xslt_obj(packet.data) log.info("XSLT Transform OK") return packet
- 38. [etl] chains = input_xml_file|my_filter|output_std [input_xml_file] class = inputs.fileinput.XmlFileInput file_path = input/cities.xml # My custom component [my_filter] class = my.myfilter.MyFilter [output_std] class = outputs.standardoutput.StandardXmlOutput class MyFilter(Filter): # Constructor def __init__(self, configdict, section): Filter.__init__(self, configdict, section, consumes=FORMAT.etree_doc, produces=FORMAT.etree_doc) def invoke(self, packet): log.info("CALLING MyFilter OK!!!!") return packet Your Own Components Stetl concepts Step 1- Define Class Step 2- Configure Class
- 39. Data Structures Stetl concepts • Components exchange Packets • Packet contains data and status • Data formats, e.g. : xml_line_stream etree_doc etree_element (feature) etree_element_array string any . .
- 40. Cases
- 42. Cases - GML to PostGIS • National Dutch Open Datasets (GML) http://nlextract.nl ✴ Topography:Top10NL, BGT ✴ Cadastral Parcels (BRK) • Ordnance Survey UK ✴ PoC Topography (OS Mastermap)
- 43. Cases - IoT/SensorWeb • SOSPilot ✴ Dutch Air Quality Data ✴ publish to PostGIS+SOS (SOS-T) ✴ EU Reporting (Jinja2 Filter) ✴ sospilot.geonovum.nl • Smart Emission ✴ AQ Sensors hosted by citizens ✴ Calibration/Aggregation ✴ publication to SOS and OGC SensorThings ✴ data.smartemission.nl
- 44. Project Status - June 24, 2016 • v1.0.9 installable via PyPi or Docker • Documentation on www.stetl.org • Real world transforms done
- 45. Project - Planned & in progress • PyWPS integration • GUI - Flask+Celery+Redis? - Node Red? - Jupiter Notebook? • More on GitHub https://github.com/geopython/stetl