Presentation by Jan Verkade, Deltares, at the Delft-FEWS International User Days 2020, during Delft Software Days - Edition 2020. Monday, 2 November 2020.
6. โScripting a Delft-FEWS configuration
1. Why at all relevant?
2. What do we mean by โscriptingโ?
3. Why script?
4. Approaches
โ what do we (not) script?
โ where do we script from?
โ what tools do we use?
5. Example: the Deltares global fluvial flow forecasting system
6. Where/ how to start?
7. Some notes
โข Scripting is very much โa field in progressโ
โข Not all questions have been asked
โข No definitive answers have been given
โข Slowly, some โbest practicesโare forming
This Photo by Unknown Author is licensed under CC BY-NC
8. Why spend time talking about scripting?
โข Delft-FEWS developments in place that allow for scripting to be done
โ templating of General Adapters, Transformations
โ storage of metadata in .csv, .dbf/.shp files
โ increased use of โfunctionsโ in location set definition, threshold definitions, etc.
โข Solves potential problems
โ efficient configuration building
โ separation of responsibilities: metadata management v. Delft-FEWS config management
โ near immediate uptake of metadata changes in forecasting applications
โ availability of configuration skills
โข Growing number of applications where scripting is used to build configurations
9. What do we mean by โscripting a Delft-FEWS configโ?
โScripting is the automated creation of parts of a Delft-FEWS configuration, based on
metadata.โ
โข Gauge locations
โข Basin models
โข Rating curves
โข Thresholds
โข โฆ
โข Workflow files
โข Map layer files
โข Topology files
โข ModuleDataSetFiles
โข โฆ
10. Why script?
1. Ensure compliance between metadata and configurations
2. Quickly include metadata changes in a forecasting application
3. Impose uniformity on configurations
4. Cost efficiency
5. Allow for separation of tasks: management of metadata v management of configurations
6. โฆ
This Photo by Unknown Author is licensed under CC BY-SA-NC
11. What do we (not) script?
Do script:
โข config elements that are subject to frequent updates
โข elements that are repeated throughout the configuration
Do not script:
โข elements that are highly custom (explorer.xml, global.properties, โฆ)
โข elements that appear few times in your configuration only
โข elements that donโt change very often
โ this varies from application to application
12. Approaches: where to script from?
โข file based storage (e.g., .xlsx or .csv files)
โข local database (e.g., MS Access Database)
โข database on a server (e.g., SQL)
โข โฆ
Considerations includeโฆ
โข Whatโs currently available?
โข Who manages the metadata v. who builds the configuration
โข Degree of referential data integrity required
โข Costs
โข โฆ
This Photo by Unknown Author is licensed under CC BY-SA
13. Approaches: tools for scripting
โข Any scripting language. So far weโve seen scripts in Visual Basic, Python and R.
โข Export of data to .csv files is almost trivial
โข Helpful if there is a good xml library available
โ Matlab: XML toolbox
โ Python: xml.etree.ElementTree library
โ R: xml2 package
16. IMFS: Englandโs new flood forecasting system
EA database
GUI
Python
Ingest
Telemetry
Thresholds
Models
Delft-FEWS
configuration
17. IMRD: the metadata repository underlying IMFS
Database and GUI developed by JBA. Additional information from paul.wass@jbaconsulting.com
18. Database and GUI developed by JBA. Additional information from paul.wass@jbaconsulting.com
19. Database and GUI developed by JBA. Additional information from paul.wass@jbaconsulting.com
20. Scripts
โข Python based; written by Martijn Kwant
โข Produce over 5,100 configuration files
โ ~2,100 workflow files
โ ~1,800 ModuleDataSetFiles
โ ~ 80 mapLayerFiles jointly containing ~380,000 lines of metadata
โข In future, outputs will automatically be included in the operational client-server system (!)
21.
22. gloffis: the Deltares global fluvial flow forecasting system
โข patchwork of multiple models that can jointly cover the earthโs land
โข firmly based on wflow models, made possible by โthree clicks to a modelโ principle
โข models vary in
โ coverage (and may, and indeed do, overlap)
โ temporal and spatial resolution
โ model type (sbm, hbv, โฆ)
โ required wflow.exe version (2019.1, 2020.1 โฆ)
โข workflows vary in
โ model that is called
โ weather forecast product that is used
(currently, choice between various DWD ICON products: regional/global, ensemble/deterministic)
โ this is where the scripting rationale is: NWPforecasts, wflow models and workflows
24. gloffis metadata repository: MS Access database
โข Enforces referential data integrity
โข Allows for user-friendly access and editing of data using forms
โข Not centrally hosted โ pros and cons
25.
26.
27.
28. Changes that are made, are conveniently highlightedby version control software
30. Scripting: where and how to start?
โข Does it make sense for me to invest in โscriptingโ?
โข How โdynamicโ is my configuration? Where, in the config, are the dynamics?
โข How is my metadata organized?
โข Do I have expertise in scripting available?
โข โฆ
โ feel free to ask for help!
This Photo by Unknown Author is licensed under CC BY-SA
31. Whom to talk to about scripting
โข First and foremost: your โaccount managerโ
โข Any of below Deltares staff membersโฆ
Marc van Dijk
marc.vandijk@deltares.nl
Jan Verkade
jan.verkade@deltares.nl
Bart van Osnabrugge โ
de Vries
Bart.vanOsnabrugge@deltares.nl
Martijn Kwant
Martijn.Kwant@deltares.nl
32. Jan Verkade
jan.verkade@deltares.nl, +31 6 5161 6107
Questions to the audience
Have you considered moving to a โscripted configurationโ?
If so, what were/are your considerations?