SeqFEWS: A data-centric workflow manager in the era of Monte-Carlo
1. SeqFEWS: A data-centric workflow
manager in the era of Monte-Carlo
2018 AUS FEWS Users Conference
Lindsay Millard, Hydrologist
2. Presentation Outline
• Context – Seqwater’s Dams
• Quick tour of previous work
• Building blocks in FEWS, and
• How to apply it to design workflows
• Key take-aways
4. • 26 referable dams
• 4 regulated water supply dams
• 22 unregulated water supply
dams
• Catchment areas range from
10 to 7,000km2
Seqwater’s Dams
5. Question
What is a Dutch word to describe FEWS?
“Datafabriek”
“Data-centric Workflow Manager”
Why not use it for non-forecasting purposes?
It would:
– assist with task efficiency and maximising skillsets
– avoid buying and learning new software
– enable efficient data management of AR&R 2016
– centralise and allow auditing of tailored design
engineering workflows
6. Motivation
• Wrapping together requirements of AR&R 2016
workflows
• Keeping workflows efficient and archived
– python scripts, GIS extraction, etc
• Scenario management
• Auditing and Continual Improvement
• Data sharing/record of project work
• Feed forward into next flood event or project
…
Export
Import
Transform
ations
Gen.Adapter
10. Project Summary:
• Improved understanding of temporal patterns
• Created a catalogue for different duration and catchment sizes
• Complete analysis: Need for a suite of moderate (6 – 24h) duration
temporal patterns to allow improved distinction between
• PMF and PMP-DF.
• Operationalise the data: Make the information readily available to
Seqwater staff
11. Related work
Examples of completed Standalones
• GateOPS Next Gen – RTC tools module
• AWAP/Silo dataset – Storage of daily rain grids from
1900
• Somerset Physical Model – Transducers to Animation
• Stochastic Storm event database – 60 Synthetic
storms
• AWRA-L Initial Loss modelling
• Monte Carlo Hydrology/Hydraulic Interface ‘Treasury’
Next “FEW” slides used with Permission of Authors
12. Synthetic rainfall events were:
• Re-formatted and imported
into FEWS
• Exported in NetCDF-Grid
format
Stochastic Storm Database
WaterCoach - 2017
13. Seq-FEWS Implementation
• Configure ‘best fit’ equation parameters
• Develop grid and scalar displays
• Integrate initial loss estimates into existing reports*
• Finalise report and procedures
• Operational system for Initial Loss estimate
AWRA-L Analysis - 2016
15. FEWS building blocks:
• An application that manages model runs efficiently
• Management of model queue to:
– assist event and scenario runs
– maximise license/hardware utilisation
• Import/Export Timeseries:
– Point, Grid and export self-contained NC.
• Transformations of Timeseries:
– Grid-grid interpolation / 2D Lookup Tables
16. AR&R 2016 Design Event Workflow
Model Build
• Calibrate Events
• Define URBS Parameters
• Storm Hyetographs
Download
• Scrape AR&R Datahub
• Chop up BoM IFD Grids
• Preburst – Initial /Continuing Loss estimation
Design
Ensemble
• Process: Areal Reduction Factors
• Temporal patterns
• Design Scenarios
Post process
Results
• Statistical Analysis & Boxplots
• Critical Duration Best Estimate for each AEP
• Plot and Report Results (1,000s)
17. 17
SeqFEWS and Python
Link websites/models using adaptors
URBS
MIKE11
Tuflow is available in 2018.03 natively
HEC RAS 4.1 adapter
Import Translate Manipulation Storage Export Publish to Reports
Use General Adapter to execute script.py or batch
.py
[roll your own]
MIKE 1D/Flood requires Iron Python
HEC RAS 5.0.4 ??
GoldSIM / RORB
Machine Learning
Matplotlib/Pandas/Seaborn Boxplots!
18. SeqFEWS
General
Adapter
.XML
Runtime
parameters
.XML .NC
.CSV
.bat / .py
Model Input
.bat / .py
Model
Results/Log
Module Data
set
GeometryRun files
Input TS Output TS
Grid
Model .exe
ModuleConfig
RegionConfig
WorkflowFiles
Modulesmodel
ModuleDataset model.zip
0d, 1d,
2d T.S.
transformation
export import
Plots
19. Export Rainfall from
FEWS datastore
URBS output
Q h TS
Import Q h into
FEWS datastore
Compute max Q
and h from FEWS
datastore
Max Q h
Importance sampling
TPT process AEP
Calculate for Q & h
Slice for location
AEP frequency
Store AEP Q and h
in FEWS datastore
URBS input
20. Key Takeaways
20
• Variety of different types of models available in FEWS
– All stitched together using the “adapter” concept
– Python is a toolbox to overcome non-standard issues
– Models can be mixed in a single workflow for auditing
• Increasing use of distributed & complex models in workflows
– Issues: speed, database sizes, complexity, …
Keep the design workflow organised and repeatable
21. Tenets of modelling
All Models are wrong
Models are never finished, only abandoned
If you can’t make it perfect, make it adjustable
Notas do Editor
A quick overview of Seqwater and why this question is important for us. Whilst the very large dams receive a lot of attention, we also have many that - whilst smaller, have downstream communities that drop them into the Hazard category that requires a PMF to be developed for a Failure Impact Assessment – not just the PMP-DF. As we shall see there is gap in the methodology for dams on catchments of this size.