The JGrass-NewAGE system essentials for hydrology modeling

The JGrass-NewAGE system essentials
Concepts
Rigon R.
Arpae, Parma, 17 Maggio 2017
GiuseppePenone

!2
R. Rigon
Introduction
MODELS
MademoiselleRose,ca.1820.EugèneDelacroix.

!3
DataParametersEquations
Mass, momentum and
energy conservation.
Chemical
transformations
Forcings and
observables
Equation’s constant. In
time!
In space they are
usually heterogeneous
Models we are talking about are computer applications
In the past they were built as monolithic programs
R. Rigon
Which kind of models

!4
I - Once a model, design and implemented as a monolithic
software entity, has been deployed, its evolution is totally in the
hands of the original developers. While this is a good thing for
intellectual property rights and in a commercial environment, this
is absolutely a bad thing for science and the way it is
supposed to progress.
RobbedfromaCCApresentation
R. Rigon
The old way

!5
II - Independent revisions and third-party contributions are
nearly impossible and especially when the code is not available.
Models falsiﬁcation (in Popper sense) is usually impossible by
other scientists than the original authors.
III- Thus, model inter-comparison projects give usually unsatisfying
results. Once complex models do not reproduce data it is
usually very difﬁcult to determine which process or
parameterization was incorrectly implemented.
R. Rigon
The old way

!6
Q: HOW CAN WE BE MORE “GALILEIAN” ?
A:YES, PRODUCING AND PROMOTING OPEN
SOURCE MODELS. THIS HOWEVER IS NOT
ENOUGH SINCE MODELS SHOULD BE
STRUCTURALLY EASY TO UNDERSTAND,
DOCUMENT, MODIFY, MAINTAIN,AND FAVOR
PROCESSES ANALYSIS.
R. Rigon
The new way

!7
MODELLING, FOR WHO ?
Which end user do you have in mind ?
Baboon,PapiusAnubis
R. Rigon
No models for everyone

!8
Modiﬁed from Rizzoli et al., ,2005
Roles
Users
Hard
Coders
Soft
Coders
Linkers Runners Player Viewers Providers
Prime
Other End
Users
Technical
Researchers
R. Rigon
Users/Roles

!9
SO WHAT ?
R. Rigon
Solutions

!10
Component-oriented software development. Objects (models
and data) should be packaged in components, exposing for re-use
only their most important functions. Libraries of components can
then be re-used and efﬁciently integrated across modelling
frameworks.Yet, a certain degree of dependency of the model
component from the framework can actually hinder reuse.
NEW (well relatively) MODELING PARADIGMS
ModiﬁedfromRizzolietal.,2005
R. Rigon
Software Engineering Solutions

!12
A F T E R 1 0 Y E A R S , W H Y T H E S E
SOFTWARES BY COMPONENTS
I N F R A S T RU C T U R E S D I D N OT
EMERGE ?
R. Rigon
Existing Examples ?
TOO INVASIVE !
TOO MANY COMPUTER
SCIENTISTS, TOO FEW
HYDROLOGISTS ?

!13
A BLUEPRINT ?
Escher-Drawingshands,1948
R. Rigon
Infrastructure design

!14
DataParametersEquations
Mass, momentum
and energy
conservation.
Chemical
transformations
Forcings and
observables
Equation’s
constant. In time!
In space they are
heteorgeneous
Numerics,
boundary and
initial conditions
Data Assimilation.
Data Models. Tools
for Analysis.
Calibration,
derivation from
proxies
To Sum up
R. Rigon

!15
Decision making
EVALUATION OF STRATEGIES THROUGH
MODELS
STRATEGIES FOR POLICY MAKERSDATA INTERPRETATION
EVALUATION OF STRATEGIES THROUGH
MODELSEVALUATION OF STRATEGIES THROUGH
MODELS
DATA INTERPRETATION
DATA INTERPRETATION
STRATEGIES FOR POLICY MAKERS
STRATEGIES FOR POLICY MAKERS
R. Rigon

!16
PREREQUISITES General
Programming LANGUAGE NEUTRAL
PLATFORM NEUTRAL: Windows, Linux and Mac
OPEN SOURCE
TARGETED AT PERSONAL PRODUCTIVITY OF
DIFFERENT USERS
People come before program efﬁciency.
BUSINESS NEUTRAL: GPL would be ﬁne if encapsulated in
components
R. Rigon

!17
PREREQUISITES Technologies
ALLOWS WRAPPING OF EXISTING CODES BUT
PROMOTES BETTER PROGRAMMING STRATEGIES
DATA BASE AWARE
DEPLOYABLE THROUGH THE WEB or as a web-server
USES MULTICORES
COMPLIANT OF STANDARDS (OGC, CUAHSI,
OTHERS)
R. Rigon

!18
PREREQUISITES Documentation/Replicability
WITH TOOLS THAT HELPS DOCUMENTATION
COMPLIANT TO STANDARDS FOR DEFINING VARIABLES
(e.g.VARIABLES AND PARAMETERS)
MANAGED IN A PUBLIC REVISION CONTROL
SYSTEM (e.g. GIT)
HAVING A STANDARD WAY AND PLACES TO
EXPOSE DOCUMENTATION
R. Rigon

Deployment choices
Rigon R, Formetta G., Antonello A., Franceschi S.
GiuseppePenone

!20
JAVA
OMS
GEOTools
JGrassTools
C/C++
Languages and infrastructures/libraries
Rigon et al.
Formetta et al., 2014

!21
JAVA
OMS
GEOTools
JGrassTools
C/C++
Python
FORTRAN
ESMF
A competing solution
Rigon et al.

!22
JAVA
OMS
GEOTools
JGrassTools
C/C++
OPENMI
C#
Another competing solution
Rigon et al.

!23
tion. As with any EMF, fully embracing the OMS3 architecture
requires a commitment to a structured model development process
which may include the use of a version control system for model
source code management or databases to store audit trails. Such
features are important for institutionalized adoption of OMS3 but
less critical for adherence by a single modeler.
techniques such as parameterized types, higher level data struc-
tures and/or object composition. The use of object-oriented design
principles for modeling can be productive for a speciﬁc modeling
project that has limited need for external reuse and extensibility.
Extensive use of object-oriented design principles can be difﬁcult
for scientists to adopt in that adoption often entails a steep learning
Fig. 1. OMS3 principle framework architecture.
Please cite this article in press as: David, O., et al., A software engineering perspective on environmental modeling framework design: The Object
Modeling System, Environmental Modelling & Software (2012), doi:10.1016/j.envsoft.2012.03.006
OMS
Rigon et al.
David et al., 2012

!24
deling components
nction and continue
ve characteristics of
a modeling object
interfaces to imple-
methods to override,
es to use. OMS3 uses
specify and describe
s and class methods
ode quality of using
rsus traditional API
tudy comparing the
hydrology models
pplied several code
ics of the different
non-invasive frame-
ncise model imple-
ode and lower code
ment. For example,
ite model required
quired between 450
mplementations had
e modeling results.
S model in OMS3
exchange items), and management of various execution states
within components including “Initialize/Run/Finalize” as described
by Peckham (2008).
While object-oriented methods focus on abstraction, encapsula-
tion, and localization of data and methods, their use can also lead to
simulation systems where objects are highly co-dependent. To
Fig. 2. OMS3 component architecture including data ﬂow, execution phases, and
encapsulation.
A software engineering perspective on environmental modeling framework design: The Object
Software (2012), doi:10.1016/j.envsoft.2012.03.006
Components again
Rigon et al.
David et al., 2012

!25
(Duriancik et al., 2008). RUSLE2 has historically been used as
a WindowsÔ-based desktop application to guide conservation
planning and inventory erosion rates over large areas. The model
provides a reusable computational engine that can be used without
a user interface for model runs in other applications. RUSLE2’s
water supply forecasts with sh
of distributed-parameter, phy
an Ensemble Streamflow P
primary ESP model base (Leave
and the PRMS hydrological wa
be used to address a wide var
information on the volume an
improve water supply forecas
ology is a modified version of
National Weather Service (D
synthesized meteorological da
timeseries data used as model
A visualization tool runn
visual display of user-selec
performs a frequency analys
the simulated hydrograph tr
historic years used with thei
ance. Different options are
analysis. One assumes that all
an equal likelihood of occ
weighting user-defined perio
a priori information, are also b
and Pacific Decadal Oscillatio
fied in the ESP procedure, and
separately for analysis. The P
will provide timely foreca
community in the western
a major source of water supp
Another modeling applicati
the OMS3 framework is the c
Ecosystem-Watershed) modeFig. 4. Cloud Services Innovation Platform (CSIP) software architecture.
Please cite this article in press as: David, O., et al., A software engineering perspective on environmental mo
Modeling System, Environmental Modelling & Software (2012), doi:10.1016/j.envsoft.2012.03.006
CSIP
Rigon et al.
David et al., 2012

!26
ulation of water quantity and quality in large watersheds
ough et al., 2010). AgES-W consists of Java-based simulation
modeling frameworks are currently under development wor
with the primary purpose of integrating existing and futur
Fig. 5. CSIP/OMS3-based mobile RUSLE2 erosion model application.
O. David et al. / Environmental Modelling & Software xxx (2012) 1e13
CSIP
Rigon et al.
David et al., 2012

!27
Other companions
Rigon et al.

!28
http://geoframe.blogspot.com
Rigon et al.

Hydrology
GiuseppePenone
Rigon R, Formetta G. Bancheri M., Serafin F., Abera W.

!30
Kriging
• Ordinary Kriging and detrended kriging and their local
versions: results are in form of raster maps or shapeﬁles for
selected points
Based on the in situ data, it selects the best variogram
(VGM) model, without any human decision, and optimises
VGM parameters automatically at each time steps.
Selection ofVGM model is NOT efﬁcient (so far).
What is there
Rigon et al.
Formetta, 2013

!31
• Separate rain from snow based on temperature: results
are in form of raster maps or shapeﬁles for selected points
It can be used conjointly with calibrators and satellite (e.g.
MODIS) data to obtain local estimates of the parameters.
RainSnow
What is there
Rigon et al.
Formetta et al. 2014

!32
• Implements degree-day, Casorzi-Dalla Fontana and
Hocks methods: needs radiation components. Results are
in form of raster maps or shapeﬁles for selected points
Snow
What is there
Rigon et al.
Formetta et al. 2014

!33
• Priestley Taylor, FAO and Penman-Monteith versions.
Various strategies were adopted to calibrate parameters.
Only PT has been throughly tested and applied.
ET
What is there
Rigon et al.
Formetta, 2013

!34
Adige
• Implements Hymod and separation of basin area in sub-
catchments numbered according to a modiﬁcation of
the Pfastetter algorithm.
Probably next version needs to be split apart into two or
three components.
What is there
Rigon et al.

!35
LWRB
SWRB
• Shortwave and longwave radiation estimation. Contains
algorithms for estimating shadows according to the
geometry of complex terrain. They also have
parameterisation for cloud cover.
What is there
Rigon et al.
Formetta et al., 2013 Formetta et al., 2016

!36
LUCA
Particle Swarm
• Calibration tools. The ﬁrst implements classic shufﬂe-
complex evolution tools. They are part of OMS core.
What is there
Rigon et al.
David et al., 2012

!37
deSaintVenant
• Integration of de Saint-Venant 1D equation (part of
Jgrasstools)
What is there
Rigon et al.
http://abouthydrology.blogspot.it/search/label/de%20Saint-Venant%20equation

!38
A - AGEs
To be checked
B- JGrass-NewAGE (https://github.com/geoframecomponents)
[Adige]
BP- Backward probabilities
Clearness Index
ET
FP -Forward probabilities
[Kriging]
NetRadiation
LWRB -
RainSnow
SWB (Simple Water Budget)
SWRB
Snow
C - JGrassTools (http://moovida.github.io/jgrasstools/)
More than 50 components
An index
Rigon et al.

!39
D - OMS (https://alm.engr.colostate.edu)
LUCA
Particle Swarm
And the whole infrastructure for running them all
An index
Rigon et al.

Case studies and Use cases
GiuseppePenone
Abera W., Formetta G., Bancheri M., Serafin F., Abera W., Rigon R.

!41
(4.1)
@t
= Jk(t)+
i
Qki(t)° ETk(t)°Qk(t)
for an appropriate set of elementary control volumes connected together. In Eq.(5.1),
S [L3
] represents the total water storage of the basin, J [L3
T°1
], ET [L3
T°1
], and Q
[L3
T°1
] are precipitation, evapotranspiration, and runoff (surface and groundwater)
respectively. The Qis represent input fluxes, of the same nature of Q, coming from
adjacent control volumes.
a
b
Figure 4.1: The location of the Posina basin in the Northeast of Italy (a) and DEM elava-
tion, location of rain gauges and hydrometer stations, subbasin-channel link partitions
used for this modelling (b).
It is clear that Eq.(5.1) is governed by two types of terms, which can be easily identi-
fied as “inputs" and “outputs". The outputs are certainly evapotranspiration, ET, and
discharges, Q, including the Qis, because they come from the assembly of control volumes.
The inputs are J(t), but this term has to be split into rainfall and snowfall. Moreover,
other inputs are ancillary to the estimation of outputs, in particular temperature, T and
radiation Rn. Another input of the equation is the definition of the domain of integration
and its“granularity", i.e. its partition into elements for which a singe value of the state
variables is produced.
In this paper we discuss the estimation of all of these input quantities, with the
Posina
A small (114 km2) basin in Vicenza province,
flowing into the Brenta river
Abera et al.
A small basin
Abera, 2016

!42
method; Isaaks et al., 1989), based on removing one data point at a time and performing
the interpolation for the location of the removed point using the remaining meteo-stations.
Finally, for this paper, kriging is used to generate time series of meterological forcings
for the centroid of each HRU. These forcings, for the purposes of this paper, are kept
constant over the whole HRU area.
Figure 4.3: The Spatial interpolation component of the NewAge system (SI-NewAge).
The ﬁgure shows how different components are connected together, here the variogram
(semivariogram) component solves for the spatial structure of measured data in the
form of an experimental variogram. The particle swarm optimization algorithm uses
the experimental variogram to identify the best theoretical semivariogram and optimal
parameter sets for each time step. Lastly, Kriging uses the best semivariogram model
Calibration of Kriging parameters
Abera et al.
Schemes of work
Abera, 2016

!43
value of Ωrank, the higher the correlation between Js and snow albedo. Those parameters
producing the highest Ωrank are used to model the hourly time steps of snowfall for each
HRU.
The derivation of snow separation parameters for each HRU is possible, however, as
is pertinent to the overall analysis of other components of the study, single, global and
optimized values of Eq.(4.3) parameters are derived.
Figure 4.4: The Snow separation component, outlining how the MODIS snow products
are used to calibrate the spatial snow accumulation ( Eq. 4.3). The dashed line shows the
iterative (calibration) process to optimize the equation. Due to the time step differences
between MODIS and the separation model output, the manual calibration is preferred
in this case.
Calibration of snow-rainfall separation
Abera et al.
Schemes of work
Abera, 2016

!44
basin outlet, but in this application we excluded it because at these scales (of around ten
kilometers) travel time in channels is irrelevant (D’Odorico and Rigon, 2003). Eventually
the Hymod component provides an estimate of the discharge at each link of the river
network of the watershed, downstream to the HRUs.
ADIGE
Figure 5.2: The HYmod component of NewAge system and its input providing compo-
nents. It shows how different components are connected, here kriging, SWE, ETP, and
calibration component connected with Adige to solve the runoff at high spatial and
temporal resolution. The detail discussion about each component can be referred at its
respective section.
Calibration of the overall system
Abera et al.
Schemes of work
Abera, 2016

!45
CHAPTER 5. ESTIMATING WATER BUDGET MODELLING OUTPUTS AND
STORAGE COMPONENT
0
1000
2000
3000
Prainfall
Psnow
Precipi,J(mm)
0
1000
2000
94/5
95/6
96/7
97/8
98/9
99/00
00/01
01/02
02/03
03/04
04/05
05/06
06/07
07/08
08/09
09/10
10/11
11/12
Q
AET
S
Watercomponents,AET,S(mm)
Hydrological years
Figure 5.11: Water budget components of the basin and its annual variabilities from
1994/95 to 2011/2012. It shows the relative share (the size of the bars) of the three
components (Q, ET and S) of the total available water J.
Annual budget
Abera et al.
The idea is that JGrass-NewAGE obtain water budgets
Aberaetal,inpreparation,2016b

!46
CHAPTER 5. ESTIMATING WATER BUDGET MODELLING OUTPUTS AND
STORAGE COMPONENT
This could have been deduced from the data alone, However, seeing it with the other
budget components enlighten the complexity of the interactions actually in place.
0
100
200
300
400
500
01-2012
02-2012
03-2012
04-2012
05-2012
06-2012
07-2012
08-2012
09-2012
10-2012
11-2012
12-2012
Date(month)
Q,ET,S(mm/month)
Q
ET
S
0
100
200
300
J(mm/month)
Figure 5.12: The same as ﬁgure 5.11, but monthly variability for the year 2012.
Monthly budget (temporal)
Abera et al.

!47
J
80 120 160 200
Q
40 80 160
ET
20 40 60
S
JanAprJulOct
−150 −100 −50 0 50
Figure 5.13: The spatial variability of the long term mean monthly water budget com-
ponents (J, ET, Q, S). For reason of visibility, the color scale is for each component
separately.
Monthly budget (spatial)
Abera et al.

!48
Events
Abera et al.
But events are equally likely well reproduced

!49
6.1. INTRODUCTION
10
20
30 40 50
Long
Lat
a
8
9
10
11
12
13
36 38 40
Long
Lat
1000
2000
3000
4000
Elevation(m)
Lat
Station
Lake Tana
b
Figure 6.1: The geographic location of Upper Blue Nile basin in the Nile basin (a) and
digitale elevation model of the basin (b). The points in ﬁgure b are the meteorological
stations used for this study.
Several validation studies of SREs have been conducted in the Ethiopian UBN basin
(Dinku et al., 2007, 2008; Haile et al., 2013; Gebremichael et al., 2014; Worqlul et al.,
2014; Romilly and Gebremichael, 2011; Hirpa et al., 2010; Habib et al., 2012). For
instance, two comparative studies by Dinku et al. (2007) and Dinku et al. (2008) on high
Blue Nile
(175000 Km2)
Abera et al.
Larger rivers
Aberaetal,inpreparation,2016c

!50
CMORPH is better in estimating ground-gauge rainfall using the two previous statistics
(i.e., r and RMSE), it is underestimating by 72%, thus being the most biased product of
the five SREs. This could be because CMORPH is only based on satellite products, and
not corrected using ground data as 3B42V7. TAMSAT, on average, is underestimating
rainfall by 30%.
CorrelationRMSEBIAS 3B42V7 CMORPH CFSR SM2R-CCI TAMSAT
8
9
10
11
12
13Lat
Correlation
<0.2
(0.2,0.3]
(0.3,0.4]
(0.4,0.5]
(0.5,0.6]
(0.6,0.7]
8
9
10
11
12
13
Lat
RMSE(mm/day)
[4, 6]
(6, 8]
(8, 10]
(10, 12]
(12, 14]
>14
8
9
10
11
12
13
36 38 40 36 38 40 36 38 40 36 38 40 36 38 40
Long
Lat
BIAS
(-0.9,-0.6]
(-0.6,-0.3]
(-0.3,-0.1]
(-0.1,0.1]
(0.1,0.3]
(0.3,0.6]
(0.6,1.4]
Figure 6.4: The spatial distribution of GOF values for different SREs: correlation coeffi-
cient (first row), RMSE (second row) and Bias (third row).
The spatial distribution of the the three GOF values (r, RMSE, BIAS) are presented
in figure 6.4. Overall the distribution of the statistics can depict a spatial pattern, i.e., the
correlations in the eastern and northeastern part of the basin are higher than western
and southwestern part. Similar pattern can be inferred from the RMSE and BIAS
Satellites products comparison
Abera et al.
Approached with satellite data
Aberaetal,2016a

!51
6.5. RESULTS AND DISCUSSIONS
A.Mehal Meda B.Debre Markos C.Assosa
0
1000
2000
3000
0 100 200 300 0 100 200 300 0 100 200 300
SREs
Gauge observations
CFSR
CMORPH
SM2R-CCI
TAMSAT
3B42V7
MeanCumulativerainfall(mm)
Days of year
Mehal_Meda
Debre_Markos Assosa
Figure 6.6: Annual mean cumulative rainfall estimations based on ﬁve SREs and gauges
data.
these two kinds of SREs (e.g., SM2R-CCI and CMORPH or 3B42V7 or TAMSAT).
Among the ﬁve SREs, TAMSAT has the highest detection capacity for lowest rainfall
intensities (91%). For all classes, TAMSAT has the highest missing rate and the highest
recorded is for the 0.1-2 mm observed rainfall class (54%), while the systematic bias
Big Bias
Abera et al.
Which are not always good
Aberaetal,2016a

!52
function of basin water storage, for instance Q and ET, good estimation of water storage
of a model has inference to its reasonable computation of other ﬂuxes as well (Döll et al.,
2014). GRACE data is an extraordinary resource to assess the over all performance of
the simulation, at least at the basin scale.
8
9
10
11
12
35 36 37 38 39 40
long
lat
3.0
3.5
4.0
4.5
5.0
Precip(mm/day)
8
9
10
11
12
35 36 37 38 39 40
long
lat
1000
1200
1400
1600
1800
Precip(mm/year)a b
Figure 7.4: The spatial distribution of daily mean (a) and annual mean rainfall estimated
from long term data (1994-2009).
Final rainfall estimates
Abera et al.
but can be corrected
Aberaetal,2016a

!53
We divide the UBN basin into 402 subbasins and channel links as shown in figure 7.2.
This spatial partitioning may not be the finest scale possible, however, considering the
size of the basin, it can be considered an acceptable compromise to capture the water
budget spatial variability.
ADIGE: Rainfall-runoff
Figure 7.3: Workflow with a list of NewAge components (in white), and remote sensing
data processing parts (gray shaded, not yet included in JGrass-NewAGE but performed
with R tools) used to derive the water budget of UBN. It does not include the components
used for the validation and verification processes.
The Modelling Solution
calibration phase
Abera et al.
Schemes of work

!54
Discharges
Abera et al.
At daily time scale

!55
Abera et al.
ET (spatial)

!56
Abera et al.
The water budget (spatial)

!57
JGRASS-NEWAGE MODEL SYSTEM AND SATELLITE DATA
0
100
200
Precip[mm/month]
−100
0
100
01 02 03 04 05 06 07 08 09 10 11 12
Months
Fluxes(Q,ET,S)[mm/month]
ET
Q
S
Figure 7.16: Basin scale long term monthly mean Water budget components based on
estimates from 1994 to 2009. It shows the relative share of the three components (Q, ET
and S) of the total available water J.
160
Abera et al.
The water budget (temporal)

!58
based on the NewAge modelling at subbasin scale, and GRACE grid resolution of 10
. Due
to the possible high leakage error introduced at high spatial resolution (Swenson and
Wahr, 2006), statistical comparison at subbasin level is not performed. However, focusing
on maps of the sample months, some level of similar spatial and temporal pattern is
revealed (ﬁgure 7.12).
−100
0
100
200
2004 2005 2006 2007 2008 2009 2010
Date
TWSC(mm/month)
NewAge
GRACE
Correlation = 0.84
Figure 7.11: Comparison between basin scale NewAge ds/dt and GRACE TWSC from
2004-2009 at monthly time step.
7.5.2 Water budget closure
The water budget components (J, ET, Q, ds/dt) of 402 subbasin of UBN is simulated for
duration of 1994-2009 at daily time series. Figure 7.13 is long term monthly mean water
JGrassNewAGE—GRACE comparison
Abera et al.
Storage variations

!59
Adige
(12000 Km2)
This is a work in progress
Abera et al.
Ongoing

!60
Ongoing
Forecasting positions
arm
courtesy of Stefano Tasin
Abera et al.

!61
Abera et al.
Infos
Introduction to JGrass-NewAGE
http://abouthydrology.blogspot.it/2015/03/jgrass-newage-essentials.html
Documentation
http://geoframe.blogspot.it/

!62
Find this presentation at
http://abouthydrology.blogspot.com
Ulrici,2000?
Other material at
Questions ?
R. Rigon
http://www.slideshare.net/GEOFRAMEcafe/parma-20160517-jgrassnewage-some-
about-the-state-of-art

The JGrass-NewAGE system essentials for hydrology modeling

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a The JGrass-NewAGE system essentials for hydrology modeling

Semelhante a The JGrass-NewAGE system essentials for hydrology modeling (20)

Mais de Riccardo Rigon

Mais de Riccardo Rigon (20)

Último

Último (20)

The JGrass-NewAGE system essentials for hydrology modeling