An overview of Terradue's solutions supporting Earth Observations (EO) Exploitation Platforms across multiple domains.
Presentation done as part of the Open Geospatial Consortium (OGC) Technical Committee ad-hoc meeting for the setup of a new domain working group on EO Exploitation Platforms.
Axa Assurance Maroc - Insurer Innovation Award 2024
Application packaging and systematic processing in earth observation exploitation platforms
1. Application Packaging and
Systematic Processing in EO
Exploitation Platforms
Earth Observation
Exploitation Platform ad hoc
OGC Technical Committee
#OGC18FR, Orleans, France 20 March 2018
3. ■ Specific fields in the domain
○ Geohazards domain
■ Earthquakes
■ Volcanoes
■ Landslides
○ Hydrology domain
■ Hydrological modelling
■ Flood mapping
■ Water quality monitoring
■ Data-intensive research
○ Targeted data
○ Targeted software
THEMATIC Exploitation Platforms
4. ■ Very fast access to large
volumes of data
○ EO data
○ In-situ data
■ Distributed processing
○ Hybrid Cloud infrastructure
○ Access to GRID resources
Thematic EXPLOITATION Platforms
5. ■ Collaborative tools
○ Referencing Publications
○ Sharing processing results
■ Different roles
○ Community scientists
○ Data providers
○ Resource providers
Thematic Exploitation PLATFORMS
6. An Ecosystem of Resources Providers
APIs of
Other Cloud
Services
APIs of
Other Cloud
Services
Data Providers
Data Providers
Tools & Services
for Exploitation
TEP
resources
Tools & Services
for Integration
EO Data
Exploitation
New EO Service
Development
New EO Product
Development
Data
Provisioning
Cloud
Provisioning
APIs of
Other Cloud
Services
Results sharing
9. Platform Data Agency
Metadata and Data Brokering
Sentinel-2 products
Amazon S3
Sentinel-1/2/3
products
Copernicus Scihub
Sentinel-1 products
Alaska Satellite
Facility
Data Agency
Terradue
Sentinel-1 products
(Sentinel 2/3
upcoming)
CNES PEPS
SHB
AS
F
PEPS
AWS
T2
Coll
GS
Landsat-8 products
EarthExplorer
USGS
Sentinel-1/2/3
products
National hubs (AT, GR, FI)
10. Cloud Processing Capacities
Federation of data centres
■ EGI.eu (BELNET, INFN RECAS BARI)
■ CNR IREA Computing Centre
■ Poznań Supercomputing and
Networking Center
Federation and Cloud Bursting
… and Cloud providers:
■ Hetzner / Terradue Private Cloud
■ EO IPT.PL
■ Amazon AWS
EC2 - powered by jclouds
Amazon Web Services
AWS
OCCI
EGI Federated Cloud
EGI
Openstack API -
powered by libcloud
IPT.PL
IPT
Openstack API -
powered by jclouds
Poznań Supercomputing
and Networking Center
PSNC
OpenNebula
registered partner
Terradue
T2
Apache CloudStack - powered
by *cloudmonkey*
Interoute
IR
Cloudsigma API
[upcoming]
CloudSigma
CS
11. Systematic SAR Data Processing
● Systematic processing using the InSAR Browse of DLR.
Each tile has the extent of an S-1 footprint.
● Providing amplitude (cyan) and
coherence (red) of all consecutive
S-1 pairs over the world tectonic mask.
● Current production rate: 150 pairs/day
● About 1TB data cached and processed daily
● Published results available freely to any user
Systematic production of S-1 InSAR pairs
12. Systematic Optical Data Processing
Surface Temperature Map developed and integrated by INGV,
processes systematically Landsat-8 data
to measure the temperature for areas
identified as «thermally active» in the
volcanic areas (22 volcanoes worldwide)
STEMP L8 – Surface Temperature Map
14. Monitoring Systematic Productions
Monthly consumption of 7K CPU/hours for processing generating more
than 15K products per month (even before entering in pre-operations)
Full-fledged dashboard with custom metrics to monitor processing
16. GEP and EPOS
A long-term plan to facilitate integrated
use of data, data products, and facilities
from distributed research infrastructures
for solid Earth science in Europe
● GEP is providing EO data processing expertise in exchange for
access to the EPOS in-situ data network
● GEP will be available to EPOS IP users and provide the interface
for the Satellite Data Thematic Core Services towards the EPOS
central hub.
EPOS - European Plate Observing System
17. The GEP and international initiatives
● Taking lessons from the GEP experience, a broader approach
developed jointly with space agencies ESA, CNES, DLR and ASI
○ aims to address priorities of the Sendai Framework for Disaster Risk Reduction 2015-
2030 taking advantage of satellite EO
○ goal: provide an EO processing & e-collaboration environment to exploit EO data to
assess geohazards and their impact
○ elaborated in the framework of CEOS WG Disasters and accepted by CEOS
○ a resource to support CEOS WG Disasters Pilots (e.g. Seismic Hazards, Landslides,
Volcanoes, Recovery Observatory) and GEO DARMA
● The GEP is also selected by the CEOS WGISS alongside the Data
Cube framework
Geohazards Lab Initiative
18. GEP & Disaster Risk Management Services
● International Charter Space and Major Disasters
selected a GEP Thematic Application for testing the
support to the Project Manager and Value Adding
partners of Charter activations:
○ Faster access to data and ancillary datasets (e.g. DEM, OSV, etc.)
○ On-line pre-processing of images (co-registering multi-sensor collections),
○ On-line image analysis (e.g. change detection, a classifier to generalize some
processing)
● ESA initiated a special contract to develop the GEP based prototype
to evaluate Exploitation Platform potential contribution to disaster
response
Charter Platform prototype
19. Ongoing Activities
● Thematic Apps: rule-based bundling of data collections and
processing services
○ Targeting specific usage scenarios
○ Processing services accessed on the GEP Portal via “Discover Thematic Apps”
○ My data uploads App: for user uploads on Data Gateway, with automated cataloguing
● Terradue Data Agency
○ Automatic multi-sourcing
○ Programmed cache
○ Personal storage (repository)
○ Production staging
○ Data usage accounting
New exploitation platform capabilities agreed upon with ESA
21. Open Cloud Strategy
■ Open APIs
Embrace Cloud bursting APIs that can
be easily plugged into the Platform’s
codebase, so to expand the Platform
offering with Providers offering
complementary strategic advantages for
different user communities.
■ Developers community
Support and nurture Cloud communities
that collaborate on evolving open
source technologies, including at the
level of the Platform engineering team,
when it comes to deliver modular
extensions.
Roadmap for interoperability
■ Self-service provisioning &
management of resources
The Platform’s end-users are able to self-
provision their required ICT resources
and to work autonomously.
■ Users rights to move data
as needed
By supporting distributed instances of
its EO Data management layer, the
Platform delivers the required level of
data locality to ensure high performance
processing with optimized costs.
22. Earth Observation Exploitation Platform
● This Domain Working Group charter defines the role for OGC
activities within the Earth Observation Data Exploitation platform
domain
● Provide an open forum for the discussion
● Presentation of interoperability requirements, use cases, pilots,
and implementations of OGC standards in this domain
● Promote OGC standards
It will not define a new platform or a new architecture
OGC Domain Working Group
23. Earth Observation Exploitation Platform
Proponents:
Nuno Catarino (DEIMOS)
Chris Holmes (PLANET)
Stan Tillman (Hexagon)
Pedro Gonçalves (Terradue)
OGC Domain Working Group
Thematic because it focuses on a specific field of the domain and thus provide a complete work environment for the scientists, enabling them to effectively perform data-intensive research by running dedicated processing software, thereby avoiding spending non-research time on developing ICT tools, sourcing data, etc.
In this case of this portal thematic, data and tools are specifically targeted for Geohazards community with data and tool to do rapid response processing, study the consequences of natural disasters or global strength.
Maybe for giving more emphasis that we support different themes we should indicate here also the Hydrology domain? (hydrological modelling, flood mapping/forecasting, water quality monitoring, soil moisture assessment, water extent and level monitoring, irrigation services, urban and agricultural water demand modelling, ...)
Scenario 1) EO Data Exploitation which allows a user to discover/select data and pre-existing processing service; process data; and visualize/analyse results, or select and apply data manipulation tools to the result; share his/her results
Scenario 2) New EO Service Development which allows a user to discover/select a data sample and software components; engineer (or upload) and validate an application (such as a processor); and deploy an application on the platform for use also by other users.
Scenario 3) New EO Product Development which allows a user to discover/select data, define data ingestion triggers on newly uploaded / deployed processor; setup and monitor the data processing tasks; and publish results as new information layers.