RSS Sentinel Toolbox

1. Research & Service Support
Delft Software Days 2015, Deltares
28/10/2015
Processing services for Sentinel-1 data
exploitation
RSS Team member: J. Manuel Delgado

2. Outline
1.Service description
2.Processing services
a. RSS processing on demand
b. RSS Cloud Toolbox
3.SNAP / Sentinels Toolbox
4.Examples of Sentinel-1A data processing

3. Service Objectives
• Support the Earth Observation community in exploiting EO data
• data provisioning
• data access
• data processing
• Empower effective and efficient research process
• Support exploitation environments
• enabling product generation
• specific derived information
instead of bringing “Big Data” to “users”
we bring “users” to “Big Data”

4. User community
RSS user community includes:
• Scientists (research centers, universities)
• Service developers (sme or industry developing technology projects)
• European or international institutions
who have started their work locally on their workstations with some data
samples and want to “scale up” to massive data processing or wide
service exposure.

5. Support in the Research Process

6. Support options
Provision of customized Sandboxes and Cloud Toolboxes
Customized Virtual Machines with pre-installed software (e.g. NEST, Sentinel
Toolbox, Matlab, IDL, other) to support user-specific processing requirements
can be provided on researcher request
Provision of EO data of interest
Data can be provided either in the sandbox/toolbox or on a dedicated Data
Farm for further processing. Reference datasets from ESA and Third Party
Earth Observation data on required areas of interest and time intervals can be
made available for download to the researcher's desktop
Processor Hosting / Processing
Algorithm integration and hosting service of selected processors on RSS
virtual infrastructure and support to processing campaigns on required areas
of interest and time intervals. Flexible processing resources (cloud) are used.
High Level Scientific and IT Competences
RSS team can support different stages of the algorithm/service development
with highly skilled competences
Research and Service Support

7. Sandboxes, Cloud toolboxes based on Customized Virtual Machines
with pre-installed software, mainly thought for post-processing and
analysis activities.
• Can be provided with the software required by the user
• Flexible amount of CPUs, RAM and dedicated storage, to be
tailored on the actual user needs and the type of machine
template required (virtual toolboxes)
Processing Resources
Grid and Cloud Processing On-Demand: g-pod
Generic GRID and Cloud-based processing on demand environment
with high-performance and sizeable computing resources.
Provides the necessary flexibility for building an application virtual
environment with quick accessibility to data, computing resources and
results.

8. RSS processing on demand
http://gpod.eo.esa.int

9. 1. The Web Portal is the user access point where he/she can
a. Submit processing requests
b. Check the status of the task
(created/pending/active/completed/failed)
c. Perform some basic operations: abort/resubmit/copy the task
2. Catalogue Access Service (CAS) is the catalogue where the input files and
auxiliary data information (metadata and URL) are registered as well as the
path/address from which the data is stored
3. The Light Grid Engine (LGE) is the orchestrator of the GRID
a. It stores the applications (PI algorithms)
b. It receives processing requests from the G-POD web portal and
sends processing tasks to the computing clusters
4. Stores and datafarm are where the data belonging the different available
missions is physically stored
5. Grid and Cloud computing resources are where the processing are
performed
Architecture

10. RSS Flexible Resources
On-demand processing service:
Platform
Infrastructure
G-POD
ESRIN
- 172 cores
- 400 TB
UK-PAC
- 96 cores
- 300 TB
Flexible/
Unlimited
Infrastructure
- 10-200 cores
- 1-10 TB
EO Scientists
Principal
Investigators
deliveryProcess
EO data
Volume accessed by PI projects in 2014:
• Total Number Submitted Jobs 39,456
• Average Number of Products per Job: 48
• Average Product Size: 700 MB
• Total Size Data Processed: 1326 TB

11. Catalogue

12. S1TBX service on G-POD

13. S1TBX service on G-POD

14. RSS CloudToolbox

15. RSS CloudToolbox Service Portal
http://eogrid.esrin.esa.int/cloudtoolbox

16. If approved the
administrator creates the
CloudToolbox
CloudToolbox request

17. start/stop button: the user
manages the VM. Commands are
launched through the APIs
User can require a machine update
CloudToolbox upgrade

18. 1. Via SSH
a. Supporting the command lines operations
2. Via Desktop viewer (VNC viewer)
a. The user experience is not affected by the unpleasant
feeling of delay
3. Users are provided with detailed instructions on how to
connect
4. VMs are configured with an automatic shutdown after
30 minutes of inactivity.
How to connect

19. SNAP / Sentinels Toolbox

20. 1. The Sentinel Toolboxes and full source code is distributed freely under the GNU
GPL license. The source code is available on Github.
2. The SENTINEL-1 Toolbox (S1TBX) consists of a collection of processing tools,
data product readers and writers and a display and analysis application to
support the large archive of data from:
a. ESA SAR missions including :
– SENTINEL-1, ERS-1 & 2 and ENVISAT, as well as
b. Third Party SAR data :
– ALOS PALSAR, TerraSAR-X, COSMO-SkyMed and RADARSAT-2.
c. The various processing tools could be run independently from the
command-line and also integrated within the graphical user interface.
d. The Toolbox includes tools for calibration, speckle filtering,
coregistration, orthorectification, mosaicking, data conversion,
polarimetry and interferometry.
3. The SENTINEL-1 Toolbox is being developed for ESA by Array in partnership
with DLR, Brockmann Consult and OceanDataLab.
Description

21. 1. Shares the core architecture with the ESA BEAM and NEST toolbox;
2. Java implementation
3. Display and Analysis Tool (DAT): integrated graphical user-friendly
interface;
4. Graph Processing Framework (GPF): for creating user-defined processing
chains;
5. Graphical or command-line execution;
6. Tiled memory management for working with very large data products;
7. Data abstraction models to handle all SAR missions in a common way
8. Modular design for easy modifications and upgrades
9. Users are able to add their own modules via APIs
10.Multithreading and Multi-core processor support
Architecture Highlights

22. S1TBX Architecture: Data flow

23. Release of Sentinel Toolboxes
Sentinel Toolboxes
• Multi-mission Scientific Toolboxes
• Developed as open source software
• Common architecture (SNAP)
• Java/Python API and Java/Python plugins
• Portable engine to a Cloud infrastructure
Download https://sentinel.esa.int
Toolbox Service
Developer
Toolbox Engine
Middle
ware
Processor
Plug-in A
Plug-in B
Plug-in C
Executable ZPlug-in C
Plug-in A
Plug-in B
Plug-in C
Plug-in C
Executable X
Executable Y
Executable Z
Visualisation
User Interface
Analysis
User Interface
Processing
User Interface
Data I/O
User Interface
Toolbox Common Library and Extensibility Framework
Wrapper API(s)
Stand-alone Tool
Adapter
Native API
new processors
(= new WPS capabilities)
are deployed in the cloud
with the Toolbox Engine
as platform.
Web
Browser
Mobile
Device
User’s
Toolbox
service
call
Data Archives
cloud
service
interfaces
WPS
Toolbox “Server Configuration”
STEP
• EO science collaborative platform
• Technical forum and community animation
• Gathering user feedback and usage
• Communicating on results
• Cloud demonstrator (e.g. Land Training 2015)
Website http://step.esa.int
SentiNel Application Platform

24. Display and analysis tool: DAT

25. Graph Processing Tools

26. Readers

27. Utilities

28. SAR Tools
1. Absolute calibration (Sentinel-1A, Envisat ASAR, ERS 1&2,
ALOS, Radarsat-2, TerraSAR-X, Cosmo-skymed )
2. Coregistration of detected and complex products
3. Terrain Correction
4. Multilooking
5. Speckle filtering
6. Debursting of Sentinel IW SLC and ASAR WSS products
7. TOPSAR support
a. Split products in subswaths
b. Assembling consecutive products (same orbit and pass)
c. TOPSAR coregistration and interferogram formation

29. Example: Sentinel-1A data processing

30. Sentinel-1A data processing
1. Example 1:
a. TOPSAR Interferogram over The Netherlands
b. Georeferencing of Sentinel-1A derived products
2. Example 2:
a. Assembling of several Sentinel-1 slices (consecutive
products belonging to same orbit acquired during the same
pass)
b. Calibration
c. Thermal Noise Removal
d. Terrain correction
e. Subset
ESA UNCLASSIFIED – For Internal Use

31. Example 1
1. Using RSS processing on-demand:
a. Submission of a TOPS Interferogram
formation in G-POD using Sentinel-
1A IW SLC products over The
Netherlands
2. Using the RSS CloudToolbox:
a. Download of resulting interferogram
products and
– post-processing on RSS
CloudToolbox using the SNAP/
Sentinel-1 Toolbox.
– Visualization of results
TOPSAR Interferogram

32. TOPSAR Interferogram

33. 1. Batch processing using Sentinel-1 Toolbox with xml graph (see figure).
TOPSAR Interferogram

34. S1TBX GUI on the RSS Cloud Toolbox

35. Georeferencing of Sentinel-1A
derived products

36. Georeferencing of Sentinel-1A
derived products

37. Sentinel-1A derived data
visualization

38. Sentinel-1A derived data
visualization

39. Example 2
1. By using the graph showed below, which has been created using the
Graph Builder module of SNAP/S1TBX, we obtain as output a subset
of two consecutive Sentinel products assembled, calibrated, thermal
noise free and terrain-corrected
Customized processing chain

40. Example 2

41. Example 2

42. Example 2

43. Example 2

44. Example 2

45. Example 2

46. RSS contacts
For general information:
RSS Website: rssportal.esa.int
Join&Share: wiki.services.eoportal.org
For feedback or enquiries:
RSS Team: rss_team@esa.int
G-POD Team: eo-gpod@esa.int