The presentation will cover GeoSolutions experience in setting up GeoServer based production systems providing access to earth observation products, with indications of technical challenges, solutions, and deployment suggestion. The presentations will cover topics such as setting up a single unified mosaic from all the available data sources, tailoring access to it to different users, determining the most appropriate stacking order, dealing with multiresolution, different coordinate systems, multiband data, SAR integration, searching for the most appropriate products using a mix of WFS, CSW and so on, serving imagery with high performance WMS and WMTS, performing small and large data extractions with WCS and WPS, closing up with deployment examples and suggestions.
Serving earth observation data with GeoServer: addressing real world requirements - FOSS4G EU 2017
1. Serving EO Data
with GeoServer
Addressing real-world EO requirements
With GeoServer
Ing. Simone Giannecchini
Ing. Daniele Romagnoli
Ing. Andrea Aime
GeoSolutions
2. GeoSolutions
Founded in Italy in late 2006
Expertise
• Image Processing, GeoSpatial Data Fusion
• Java, Java Enterprise, C++, Python
• JPEG2000, JPIP, Advanced 2D visualization
Supporting/Developing FOSS4G projects
GeoServer, MapStore
GeoNetwork, GeoNode, Ckan
Clients
Public Agencies
Private Companies
http://www.geo-solutions.it
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
4. ImageMosaic – the basics
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
5. Terminology
Granule/Tile
The individual raster element composing the
mosaic
(Granule) Index
The collection of metadata records describing
the location, spatial coverage and other
attributes of each single granule
Dimensions/Domains
The dimensions besides the spatial ones used
to distinguish individual granules
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
6. Complete freedom
Granules can:
Overlap as they please, can have different
resolutions
Be in different file formats (faster if the format is
uniform)
Have different color models, RGB, gray, paletted
(needs JAI-EXT enabled)
Be in different coordinate reference systems
(needs GeoServer 2.12 to work best)
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
7. The mosaic index
Always present
A table of granules, their footprint, location,
and extra attributes
Used to look up which granules to mosaick for
a given request
Currently supported/tested sources
PostGIS (JNDI)
Oracle (JNDI) it’s been a nightmare because of
naming!
Shapefile (no attr indexing, no timestamp)
H2 (local embedded db)
Can be customized to support custom
granule indexes (e.g. legacy catalog)
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
8. The Dimensions
Maps to alphanumeric attributes in the index
TIME and ELEVATION receive special
treatment for WMS and WCS
Custom/Additional dimensions
Everything besides TIME & ELEVATION
They can be dynamically discovered
from GetCapabilities/DescribeCoverage
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
9. Filling the index
Available options
Have GeoServer do it for you from the UI
(one shot setup)
Use the REST API to add granules and
make it populate the index
Directly write the index (common for
moving window of data)
Use the OpenSearch for EO extension
(later in the presentation)
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
10. Grabbing attributes from file names
elevationregex.properties file
regex=(?<=_)(d{3})(?=_)
elevationregex.properties file
regex=[0-9]{8}T[0-9]{9}Z(?!.*[0-9]{8}T[0-9]{9}Z.*)
NCOM_wattemp_020_20081031T0000000_12.tiff
While building the index GeoServer can
fetch attributes from file name
Or from within the file if needed
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
11. REST configuration of mosaic contents
Granule Index CRUD Operations via REST
Create mosaic
Read its index schema
Read its index contents
Update/Delete/Insert granules
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
GS
Pixels Index
Client
12. Documentation links
GeoServer reference docs:
http://docs.geoserver.org/stable/en/user/data/raster/image
mosaic/index.html
http://docs.geoserver.org/stable/en/user/data/raster/image
mosaic/tutorial.html
http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y
aml
GeoSolutions training:
http://geoserver.geo-
solutions.it/edu/en/multidim/imagemosaic/index.html
http://geoserver.geo-
solutions.it/edu/en/multidim/accessing_multidim/index.html
http://geoserver.geo-
solutions.it/edu/en/multidim/rest/index.html
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
13. ImageMosaic – the fun stuff
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
14. Filtering and sorting
The mosaic index can be filtered upon with CQL
Can be sorted on using WFS like syntax
More possibilities than just Time and Elevation
SENSOR = SAR AND SATELLITE=XYZ
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
SORTBY = TIME D
15. Masking Support
Mask allow finding valid data
Alternative to NODATA for lossy compressions
Vector and raster masks supported
Raster masks can be embedded (GDAL style)
Vector ones can be WKB or shapefiles
API to plug your own mask provider
Available for image mosaic and GDAL based data
sources
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
16. Coverage Views
NetCDF/Grib/Mosaic: one reader, many coverages
Sometimes the coverages are strictly related
E.g., U and V components of wind/current vectors
Merge them back into a single coverage as bands
Allows rendering engine to work with them both as one
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
17. Pluggability
Lots of extension points and pluggable bits
PropertyCollectors: extract index attributes from
granules
CoverageNameCollector: when you have a hash of files
from multiple coverages
GranuleHandler: pre-process granules before getting
them in the mosaic
GranuleAcceptor: decide if the granule is actually part
of the mosaic
SubMosaicProducer: custom way to mosaic granules
GranuleCatalog: how you store the index
See more examples at the end of presentation
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
19. NetCDF format support
NetCDF support
Support COARDS/CF* conventions loosely
Expose NetCDF internal data as a set of 2D slices
Fast 2D (time, elevation) slice extraction
Supports NCML
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
20. One or more variables
Polyphemus Sample Dataset
1 File Multiple Coverages!
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
21. NetCDF GridMapping to EPSG codes
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
22. NetCDF WCS output
WCS can output NetCDF for any multidimensional source
Several customizations to control output and preserve info
if the source is a NetCDF itself
Source has to be multidimensional (current restriction)
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
27. GRIB/GRIB2 Format Support
Based on same UCAR NetCDF Java libs used
by NetCDF Format
Same capabilities of NetCDF Format
Same indexing logic
Same ImageMosaic support
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
28. Documentation links
GeoSolutions training:
http://geoserver.geo-
solutions.it/edu/en/multidim/mosaic_config/netc
df_mosaic.html
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
29. The services - visualize
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
30. WMS-T support
TIME, ELEVATION & Custom dimensions
http://localhost:8080/geoserver/geosolutions/wms?...
&time=2013-03-1T00:00:00.000Z
&elevation=35.0
&DIM_FILEDATE=2013-03-01T00:00:00.000Z
&DIM_UPDATED=2013-04-08T08:18:41.597Z
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
32. WMS – custom filtering and sorting
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
&CQL_FILTER=platform=SENTINEL2
&sortBy=time D &sortBy=time D
33. WMS-EO
Extending LayerGroup
concept
Support same style on
both raster and vector
data
Support custom
dimensions
Alter map on the fly to
support band
combination
Wizard to configure EO
layer groups
Product layer tree
Mix of vector and raster info
Raw data, flags (clouds,
snow), browse
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
34. WMS-EO
Helper UI to build the tree
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
35. Rendering transformations
SLD Based transformations
On-the-fly contouring
On-the-fly poligonalization
Current arrows
Wind Barbs
Pluggable: add your own
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
37. The services - download
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
38. WCS 2.0
Basics
Core service
KVP binding
XML binding
Common extensions
CRS
Scaling
Interpolation
Range subsetting
GeoTiff
GML
NetCDF
Add the output format extensions
GeoTIFF
GML Grid
NetCDF
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
39. WCS 2.0 processing chain
Crop
RangeSubset
Scale & Interpolate
Reproject & Interpolate
Encode
Core
Extension
Extension
Extension
Extension
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
40. WCS 2.0 DescribeCoverage
4D BBOX
TIME details
as extension
ELEVATION
details as
extension
CUSTOM
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
41. WCS – custom filtering and sorting
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
&CQL_FILTER=platform=SENTINEL2
&sortBy=time D &sortBy=time D
43. WCS EO
Add support for WCS EO metadata in readers
Associate each file with EO metadata
Include such information in
DescribeCoverage/DescribeEODataset
EODataset: multidimensional mosaic or NetCDF file
Describe the inner structure (granules listing) in
DescribeEODataSet
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
45. Putting it all together (and finding it)
Use case
Lots of satellite imagery
Different satellites
Different sensor
Deep time and space
distribution
What to do about it?
“OpenSearch for EO“ to
the rescue
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
OpenSearch
OGC OpenSearch Geo and
Time
OGC OpenSearch for EO
46. OpenSearch for EO
Geo and Time: Extension to OpenSearch
adding search keywords for time and space
Earth Observation: extension to the above
adding search keywords for and Earth
Observation properties
E.g.
Cloud cover
Snow cover
Off nadir
…
Simpler alternative to CSW with ebRIM
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
47. General information schema
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
Collection
Product
Granule
Search
Search
Mosaic/OGC
48. Two steps search
First find collection of
similar products (by
satellite/sensor), e.g.
Optical or SAR?
Supported resolutions
Availability of data in
particular timeframe
Then search products in it,
by
Time
Space
Cloud cover
…
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
49. Simple search
http://cloudsdi.
geo-solutions.it/
geoserver/oseo/search?
parentId=SENTINEL2
&cloudCover=30]
Output is RSS with
description,
thumb/quicklook and
links
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
50. Metadata and cross linking
Links to metadata:
ISO metadata for collection
GML O&M for products
Links to OGC services (cross linking)
WMS/WMTS to see collection/product
WFS to get “masks” (validity, sea, cloud, snow,
…)
WCS to extract raster
Direct download to get original package
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
51. Tight integration with mosaic possible
The system creates one “feature type” per
collection
Can be used as the index to a mosaic
One mosaic per collection
Filter using CQL and sort by attribute on the
WMS/WCS service
Using all the attributes already searchable
by OpenSearch
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
52. REST API to manage OpenSearch
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
Swagger documentation
53. Real World Use cases
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
54. Use Case 1 – Light Integration
GeoServer + MapStore + Legacy Ingestion
Maritime Data
SAR Time Series
Oil Spill + other vectors
Single Mosaic + CQL + Custom Access
Manager
Integrated SSO with MapStore
Precooked Download of Scenes
Fine grain control over data access
Download
Visualization
Clever usage of Mosaic Indexes via WFS
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
55. Use Case 1 – Light Integration
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
56. Use Case 1 – Light Integration
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
57. Use Case 1 – Light Integration
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
58. Use Case 2 – Deep Integration
High Res Imagery
SSO with CAS
Legacy Ingestion
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
59. Use Case 2
Database of satellite imagery, one satellite collect
(strip) materialized as many files (granules)
Access filtering based on current user roles (custom
data store implementation)
Custom stacking order (SORT BY) also user based if
not already provided in the request
Mosaic
POSTGIS
Custom store wrapper
Postgis store
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017
60. Use Case 3 – Let’s go Deeper!
XXX
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017