GRASS GIS as high-performance remote sensing toolbox
GRASS GIS has contained remote sensing tools for decades, catering to the needs of users since the first generations of Landsat satellites. In this talk we will present the current efforts of integrating modern data sources and modern approaches into GRASS GIS. Tools exist for pre-processing recent very high resolution images, object-based image analysis (OBIA), Lidar data handling, etc. At the same time, efforts have gone into ensuring the scalability of tools for huge data sets. The presentation will provide a short brief on the general state of GRASS GIS development, go on to an overview of the remote sensing tools available, to end with a use case on how to use GRASS GIS for time series processing in a high performance cluster/grid computing environment. GRASS GIS has been existing for over 30 years by now and provides a very large and diverse set of state-of-the-art tools for the analysis of spatial data. Less known by many, remote sensing tools have been part of it almost from the beginning. GRASS GIS provides a series of imagery analysis tools for pre-processing (radiometric correction, cloud detection, pansharpening, etc), creating derived indices (vegetation indices, texture analysis, principal components, fourier transform, etc), classifying (management of training zones, different classifiers, validation tools), and producing other derived products such as evapotranspiration and energy balance models. Next to these tools for satellite images, other tools exist for the handling of aerial photography for creation of orthophotos, and for the import and analysis of Lidar data. In addition to these tools, efforts have gone into integrating current state-of-the-art methods such as object-based image analysis and machine learning. A complete toolchain exists to segment images using different algorithms, to create superpixels, to collect statistics characterizing the resulting objects, and to apply machine learning algorithms for classification. New modules also include unsupervised segmentation parameter optimization and active learning. Options for pixel-based classification have also been enlarged to a host of machine learning algorithms. A specific aspect of treating the rapidly increasing amounts of satellite data is the scalability of tools. Through its modular structure, GRASS GIS also allows to easily parallelize certain operations, thus opening the door to the use of cluster/grid computing environments. Speakers - Markus Neteler - Moritz Lennert - Markus Metz Source + video registration: https://fosdem.org/2018/schedule/event/geo_grass
Recomendados
Recomendados
Mais conteúdo relacionado
Mais de Markus Neteler
Mais de Markus Neteler (11)
Último
Último (20)
GRASS GIS as high-performance remote sensing toolbox
- 1. GRASS GIS in the sky GRASS GIS as high- performance remote sensing toolbox Markus Neteler, Markus Metz, Moritz Lennert https://grass.osgeo.org/
- 2. FOSDEM 2018 GRASS GIS in the sky 2/20 GRASS GIS Intro r3.flow Skyview factor from LiDAR g.gui.tplot ● Geographic Resources Analysis Support System ● Open Source GIS ● developed since 1984, since 1999 GNU GPL ● Portable code (many operating systems, 32/64bit) ● Your GIS backbone – linkable to:
- 3. FOSDEM 2018 GRASS GIS in the sky 3/20 GRASS GIS and Python Using GRASS GIS from “outside” through “grass-session” pip install git+https://github.com/zarch/grass-session.git # in future, the stable "grass-session" release # will be available with: # pip install grass-session Finally easy use of GRASS GIS as a processing backend in Python! Combine with GDAL, OTB, ...
- 4. FOSDEM 2018 GRASS GIS in the sky 4/20 GRASS GIS 7.4 Release ● New: Get demo data at start screen ● GUI: data catalog improved ● Most of ortho-rectification brought back ● r.in.gdal + r.external: provide support for import of raster maps exceeding 90N or 90S or with an EW extent larger than 360 degrees ● r.out.gdal: possibility to create overviews which enhances the compatibility with other GIS software packages ● v.clip added for easy vector clipping ● … (480 fixes and improvements with respect to 7.2.0) https://trac.osgeo.org/grass/wiki/Grass7/NewFeatures74
- 5. FOSDEM 2018 GRASS GIS in the sky 5/20 GRASS GIS 7.4 Release ● Atmospheric correction updated with new satellites ● MODIS product processing made easy
- 6. FOSDEM 2018 GRASS GIS in the sky 6/20 Remote sensing in GRASS GIS : a long history ● Imagery submodule already available for GRASS 1.1 in 1986 ● i.* modules fully integrated in version 3.0 in 1988 ● Steady improvement and additions ever since ● From simple text terminal to X- monitors to a modern GUI ● Memory efficient modules
- 7. FOSDEM 2018 GRASS GIS in the sky 7/20 Preprocessing i.landsat.toar i.pansharpen i.atcorr i.landsat.acca i.topo.corr Classification i.maxlik i.smapr.learn.ml r.kappa g.gui.iclass ● Pixel-based tools for satellite and aerial imagery ● Most state-of-the-art methods implemented ● Complete toolchain from pre- processing to classification ● Many highly specialized tools Transformation, edge detection, index extraction i.pca i.fft i.cca i.oifi.tasscap i.wavelet i.zci.edge i.cva Remote sensing in GRASS GIS : pixel-based techniques Specialized modules i.evapo.* i.biomassi.eb.* i.albedo i.gravityi.feotio2 i.water i.lswt
- 8. FOSDEM 2018 GRASS GIS in the sky 8/20 ● Complete toolchain from segmentation to classification ● Including – unsupervised segmentation parameter optimization – high performance object statistics calculation – module-level parallelization ● Recently created module for SLIC superpixel creation source : http://dx.doi.org/10.3390/rs9040358 Remote sensing in GRASS GIS : object-based image analysis
- 9. FOSDEM 2018 GRASS GIS in the sky 9/20 ● Suite of LiDAR data tools ● Suite for creation of orthophotos ● Current developments : – convolutional neural networks – cutlines for semantically sensitive tiling – etc, etc ● Constantly growing list of extensions ● Permanent work on performance improvements Remote sensing in GRASS GIS : plus so much more !
- 10. FOSDEM 2018 GRASS GIS in the sky 10/20 High-performance computing Example: NDVI time series Harmonic Analysis of Time Series (HANTS)
- 11. FOSDEM 2018 GRASS GIS in the sky 11/20 High-performance computing Components – master with job/queue manager – compute nodes disk space per CPU core RAM per CPU core ideally one SSD per node master (compute node) General HPC layout compute node compute node compute node Parallelization : several GRASS commands running at the same time
- 12. FOSDEM 2018 GRASS GIS in the sky 12/20 High-performance computing Chunks for parallel processing Temporal processing ● spatial chunks need to be mosaiked at the end or ● temporal chunks need overlap t 1 2 3 4 2 1 overlap
- 13. FOSDEM 2018 GRASS GIS in the sky 13/20 High-performance computing Spatial chunks computational region ● North, South, West, East ● rows, columns pre-defined regions, one for each chunk (tile) Alternative ● create X tiles from one raster map (r.tile) 1 2 3 4
- 14. FOSDEM 2018 GRASS GIS in the sky 14/20 High-performance computing Chunks for parallel processing Spatial processing ● each time step as one chunk or ● spatial chunks not recommended → spatial discontinuities t
- 15. FOSDEM 2018 GRASS GIS in the sky 15/20 High-performance computing GRASS installation setup – environmental variables – paths GRASS session setup – variable GISRC for rc file ● GRASS database ● location ● mapset script 1 GRASS commands script 2 1. create unique GISRC, unique mapset 2. run script 1 3. copy results 4. delete GISRC, mapset script 3 job manager settings run script 2 always check return codes
- 16. FOSDEM 2018 GRASS GIS in the sky 16/20 High-performance computing temporary GRASS GIS session GISRC: name of file with GRASS variables GISRC contents: GISDBASE: name LOCATION_NAME: name MAPSET: name LOCATION_NAME: sub-directory of GISDBASE MAPSET: sub-directory of LOCATION_NAME export GISRC=/path/to/tmpgisrc Ideally all on a SSD scratch disk script 2 Arguments : first and last time step 1. create temporary GISRC, mapset with unique names, using a pre-defined mapset template 2. run script 1 in temporary mapset 3. copy results, one for each time step, from temporary mapset to final mapset 4. delete temporary GISRC, mapset rm -rf $GISRC /path/to/temp_mapset directories http://grass.osgeo.org/wiki
- 17. FOSDEM 2018 GRASS GIS in the sky 17/20 High-performance computing job/queue manager – Select / create a queue – submit a job to a queue – start a job when hardware resources are available – redirect stdout and stderr to unique files job with task(s), here task = script 2 job 1 : running job 2 : running job 3 : running job 4 : running job 5 : waiting job 6 : waiting job 7 : waiting job 8 : waiting waiting : no hardware resources available
- 18. FOSDEM 2018 GRASS GIS in the sky 18/20 High-performance computing Collect results – Copy results to one common GRASS mapset → this is the I/O bottleneck try nice / ionice cluster file systems job 1 results job 2 job 3 job 4 job 5 job 6 job 7 job 8job 9job 10job 11 job 12 job 13 job 14
- 19. FOSDEM 2018 GRASS GIS in the sky 19/20 High-performance computing MODIS Land Surface Temperature temporal + spatial processing
- 20. FOSDEM 2018 GRASS GIS in the sky 20/20 High-performance computing Most important Have a good admin that fixes the system after you broke it Neteler M, Metz M, 2011 – 2018, pers. com.