1.
State of JAI
Java Image Processing for GeoSpatial Community

2.
Java Advanced Imaging

3.
What is Java Advanced Imaging?
● Image Processing Library for Java Applications
○ a “pure java” solution for image processing
● Tiling computational model
○ Images accessed tile-by-tile
○ Centrally-maintained cache for performance
○ Facilitates multi-threaded processing
● Deferred execution
○ Construct a chain of operations
○ Only compute as needed
● Threaded computation
● Object-oriented extensibility
○ Register your own operations
○ Or override the built-in operations
● Remote imaging
○ Evaluate operator chains on remote hosts
○ Transfer images tile by tile
● Border extension
● Image warping
● Pixel interpolation
● Graphics2D Drawing
● Regions of Interest (ROI)
● Image file handling
○ Using JAI ImageIO project

4.
Demo!

5.
What else was cool in 1999

6.
How JAI has been used for GeoSpatial

7.
JAI-EXT
JAI-Ext has been developed by GeoSolutions as an internal effort to replace Oracle JAI
operation implementations (remember deferred binding?).
The result is a strong open source project with an Apache 2.0 License.
Improves on JAI in three different ways:
● Adding more features to existing operations, like support for “no data” and ROI
● Improving performance of some operations
● Developing new operations

8.
JAI-EXT Functionality
● Operations implemented supporting nodata
○ Affine*
○ BandMerge
○ Border
○ Crop
○ Lookup
○ Mosaic
○ Null
○ Rescale
○ Scale*
○ Statistic
○ Translate
○ Warp*
○ ZonalStatistics
* Support nearest-neighbor, bilinear, bicubic interpolation
● ConcurrentCache
○ Replacement JAI TileCache
○ Offering better concurrency
○ Uses Guava cache internally
● And more
○ Public roadmap

9.
JAI-Tools
Project by Michael Bedward with a number of great ideas:
● Contour
● Image generation
● Kernel stats
● Rangelookup
● Regionalize
● Vectorize
● Zonal Stats
Project is no longer active, functionality migrating as needed to JAI-EXT.

10.
Jiffle
Map algebra language by Michael Bedward:
● provides a “raster calculator” language
for dynamic processing of bands
● Integrated as a JAI Operation
Project has migrated to JAI-EXT and it is
included starting version 1.1.0

11.
Jiffle Example
nir = src[7];
vir = src[3];
dest = (nir-vir)/(nir+vir);
Sentinel 2 dataset, 13
bands
Jiffle script On the fly NDVI index display

12.
GeoTools ImageWorker Helper Class
Utility builder helping to run operations on images
Hides some of the complexity, optimizes repeated operations, handles differences
between JAI and JAI-EXT (e.g., can you talk about no-data, or not?)
ImageWorker worker = new ImageWorker(image);
ImageWorker = worker.scale( percent, percent, 0,0,
bilinear).
RenderedImage result = worker.getRenderedImage();

13.
GeoTools GridCoverage
AbstractGridFormat format = GridFormatFinder.findFormat( file );
Hints hints = null;
if (format instanceof GeoTiffFormat) {
hints = new Hints(Hints.FORCE_LONGITUDE_FIRST_AXIS_ORDER, Boolean.TRUE);
}
GridCoverage2DReader gridReader = format.getReader(file, hints);
GridCoverage2D gridCoverage = gridReader.read(null);
AWT RenderedImage
Grid2World (AffineTransform)
CoordinateReferenceSystem
GridCoverage
The pixels
From pixel coords to real world coords
The “meaning” of the real world coords

14.
GeoTools CoverageProcessor
CoverageProcessor processor = CoverageProcessor.getInstance();
final ParameterValueGroup param =
processor.getOperation("CoverageCrop").getParameters();
param.parameter("Source").setValue(gridCoverage);
param.parameter("Envelope").setValue(envelope);
GridCoverage2D result = processor.doOperation(param);
AWT RenderedImage
Grid2World (AffineTransform)
CoordinateReferenceSystem
AWT RenderedImage
Grid2World (AffineTransform)
CoordinateReferenceSystem
CROP

15.
GeoTools Operations
Equivalent of ImageWorker, but helps applying operations at the coverage (georeferenced) level
Operations ops = new Operations(null);
double scale = 0.5; // 50%
GridCoverage scaledCoverage = ops.scale(coverage, scale, scale, 0, 0);

16.
JAI Licensing and Distribution
Sun Distribution License
● Free to Distribute
● Not open source!
Java Research License
● Access to source code
for research purposes only!
Distributed as “standard extension” to Java
● Additional install into JRE
● An early attempt at Java “modules”
Optional “native” acceleration mediaLib binaries
Java Runtime Environment
Core system classes
- java.lang.*
- java.io.*
- java.awt.*
- ...
Extension
javax.media.j3d.*
Extension
javax.media.jai.*

17.
Making Peace with Open Source

18.
Two problems with JAI
● The Binary License / Research License is not open source
○ Upset projects like OSGeo Live that would like to be purely open source
○ Blocker for uDig graduating from LocationTech
● The project has been abandoned by Oracle

19.
OpenCV
● Open Source Computer Vision Library
○ Roughly comparable to JAI
○ BSD license
○ Memory bound? Does it have ability to work with images larger than memory?
○ C/C++ codebase with bindings for Java
■ Some overhead taking data across JNI divide
■ Intended use is to set up a processing chain and visualize the result
● Features
○ Can use OpenCL to share workload between CPU/GPU

20.
ImageJ
● “Image Processing and Analysis in Java”
○ Much better comparison with JAI
○ Open source? Not really it is public-domain which causes problem…
○ Pure-java solution, runs anywhere
○ Memory bound, wants to load images into memory for processing
● Community
○ imagej.net
○ Popular in the scientific and health community
■ Started in the 70s in Pascal
■ Large base of community developed plugins
■ FIJI (Fiji is just ImageJ) offers a distribution of plugins that work
○ Author has now retired from National Institute of Mental health, but continues to develop
■ imagej.nih.gov/ij

21.
Apache Commons Imaging
● “a Pure-Java Image Library”
○ Looks to focus on great image format support
○ Open source! Apache License
○ Pure-java solution, runs anywhere
○ Memory bound!
● Community
○ No stable release available because there are lots of outstanding issues
○ Version 0.97 released in 2009 :(

22.
Emerging Requirements
● Pure-java image processing
● Extendable
○ Enjoy the ability to create custom operations
○ Opens the door to native operation (example use of OpenCV)
● no artificial limit on number of bands

23.
● Long term goal is to replace JAI? Is it in
position to do so?
● Pros:
○ Large portion of operations are rewritten
○ Key components like tile cache are rewritten
○ Integrates with JDK image support, does not
reinvent the wheel
● Cons:
○ Makes use of JAI Interfaces,
(which are still not open source)
○ Usability issues around JAI Interfaces
(programming style has changed since 1999)
JAI-EXT as an Alternative ?
It is too bad we are really stuck, if only Oracle would
make the interfaces open source we are this close to
being able to do the rest!
Oracle has made the vast majority of Java open
source, but for whatever reasons JAI missed out!

24.
● Updated approach
○ wrapper implementation at OSGeo
■ Delegate to existing JAI / JAI-Ext
○ Migrate GeoTools to RPE Interfaces
○ Provide start-up switch to change between
■ Wrapper
■ Clean room
● Cons:
○ This is an idea that requires development.
■ “I tried to look at the source code, but
apparently there is not much”
○ Will not integrate with Java native raster types
Raster Processing Engine
● Raster processing engine:
○ Modern Java API using Java 8 constructs,
literate programing style, as appropriate
○ Pure Java implementation
○ Ability to stage larger rasters as tiles in
memory and process tiles in parallel
○ Clear image processing operations, allowing
installations to use native libs to accelerate
processing if available
● Initial strategy
○ Provide new “clean room” interfaces
○ Initial implementation
■ Migrate JAI-EXT operations
○ Incubation at LocationTech
■ Benefit from strong IP practices
■ fundraising opportunities

25.
Bonn Code Sprint
Raster Processing Engine Scoping Exercise

26.
Bonn Code Sprint

27.
Bonn Sprint Objectives
● Scope the functionality required for Raster Processing Engine
○ If JAI can not be fully brought into Open Source, what do we have to replace?
● Prototype user facing interfaces
● Quickly morphed into:
○ Wait how does JAI actually work?
○ Wait what are the important interfaces

28.
We don’t write docs this way anymore

29.
Bonn Sprint: Code Archeology
“Primitive civilizations must have used this to … enumerate possible categorical values?”

30.
Relationship with AWT
Image: RenderedImage
● Interface
● Good for final result
● Tiled
● Relative to a larger canvas
○ Offset to (0, 0)
○ .getMinX()
○ .getMinY()
● Tile grid to larger canvas
○ Offset to (0,0)
○ .getTileGridXOffset()
○ .getTileGridYOffset()
● Tiles allow working with larger
than memory rasters.

31.
Relationship with AWT
Image: PlanarImage
● Abstracts over
○ Image in memory
○ Image source
○ Operation DAG
● Does not expose TileCache
● Does not expose TileScheduler
● Execution does not happen until
○ .getData(bbox)
○ .getTile(x, y)
● Exposes Direct Acyclic Graph for
deferred Operations
○ .getSources
○ .getSinks
JAI

32.
Deferred Execution
Read CropWarpSource
Encode
PNG/JPEG
Tile cache
One tile at a time: handle images larger than memory
Allows also to compute multiple tiles in parallel, one per thread
Adds the basis for distributed computation (given the right tile cache)

33.
Deferred Binding
Read OpImageOpImage
Source

34.
Deferred Binding
CropOpImageWrapOpImage
Read OpImageOpImage
Source
Encode
PNG/JPEG

35.
JAI/JAI-EXT Entanglements
How deep are the roots of this thing?
What are we going to snag when we pull on it?
● jai-ext source:
○ public abstract class WarpOpImage extends javax.media.jai.WarpOpImage
● jai-ext implements concurrent TileCache
● TileScheduler is not concurrent
● Use of TileCache and TileScheduler is of course hidden from jai-ext ops
○ Most operations are driven by RenderedImage.getData
● TileScheduler is also hidden from jai-ext
○ Although … it is not not use Java Executors.
● So it is … possible

36.
JAIRPE

37.
RPE

38.
JAI ParameterBlock Example
FileSeekableStream stream = new FileSeekableStream(args[0]);
RenderedOp image1 = JAI.create(“stream”, stream);
ParameterBlock params = new ParameterBlock();
params.addSource(image1);
params.add(2.0F); // x scale factor
params.add(2.0F); // y scale factor
params.add(0.0F); // x translate
params.add(0.0F); // y translate
params.add(Interpolation.INTERP_BILINEAR);
/* Create an operator to scale image1. */
RenderedOp image2 = JAI.create(“scale”, params, hints);

39.
FileSeekableStream stream = new FileSeekableStream(args[0]);
RenderedOp image1 = JAI.create(“stream”, stream);
RenderedOp image2 =
ScaleDescriptor.create(image1,2.0F, 2.0F, 0.0F, 0.0F,Interpolate.INTERP_BILINEAR, hints);
JAI Descriptor Example

40.
Raster Processing Engine Example
FileSeekableStream stream = new FileSeekableStream(args[0]);
Operation image1 = ImageRead.stream(fileSeekableStream)
Operation image2 = Affine.source(image1)
.scale(2.0F,2.0F)
.interpolation(Interpolation.BILINEAR).create();

41.
Interaction with Java SWT Image

42.
Audit of GeoTools and GeoServer Use

43.
But wait ...
Oracle is using JAI inside their database product!Oracle has just donated Java Enterprise Edition to
the Eclipse Foundation, forming the Jakarta project.
I wonder if Oracle would donate the JAI interfaces to
us? It would save a lot of time.
Answer is perhaps, just one more release?
Huh?
“1.10 Oracle Multimedia Architecture”
Oracle Database Online Documentation 11g Release 1 (11.1)

44.
Questions and Discussion

45.
Q: Will JAI Work with Java 11?
● Pure java implementation on the CLASSPATH should work
○ We have not tried it with the module system
● The JRE “standard extensions” system was removed in Java 9
○ Replaced by “jigsaw” module system

46.
JAI Descriptor Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
RenderedOp image1 =
JAI.create(“stream”, stream);
RenderedOp image2 = ScaleDescriptor.create(
image,
2.0F, 2.0F, 0.0F, 0.0F,
Interpolate.INTERP_BILINEAR,
null);
JAI and RPE Compared
RPE Operation Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
Operation image1 =
ImageRead.stream(fileSeekableStream);
Operation image2 =
Affine.source(image1)
.scale(2.0F,2.0F)
.interpolation(Interpolation.BILINEAR)
.create();

47.
JAI and RPE Compared
JAI ParameterBlock Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
RenderedOp image1 =
JAI.create(“stream”, stream);
ParameterBlock params = new ParameterBlock();
params.addSource(image1);
params.add(2.0F); // x scale factor
params.add(2.0F); // y scale factor
params.add(0.0F); // x translate
params.add(0.0F); // y translate
params.add(Interpolation.INTERP_BILINEAR);
RenderedOp image2 = JAI.create(“scale”, params);
RPE Operation Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
Operation image1 =
ImageRead.stream(fileSeekableStream)
Operation image2 =
Affine.source(image1)
.scale(2.0F,2.0F)
.interpolation(Interpolation.BILINEAR)
.create();

48.
References
- Java Advanced Imaging API Home Page (Oracle)
- Introduction to the Java(TM) Advanced Imaging API (SlideShare)