The document discusses efficient image processing techniques for Android, focusing on the RenderScript framework. It provides an overview of RenderScript, how to write kernels in C, and how to call them from Java. Examples are given for common image processing tasks like grayscale conversion, bloom effects, and local adjustments. Caching strategies and handling low-memory devices are also covered to ensure performance across all hardware.

1. EFFICIENT IMAGE PROCESSING
ON ANDROID
Nicolas Roard

2. EFFICIENT IMAGE PROCESSING
•
Works well on all hardware
•
Fast, ideally realtime interaction
•
Handles complex and flexible processing
•
Handles large images
•
Minimize memory usage

3. WE WANT TO HAVE OUR CAKE
AND EAT IT TOO

4. ANDROID KITKAT
PHOTO EDITOR
•
Non-destructive edits
•
Full-size images processing
•
Combine effects freely
•
Easy to use, yet powerful: grow with the user

5. NON-DESTRUCTIVE EDITS
•
Effects are modifiable or reversible without quality loss
•
Allow re-edits of processed images

6. RENDERSCRIPT (RS)
•
Cool thingy that let you do fast image processing

7. TIMELINE
3 VERSIONS, 10 MONTHS

8. 4.2 - NOVEMBER 2012
•
Color FX (9 looks)
•
11 Borders
•
Geometry: Straighten, Rotate, Crop, Mirror
•
Filters & Tools
•
Autocolor, Exposure, Vignette, Contrast, Shadows, Vibrance,
Sharpness (RS-based), Curves, Hue, Saturation, BW Filter
•
Non-destructive edits (in the editor -- save create a copy)
•
Exposed history

9. G+ EDITOR - MAY 2013
•
RenderScript implementations of Snapseed filters
•
Frames, Film, Drama, Retrolux
•
Non-destructive
•
Cloud-based (local processing only used for caching and UI
interactions)

10. 4.3 - JULY 2013
•
Move to RenderScript
•
New 16 image-based borders (ported from Snapseed, RS-based)
•
Filters & Tools
•
•
•
Highlights, Improved Vignette
Local adjustment (ported from Snapseed, RS-based)
New Tablet UI, refined UI, introduction of the state panel instead of
the history panel

11. 4.4 - SEPTEMBER 2013
•
Filters & Tools
•
Custom borders, Drawing tool, negative, posterize
•
RS filters: Graduated filter, Vignette, per channel saturation,
sharpness/structure
•
Refined UI (animations, etc.)
•
Pinch to zoom enabled (full-res zoom)
•
Re-edits enabled
•
Background save service, export, print support

12. DEMO

15. SOME ADDITIONAL INFOS
•
Phone and Tablet UI
•
Filters in C & RenderScript
•
Works on Full Size images -- largest tried was a 278MP image on a
Nexus 7 2nd gen. Limited by available RAM.
•
Nearly all of the editor is in AOSP!

16. IMAGE PROCESSING

17. PIPELINE
Original
Image

18. PIPELINE
Original
Image
Filter

19. PIPELINE
Original
Image
Filter
Processed
Image

20. IMAGE PROCESSING
•
In Java
•
In native code (JNI calls to C/C++)
•
In OpenGLES2
•
RenderScript

21. JAVA
•
Use getPixel()
•
Use getPixels()
•
Use copyPixelsToBuffer() [premultiplied]
•
GC calls. GC calls everywhere.

22. NATIVE CODE
•
Pass a Bitmap through JNI to C/C++
•
Quite fast & pretty easy to work with (pointer to the bitmap -- and
no GC!)
•
JNI / Native can be fastidious
•
Handling different CPU architectures can be an issue
•
Optimizations can be complicated
•
JNI management

23. OPENGL ES 2.0
•
Fast -- can write interactive processing
•
Hard to ensure the shaders will perform well on all devices
•
Limited in size (max texture size...)
•
Needs adhoc shaders, i.e. fixed pipelines.
•
Expensive to retrieve processed image

24. RENDERSCRIPT

25. “RENDERSCRIPT IS A FRAMEWORK FOR RUNNING
COMPUTATIONALLY INTENSIVE TASKS AT HIGH PERFORMANCE ON
ANDROID. RENDERSCRIPT IS PRIMARILY ORIENTED FOR USE WITH
DATA-PARALLEL COMPUTATION, ALTHOUGH SERIAL
COMPUTATIONALLY INTENSIVE WORKLOADS CAN BENEFIT AS WELL.”

26. •
Write “kernels” in a C99-like language with vector
extensions and useful intrinsics
•
RenderScript executes them in parallel, on the GPU or
CPU
•
Java used to manage lifetime of objects/allocations and
control of execution
•
Portability

27. RENDERSCRIPT
•
Fast -- through LLVM optimizations and Parallelization
•
Supports CPU / GPU
•
Compatibility Library
•
Easy to offload to the background
•
Pretty easy to write

28. RENDERSCRIPT
•
Cannot allocate memory from kernels, need to do it from outside
•
RenderScript can be called from Java or from Native
•
Compatibility library!

32. HOW?
•
Optimized Math library
•
Optimizations on the device
•
Easier to read & maintain (vector math library helps)

33. ALLOCATIONS
•
Bound to Scripts
•
Can be bound to SurfaceTexture (producer & consumer)
•
Can share memory between Allocation and Bitmap

34. HOW TO USE IT

35. 1. SCRIPT
#pragma version(1)
#pragma rs java_package_name(com.example.rsdemo)
uchar4 __attribute__((kernel)) color(uchar4 in) {
return in;
}

36. 2. CREATE CONTEXT
RenderScript mRS = RenderScript.create(mContext);
ScriptC_filter filter = new ScriptC_filter(mRS,
mResources, R.raw.filter);

37. 3. LOAD BITMAP
Bitmap bitmapIn = BitmapFactory.decodeResource(
getResources(),
R.drawable.monumentvalley);
Bitmap bitmapOut = bitmapIn.copy(bitmapIn.getConfig(),
true);

38. 4. CREATE ALLOCATIONS
Allocation in = Allocation.createFromBitmap(mRS, bitmapIn,
Allocation.MipmapControl.MIPMAP_NONE,
Allocation.USAGE_SCRIPT
| Allocation.USAGE_SHARED);
Allocation out = Allocation.createTyped(mRS, in.getType());

39. 5. APPLY THE SCRIPT
filter.forEach_color(in, out);
out.copyTo(bitmapOut);

41. SCRIPT INTRINSICS
ScriptIntrinsicBlur blur = ScriptIntrinsicBlur.create(mRS,
Element.U8_4(mRS));
blur.setRadius(25.f);
blur.setInput(in);
blur.forEach(in);

44. READY TO USE
•
ScriptIntrinsic3DLUT
•
ScriptIntrinsicBlend
•
ScriptIntrinsicBlur
•
ScriptIntrinsicColorMatrix
•
ScriptIntrinsicConvolve3x3
•
ScriptIntrinsicConvolve5x5
•
ScriptIntrinsicLUT
•
ScriptIntrinsicYuvToRGB
•
ScriptGroup

45. PAINT IT BLACK
(OR GRAY)

46. SCRIPT
uchar4 __attribute__((kernel)) color(uchar4 in) {
return in;
}

47. SCRIPT
uchar4 __attribute__((kernel)) grey(uchar4 in) {
in.g = in.r;
in.b = in.r;
return in;
}

48. SCRIPT
uchar4 __attribute__((kernel)) grey(uchar4 in) {
in.gb = in.r;
return in;
}

50. NDK
void Java_com_example_rsdemo_ProcessImage_processBitmap(JNIEnv* env,
jobject this, jobject bitmap,
jint width, jint height) {
unsigned char* rgb = 0;
AndroidBitmap_lockPixels(env, bitmap, (void**) &rgb);
int len = width * height * 4;
int i;
for (i = 0; i < len; i+=4) {
int red = rgb[i];
rgb[i+1] = red;
rgb[i+2] = red;
}
AndroidBitmap_unlockPixels(env, bitmap);
}

51. LOCAL EFFECT

53. SCRIPT
uchar4 __attribute__((kernel)) grey(uchar4 in) {
in.g = in.r;
in.b = in.r;
return in;
}

54. SCRIPT
uchar4 __attribute__((kernel)) grey1(uchar4 in,
uint32_t x, uint32_t y) {
in.g = in.r;
in.b = in.r;
return in;
}

55. SCRIPT
uchar4 __attribute__((kernel)) grey1(uchar4 in,
uint32_t x, uint32_t y) {
float range = (float) x / width;
uint32_t grey = (1 - range) * in.r;
in.r = grey;
in.g = grey;
in.b = grey;
return in;
}

57. SCRIPT
uchar4 __attribute__((kernel)) grey1(uchar4 in,
uint32_t x, uint32_t y) {
float range = (float) x / width;
uint32_t grey = (1 - range) * in.r;
in.r = grey;
in.g = grey;
in.b = grey;
return in;
}

58. SCRIPT
uchar4 __attribute__((kernel)) grey2(uchar4 in,
uint32_t x, uint32_t y) {
float range = (float) x / width;
uint32_t grey = (1 - range) * in.r;
in.r = (in.r * range) + grey;
in.g = (in.g * range) + grey;
in.b = (in.b * range) + grey;
return in;
}

59. SCRIPT
uchar4 __attribute__((kernel)) grey3(uchar4 in,
uint32_t x, uint32_t y) {
float range = (float) x / width;
float4 pixel = rsUnpackColor8888(in);
float grey = (1 - range) * pixel.r;
pixel.r = pixel.r * range + grey;
pixel.g = pixel.g * range + grey;
pixel.b = pixel.b * range + grey;
return rsPackColorTo8888(
clamp(pixel, 0.f, 1.0f));
}

60. SCRIPT
uchar4 __attribute__((kernel)) grey4(uchar4 in,
uint32_t x, uint32_t y) {
float range = (float) x / width;
float4 pixel = rsUnpackColor8888(in);
float grey = (1 - range) * pixel.r;
pixel.rgb = pixel.rgb * range + grey;
return rsPackColorTo8888(
clamp(pixel, 0.f, 1.0f));
}

61. EXAMPLE: BLOOM
•
Select the bright pixels
•
Blur the result
•
Add the blurred bright pixels back to the image

64. WITH JAVA
private void brightPass(int[] pixels, int width, int height) {
int threshold = (int) (brightnessThreshold * 255);
int r;
int g;
int b;
int luminance;
int[] luminanceData = new int[3 * 256];
// pre-computations
for (int i = 0; i <
luminanceData[i
luminanceData[i
luminanceData[i
}
for conversion from RGB to YCC
luminanceData.length; i += 3) {
] = (int) (i * 0.2125f);
+ 1] = (int) (i * 0.7154f);
+ 2] = (int) (i * 0.0721f);

65. int index = 0;
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++) {
int pixel = pixels[index];
// unpack the pixel's components
r = pixel >> 16 & 0xFF;
g = pixel >> 8 & 0xFF;
b = pixel
& 0xFF;
// compute the luminance
luminance = luminanceData[r * 3] + luminanceData[g * 3 + 1] +
luminanceData[b * 3 + 2];
// apply the treshold to select the brightest pixels
luminance = Math.max(0, luminance - threshold);
int sign = (int) Math.signum(luminance);
// pack the components in a single pixel
pixels[index] = 0xFF000000 | (r * sign) < < 16 |
(g * sign) << 8 | (b * sign);
index++;
}
}
}

66. WITH GL SHADER
uniform sampler2D baseImage;
uniform float brightPassThreshold;
void main(void) {
vec3 luminanceVector = vec3(0.2125, 0.7154, 0.0721);
vec4 sample = texture2D(baseImage, gl_TexCoord[0].st);
float luminance = dot(luminanceVector, sample.rgb);
luminance = max(0.0, luminance - brightPassThreshold);
sample.rgb *= sign(luminance);
sample.a = 1.0;
gl_FragColor = sample;
}

67. WITH RENDERSCRIPT
float brightPassThreshold;
uchar4 __attribute__((kernel)) brightPass(uchar4 in) {
float3 luminanceVector = { 0.2125, 0.7154, 0.0721 };
float4 pixel = rsUnpackColor8888(in);
float luminance = dot(luminanceVector, pixel.rgb);
luminance = max(0.0f, luminance - brightPassThreshold);
pixel.rgb *= sign(luminance);
pixel.a = 1.0;
return rsPackColorTo8888(clamp(pixel, 0.f, 1.0f));
}

68. JAVA-SIDE
ScriptIntrinsicBlur blur =
ScriptIntrinsicBlur.create(mRS, Element.U8_4(mRS));
ScriptIntrinsicBlend blend =
ScriptIntrinsicBlend.create(mRS, Element.U8_4(mRS));
filter.set_brightPassThreshold(0.15f);
filter.forEach_brightPass(in, out);
blur.setRadius(25.f);
blur.setInput(out);
blur.forEach(out);
blend.forEachAdd(in, out);
out.copyTo(bitmapOut);

69. WORKING WELL EVERYWHERE

70. WORKING WELL ON ALL
HARDWARE
•
Architect for the worst
•
Scale with the device capabilities
•
Screen size / dpi
•
Available memory
•
Available CPU / GPU
•
Think about what is a downgraded experience

71. LOADING
•
Load in the background
•
•
AsyncTask, or use a background thread
Bitmaps loading
•
query the size
•
inSampleSize
•
reuseBitmap
•
BitmapRegionDecoder

72. QUERY THE SIZE
BitmapFactory.Options options =
new BitmapFactory.Options();
options.inJustDecodeBounds = true;
BitmapFactory.decodeResource(
getResources(), R.id.myimage, options);
int imageHeight = options.outHeight;
int imageWidth = options.outWidth;
String imageType = options.outMimeType;

73. INSAMPLESIZE
•
Only load what you need
•
needs to be a power of two, so for a 2048x2048 image,
•
insamplesize=2
=> 1024x1024 image
•
insamplesize=4 => 512x512 image

74. CALCULATE INSAMPLESIZE
if (bounds.width() > destination.width()) {
int sampleSize = 1;
int w = bounds.width();
while (w > destination.width()) {
sampleSize *= 2;
w /= sampleSize;
}
options.inSampleSize = sampleSize;
}

75. REUSE BITMAP
BitmapFactory.Options options;
(...)
Bitmap inBitmap = ...
(...)
options.inBitmap = inBitmap;
API level 11 (Android 3.0)
Before API level 19 (Android 4.4) only same size

76. BITMAP REGIONDECODER
InputStream is = ...
BitmapRegionDecoder decoder =
BitmapRegionDecoder.newInstance(is, false);
Rect imageBounds = ...
Bitmap bitmap =
decoder.decodeRegion(imageBounds, options);
API level 11 (Android 3.0)

77. PIPELINE

78. PIPELINE
•
Run in a background service (used when saving too)
•
Mix C, RenderScript, java filtering (canvas draw)
•
multiple pipelines in parallel (direct preview, highres, icons, full res,
geometry, saving)

79. FLEXIBLE PROCESSING
•
Unbounded pipeline
•
No fixed order
•
Complex filters
•
Geometry-based
•
Global
•
Local

80. FILTER TYPES
Color Fx
Geometry
*
Borders
Color
transforms
Crop
Contrast
Image-based
Straighten
Saturation
Parametric
Rotate
Local
Mirror
Vignette

81. Vintage
Instant
Washout
X-Process
COLOR FX - 3D LUT

82. CACHING
•
Cache RS scripts, allocations
•
Cache original bitmaps
•
Aggressively destroy/recycle resources in filters to keep memory low
•
If possible, filters should process the input bitmap directly
•
N-1 cache

83. MEMORY USAGE
•
Bitmap cache heavily reusing bitmaps
•
LruCache class (available in support lib too)
•
Pick image sizes depending on the device resolution / DPI
•
Have a path ready for a downgraded experience

84. DEVICE CAPABILITIES
•
Ask the system
•
Runtime.getRuntime().maxMemory()
•
New isLowRamDevice() API
•
Handles low-memory signals
•
Handles java.lang.OutOfMemory exceptions

85. PIPELINE CACHE
Original
Image

86. PIPELINE CACHE
Original
Image
Filter
Processed
Image

87. REALISTICALLY...
Original
Image

88. REALISTICALLY...
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

89. PROCESSING
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

90. PROCESSING
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

91. PROCESSING
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

92. N-1 CACHING
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

93. N-1 CACHING
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

94. N-1 CACHING
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

95. N-1 CACHING
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

96. N-1 CACHING
Original
Image
Filter
Filter
Filter
Filter
Processed
Image

97. N-1 CACHING
•
No silver bullet
•
Only really useful when the user manipulates the last filters of the
pipeline...
•
...but this is after all the more common scenario!

98. REALTIME INTERACTION

99. REALTIME INTERACTION
•
Background processing
•
Minimize allocations -- Careful with Garbage Collection!
•
Optimized filters, RenderScript helps
•
Caching in the pipeline
•
Low/High resolution preview

100. PREVIEW SYSTEM
Bitmap
Bitmap
Bitmap
Preset
Preset
Preset
UI Thread
Processing Thread

101. PREVIEW SYSTEM
Bitmap
Bitmap
Bitmap
Preset
Preset
Preset
UI Thread
Processing Thread

102. PREVIEW
Continuous
preview
High-res
preview
Full resolution

103. PREVIEW SYSTEM
Low-res preview
Request
High-res preview
Full-res preview
If new request, delay more

104. HOW TO CHEAT
•
The triple-buffer preview can have a low resolution
•
The UI elements and controls are animated and manipulated at 60 fps
on the UI thread
•
The rendering pipeline can (and needs to) be interrupted

105. INTERRUPTION IN
RENDERSCRIPT
LaunchOptions options = new LaunchOptions();
options.setX(xStart, xEnd);
options.setY(yStart, yEnd);
mScript.forEach_vignette(in, out, options);

106. LaunchOptions options = new LaunchOptions();
boolean even = true;
int tile = 128;
int height = bitmapIn.getHeight();
int width = bitmapIn.getWidth();
for (int yStart = 0; yStart < height; yStart += tile) {
for (int xStart = 0; xStart < width; xStart += tile) {
int xEnd = xStart + tile;
int yEnd = yStart + tile;
options.setX(xStart, xEnd);
options.setY(yStart, yEnd);
if (even) {
filter.forEach_grey(in, out, options);
} else {
filter.forEach_color(in, out, options);
}
even = !even;
}
}

108. FULL RESOLUTION PREVIEW

109. FULL RESOLUTION
PREVIEW
•
Ideally, we should use a tile-based rendering
•
At the moment, we use BitmapFactory region decoder instead
•
Filters need to be able to handle partial regions
•
future payoff: streaming save

110. FUTURE

111. FUTURE -- IN THE PHOTO
EDITOR
•
Merging filters
•
•
•
RenderScript
At the pipeline level (color cubes...)
Streaming saving
•
Some code is there (AOSP), but not used

112. FUTURE -- IN ANDROID
FRAMEWORK
•
TileView
•
Adding filtering pipeline in android framework
•
Image loader improvements
•
RAW support? Color correction?

113. 13223x5598
70MP image
~60 tiles, ~15Mb

114. 13223x5598
70MP image
~60 tiles, ~15Mb

115. TILEVIEW
TileView
TileGrid
Tile
TileViewAdapter
ImageTileViewAdapter
TileCache
TestTileViewAdapter

116. TILEVIEW ADAPTER
public interface TileViewAdapter {
public int getTileSize();
public int getContentWidth();
public int getContentHeight();
public void onPaint(float scale,
float dx, float dy, Bitmap bitmap);
public Bitmap getFullImage(int max);
void setDebug(boolean debug);
}

117. QUESTIONS, SUGGESTIONS?
•
RenderScript documentation:
•
•
http://developer.android.com/guide/topics/renderscript/
compute.html
contact: nicolasroard@google.com