The document discusses hacking images for faster OpenCV processing in an iOS application. Specifically, it explores sharing memory between OpenCV's Mat class and Core Graphics contexts and images to avoid unnecessary copying. It shows how to expose the Mat data as a Swift-compatible type and create a CGContext and CGImage directly from the Mat's memory pointer. This allows operating directly on the image data without copying, simplifying operations and improving performance. In testing, a performance gain was not achieved likely due to other processing needs, but the simpler data flow was beneficial.

1. Hacking images for
Faster OpenCV
/ LINE Fukuoka(@_ha1f)

2. ● 2017’s new graduate, iOS Developer
● Current Project: LINE
● Previous Project: LINE Creators Studio
(@_ha1f)

3. LINE Creators Studio
Create & Publish LINE Stickers

4. LINE Creators Studio
Create & Publish LINE Stickers

5. As-Is
● The same path drew
● We can modify but…

6. New feature: Grabcut
Interactive foreground extraction

7. ● Estimate segmentation from input marker
● OpenCV has the implementation, so it’s
easy to integrate!
Grabcut

8. ● Estimate segmentation from input seed
● OpenCV has the implementation, so it’s
easy to integrate!
Grabcut

9. Convenience
● We’re using application instead of WEB for “convenience”
● Performance is essential
C++
● We cannot catch errors (crashes immediately)
● We cannot share types (Only Obj-C can bridge)
Difficulties

10. MatToUIImage, UIImageToMat
Copy memory every time…
https://github.com/opencv/opencv/blob/master/modules/imgcodecs/src/ios_conversions.mm

11. Grabcut flow
Original Image Input Image
Input Mask
Grabcut
Initial Path
Marker
convert

12. Grabcut flow
Mask Image
Original Image Input Image
Input Mask
Grabcut
Initial Path
Marker
We have to copy every
time we add markers
convert
convert

13. Hacking images
What is “image”?

14. var bits = [UInt8](repeating: 0, count: width * height * 4)
let context = CGContext(
data: &bits,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: 8 * width,
space: CGColorSpaceCreateDeviceRGB(),
bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue)
image.draw()
print(bits)
We can read as [UInt8], without copying
Hacking CGContext

15. Mat(
int _rows,
int _cols,
int _type,
void* data
)
Needless to say, it uses pointer
Hacking cv::Mat

16. Similar format.

17. Why not share the memory?

18. @property cv::Mat rawImage;
- (unsigned char *)data {
return self.rawImage.data;
}
- (size_t)bufferSize {
return self.rawImage.total() * self.rawImage.elemSize();
}
Expose as Swift-compatible types
In Objective-C++

19. CGContext(
data: image.data,
width: image.cols(),
height: image.rows(),
bitsPerComponent: image.elemSize1() * 8,
bytesPerRow: image.step[0],
space: CGColorSpaceCreateDeviceGray(),
bitmapInfo: CGImageAlphaInfo.none.rawValue
)
※depends on color mode, this is case for Mask
Create CGContext

20. let bufferSize = context.bytesPerRow * context.height
let dataProvider = CGDataProvider(data: Data(bytesNoCopy: data, count: bufferSize, deallocator: .none) as
CFData)!
let maskImage = CGImage(maskWidth: context.width,
height: context.height,
bitsPerComponent: context.bitsPerComponent,
bitsPerPixel: context.bitsPerPixel,
bytesPerRow: context.bytesPerRow,
provider: dataProvider,
decode: nil,
shouldInterpolate: false)!
※depends on color mode, this is case for Mask
Read from CGContext as CGImage

21. We can share memory between
cv::Mat,
CGContext,
CGImage
Conclusion

22. ● We can clone CGImage using
`pointer.copyMemory(from:byteCount:)`
● We can reset mask using `assign(repeating:count:)`
Easy undo/redo/initialize
Simpler image operations

23. ● Performance was not so improved…
● We had to build masks to show
● We had to copy to push undo stack
● Simple & Clear data-flow made us happy!
Result

24. THANK YOU