More Related Content Similar to Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python (20) More from DataWorks Summit (20) Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python1. 1 © Hortonworks Inc. 2011–2018. All rights reserved.
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Open Source Computer Vision with TensorFlow, MiniFi,
Apache NiFi, OpenCV, Apache Tika and Python
Timothy Spann, Solutions Engineer
Hortonworks @PaaSDev
Vision Thing
2. 2 © Hortonworks Inc. 2011–2018. All rights reserved.
Disclaimer
• This document may contain product features and technology directions that are under
development, may be under development in the future or may ultimately not be
developed.
• Technical feasibility, market demand, user feedback, and the Apache Software
Foundation community development process can all effect timing and final delivery.
• This document’s description of these features and technology directions does not
represent a contractual commitment, promise or obligation from Hortonworks to deliver
these features in any generally available product.
• Product features and technology directions are subject to change, and must not be
included in contracts, purchase orders, or sales agreements of any kind.
• Since this document contains an outline of general product development plans,
customers should not rely upon it when making a purchase decision.
3. 3 © Hortonworks Inc. 2011–2018. All rights reserved.
What?
• For processing images from IoT devices like Raspberry Pis and NVidia Jetson TX1 and
more that are equipped with attached cameras or external USB webcams, we use
Python to interface via OpenCV and PiCamera. From there we run image processing at
the edge on these IoT device using OpenCV and TensorFlow to determine attributes and
image analytics. Apache MiniFi coordinates running these Python scripts and decides
when and what to send from that analysis and the image to a remote Apache NiFi server
for additional processing.
• At the Apache NiFi cluster, in the cluster it routes the images to one processing path and
the JSON encoded metadata to another flow. The JSON data (with its schema referenced
from a central Schema Registry) is routed using Record Processing and SQL. This data in
enriched and augmented before conversion to Apache ORC to be stored in HDFS with
Apache Hive tables.
4. 4 © Hortonworks Inc. 2011–2018. All rights reserved.
Agenda
• OpenCV
• TensorFlow
• Apache Tika
• Apache NiFi and MiniFi
• Demos
• Questions
5. 5 © Hortonworks Inc. 2011–2018. All rights reserved.
So Why Am I Ingesting Images From Edge Devices?
Use Cases
Object Recognition
• WebCam Security
• Anomaly Detection
• Logging
• Metadata about images
• Customer Analysis
Image Classification Motion Estimation
• Movement tracking
• Security
• Occupied Room
Active Archive
• Store all images
• Training datasets
• Joining With Other Data
• Cameras Everywhere
7. 7 © Hortonworks Inc. 2011–2018. All rights reserved.
Ingestion
Simple Event Processing
Engine
Stream Processing
Destination
Data Bus
Build
Predictive Model
From Historical Data
Deploy
Predictive Model
For Real-time Insights
Perishable Insights
Historical Insights
Open Computer Vision Flow
8. 8 © Hortonworks Inc. 2011–2018. All rights reserved.
Open Source Image Analytical Components
Streaming Analytics
Manager
Image Ingest
Distributed queue
Buffering
Process decoupling
Routing and Pre-Processing
Orchestration
Queueing
Simple Event Processing
Image Capture
Image Processing
9. 9 © Hortonworks Inc. 2011–2018. All rights reserved.
Streaming Analytics
Manager
Part of MiniFi C++ Agent
Detect metadata and data
Extract metadata and data
Content Analysis
Deep Learning Framework
Open Source Image Analytical Components
Enabling Record Processing
Schema Management
10. 11 © Hortonworks Inc. 2011–2018. All rights reserved.
Aggregate all data from sensors, geo-location devices, machines and social
feeds
Collect: Bring Together
Mediate point-to-point and bi-directional data flows, delivering data
reliably to Apache HBase, Apache Hive, Slack and Email.
Conduct: Mediate the Data Flow
Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather,
location, and TensorFlow.
Curate: Gain Insights
11. 12 © Hortonworks Inc. 2011–2018. All rights reserved.
{
"imagefilename" : "/opt/demo/images/2018-04-
17_1127.jpg",
"yaw" : 100.0,
"host" : "sensehatmovidius",
"top3" : "n06874185 traffic light, traffic signal,
stoplight",
"top5" : "n03773504 missile",
"humidity" : 31.2,
"uuid" : "uuid_json_20180417152727.json",
"ipaddress" : 192.168.1.104,
"top2" : "n04286575 spotlight, spot",
"top3pct" : "6.199999898672104",
"top2pct" : "10.199999809265137",
"cputemp2" : 56.92,
"z" : 1.0,
"diskfree" : "4152.5 MB",
"top1pct" : "13.79999965429306",
"currenttime" : "2018-04-17 15:27:37",
"label2" : "n04592741 wing",
"pitch" : 360.0,
"pressure" : 1026.2,
"roll" : 1.0,
"label1" : "n04286575 spotlight, spot",
"top5pct" : "4.30000014603138",
"label4" : "n06874185 traffic light, traffic signal, stoplight",
"y" : 0.0,
"label3" : "n04009552 projector",
"cputemp" : 58,
"top1" : "n02930766 cab, hack, taxi, taxicab",
"top4pct" : "5.000000074505806",
"tempf" : 75.81,
"memory" : 56.5,
"top4" : "n03345487 fire engine, fire truck",
"starttime" : "2018-04-17 15:27:25",
"runtime" : "12",
"label5" : "n09229709 bubble",
"temp" : 35.45,
"x" : 0.0
}
Example Data
13. 14 © Hortonworks Inc. 2011–2018. All rights reserved.
What Is OpenCV?
• OpenCV is a an open source computer vision library
• Nearly 20 years old
• Started by Intel
• Current Version 3.4.1
• C++, Python and Java Interfaces
• Runs on Windows, Linux, Mac, BSD, iOS, and Android.
• Can be built from source with Make
• Runs on Raspberry PIs
https://en.wikipedia.org/wiki/OpenCV
14. 15 © Hortonworks Inc. 2011–2018. All rights reserved.
What Can I Do with OpenCV?
• Facial Recognition
• Image Capture From Cameras
• Object Identification
• Motion Tracking
• Pixel Manipulation
• Image Properties
• Image Data Manipulation
• Image Processing including filtering, color conversion and histograms
• Image Labelling
https://github.com/jdye64/nifi-opencv
https://www.learnopencv.com/
https://docs.opencv.org/3.4.0/d9/df8/tutorial_root.html
15. 16 © Hortonworks Inc. 2011–2018. All rights reserved.
What Does OpenCV Python Code Look Like?
https://community.hortonworks.com/articles/182850/vision-thing.html
https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html
https://github.com/aruizga7/Self-Driving-Car-in-DSX/tree/master/1.%20Line%20Lane%20Detection
import cv2
cap = cv2.VideoCapture(0)
ret, frame = cap.read()
filename = ‘images/ilovedataworkssummit.jpg’
cv2.imwrite(filename, frame)
img = cv2.cvtColor(cv2.imread(filename),cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (224, 224))
cv2.rectangle(image, (x, y), (x + w, y + h), (255, 255, 0), 2)
17. 18 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache NiFi Integration with TensorFlow Options
• TensorFlow (C++, Python, Java)
via ExecuteStreamCommand
• TensorFlow NiFi Java Custom Processor
• TensorFlow Running on Edge Nodes (MiniFi)
18. 19 © Hortonworks Inc. 2011–2018. All rights reserved.
python classify_image.py --image_file /opt/demo/dronedata/Bebop2_20160920083655-0400.jpg
solar dish, solar collector, solar furnace (score = 0.98316)
window screen (score = 0.00196)
manhole cover (score = 0.00070)
radiator (score = 0.00041)
doormat, welcome mat (score = 0.00041)
bazel-bin/tensorflow/examples/label_image/label_image --
image=/opt/demo/dronedata/Bebop2_20160920083655-0400.jpg
tensorflow/examples/label_image/main.cc:204] solar dish (577): 0.983162I
tensorflow/examples/label_image/main.cc:204] window screen (912): 0.00196204I
tensorflow/examples/label_image/main.cc:204] manhole cover (763): 0.000704005I
tensorflow/examples/label_image/main.cc:204] radiator (571): 0.000408321I
tensorflow/examples/label_image/main.cc:204] doormat (972): 0.000406186
TensorFlow via Python or C++ Binary (Java Library Is New!)
19. 20 © Hortonworks Inc. 2011–2018. All rights reserved.
TensorFlow Python ExecuteStreamCommand NiFi
https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html
20. 21 © Hortonworks Inc. 2011–2018. All rights reserved.
Run TensorFlow on YARN 3.0
https://community.hortonworks.com/articles/83872/data-lake-30-containerization-erasure-coding-gpu-p.html
21. 22 © Hortonworks Inc. 2011–2018. All rights reserved.
Why TensorFlow? Also Apache MXNet, PyTorch and DL4J.
• Google
• Multiple platform
support
• Hadoop integration
• Spark integration
• Keras
• Large Community
• Python and Java APIs
• GPU Support
• Mobile Support
• Inception v3
• Clustering
• Fully functional demos
• Open Source
• Apache Licensed
• Large Model Library
• Buzz
• Extensive Documentation
• Raspberry Pi Support
22. 23 © Hortonworks Inc. 2011–2018. All rights reserved.
TensorFlow Java Processor in NiFi
https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html
https://github.com/tspannhw/nifi-tensorflow-processor
23. 24 © Hortonworks Inc. 2011–2018. All rights reserved.
TensorFlow Running on Edge Nodes (MiniFi)
24. 25 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache Tika with Apache NiFi
• Detection
• Parsing
• Output Formats including Text and HTML
• Translation
• Language Identification
• Apache NiFi - Bundled ExtractMediaMetadata Processor
• Apache NiFi - Extract the content metadata from flowfiles
https://tika.apache.org/1.18/gettingstarted.html
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-media-nar/1.6.0/org.apache.nifi.processors.media.ExtractMediaMetadata/
25. 26 © Hortonworks Inc. 2011–2018. All rights reserved.
HORTONWORKS DATA FLOW
NIFI
1.2.0HDF 3.0
Jul 2017
1.0.0
HDF 2.0
Mar 2016
1.1.0
NiFiRegistry
Ranger
0.7.0
0.5.0
0.6.0
Ambari
2.5.1
2.4.0
2.4.2
Kafka
0.10.1.0
0.9.0
0.10.0
Zookeeper
3.4.6
3.4.6
3.4.6
Storm
1.1.0
1.0.1
1.0.2SAM
0.5.0
SchemaRegistry
0.3.0
HDF 2.1
Aug 2016
Ongoing Innovation in Apache
HDF 1.0
Dec 2014
0.3.0
0.6.1
HDF 1.2
Oct 2015
MiNiFiC++andJava0.2.0
Ongoing Innovation in OpenSource
1.0.0
0.0.1
0.10.0
HDF 3.1.1
Jan 2018 1.5.0 0.1.0 0.7.02.6.11.0.0 3.4.61.1.10.6.0 0.5.00.4.0
SECURITYSTREAMING & INTEGRATION OPERATIONS
Hortonworks Data Flow 3.1.1
https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_release-notes/content/ch_hdf_relnotes.html
26. 27 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache Tika Supported File Formats
• HTML, XML
• Microsoft Word, Excel, PowerPoint, Outlook
• OpenOffice
• RSS
• RTF
• Zip, Tar, 7zip, Gzip, RAR
• PDF
• MP3, WAV, MIDI
• MP4, FLV
• TIFF, JPEG, PNG, BMP, GIF
• And more!
27. 28 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache Tika with Apache NiFi
https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html
https://community.hortonworks.com/articles/81694/extracttext-nifi-custom-processor-powered-by-apach.html
https://community.hortonworks.com/articles/76924/data-processing-pipeline-parsing-pdfs-and-identify.html
https://github.com/tspannhw/nifi-extracttext-processor
https://community.hortonworks.com/content/kbentry/177370/extracting-html-from-pdf-excel-and-word-
documents.html
28. 29 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache Tika with Apache NiFi
29. 30 © Hortonworks Inc. 2011–2018. All rights reserved.
HDF Data-In-Motion Platform – with HDF 3.1
30. 31 © Hortonworks Inc. 2011–2018. All rights reserved.
Why Apache NiFi?
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Hundreds of processors
• Visual command and
control
• Over a fifty sources
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering
• Version Control
31. 32 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache MiNiFi
• NiFi lives in the data center. Give it an
enterprise server or a cluster of them.
• MiNiFi lives as close to where data is born
and is a guest on that device or system
“Let me get the key parts of NiFi close to where data begins and provide
bidirectional data transfer"
32. 33 © Hortonworks Inc. 2011–2018. All rights reserved.
Edge Intelligence with Apache MiNiFi
Guaranteed delivery
Data buffering
‒ Backpressure
‒ Pressure release
Prioritized queuing
Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
Data provenance
Recovery / recording a rolling log
of fine-grained history
Designed for extension
Different from Apache NiFi
Design and Deploy
Warm re-deploys
Key Features
33. 34 © Hortonworks Inc. 2011–2018. All rights reserved.
Custom Apache NiFi Processors for Open Source Computer Vision
34. 35 © Hortonworks Inc. 2011–2018. All rights reserved.
TensorFlow with MiniFi
https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html
https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html
https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
35. 36 © Hortonworks Inc. 2011–2018. All rights reserved.
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html
https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html
https://community.hortonworks.com/articles/182850/vision-thing.html
https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html
Image Analytics
36. 37 © Hortonworks Inc. 2011–2018. All rights reserved.
NiFi and Kafka Are Complementary
NiFi
Provide dataflow solution
• Centralized management, from edge to core
• Great traceability, event level data provenance
starting when data is born
• Interactive command and control – real time
operational visibility
• Dataflow management, including prioritization,
back pressure, and edge intelligence
• Visual representation of global dataflow
Kafka
Provide durable stream store
• Low latency
• Distributed data durability
• Decentralized management of producers &
consumers
+
37. 38 © Hortonworks Inc. 2011–2018. All rights reserved.
Integrated Provisioning and Security
Kafka 1.0 Support
To enhance data governance and lineage, users can
now manage access control policies using resource or
tag-based security in Ranger for Kafka 1.0 clusters.
Users can now install, configure, manage, upgrade,
monitor, and secure Kafka 1.0 clusters with Ambari.
New processors in NiFi and Streaming Analytics
Manager support Kafka 1.0 features including message
headers and transactions.
38. 39 © Hortonworks Inc. 2011–2018. All rights reserved.
What Is Apache Kafka?
• Distributed streaming platform that allows
publishing and subscribing to streams of
records
• Streams of records are organized into
categories called topics
• Topics can be partitioned and/or replicated
• Records consist of a key, value, and
timestamp
http://kafka.apache.org/intro
Kafka
Cluster
producer
producer
producer
consumer
consumer
consumer
APACHE KAFKA
40. 41 © Hortonworks Inc. 2011–2018. All rights reserved.
https://community.hortonworks.com/articles/177349/big-data-devops-apache-nifi-
hwx-schema-registry-sc.html
41. 42 © Hortonworks Inc. 2011–2018. All rights reserved.
Completion of Schema Lifecycle:
Merged Schema from Dev Branch to Master
42. 43 © Hortonworks Inc. 2011–2018. All rights reserved.
Schema Registry Support for Different “States”:
Enable, Disable, Archive
43. 44 © Hortonworks Inc. 2011–2018. All rights reserved.
SAM and Schema Registry Integration
• Streaming Apps Require a Schema
• Unlike NiFi, SAM requires a schema to build
streaming analytics applications.
• Every SAM builder component requires a schema to
function.
• SAM’s primary mechanism for connecting to a stream
of data is Kafka, but Kafka does not have a schema.
• This is where HDF’s Schema Registry component
becomes incredibly valuable.
• SAM’s Kafka Source Component integrated
with Schema Registry
• When you configure a Kafka source and supply
kafka topic, SAM calls the Schema Registry.
• Using the Kafka topic as the key, SAM will retrieve
the schema.
• This schema is then displayed on the tile component,
and is passed to downstream components.
44. 45 © Hortonworks Inc. 2011–2018. All rights reserved.
Streaming Analytics Manager
45. 46 © Hortonworks Inc. 2011–2018. All rights reserved.
Streaming Analytics Manager
46. 47 © Hortonworks Inc. 2011–2018. All rights reserved.
Contact
https://community.hortonworks.com/users/9304/tspann.html
https://dzone.com/users/297029/bunkertor.html
https://www.meetup.com/futureofdata-princeton/
https://twitter.com/PaaSDev
https://dzone.com/refcardz/introduction-to-tensorflow
47. 48 © Hortonworks Inc. 2011–2018. All rights reserved.
Hortonworks Community Connection
Read access for everyone, join to participate and be recognized
• Full Q&A Platform (like StackOverflow)
• Knowledge Base Articles
• Code Samples and Repositories
48. 49 © Hortonworks Inc. 2011–2018. All rights reserved.
Community Engagement
Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2018. All Rights Reserved
20,000+
Registered Users
55,000+
Answers
87,000+
Technical Assets
One Website!
Editor's Notes Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined. Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined. Kafka
Reads events in memory and write to distributed log
https://github.com/USCDataScience/dl4j-kerasimport-examples/tree/master/dl4j-import-example
Also: https://github.com/adatao/tensorspark https://arimo.com/machine-learning/deep-learning/2016/arimo-distributed-tensorflow-on-spark/
https://caffe2.ai/docs/AI-Camera-demo-android
https://community.hortonworks.com/articles/54954/setting-up-gpu-enabled-tensorflow-to-work-with-zep.html Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined. Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined. Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined. Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined. Flow Management – the key is edge to anywhere with intelligence. This means the crux of it is being able to connect anything with anything else, from anywhere. This is a guiding principle for the roadmap on this aspect of data in motion.
Stream Processing – the key is time to insight. This means the crux of this is to be able to extract actionable information as quickly and easily as possible. This is the guiding principle for this aspect of data in motion.
Enterprise services – goes without saying, this has to all work together, efficiently, reliable, effectively. TALK TRACK
Apache MiNiFI is a sub project of Apache NiFi.
It is designed to solve the difficulties of managing and transmitting data feeds to and from the source of origin, enabling edge intelligence to adjust dataflow behavior with bi-directional communication, out to the last mile of digital signal.
It has a very small and lightweight footprint*, and generate the same level of data provenance as NiFi that is vital to edge analytics and IoAT (Internet of Any Thing)
It’s a little bit diferent from NiF in that is is not a real-time command and control interface – in fact – the agent, unlike NiFi doesn’t have a built in UI at all.
MiNiFi is designed for design and deploy situations and for “warm re-deploys”.
HDF 2.0 supports the java version of the MiNiFi agent, and a C++ version is coming soon as well.
Apache Ambari support for Kafka 1.0 – Users can now install, configure, manage, upgrade, monitor, and secure Kafka 1.0 clusters with Ambari.
Apache Ranger support for Kafka 1.0 – To enhance data governance and lineage, users can now manage access control policies using resource or tag-based security for Kafka 1.0 clusters.
New NiFi and SAM processors for Kafka 1.0 – New processors in NiFi and Streaming Analytics Manager support Kafka 1.0 features including message headers and transactions.
Data Science Cheat Sheet - https://hortonworks.app.box.com/file/234426455072
White Paper - https://hortonworks.app.box.com/file/151460926459
Videos - https://www.youtube.com/user/Hortonworks/feed?activity_view=1