Summarizes new capabilities added to Apache NiFi 1.2.0 (soon to be released). Disclaimer: - The contents in this slide deck are derived from Apache NiFi JIRA issues which is labeled with next release target 1.2.0 and source code available at Github (already merged into master branch), however it does NOT mean these are guaranteed to be released and still are subjects to change. - The motivation of this presentation is share what have been introduced into the project since the latest Apache NiFi 1.1.2 release. - The contents are created from information available under Apache NiFi project, however, the way summarize it is solely done with my personal thoughts and not a consensus built among Apache NiFi community.

1. What will be new in
Apache NiFi 1.2.0
(was not released as of this writing, but released on May 8th)
Apr 28th, 2017
Apache NiFi committer: Koji Kawamura (ijokarumawak)

2. Disclaimer: See release note for detail!
• Apache NiFi 1.2.0 has been released on May 8th, 2017!! Please see the
official release note for official what’s new.
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#Release
Notes-Version1.2.0
• The contents in this slide deck are derived from Apache NiFi JIRA issues
which is labeled with next release target 1.2.0 and source code available at
Github (already merged into master branch), however it does NOT mean
these are guaranteed to be released and still are subjects to change.
• The motivation of this presentation is share what have been introduced
into the project since the latest Apache NiFi 1.1.2 release.
• The contents are created from information available under Apache NiFi
project, however, the way summarize it is solely done with my personal
thoughts and not a consensus built among Apache NiFi community.

3. Themes
• Schema Registry, Record Reader/Writer
• Multiple version of Nar
• Support EL for various Processor properties
• Performance Improvement
• CDC (Capture Data Change)
• Rollback on Failure
• Flow control
• UX
• Security

4. Schema Management is a pain…
We need schema to:
- Analyze data
- Convert one format to another
- Validation
- … etc
- Centralized schema management is needed..

5. Schema?

6. Schema Registry
• AvroSchemaRegistry
• Provides a service for registering and accessing schemas. You can register a
schema as a dynamic property where 'name' represents the schema name
and 'value' represents the textual representation of the actual schema
following the syntax and semantics of Avro's Schema format.
• HortonworksSchemaRegistry
• Provides a Schema Registry Service that interacts with a Hortonworks Schema
Registry, available at https://github.com/hortonworks/registry

7. Record Reader/Writer

8. Schema Registry and Reader/Writer
• ConvertRecord
• Converts records from one data format to another using configured Record Reader
and Record Write Controller Services.
• SplitRecord
• Splits up an input FlowFile that is in a record-oriented data format into multiple
smaller FlowFiles
• PutDatabaseRecord
• The PutDatabaseRecord processor uses a specified RecordReader to input (possibly
multiple) records from an incoming flow file.
• QueryRecord
• Evaluates one or more SQL queries against the contents of a FlowFile. The result of
the SQL query then becomes the content of the output FlowFile. This can be used,
for example, for field-specific filtering, transformation, and row-level filtering.

9. Multiple versions of Nar
MANIFEST.MF

10. New Processors: CDC
• CaptureChangeMySQL
Retrieves Change Data Capture (CDC) events from a MySQL database.
CDC Events include INSERT, UPDATE, DELETE operations. Events are
output as individual flow files ordered by the time at which the
operation occurred.

11. How CDC works

12. Rollback on Failure
• PutSQL
• PutHiveQL
• PutHiveStreaming
• PutDatabaseRecord

13. PutSQL : default behavior
2 13 PutSQL RDBMS
Input FlowFiles
1 3
1
3 2
success failure

14. PutSQL : Rollback on Failure
2 13 PutSQL RDBMS
Input FlowFiles
success failure
Rollback!
Modified records are rolled back
No output FlowFile, those will be kept in the input queue

15. New Processors: Flow Control
• EnforceOrder
• Enforces expected ordering of FlowFiles those belong to the same data group.
• Wait
• Routes incoming FlowFiles to the 'wait' relationship until a matching release
signal is stored in the distributed cache from a corresponding Notify
processor. When a matching release signal is identified, a waiting FlowFile is
routed to the 'success' relationship, with attributes copied from the FlowFile
that produced the release signal from the Notify processor.
• Notify
• Caches a release signal identifier in the distributed cache, optionally along
with the FlowFile's attributes. Any flow files held at a corresponding Wait
processor will be released once this signal in the cache is discovered.

16. Flow Control using Wait/Notify

17. New Processors GCS
• DeleteGCSObject
• FetchGCSObject
• ListGCSBucket
• PutGCSObject

18. New Processors
• ConsumeEWS
• Consumes messages from Microsoft Exchange using Exchange Web Services. The
raw-bytes of each received email message are written as contents of the FlowFile
• ConvertExcelToCSVProcessor
• Consumes a Microsoft Excel document and converts each worksheet to csv.
• ExtractCCDAAttributes
• Extracts information from an Consolidated CDA formatted FlowFile and provides
individual attributes as FlowFile attributes.
• ExtractGrok
• Evaluates one or more Grok Expressions against the content of a FlowFile, adding the
results as attributes or replacing the content of the FlowFile with a JSON notation of
the matched content

19. New Processors
• FetchHBaseRow
• Fetches a row from an HBase table.
• FuzzyHashContent
• Calculates a fuzzy/locality-sensitive hash value for the Content of a FlowFile and puts
that hash value on the FlowFile as an attribute whose name is determined by the
<Hash Attribute Name> property.
• ISPEnrichIP
• Looks up ISP information for an IP address and adds the information to FlowFile
attributes.
• ListenBeats
• Listens for messages sent by libbeat compatible clients (e.g. filebeats, metricbeats,
etc) using Libbeat's 'output.logstash', writing its JSON formatted payload to the
content of a FlowFile.This processor replaces the now deprecated ListenLumberjack

20. New Processors
• ExecuteScript
• ClojureScriptEngine is added!
• UpdateCounter
• This processor allows users to set specific counters and key points in their flow. It is
useful for debugging and basic counting functions.
• AttributeRollingWindow
• Track a Rolling Window based on evaluating an Expression Language expression on
each FlowFile and add that value to the processor's state.
• GetTCP
• Connects over TCP to the provided endpoint(s). Received data will be written as
content to the FlowFile
• QueryDatabaseTable
• MSSQL2008DatabaseAdapter, MSSQLDatabaseAdapter

21. Deep Linking!

22. Align Components!

23. More context menu at root Process Group
BTW, did you know you can
‘Refresh’ flow by ‘Cmd + r’?

24. … and more!!
1.2.0 is discussed to be released soon! Released and available for download!
Thank you :)
https://nifi.apache.org/download.html