This document introduces NiFi Registry, a new component of Apache NiFi that allows for versioning and centralized management of NiFi flows. It provides capabilities for deploying flows between environments through version control and management of parameterized variables. The architecture involves a metadata database and flow persistence provider to store versions of flows and their associated metadata. Examples of deployment scenarios and existing tools for automation are also presented.

1. Introduction to NiFi Registry
Bryan Bende & Kevin Doran
April 17th 2018

2. 2 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Background
à Most frequently asked question…
– How do I deploy my flow?
à In theory….
– Avoid long turn around time of deployments
– Quickly modify dataflow via NiFi UI
à In reality…
– Classic development lifecycle (dev -> int -> prod)
– Promote changes after successful testing
– Hands-off UI in production

3. 3 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Previous Deployment Approaches
à Copy flow.xml.gz between environments
– Requires copying entire data flow for any change
– Can’t tell what changed, hard to diff if you put in version control
– Requires all environments use the same encryption key for sensitive properties
à Make templates of process groups
– Script creation of template and deployment to next environment
– Requires stopping flow and removing components, then re-instantiating template
– No easy way to see changes, hard to rollback

4. 4 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Previous Variable Handling
à Need to parametrize values in the flow per environment
– Connection strings, URLs, File System paths, etc.
à Can set variables in bootstrap.conf
– -Dmy.var=foo
à Can set a properties file in nifi.properties
– nifi.variable.registry.properties=production.properties
à Both require command line access
à Both require restart of entire NiFi instance to pick up changes

5. 5 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Enter NiFi Registry
à NiFi Registry - sub-project of Apache NiFi
– https://github.com/apache/nifi-registry
– https://issues.apache.org/jira/projects/NIFIREG
à Complimentary application, central location for
storage/management of “versioned” resources
à Initial capability to store and retrieve “versioned
flows”
à Integration on NiFi side
– Start/Stop version control of a process group
– Change version (upgrade/downgrade)
– Import new process group from a version

6. 6 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Enter Variable Registry
à Parameterize configuration like connection
strings, file paths, etc.
à Referenced via Expression language
– Kafka Brokers = ${kafka.brokers}
à Variables associated with a process group
à Right-click on canvas to view variables for
current process group
à Hierarchical order of precedence, resolve
closest reference to component
à Editing variables automatically restarts
any components referencing the variables!
Level 1
Level 2
Vars
Vars

7. 7 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
NiFi Registry Terminology
à Buckets
– Logical place to store versioned items/resources
– Permissions assigned at bucket level
• READ - Retrieve any item from the bucket
• WRITE – Create new items in the bucket
à Versioned Flows
– Currently the only type of item
– Each versioned flow has name, description, and 1 or more “snapshots”
à Versioned Flow Snapshot
– Each snapshot has metadata and contents
– Metadata contains a one-up version #, commit message, author, & create date
– Content contains a Versioned Process Group i.e. snapshot of PG from NiFi

8. 8 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Architecture
à Metadata Database
– Metadata for each bucket
– Knowledge of which items are in each bucket
– Metadata for each versions of an item
– Currently uses embedded H2 DB
à Flow Persistence Provider
– Stores/retrieves serialized bytes of a versioned flow
– Pluggable extension point
– Provided implementation uses local file system
– NIFIREG-162 Git-based implementation
• https://github.com/apache/nifi-registry/pull/112
Client UI (Angular)
Server (Spring Boot)
REST API (Jersey)
Service Layer (Spring, Java)
Metadata Database
Flow Persistence
Provider
H2 File System
= Extension Point

9. 9 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
StagingDev
Deployment Scenario 1 – Shared Registry with Multiple NiFi’s
NiFi Registry
NiFi NiFi
Production
NiFi
1. Start version
control and save
new versions
2. Import flow
from Dev and
Perform testing
3. Import flow
that passed tests
in staging
LDAP

10. 10 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Deployment Scenario 2 – Separate Environments
StagingDev
NiFi Registry
NiFi NiFi
Production
NiFi
1. Start version
control and save
new versions
3. Import flow
version & test 4. Manually
move specific
version to prod
registry
NiFi Registry NiFi Registry
LDAP LDAPLDAP
2. Manually
move specific
version to
staging registry
5. Import flow
version and start

11. 11 © Hortonworks Inc. 2011 – 2018. All Rights Reserved
Tools & Automation
à NiPyApi
– Python wrapper for NiFi & NiFi Registry REST APIs
– https://github.com/Chaffelson/nipyapi
à NiFi CLI
– Part of nifi-toolkit released with Apache NiFi 1.6.0
– https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli
– README file at above location has documentation to get started