Flink Forward San Francisco 2022. Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A by James Busche & Ted Chang

1. The Apache Flink® Conference
San Francisco 2022
August 3

2. Using the New Apache Flink
Kubernetes Operator in a
Production Deployment
Jim Busche, IBM
Ted Chang, IBM

3. Agenda
1. Introduction
○ Problems we are we solving.
2. Overview of Kubernetes operators and their benefits.
○ Five levels of the operator maturity model.
○ Introduce the newly released Apache Flink Kubernetes Operator.
3. Flink Kubernetes Operator Container Image modifications
○ UBI images
○ IBM Java
4. Enhancements we're making in:
○ Versioning/Upgradeability/Stability
○ Security
5. Demo OLM managed Custom Flink Operator in action
6. Q&A

4. Problems we Are Solving
❖ How to support products requires
different Flink versions?
❖ How to upgrade Flink Operator
running in production?
❖ How to make Flink more secure?
❖ Reduce duplicate effort among

5. What is Kubernetes operators
The Operator pattern aims to
❖ Capture the key aim of a human
operator who is managing a
service or set of services.
➢ Operands: The Managed
Services. Typically backing
services.
❖ Have deep knowledge how the
service ought to behave, how to
deploy it.
❖ React if there are problems.
“The Twelve-Factor App: IV. Backing services” https://12factor.net/backing-services

6. How are operators helpful
An operator makes a service self-managing
❖ Embeds domain-specific knowledge about the service and its lifecycle
❖ Installs the service, upgrades it, keeps it running, tracks metrics, etc.
Hybrid cloud: The service is hosted in the customer’s cluster
❖ The customer is responsible for managing the service
❖ Customer may lack staff, skills, interest
❖ A service manages itself is much more appealing

7. Operator Maturity Model
Operator Maturity Model
Level 1: Basic Install
Level 2: Seamless Upgrades
Level 3: Full Lifecycle
Level 4: Deep Insights
Level 5: Auto Pilot
https://docs.openshift.com/container-platform/4.10/operators/understanding/olm-what-operators-are.html#olm-maturity-model_olm-what-operators-are

8. Apache Flink Kubernetes Operator and CR
The Flink Kubernetes Operator extends the Kubernetes API with the ability to manage and operate Flink Deployments. The operator
features the following amongst others:
● Deploy and monitor Flink Application and Session deployments
● Upgrade, suspend and delete deployments
● Full logging and metrics integration
● Flexible deployments and native integration with Kubernetes tooling

9. Benefit of Using the Flink Kubernetes Operator
Operator handles the following
tasks declaratively using the the
CR
❖ Deploying jobs
❖ Configuration
❖ Upgrade
❖ Backup
❖ Restore
❖ High Availability
❖ Metrics Monitoring

10. Flink in our products
Watson AIOPs
Flink is used by the log anomaly
detection data prep pipeline and for
the event lifecycle.
https://www.ibm.com/cloud/blog/watson-aiops-bringing-ai-to-it-operations-management
Business Automation Insights
Data ingestion and processing
relies on Apache Flink data
processing framework

11. Software Compliance in Production
✓ Platform: Red Hat OpenShift
✓ Container Image: UBI
✓ Security
✓ Package Management: Operator
Lifecycle Management (OLM)
Before we can Flink into our product, certain requirements
must be met. For example:

12. Red Hat OpenShift
❖ Enterprise-grade Kubernetes features
including 7/24 Support
❖ Built-in Security Context Constraint (SSC)
provides default execution policies,
increasing the entire Kubernetes cluster security
level.
❖ Role-based access control (RBAC) in OpenShift
is a non-optional feature, enabling role-based
permissions as required.
❖ Containers required to run as non-root

13. Dockerfile modifications
❖ If building your own image, make sure you do
latest OS update to get the latest security patches.
For example:
➢ https://github.com/apache/flink-
kubernetes-
operator/blob/main/Dockerfile#L62
➢ ARG SKIP_OS_UPDATE=true Change to
ARG SKIP_OS_UPDATE=false
❖ You can use/create your own DockerFile/base
image, for example we used the Red Hat UBI
image with IBM Java 11
❖ What benefit do we have for swapping UBI base?
For production environments that require an
enterprise version of Linux.

14. Security Pipeline
❖ Tekton based Security Scan Pipeline running the following
➢ Twistlock vulnerability and compliance
■ Scans containers for vulnerabilities.
➢ Whitesource/Mend
■ WhiteSource is an open-source vulnerability scanner that scans source code for known OSS
vulnerabilities and for compliance.
■ Recommend fixes.
➢ Scorecard
scorecard --local=./flink-kubernetes-operator
RESULTS
-------
Aggregate score: 5.0 / 10

15. Operator Lifecycle Manager
Open source toolkit to manage
Operators in a Kubernetes Cluster
❖ Over-the-Air Updates and Catalogs
❖ Dependency Model
❖ Discoverability
❖ Cluster Stability
❖ Declarative UI controls

16. Demo of the Apache Flink Kubernetes Operator in-action
See the prepared recording
❖ Demo of the Apache Flink Operator in-action.
➢ Apply the catsrc
➢ Apply the Subscription (Recommended installPlanApproval: Manual)
➢ Approve the install plan
➢ Check the csv and Flink Kubernetes Operator

17. Summary, and Q&A
- Q&A
- Thank you for attending!