Resource placement is a policy-rich problem, particularly across multi-cluster, multi-geography and multi-cloud environments. Placement may be based on company conventions, external regulation, pricing, performance requirements, or complex combinations of those. Furthermore, placement policies evolve over time and vary across organizations. As a result, it is very difficult to anticipate the policy requirements of all users. In this presentation, Torin Sandal (Lead Engineer of Open Policy Agent) will present, along with Irfan Ur Rehman, and demonstrate the work they've done integrating OPA into the Kubernetes Cluster Federation Control Plane. This enables high level policies to be expressed in a easy to understand policy language, and automatically enforced across federations of Kubernetes clusters.

1. Policy-based Resource Placement
...across Hybrid-Cloud Federations of Kubernetes Clusters
Irfan UR Rehman (Huawei)
@irfanurrehman
Torin Sandall (Styra)
@sometorin

2. Federation: Overview
federation-apiserver
federation-controller
app app
Federation Control Plane
Federated
Deployment
etcd
Federated
Clusters

3. Federation: Placement
federation-apiserver
federation-controller
kind: ReplicaSet
metadata:
annotations:
replica-set-preferences: |
{“clusters”: {
“us-west-1”: {“weight”:1},
“asia-se-2”: {“weight”:1}
}}
● Placement can be controlled
per-resource via annotations
● 2 annotations supported:
federation.kubernetes.io/replica-set-preferences
federation.alpha.kubernetes.io/cluster-selector
● federation-controller evaluates
annotations to produce final
placement
ReplicaSet
created
1
2
3
4

4. Policy-based Placement
federation-apiserver
federation-controller
● Resource placement is a “policy-rich” problem space
○ Legal regulation, cost, technical constraints, internal
conventions, etc.
● Goal: give admins greater control and flexibility
○ Automated & programmable
○ Expressiveness
○ Leverage context
○ Ease management
● Decouple developer intent from admin policy
○ Avoid duplication
○ Prevent (and detect) violations
○ Abstract policy implementation
● Policy Engine decides which clusters app runs on
○ Pluggable
○ Simple interface
opa

5. Architecture
federation-apiserver
federation-controller
opaadmission controller
kind: ReplicaSet
metadata:
name: my-app
annotations:
customer: acmecorp
● Admission Controller inside federation-apiserver
queries Policy Engine when resources are
created or updated
● Admission Controller implements “fail-closed”
model in case query fails.
POST /v1/data/k8s/placement
input:
resource:
metadata:
name: my-app
annotations:
customer: acmecorp
...
HTTP/1.1 200 OK
result:
annotations:
replica-set-preferences:
clusters:
us-west-1: {weight: 1}
asia-se-3: {weight: .5}
Example Query

6. Architecture
host cluster
pod
federation-apiserver
federation-controller opa
admission controller
etcd
sidecar
external state
policy definition

7. Example
Apps labelled with customer name. Customers are associated with a
jurisdiction. Furthermore, apps may be labelled with criticality. If low then public
cloud clusters may be used, otherwise, on-prem clusters must be used.

8. Visibility & Remediation
federation-apiserveropa sidecar
which of customer
x’s apps are
deployed in EU?
which customers are
affected if cluster X is
no longer PCI
certified?
policy
clusters, deployments,
services, ...

9. Conflicts
● Developers could accidentally specify
conflicting intent (result: empty set)
● Developers could explicitly request invalid
clusters (result: error)
● Resolve conflicts within policy engine
whenever possible
○ Policy is the only place where all intent is known
kind: ReplicaSet
metadata:
annotations:
customer: acmetel-US
criticality: low
replica-set-preferences:
clusters:
- us-west-2
- eu-central-1
...
not_allowed[cluster] {
requested_clusters[cluster]
not allowed_clusters[cluster]
}
errors[“invalid cluster(s)”] { not_allowed != set() }

10. Future Work
● Improve policy management
○ Current: policies stored as ConfigMaps in the
federation-apiserver
○ Future: policies represented as first-class API objects
○ Cleaner mechanism for reporting policy enforcement
status
■ Installed, errors, etc.
● Demonstrate new use cases
○ Cost-based policies
■ Replicate external data representing resource
pricing (e.g., cpu, memory, etc.)
■ Pick clusters based on pricing data
■ Cluster inter-connect may be expensive
federation-apiserver
federation-controller
opa

11. ● Kubernetes Federation enables hybrid-cloud
deployments for a variety of use cases
● Resource placement is a policy-rich problem
that must address important business
requirements
● Policy solution should empower admins with
greater control and flexibility
Conclusion
federation-apiserver
federation-controller
opa

12. Thank You!
SIG-Federation
Open Policy Agent (OPA)
github.com/open-policy-agent/opa