Deploying Cloud-Native Red Team Infrastructure with Kubernetes, Istio, and Envoy

1. Deploying Cloud Native Red
Team Infrastructure with
Kubernetes, Istio and Envoy

2. Who we are
◎Larry Suto - Larry is an independent
security consultant based out of Oakland,
CA. He spends a lot of time researching
using cloud infrastructure for all types of
security testing. He spends some time on
Windows security as well.
Twitter: @larrysuto
2

3. Who we are
◎Jeff Holden – Works at the largest college
system in the united states. CISO by day,
hacker by night. Currently living in an RV
with 3 dogs, 1 wife, and 1 kid.
◎@jeffh

4. Goals
◎Working in AWS, GCE, Azure
◎Automated
○ Set domains, access keys, etc
◎Portable
◎Scriptable

5. Why Kubernetes?
◎Infrastructure as code
◎Portable
◎Speed
◎Efficiency

6. What is this Docker thing?

7. Docker

8. Kubernetes
Open source container-orchestration system for automating application
deployment, scaling, and management.

9. Creating a Kubernetes Deployment
◎Configmap/Secret
◎Deployment yaml
◎Service yaml
◎Helm chart (optional)

10. What is this Kops thing?
• Takes care of the infrastructure grunt work
• Creates Network
• Creates Base VMs
• Creates load balancer
• Creates Access list
• Supports
• AWS
• GCE

11. Kops
◎kops create cluster --zones us-west-2a,us-
west-2b --topology private
◎--networking calico --master-size t3.micro -
-master-count 3 --node-size
◎ t3.large --name <kube cluster name>
◎--state=s3://<Your S3 Bucket Name> --yes

12. External DNS
◎Support all major cloud services
◎ Pod adds and updates DNS entries to
cloud providers DNS
○ Need to use cloud provider specific tool to create
hosted zone
○ Can automate domain registration
12

13. SSL Cert Manager
13

14. SSL Cert Manager
◎https://docs.cert-manager.io/en/latest/
◎Automates LetsEncrypt Certificate
Management
◎TLS Passthrough
14

15. Background
◎Placing Apache mod_rewrite in front of C2 infrastructure has been common
place for many years now
◎Many advances have been made in redirection technology recently driven
by the move to container-based cloud systems such as Kubernetes
◎Adoption of Docker by the red team is common but we have seen little in
the way of container management systems and service mesh technology
which provide sophisticated mechanisms for ingress and egress
manipulation
◎Service mesh technology holds a lot of promise for sophisticated c2
redirection and traffic manipulation and can leverage multiple clouds with
many simultaneous ingress points
◎With Kubernetes its quite straight forward to introduce new redirection
point dynamically

16. Recipes for Containerizing C2
◎Candidates
○ Cobalt Strike
○ Merlin (http/2)
○ Many others (Empire, Faction, silver, Covenant,..)

17. Dockerizing Cobalt Strike
docker build --build-arg cskey="$(cat license)" -t cobalt/ub18:1.0

18. FROM ubuntu:18.04 as base
MAINTAINER kubered
LABEL version=”1.0"
LABEL description="CobaltStrike."
ARG cskey
ENV cs_key ${cskey}
SHELL ["/bin/bash", "-c"]
RUN apt-get update &&
apt-get install -y wget curl net-tools sudo software-properties-common apt-utils --no-install-recommends &&
apt install -y openjdk-11-jdk &&
update-java-alternatives -s java-1.11.0-openjdk-amd64 &&
rm -rf /var/lib/apt/lists/*
RUN var=$(curl 'https://www.cobaltstrike.com/download' -XPOST -H 'Referer: https://www.cobaltstrike.com/download' -H 'Content-Type:
application/x-www-form-urlencoded' -H 'Origin: https://www.cobaltstrike.com' -H 'Host: www.cobaltstrike.com' -H 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -H 'Accept-Language: en-us' -H 'User-Agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5' --data
"dlkey=$cs_key" | sed -n 's/.*href="([^"]*).*/1/p' | grep /downloads/ | cut -d '.' -f 1) &&
cd /opt &&
wget https://www.cobaltstrike.com$var.tgz &&
tar xvf cobaltstrike-trial.tgz &&
cd cobaltstrike &&
echo $cs_key > ~/.cobaltstrike.license &&
./update
RUN apt-get -y clean &&
apt-get -y autoremove
COPY cobalt-kube.profile /opt/cobaltstrike/profiles/
# set entry point
WORKDIR "/opt/cobaltstrike"
ENTRYPOINT ["./teamserver"]

19. Dockerizing Merlin
FROM golang:stretch
MAINTAINER @audibleblink
RUN apt-get update && apt-get install -y git make
RUN go get github.com/Ne0nd0g/merlin/...
WORKDIR $GOPATH/src/github.com/Ne0nd0g/merlin
VOLUME ["data/temp"]
EXPOSE 443
CMD ["go", "run", "cmd/merlinserver/main.go", "-i", "0.0.0.0"]

20. Configmap and Secret
kubectl create –f cobalt-config.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: cobalt-password
data:
password: cbs76654A
Configmaps allow you modify application behavior without respinning an
image
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-{{ .Values.malConfig }}
data:
randomized.profile: |-
{{ .Files.Get "randomized.profile" | indent 4 }}

21. Deployment yaml
(Helm template)
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-deployment
namespace: {{ .Values.default }}
spec:
selector:
matchLabels:
app: {{ .Release.Name }}
replicas: 1
template:
metadata:
labels:
app: {{ .Release.Name }}
spec:
securityContext:
runAsUser: 0
imagePullSecrets:
- name: {{ .Values.pullSecret }}
containers:
- name: {{ .Release.Name }}
image: {{ .Values.image }}:{{ .Values.tag }}
imagePullPolicy: IfNotPresent
volumeMounts:
- name: {{ .Release.Name }}-{{ .Values.malConfig }}
mountPath: {{ .Values.malc2path }}
subPath: {{ .Values.subpath }}

22. Deployment yaml cont
ports:
- containerPort: 80
- containerPort: 50050
- containerPort 443
env:
- name: COBALT_PASSWD
value: {{ .Values.password | quote }}
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MAL_C2_PATH
value: {{ .Values.malc2path | quote }}
args: ["$(MY_POD_IP)", "$(COBALT_PASSWD)",
$(MAL_C2_PATH)]
volumes:
- name: {{ .Release.Name }}-{{ .Values.malConfig }}
configMap:
name: {{ .Release.Name }}-{{ .Values.malConfig }}

23. Service Definition (teamserver mgmt internal)
apiVersion: v1
kind: Service
metadata:
name: cobalt-console
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
spec:
ports:
- port: 50050
protocol: TCP
targetPort: 50050
selector:
app: teamserver1-deployment
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: a54cae28bd42b11e7b2c7020a3f15370-27798109.us-west-2.elb.amazonaws.com

24. Service Definition (Cobalt Strike Listener)
apiVersion: v1
kind: Service
metadata:
name: cobalt-listener
labels:
app: team-server
spec:
ports:
- port: 80
protocol: TCP
selector:
app: team-server

25. Mounting SSL Certificates to Containers pt 1
apiVersion: v1
data:
merlin.key:
LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2QUlCQURBTkJ
na3Foa2lHOXcwQkFRRUZBQVNDQktZd2dnU2lBZ0VBQW9JQkFRQ1lrL
2hMaEMzalh2Y3kKUHY1VDdNcU1OMWR5STlQNVM5MlpUUllNT1VZb2
JiUXREeE1KbWxMd3g4c0owQURlWjVzTWRSQkYwWjJzNVBrMApHL3V
2d2c2c2JpSTFCaXVqaVBzdnRwWVpIaC9nZVdJUG5zS....
merlin.crt:
S0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHRENDQWdBQ
0NRRHJDajdxWHFhR1VqQU5CZ2txaGtpRzl3MEJBUXN….
kind: Secret
metadata:
name: merlin-ssl
type: Opaque
$ cat ssl.secret

26. Mounting SSL Certificates to Containers pt 2
spec:
containers:
- image: merlin
name: merlin
volumeMounts:
- mountPath: "/opt/merlin/data/x509"
name: merlin-ssl
readOnly: true
ports:
- containerPort: 443
volumes:
- name: merlin-ssl
secret:
secretName: merlin-ssl

27. Merlin Deployment (Helm)

28. Merlin C2 Console (Helm Deployment)

29. Silenttrinity Deployment (Helm)

30. Silenttrinity Console

31. Envoy Proxy
◎It can proxy any TCP protocol.
◎It can do SSL. Either direction.
◎Full support for HTTP/2 and can translate between HTTP/2 and
HTTP/1.1 (either direction).
◎It has good flexibility around discovery and load balancing.
◎It’s a sidecar process, so it’s completely agnostic to you service
implementation language

32. Envoy xDS API
◎Listener Discovery Service (LDS) - an API that allows Envoy to query what
listeners should be exposed on this proxy
◎Route Discovery Service (RDS) - a part of the configuration for listeners that
specifies which routes to use; this is a subset of LDS for when static and
dynamic configuration should be used
◎Cluster Discovery Service (CDS) - an API that allows Envoy to discover what
clusters and respective configuration for each cluster this proxy should have
◎Endpoint Discovery Service (EDS) - a part of the configuration for clusters
that specifies which endpoints to use for a specific cluster; this is a subset of
CDS
◎Secret Discovery Service (SDS) - an API used to distribute certificates
◎Aggregate Discovery Service (ADS) - a serialized stream of all the changes to
the rest of the APIs; you can use this single API to get all of the changes in
order

33. Envoy Docker Container and httpbin App

34. Envoy Sample Config

35. Running Envoy

36. Envoy Headers

37. Envoy Stats

38. Envoy Based Ingress Controllers
◎Gloo
◎Istio
◎We also will start providing examples for
Traefik as it is a very simple and easy to
deploy edge proxy written in golang.

39. Envoy based Gateways
39
◎Gateway describes a load balancer operating at the
edge of the mesh receiving incoming or outgoing
HTTP/TCP connections.

40. Envoy based Virtual Services
◎A Virtual Service defines a set of traffic routing rules to
apply when a host is addressed. A routing rule defines
matching criteria for traffic of a specific protocol.
◎If the traffic is matched, then it is sent to a named
destination service (or subset/version of it) defined in the
registry.
◎Supports many protocols : http, https, http/2

41. gloo Envoy Gateway
◎Ingress Gateway (Envoy)
◎Autodiscovers Kubernetes services as Upstreams

42. gloo Routing Rules

43. gloo Gateway
◎Installs with Helm:
○ helm repo add gloo https://storage.googleapis.com/solo-public-
helm
○ helm install gloo/gloo --namespace my-namespace
◎Installs with default ports 80 and 443 but this can be
customized with Helm
https://gloo.solo.io/installation/gateway/kubernetes/
◎Or kubectl edit gateway-proxy-v2 -n gloo-system
◎Also can be configured as a pure TCP Proxy
○ https://gloo.solo.io/gloo_routing/tcp_proxy/

44. gloo Upstreams

45. gloo VirtualService

46. gloo Setting up Server TLS
kubectl create secret tls gateway-tls --key tls.key --cert tls.crt --namespace default
glooctl create secret tls gateway-tls --certchain $CERT --privatekey $KEY
glooctl edit virtualservice --name default --namespace gloo-system --ssl-secret-name gateway-tls
--ssl-secret-namespace default

47. gloo Multiple TLS VirtualServices with SNI
kubectl create secret tls teamserver2-cert --key tls.key --cert tls.crt --namespace default
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj
"/CN=teamserver2.ccco.net"
glooctl create virtualservice --name teamserver2 --domains teamserver2.ccco.net
glooctl edit virtualservice --name teamserver2 --namespace gloo-system
--ssl-secret-name teamserver2-cert --ssl-secret-namespace default --ssl-sni-domains teamserver2.ccco.net
You will get errors because the upstream listener is serving
https

48. gloo Setting up Client TLS (mTLS)
glooctl edit upstream --name default-teamserver2-server-443 --namespace gloo-
system
--ssl-secret-name teamserver2-upstream-tls --ssl-secret-namespace default
kubectl create secret tls teamserver2-upstream-tls --key key.pem
--cert cert.pem --namespace default
kubectl get upstream -n gloo-system default-teamserver2-server-443 -
o yaml
Create Upstream Secret (or use the one for the gateway)
Update the Upstream with Secret
Dump the Updated Upstream

49. gloo Updated Upstream with TLS

50. Istio Service Mesh
Mixer Istio-AuthPilot
Envoy Envoy
SvcA SvcB
Envoy
Istio-Ingress
(Gateway)
Service A (C2) Service B
http, https,
http/2 http, https,
http/2
http, https,
http/2

51. Installing Istio
◎kubectl create namespace istio-system
◎Install CRD with kubectl apply
◎helm install install/kubernetes/helm/istio --name istio --namespace istio-system
--values install/kubernetes/helm/istio/values-istio-demo.yaml
◎Inject the Istio/Envoy sidecar manually or using injection webhook
○ istioctl kube-inject -f samples/sleep/sleep.yaml | kubectl apply -f -
○ kubectl label namespace default istio-injection=enabled –overwrite
◉ Injection occurs when the pod restarts

52. Istio Ingress
◎kubectl get svc istio-ingressgateway -n istio-system
◎Handles http or tcp ingress
◎Overcomes weaknesses of Kubernetes Ingress
○ Kubernetes Ingress mostly focused on http/https
◎Generally auto-provisions a load balancer
52

53. Multiple Ingress (Redirectors)
C2 Services
Kubernetes
cluster
istioingressC
glooGateway
istiongressA
istioingressB
IstioegressA
C2 Services

54. Modifying the Ingress Gateway
54
istio-ingressgateway:
enabled: true
…
gateway.
image: node-agent-k8s
…
labels:
app: istio-ingressgateway
istio: ingressgateway
…
ports:
- port: 15020
targetPort: 15020
name: status-port
- port: 80
targetPort: 80
name: http2
nodePort: 31380
- port: 443
name: https
nodePort: 31390
- port: 50050
name: tcp
nodePort: 31400
This file can be modified to add multiple ingress points:
install/kubernetes/helm/istio/charts/gateways/values.yaml

55. Deploy your own Ingress Gateway
55
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: istio-ingressgateway
namespace: istio-system
spec:
replicas: 1
template:
metadata:
labels:
app: istio-ingressgateway
istio: ingressgateway
visibility: internal # put a custom label here.
annotations:
sidecar.istio.io/inject: "false"
scheduler.alpha.kubernetes.io/critical-pod: ""
spec:
...

56. Istio Ingress Private Load Balancer
kubectl get svc istio-ingressgateway -n istio-system -o yaml > private-ingressgateway.yaml
• Add annotation: service.beta.kubernetes.io/aws-load-balancer-internal=“true”
• Change the name to anything: IE from istio-ingressgateway to
private-ingressgateway
• Change the app label to anything: IE istio-ingressgateway to
private-ingressgateway
• Change the istio label to anything: IE ingressgateway to private-
ingressgateway
• Change selector configuration section and update the app and
istio label to match the values you defined in the metadata section
• Remove all the nodePort values from the ports configuration so
new ports can be allocated automatically

57. Istio Ingress Private Loadbalancer(cont)

58. Exposing Services via Istio-ingeress gateway
◎To expose a service using ingressgateway you have to create at
least 2 objects
○ Gateway
○ Virtual Service

59. Istio Gateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: teamserver-gateway
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 80
name: http-system
protocol: HTTP
hosts:
- "*"
- port:
number: 50050 # exposes teamserver admin port
name: tcp-admin
protocol: TCP
hosts:
- "*"
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: PASSTHROUGH
hosts:
- "*"

60. Istio Virtual Service Example
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: teamserver1-basic
spec:
hosts:
- kube.cccco.net
gateways:
- teamserver-gateway
tcp:
- match:
- port: 80
route:
- destination:
host: teamserver1-service
port:
number: 80

61. Virtual Service -Multiple Conditions
61
gateways:
- teamserver-gateway
http:
- match:
- headers:
user-agent:
regex: "Trident/7.0;srv:11.0"
uri:
prefix: "/zC"
route:
- destination:
host: teamserver1-service
- route:
- destination:
host: www.google.com

62. StringMatch for HTTP Headers
Field Type Description
exact string (oneof ) exact string match
prefix string (oneof ) prefix-based match
regex string (oneof ) ECMAscript style regex-based match
Case sensitive

63. Istio ingressgateway Tips
◎A selector is used by istio to select the ingressgateway.
This is important when there are multiple
ingressgateways
◎istio uses port naming in some routing logic
◎Port names are of the form protocol-suffix with grpc,
http, http2, https, mongo, redis, tcp, tls or udp as the
protocol.

64. Routing to External Destinations
◎Envoy passthrough to external services
○ On by default
○ global.outboundTrafficPolicy.mode option set to ALLOW_ANY
○ kubectl get configmap istio -n istio-system -o yaml | grep -o "mode:
ALLOW_ANY"

65. Notes on Ingress Capabilities
◎A mesh can have any number of gateways, and multiple different implementations of the
gateway can coexist
◎The Kubernetes Ingress API cannot express the routing needs of Istio. Kubernetes Ingress
looks for a common intersection between different HTTP proxies. It only supports the most
basic of HTTP routing.
◎Kubernetes Ingress itself does not support the TCP protocol. Kubernetes Ingress cannot be
set up to configure an NGINX Ingress Controller for TCP load balancing – requires a special
configmap
◎Istio Gateway has overcome the above shortcomings of Ingress by separating the L4-L6
configuration from the L7 configuration
◎Gateway is only used to configure L4-L6 functions
○ exposed ports, TLS configuration
◎A VirtualService is bound a Gateway in order to control inbound TCP and HTTP traffic

66. Istio Ingress Multiple SNI Hosts (Cobalt Strike)
Beacon
ls1.ccco.net:443
ls2.ccco.net:8443
Teamserver1-service
Istio Ingress

67. Istio Ingress Multiple SNI Hosts (cont)

68. TLS Ingress Gateway Non-Passthrough
Create a Kubernetes secret to hold the server’s certificate and private key.
Use kubectl to create the secret istio-ingressgateway-certs in namespace istio-
system . The Istio gateway will load the secret automatically.
create -n istio-system secret tls istio-ingressgateway-certs –key key.pem --cert cert.pem
Create a Gateway and a VirtualService with the desired routing rules

69. Istio Gateway

70. Istio TLS VirtualService

71. Istio TLS Ingress Gateway Multiple Hosts
Create the secret for the additional backend
kubectl create -n istio-system secret tls istio-ingressgateway-teamserver2service-certs --key key.pem --cert .cert.pem
To include a volume mounted from the new created secret, patch the istio-ingressgateway deployment:
patch.json
kubectl -n istio-system patch --type=json deploy istio-ingressgateway -p "$(cat patch.json)"

72. Traefik

73. Installing Traefik
◎https://docs.traefik.io/v2.0/user-
guides/crd-acme/
◎Deploy Cluster Resources – CRDs
○ IngressRoute and the Middleware kinds
○ RBAC definitions
◎Traefik Service
◎Traefik Deployment
◎Traefik Ingress Routes

74. Traefik Services

75. Traefik Deployment

76. Traefik Entrypoints

77. Drone
◎Drone is a modern CI/CD platform built with
containers as first class citizens. Pipelines are
configured using a yaml file that you check-in to your
git repository.
◎Can be deployed as a Docker container to a VM or a
Kubernetes cluster
◎Integrates with git repository and can deploy pipelines
to a Kubernetes cluster

78. Using Drone
78
pipeline:
build:
image: gradle:4.5.1-jdk9
commands: gradle --help
docker:
image: plugins/docker
secrets: [ docker_username, docker_password ]
repo: komljen/test-image
tags: ${DRONE_COMMIT_SHA}
kubectl:
image: komljen/drone-kubectl-helm
secrets: [ kubernetes_server, kubernetes_cert,
kubernetes_token ]
kubectl: "get pods"
helm:
image: komljen/drone-kubectl-helm
secrets: [ kubernetes_server, kubernetes_cert,
kubernetes_token ]
helm: list
notify:
image: plugins/slack
channel: drone-notification
secrets: [ slack_webhook ]

79. Monitoring C2 with Prometheus
◎Open source metrics based monitoring system
◎Can be used to instrument application
◎Applications without instrumentation support can be monitored
using exporters and other legacy methods
○ extract whitebox monitoring data from application logs for collection
in a timeseries database
○ Google mtail
○ Grok Exporter
○ JMX (Cobalt Strike teamserver is a Java application so theoretically it
could be supported)
◎C2 authors and developers can instrument code to provide fine
grained monitoring support

80. Google mtail
◎Can be used to create timeseries data from standard logs
○ https://github.com/google/mtail/blob/master/docs/Programming-
Guide.md
◎Prometheus exporter is available
◎Cobalt Strike teamserver logs are in the Cobalt Strike working
directory:
○ Ie /opt/cobaltstrike/logs
○ Weblogs and beacon session logs are available
◎Use mtail to create counters by matching events in log with
regular expressions

81. Grok Exporter
◎https://github.com/fstab/grok_exporter
◎Can be used to convert arbitrary logs into
Prometheus compatible time series data
◎An example would be the Cobalt Strike
beacon log checkin entry
○ This interval can be collected as a metric to
monitor beacon health
◎Grok Exporter uses same language as

82. Prometheus Visuals (with Sysdig)

83. References and Thanks
◎https://bluescreenofjeff.com/
◎https://istio.io/docs/concepts/traffic-
management/
83

84. Questions?
◎https://github.com/cloudc2/kubered
84