1.
FULLSTACK TECH RADAR DAY
The #2nd_half
Scaling to the #next2
Haggai Philip Zagury, DevOps Group & Tech Lead

2.
FULLSTACK TECH RADAR DAY
Who we are
Tikal helps ISV’s in Israel & abroad in their technological
challenges.
Our Engineers are Fullstack Developers with expertise in Android,
DevOps, Java, JS, Python, ML
We are passionate about technology and specialise in
OpenSource technologies.
Our Tech and Group leaders help establish & enhance existing
software teams with innovative & creative thinking.
https://www.meetup.com/full-stack-developer-il/

3.
FULLSTACK TECH RADAR DAY
Haggai Philip Zagury - DevOps Group & Tech lead
My open thinking and open techniques ideology is driven by
Open Source technologies and the collaborative manner defining
my M.O.
My solution driven approach is strongly based on hands-on and
deep understanding of Operating Systems, Applications stacks and
Software languages, Networking, Cloud in general and today more
and more Cloud Native solutions.

4.
FULLSTACK TECH RADAR DAY
Planning a data pipeline (with constraints)
● An event driven application

5.
FULLSTACK TECH RADAR DAY
Planning a data pipeline (with constraints)
● An event driven application
● High Resolution Video Streams from multiple locations
● Plan for 10s of simultaneous sources …
Digest

6.
FULLSTACK TECH RADAR DAY
Planning a data pipeline (with constraints)
● An event driven application
● High Resolution Video Streams from multiple locations
● Plan for 10s of simultaneous sources …
● Data enrichment pipelines
Digest Enrich

7.
FULLSTACK TECH RADAR DAY
Planning a data pipeline (with constraints)
● An event driven application
● High Resolution Video Streams from multiple locations
● Plan for 10s of simultaneous sources …
● Data enrichment pipelines
● Output an HD display in near real-time (+- 8sec)
Digest Enrich

8.
FULLSTACK TECH RADAR DAY
@scale - point #1
● The board emphasises a perquisite that
we have unlimited resources to
complete the task …
● Which isn’t the case in real-life …

9.
FULLSTACK TECH RADAR DAY
@scale - point #2
● If you have a consistent measurable unit
● you can plan your journey to the next
square either by:
● Being more Efficient
● Doubling* your compute power
(not accurate will explain later why)

10.
FULLSTACK TECH RADAR DAY
Planning a data pipeline - starting point
● The scale is not predictable …
● As you grow so does your supporting
services …
● Dealing with debug logs in a section
of a cluster means 6 times network
traffic

11.
FULLSTACK TECH RADAR DAY
Planning a data pipeline - reality
● The scale is not predictable …
● As you grow so does your supporting
services …
● Dealing with debug logs in a section
of a cluster means 6 times network
traffic
● Deployment complexity increases …

12.
FULLSTACK TECH RADAR DAY

13.
FULLSTACK TECH RADAR DAY
The cloud is your friend
● Managed Services
● Solving Complexity of operation
● Increases/Decreases Price (in most
cases)
● Pay as You go …
● Polyglottism / “Architecture captivity“

14.
FULLSTACK TECH RADAR DAY
Solution

15.
FULLSTACK TECH RADAR DAY
Solution
How to store data for
optimised
r/w procedures
55,000 read requests per second
Hitting S3 Limit on >=10 simultaneous streams

16.
FULLSTACK TECH RADAR DAY
Solution
How to store data for
optimised
r/w procedures
55,000 read requests per second
Hitting S3 Limit on >=10 simultaneous streams
# Complexitynext2

17.
FULLSTACK TECH RADAR DAY
Solution

18.
FULLSTACK TECH RADAR DAY
Solution
Performance ?

19.
FULLSTACK TECH RADAR DAY
Solution
Performance ?
#2nd_half ?

20.
FULLSTACK TECH RADAR DAY
VM vs Container*
Performance ?
Less ComplexityMore Complexity Gpu/Tpu

21.
FULLSTACK TECH RADAR DAY
Solution

22.
FULLSTACK TECH RADAR DAY
Solution
* is on the network !

23.
FULLSTACK TECH RADAR DAY
Solve drift … with “money”
● 1 frame (of 45MB) * 30 frames per sec = 1350 MB == 1.35 GB == 10.8 Gb
> 10 Gb

24.
FULLSTACK TECH RADAR DAY
EC2 Network performance comparison
● Worth a read on ec2 network performance
○ https://docs.google.com/spreadsheets/d/1N2xQqry-
zAKnK6FtW8X5zBYhMiFFnuMySMpx7f3K60s/edit?usp=sharing
○ https://cloudonaut.io/ec2-network-performance-cheat-sheet/

25.
FULLSTACK TECH RADAR DAY
Solution
* is on the network !

26.
FULLSTACK TECH RADAR DAY
Solution
Multi-tenancy ?
who did what, when ?

27.
FULLSTACK TECH RADAR DAY
Resource Tags are your friend !
● Tagging resources
● Putting price tags on operations !
● How much did pipeline1 cost ?
● Can I do it faster ? [ Time ]
● Each enrichment service has it’s own
cluster optimised for Compute unit sizes
staging
stream
infra

28.
FULLSTACK TECH RADAR DAY
Solution

29.
FULLSTACK TECH RADAR DAY
#2nd_half or # issues …next2
Justin Ryan
https://qconsf.com/sf2019/presentation/scaling-patterns-netflixs-edge
Scaling Patterns for Netflix's Edge

30.
FULLSTACK TECH RADAR DAY
#2nd_half or # issues …next2
https://www.infoq.com/presentations/efficiency-better-software-faster/
Todd Montgomery
Todd Montgomery explores the everyday things that those with an eye to
performance and efficiency do that can be leveraged by anyone to build better software faster.

31.
FULLSTACK TECH RADAR DAY
Compute
Buffet ;)
● Taken @SFO
Nov 2019

32.
UNDER
THE
HOOD

33.
FULLSTACK TECH RADAR DAY
So building distributed systems is hard

34.
FULLSTACK TECH RADAR DAY
So building distributed systems is hard

35.
FULLSTACK TECH RADAR DAY
The #2nd half/ # solution(s)next2
● What is the 2nd half of the table problem(s)
● ***** real-life scenario !
● Scale ability -> Capacity planning in the “Container Orchestration Era”
● (in the past Marathon) Kubernetes
● FaaS (e.g. AWS Lambda)
● Nomad

36.
FULLSTACK TECH RADAR DAY
Lego for the masses
● Solves Complexity (i know debatable …)
● 1 size fit all
background image by @hagzag

37.
FULLSTACK TECH RADAR DAY
Commoditize compute unit management
kubectl get deploy -n kube-system
cert-manager 1/1 1 1 58d
cert-manager-cainjector 1/1 1 1 58d
cert-manager-webhook 1/1 1 1 58d
coredns 2/2 2 2 173d
enternal-dns-prod-external-dns 1/1 1 1 171d
kube-slack 1/1 1 1 173d
tiller-deploy 1/1 1 1 173d

38.
FULLSTACK TECH RADAR DAY
Resource Limits apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: db
image: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: "password"
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"

39.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: test-network-policy
namespace: default
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- ipBlock:
cidr: 172.17.0.0/16
except:
- 172.17.1.0/24
- namespaceSelector:
matchLabels:
project: myproject
- podSelector:
matchLabels:
Resource Limits

40.
FULLSTACK TECH RADAR DAY
Time Days Minuets Seconds Milliseconds
Cost
Agility
Scalability
https://www.slideshare.net/BryanMcAninch/the-faas-and-the-curious-86874211

41.
FULLSTACK TECH RADAR DAY
Make the best of what you have ?
● Be more accurate / precise in what you do !
● Measuring techniques (that scale)

42.
FULLSTACK TECH RADAR DAY
Guess what ?
They both get to Eilat !

43.
FULLSTACK TECH RADAR DAY
Keep Efficiency in mind
Time
Price
Complexity
Define your Quantifiable unit !

44.
FULLSTACK TECH RADAR DAY
F
● A Fixed unit size Grid == FaaS

45.
FULLSTACK TECH RADAR DAY
Solving complexity …

46.
FULLSTACK TECH RADAR DAY
Load Shifting & Peak Shaving

47.
FULLSTACK TECH RADAR DAY
Keep Efficiency in mind
Time
Price
Complexity
Define your Quantifiable unit !
Thank you for taking the Time !