Basic principles of 2nd half thinking in microservices and how "sidecar" systems need to be part of the design considering how this new microservice is going to affect its eco-system. Logging is a great example and we demonstrate how to plan one, without missing the basics ...

1. FullStack Developers Israel
SCALING I/O BOUND
MICRO SERVICES
Haggai Philip Zagury |
DevOps Group & Tech Lead @ Tikal Knowledge

2. FullStack Developers Israel
TAGGING IS SCALING

3. FullStack Developers Israel
SCALING I/O

4. FullStack Developers Israel
SCALING I/O
FIRE FIGHTING …

5. FullStack Developers Israel
SCALING I/O

6. FullStack Developers Israel
SCALING I/O
A E-MAIL SYSTEM
~60,000 mailboxes
~5 -> ~40 servers

7. FullStack Developers Israel
SCALING I/O
A E-MAIL SYSTEM
~60,000 mailboxes
~5 -> ~40 servers

8. FullStack Developers Israel
SCALING I/O
▸ Terminal multiplexer
~5 -> ~40 SERVERS IN 1Y

9. FullStack Developers Israel
SCALING I/O
▸ Terminal multiplexers
~5 -> ~40 SERVERS IN 1Y
tail -f /var/log/messages
tail -f /var/log/mail.log

10. FullStack Developers Israel
SCALING I/O
▸ Mail Size => Attachment Size
▸ As mail passed through
- Attachment size …
- Storage size
- “Sliding Windows”
LIVING WITH I/O LIMITS
postconf -e message_size_limit=52428800
postconf -e message_size_limit=26214400
postconf -e message_size_limit=10485760

11. FullStack Developers Israel
SCALING I/O
▸ Mail Size => Attachment Size
▸ As mail passed through
- Attachment size …
- Storage size
- “Sliding Windows”
LIVING WITH I/O LIMITS
postconf -e message_size_limit=52428800
postconf -e message_size_limit=26214400
postconf -e message_size_limit=10485760

12. FullStack Developers Israel
SCALING I/O
USE THE RIGHT TOOL STORE FOR THE JOB

13. FullStack Developers Israel
DemoLand

14. FullStack Developers Israel
SCALING I/O
10.10.10.310.10.10.2 10.10.10.4
default
demo-app

15. FullStack Developers Israel
SCALING I/O
10.10.10.310.10.10.2 10.10.10.4
default
demo-app
logging

16. FullStack Developers Israel
SCALING I/O
# multi-az cluster 3 master 3 nodes
kops create cluster ${NAME}
--node-count 3
--zones ${AWS_REGION}a,${AWS_REGION}b,${AWS_REGION}c
--master-zones ${AWS_REGION}a,${AWS_REGION}b,${AWS_REGION}c
--node-size t2.medium
--master-size t2.medium
SETUP - AUTOMATE !

17. FullStack Developers Israel
SCALING I/O
export NAME=ms-at-scale.tikal.io
# single-az cluster 3 master 3 nodes
kops create cluster --name ${NAME}
--node-count 3
--master-count 3
--zones ${AWS_REGION}b
--master-zones ${AWS_REGION}b
--node-size t2.medium
--networking weave
--cloud-labels "Team=Scaling-ws,Owner=Haggai Philip Zagury"
--master-size t2.medium
SETUP - AUTOMATE - DON’T FORGET TO TAG …

18. FullStack Developers IL
SCALING I/O

19. FullStack Developers Israel
Polycloud Strategy
May 2020

20. FullStack Developers Israel
TEXT
Logs, Logs , Logs …

21. FullStack Developers Israel
SCALING I/O
THE “LOG-SHIPPING” …

22. FullStack Developers Israel
SCALING I/O
THE “LOG-SHIPPING” …

23. FullStack Developers Israel
SCALING I/O
THE “LOG-STASHING” …

24. FullStack Developers Israel
SCALING I/O
▸ Planning I/O in the cloud
▸ Instance Types
▸ Block Devices
▸ Storage Type (Temp)
THE “LOG-STASHING” STORY^n … IN THE CLOUD ….

25. FullStack Developers Israel
SCALING I/O
▸ Planning I/O in the cloud
▸ Instance Types - TAG!
▸ Block Devices - TAG!
▸ Storage Type (Temp) - TAG!
THE “LOG-STASHING” STORY^n … IN THE CLOUD ….

26. FullStack Developers Israel
TEXT

27. FullStack Developers Israel
TEXT

28. FullStack Developers Israel
SCALING I/O
10.10.10.310.10.10.2 10.10.10.4
default
demo-app
logging

29. FullStack Developers Israel
SCALING I/O
10.10.10.310.10.10.2 10.10.10.4
default
demo-app
logging

30. FullStack Developers Israel
DAS (Direct Attached Storage) NAS (Network Attached Storage) SAN (Storage Area Network)
NAS
(NAS)
SAN
STORAGE TOPOLOGIES

31. FullStack Developers Israel
tail -f /var/log/messages
DAS (Direct Attached Storage)
STORAGE TOPOLOGIES

32. FullStack Developers Israel
DAS (Direct Attached Storage) NAS (Network Attached Storage) SAN (Storage Area Network)
NAS
(NAS)
SAN
STORAGE TOPOLOGIES

33. FullStack Developers Israel
SCALING I/O
PLANNING A DISTRIBUTED (STORAGE)
Analytics Acct
▸ Who pays for what ?
▸ Tags Tags Tags …

34. FullStack Developers Israel
TAGS == DATA ORIGINS & PRESENTATION
REGION POD INSTANCE *
}App Metrics
Log Data
Filter Tags & Alerts

35. FullStack Developers Israel
SCALING I/O
LOCAL DISK (NVME)
▸ time dd if=/dev/urandom of=/mnt/localtest bs=1M count=8192 status=progress
8584691712 bytes (8.6 GB, 8.0 GiB) copied, 112 s, 76.6 MB/s
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 112.066 s, 76.7 MB/s
real 1m52.068s
user 0m0.000s
sys 0m44.859s

36. FullStack Developers Israel
SCALING I/O
LOCAL DISK (NVME)
▸ mkdir /mnt/test && docker run -it --rm -v "/mnt/test:/test" archlinux /bin/bash
time dd if=/dev/urandom of=/containertest/file bs=1M count=8192 status=progress
8550088704 bytes (8.6 GB, 8.0 GiB) copied, 113 s, 75.7 MB/s
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 113.618 s, 75.6 MB/s
real 1m53.619s
user0m0.000s
sys 0m45.364s

37. FullStack Developers Israel
SCALING I/O
IPERF (BANDWIDTH CHECK) { EC2->EC2 }
▸ server1 -> iperf -s
▸ server2 -> iperf -c server1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.14.168 port 5001 connected with 172.31.8.221 port 39606
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 11.3 GBytes 9.68 Gbits/sec

38. FullStack Developers Israel
SCALING I/O
IPERF (BANDWIDTH CHECK) { EC2->CONTAINER ON REMOTE HOST }
▸ server1 -> iperf -s
▸ server2 -> iperf -c server1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.14.168 port 5001 connected with 172.31.8.221 port 39606
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 11.3 GBytes 9.68 Gbits/sec

39. FullStack Developers Israel
SCALING I/O
IPERF (BANDWIDTH CHECK) { CONTAINER->CONTAINER ON REMOTE HOST }
▸ server1 -> iperf -s
▸ server2 -> iperf -c server1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 4] local 172.31.14.168 port 5001 connected with 172.31.8.221 port 39606
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 11.3 GBytes 9.68 Gbits/sec

40. FullStack Developers Israel
Haggai Philip Zagury |
DevOps Group & Tech Lead @ Tikal Knowledge