Introduction to Delmo https://github.com/bodymindarts/delmo

1. Local testing of Containerized
Distributed Systems
Strange Group Meetup,
Justin Carter (@bodymindarts)
Berlin, Oct. 27. 2016
Stark & Wayne

2. Delmo
Delmo is a tool to test systems running
within multiple docker containers
https://github.com/bodymindarts/delmo

3. 30 staff, 7 countries
PaaS experts
Cloud Foundry experts

4. Plattforms
Dingo PostgreSQL
Delmo

5. http://www.slideshare.net/KarlIsenberg/cloud-surfing-kubernetes-on-mesos

6. The sophistication of our platform bounds
the granularity of decomposition in our
systems
(also means more complexity to setup locally)

7. How much of the cloud stack do we
want to run locally (if any)?
Platform on dev workstation? Dev env in the cloud?
Apps decoupled from platform?

8. Contract Driven Tests / Mocks add a
lot of code to maintain.
Dealing with deps?
Do we need them?

9. Cloud dev workflow
How much of the stack do we want to run?
How do we want to run our deps?
How do we automate this and integrate
into an efficient workflow?

11. When you can deploy 1000s of apps,
you’d better be able to deploy 1000s
of dbs!
Platform consideration

12. PostgreSQL in Docker
Clustered with streaming replication
Continuously backed up
Automatic failover
Dingo PostgreSQL
https://github.com/dingotiles/dingo-postgresql-release

13. Dingo PostgreSQL Architecture
H
A
P
r
o
x
y
Deployed by
BOSH
Client 1
Client 2

14. Critical Features
Re-configuration of PostgreSQL
Restart PostgreSQL
Streaming backups to S3

15. Changing Docker
Image

17. Test independently of BOSH
Test smaller system boundary
Build and test containers locally
Goals

18. Postgres
Leader
Postgres
Replica
Example Test
Will failover work?
Dingo PG
HAProxy
Query
Leader
etcd

19. Assert
Did data get replicated?
Did reconfiguration / promotion work?
Are connections being accepted?

20. Given: System definition
When: Some failure
Then: Assert expected Behaviour
Test Case

21. Delmo
Declare system with docker-compose
Declare test cases
Run tests (locally or remotely)

22. patroni1
Docker Compose
patroni2
etcd
System Tasks
Tasks are
packaged bash
scripts that
execute
commands
against the
running system

23. System
Tasks
Cache

24. Delmo Config

25. Meta
Tests
Tasks

26. Demo!
https://github.com/bodymindarts/delmo-demo

27. Initialise
Start System
Run Tasks
Cleanup
Summary

28. More
Run on a remote docker-machine
Run tasks in parallel
delmo-concourse image
(for executing in CI/CD)

29. Next Steps
Better error output / validation
Network failure / partitioning
Support for other backends (bosh)?

30. Questions?
Justin Carter
@bodymindarts
github.com/bodymindarts
github.com/bodymindarts/delmo