Evolution of Our Software Architecture

1. EVOLUTION OF
OUR SOFTWARE
ARCHITECTURE
Presented at
Aarhus Clojure Meetup, June 2017
by
Paul Lam (@Quantisan),VP Engineering, Motiva

2. A BRAIN FOR CAMPAIGN
• Motiva AI is a messaging
platform that optimizes
messaging campaigns and
improves targeting,
conversions, and
engagement - automatically

3. – https://en.wikipedia.org/wiki/Software_architecture
“Software architecture refers to
the high level structures of a software system,
the discipline of creating such structures, and
the documentation of these structures.”

4. ARCHITECTURE EVOLUTION
• How can a system be adapted to support
uncertain business goals?
• How resilient is the system to planned and
unplanned change?
• How can the quality of a deficient architecture be
improved?
http://www.sei.cmu.edu/architecture/research/previousresearch/evolution.cfm

5. WHY NOT BUILD FROM
SCRATCH EVERYTIME?
Cost
Percentage of Architecture Re-use
But! https://martinfowler.com/bliki/SacrificialArchitecture.html

7. • LAYERED
• MICROSERVICES
• EVENT-DRIVEN
• MICROKERNEL
• SPACE-BASED

8. JUST-IN-TIME
DEVELOPMENT

9. LAYERED MONOLITH

10. Database
Frontend
Business Logic
Persistence
Presentation

11. D
F
Bus
Per
Pre

12. D
F
Bus
Per
Pre TESTING

13. D
F
Bus
Per
Pre DEPLOY

14. MACHINE LEARNING?
D
F
Bus
Per
Pre

15. COUNTING
WITH CLOJURED
F
Bus
Per
Pre
https://github.com/nathanmarz/specter

16. INTERACTIVE
DEVELOPMENTD
F
Bus
Per
Pre

17. LAYERED MONOLITH
• 3 months from nothing to production
• Easy to test
• Easy to deploy
• Not one-size fit all
• Easy to get complicated
D
F
Bus
Per
Pre

18. MICRO-SERVICES

19. OUR PROBLEMS
1. Machine learning research to production process
is disconnected and duplicated
2. Interface points with external APIs are
complicated and fragile

20. MACHINE LEARNING
AS AN API

21. CODE SHARING:
RESEARCH AND PRODUCTION
Jupyter Notebook
$ pip install burt
import burt
b = burt.Burt(coll)
b.getExperimentBatch()
...
Production Server
$ pip install burt
import burt
b = burt.Burt(coll)
b.getExperimentBatch()
...

22. Database
Frontend
Business Logic
Persistence
Presentation

23. Databases
Frontend
Business Logic
Persistence
Presentation Machine Learning
Persistence
External API
Persistence

24. MICRO-SERVICES
• Using the right stack for the right task
• Easy to change
• Easy to develop
• Design overhead
• Multi-step orchestration getting tedious

25. EVENT-DRIVEN

26. NEW PROBLEM
• Increasing number of business logic steps
• Introduce self-recovery on flaky steps

27. Databases
Frontend
Business Logic
Persistence
Presentation Machine Learning
Persistence
External API
Persistence

28. Frontend
Event Mediator
Presentation Machine Learning
Persistence
External API
Persistence
Event 2 Processor
Persistence
Event 3 Processor
Persistence
Event 1 Processor

29. EVENT MEDIATOR

30. DEAD LETTER QUEUE
FOR RECOVERY
Sender Worker
Queue
Task failed
Dead letter queue
Intended Receiver

31. EVENT-DRIVEN
• Robust
• Scalable
• Easy to change
• Integration tests not easy
• Not easy to develop

32. SUMMARY
• Evolve system with just-in-time development
• Start simple and let demand drive architecture
• Divide and conquer

33. CREDITS
• Jonathan Irving, github.com/j0ni
• Matt Hill, @areamatt
• Chris Diehl, @chrisdiehl

34. CONTACT
Paul Lam
@Quantisan
github.com/Quantisan
paul@motiva.ai