4. 44
ibotta is a rewarded
shopping app built here
in Denver.
The backend began life
as a Ruby on Rails
monolith built on AWS
five years ago
5. Base Architecture
5
● At its most fundamental, the Ibotta backend
was comprised of two systems:
○ Receipt processing via OCR
○ A set of API’s to power a mobile
experience
8. Base Architecture - in
8
meijer
$1 apl
$2 oj
total
$3
Receipt
Processor
OCR
RoR Monolith
MySQL
9. Base Architecture - out
9
Offer / Bonus
Matching
Ledgers
RoR Monolith
MySQL
Notifications
10. Base Architecture -
scale?
10
So that’s simple enough…
But how do you normalize data? Per-retailer processors
And then - you start getting ‘clever’.
Optimizations for specific paths, relying on data structured by
circumstance, relying on global state, tribal knowledge grows...
11. Base Architecture -
scale?
11
So our nice lasagna has managed to
turn into spaghetti after all.
And worse, there are now hidden side-
effects and dependencies.
credit: @benorama
12. Motivation For Microservices
12
A growing company leads to:
● Growing customer base
● Growing code base
● Growing developer base
Using microservices we can:
● Build/scale services independently
● Reduce problem space for individual developers
● Develop features faster
13. Initial Attempts
13
We’re an AWS shop, so let’s try an AWS
product!
Enter Elastic Beanstalk
Docker and environment support
15. Initial Attempts
15
👎
no ability to do blue-green deploys
deploys had to be done in batches
batch failure -> total rollback
no “for free” sidecar containers (helm)
17. Mono → Micro
17
So we want to have our lovely ravioli-architecture
THIS WILL BE A LONG, DIFFICULT PROCESS!
YOU WILL NOT BE CHURNING OUT FEATURES!
18. Mono → Micro
18
OK great, you’ve decided to go to an SOA.
This means all your problems are now over!
19. Mono → Micro
19
OK great, you’ve decided to go to an SOA.
This means all your problems are now over!
just kidding…
20. Mono → Micro
20
● Agreeing on transport layers (HTTP? GRPC? SQS? MQ?)
● Agreeing on message format (JSON? Proto? Avro?)
● What does “backwards compatibility” mean?
● How do you manage inter-service dependencies?
21. Mono → Micro
21
function calls → network calls
observability and metrics become even more important
22. Mono → Micro
22
PERSONAL OBSERVATION:
µServices and SOA work because of 2 things:
1. the “secret sauce” of tooling
2. excellent monitoring and metrics
27. Starting Kubernetes
27
● Use Kops & Terraform to create cluster on AWS
● Built Staging and Production cluster
● Experiment and upgrade Staging first
● Use Daemonsets for log aggregation & metrics
● Kiam for Iam roles in pods
● Run dashboard with minimal role & service account
token
28. Helm and Service Provisioning
28
● Helm can be considered a “package manager” for K8s
● Package resource YAML in “charts”
● Template support with values.yaml
● Create standard service charts for services
● Initially wrap Helm commands with scripts
● Now using Harness.io - deploy as a service
29. Service Mesh Experience
29
● Experiments with early versions (0.3&4) of Istio were
promising
● Built tooling for Canary header, % traffic switch, metrics
to Datadog
● Issues with Pilot under production load forced us to
remove
● Now Running Nginx ingress controllers
31. Paving the Road to Production
31
we need to help developers become more self
sufficient by working with them to pave a road to
production with tools.
32. Starting New Services
32
Use Yeoman generators to template new services
This includes:
● Example outline service
● Configuration of CI
● Configuration of Deployment requirements
33. Provisioning Infrastructure
33
● We use Terraform to provision AWS resources
● Develop Terraform modules for common use cases
● Base Yeoman generator for the minimum service
● Deployment pipeline process for making changes
● Training “Infrastructure specialists” in dev groups
● Developers write their own service modules
34. Support Libraries
34
Build common libraries in standard languages for:
logging, metrics, security and other shared services
This reduces work for individual services
Promotes a standard approach
Makes it easier to upgrade
Run as internal open source so devs provide PRs
36. Pitfalls
36
● More moving pieces
● Takes time to build platform, tools
● Takes time to change mindset, learn new approach
● More explicit communication required
● Harder to test across services
● Long term service ownership
37. Pitfalls
37
● Coordinating dependencies vs independent work / deploys
● Ensuring that your events are general-purpose and robust
● “Agile” teams vs. maintaining a high-level picture and system
design
● Team structure vs service ownership
39. Benefits
39
● Teams are empowered to do what they need
● Smaller services scale independently, lower blast radius
● Kubernetes provides a lot of resilience
● Easier to take risks, try new things
● Easier to work in cross functional teams
● Easier to align closer with other groups
40. Benefits
40
● UNIX philosophy - do one thing and do it well
○ no need to shoehorn everything into 1 framework
● Isolation, and encapsulation of business logic
○ pseudo-functional approach
○ system comprised of composable units
● “Easy” canarying of individual components