06/18/2014 - Billing & Payments Engineering Meetup @ Netflix
For this Meetup, we have invited speakers from several tech companies to give a series of lightening talks on challenges related to billing & payments systems.
This event is for engineers who are interested in learning more about billing & payments systems. No previous experience with this kind of system is required to attend.
Presenters:
- Mathieu Chauvin - Engineering Manager for Payments @ Netflix
- Taylor Wicksell - Sr. Software Engineer for Billing @ Netflix
- Jean-Denis Greze - Engineer @ Dropbox
- Alec Holmes - Software Engineer @ Square
- Emmanuel Cron - Software Engineer III, Google Wallet @ Google
- Paul Huang - Engineering Manager @ Survey Monkey
- Anthony Zacharakis - Lead Engineer @ Lumos Labs
- Shengyong Li / Feifeng Yang - Dir. Engineering Commerce / Tech Lead Payment @ Electronic Arts
2. • Mathieu Chauvin – Netflix
Validating Payment Information
• Taylor Wicksell – Netflix
Flexible Billing Integrations Using Events
• Jean-Denis Greze – Dropbox
Mistakes (and Wins) Building Payments at Dropbox
• Anthony Zacharakis – Lumosity
Billing Migrations: Maintaining Your (Data) Sanity
• Alec Holmes – Square
Scaling Merchant Payouts at Square
=== Break ===
• Paul Huang – SurveyMonkey
Billing Globalization and Auto Renewal Optimization
• Emmanuel Cron – Google Wallet
Moving from SE to Host Card Emulation
• Feifeng Yang / Michael Chen – Electronic Arts
Ecommerce at EA
• Krishnan Sridhar – LinkedIn
Real-Time Analytics and Smart Routing
24. Common Event Pipeline
Current Implementation
• SQS Entry Point
• Custom routing service
• Custom code for each
endpoint integration
• Weak ordering of events
Future Implementation
• Suro / Kafka Entry Point
• Configurable Routing and
Transformation
• Pluggable endpoint
integration
• Option for strong ordering
of events
29. Mistakes (and Wins) Building
Payments At Dropbox
Jean-Denis Grèze & Dan Wheeler
June 18th, 2014
30. Backend Tips
• Not about increasing conversion
• Not about pricing
• Not about plan and feature optimizations
• Not about upselling
• Not about consumer SaaS at scale
• Not about self-serve in
SMB/SME/Enterprise
31. Pains of Scaling Payments
• Thousands of customers to millions of
customers
• SMB to Enterprise
– Custom flows!
• International expansion
– Fraud
– New payment methods (delayed settlements)
– Different price points
32. Out Of The Dark Ages
• For a long time, only 0.5 engineers
worked on payments and billing
• March 2013: consult w/ leading payment
engineers, PMs and executives on how to
build an amazing payments team
– 15+ in 1 year
33. Advice
• Build a payments + billing backend that is:
– Flexible
• Migrations (sadface)
• Requirements will change – often
– Auditable
• Always know why and state changed
– Append Only
• Never lose data
37. Migrations
• You will have to migrate
– 3rd-party vaulting to self vaulting
– New markets = new processors
– If you are a growing company, your internals will require
migrations
• Stakes are high
– Double-billing? Forgetting to charge some users?
Inadvertently moving users from one pricing category to
another?
• Old way:
– Ad hoc
– Tons of tests
38. I. Leverage Existing Code:
Equivalence
• Write equivalence between old and new
implementations (database, API, 3rd party
providers, tests, etc.)
• Run everything through both systems at
once, with equivalence being tested
• Every step of the way check that equivalence
relations hold (e.g., old-style invoice has a
new-style invoice equivalent)
• Turn off old system when everything works
for X amount of time
39. Migration Pro-Tip
• If you can migrate in both directions at
will on a per-endpoint basis, your life
will be awesome and people will love you.
41. Logging
• Dark Ages = tons of logging
• Very comprehensive, but ad-hoc = too
much effort to re-create state
• Human error
42. II. Automated Logging
• Automatically log
– Any DB write (graph, relational, etc.)
– Any 3rd party API call (and some internal calls like
email)
• Pre-log
– Any incoming 3rd party payload
• Can recreate past actions if we introduced a
regression
• LOG A REASON (and code version)
– 1 year is a long time
44. Not too much data
• Dropbox = large scale for SaaS
• Hundreds of millions of users
(provisioning is hard)
• Tons of data, but still payments data <<
file storage data
– Although we do have the benefits of amazing
infrastructure
45. III. States, Not Deltas
• Generally # states << # deltas
• States
– Pro 100 + Packrat
– DfB + Harmony Enabled
• Deltas
– Add 5 licenses, 6 licenses, etc.
– Add 20 licenses and switch from monthly to yearly
• Use states and let the system figure out how
to get from start to end
46. IV. Possibility & Transitions Use
Same Code Paths
• No difference between:
– entity.is_valid_transition(end_state)
– entity.perform_transition(end_state)
• Except that writes are turned off for the
former.
• No change for “is something possible”
logic to be different than “do the thing”
logic.
47. States Are Nice
transition_space = MoneytreeTransitionsSpace.build_cross_product(
entity=me,
gateways=ALL_GATEWAYS,
plans=[Pro100, Pro200, Pro500, DfB, DfBTrial],
schedules=[Monthly, Yearly],
currencies=ALL_CURRENCIES,
features=ALL_FEATURES,
tax_profile=[NoTax, SimpleTax, ComplexTax],
)
# …
If transition_space.supports(FEATURE_PACKRAT):
# …
48. Write Protection
• Seems dumb, but need to be careful not
to accidentally change values in payments
world. Have clearly-defined code paths
that can touch state, talk to 3rd party
components, etc.
49. V. One More Lesson
• Payments + Billing != Finance
• Business requirements don’t always
translate to what’s best/easiest in the
world of accounting. You need to flexibly
work in both worlds – can’t risk the user
experience to make your finance dep’t
happy (and vice-versa)
• Get a great PM
50. VI. Why Payments Are Cool?
• Infrastructure?
– Payments service
– Provisioning service
• Product?
– Upsells? (+50% increase in revenue per user)
– Gating features?
• Product Infrastructure!
– Build a successful structure by emphasizing
hard problems in both worlds!
51.
52. Now + The Future
• Other Cool Projects
– ML for risk/anomaly detection (e.g., for payment
methods that don’t settle immediately)
– Price AB testing (*)
– Cross-platform upsell framework
• Questions?
– dan@dropbox.com
– jeandenis@dropbox.com
• Hiring
– Get in touch!
58. Lumos Labs, Inc.
Payment system limitations
• Hard to add additional gateways
• Models don’t reflect business reality
• Not built for reporting
• Code is brittle
60. Lumos Labs, Inc.
New payment system features
• Trivial to add new payment methods
• Subscriptions are the core model
• Built with reporting in mind
• Separate, well encapsulated library
65. Lumos Labs, Inc.
Just deprecate the old one
• Don't migrate anyone, let old users
churn out naturally (will take forever)
66. Lumos Labs, Inc.
Just deprecate the old one
• Don't migrate anyone, let old users
churn out naturally (will take forever)
• Migrate everyone, but only most
critical/current subscription info
(loses history)
67. Lumos Labs, Inc.
Just deprecate the old one
• Don't migrate anyone, let old users
churn out naturally (will take forever)
• Migrate everyone, but only most
critical/current subscription info
(loses history)
• Migrate everyone + full history
(tricky, lots of edge cases)
73. Lumos Labs, Inc.
enter the
sanity check
Just make a sanity check before and
after the migration to ask questions
74. Lumos Labs, Inc.
enter the
sanity check
Both models should answer certain
questions the same way, e.g:
• How much did the user pay for a subscription?
• How many total transactions did the user make?
• Was auto-renewal enabled on X date?
75. Lumos Labs, Inc.
class SanityCheck
Methods = [:subscriber?, :transaction_count] # etc.
def initialize(record)
@record = record
@before_values = SanityCheck.values(record)
end
def self.values(record)
Methods.map { |m| [m, record.send(m)] }.to_h
end
end
enter the
sanity check
76. Lumos Labs, Inc.
class SanityCheck
...
def check
@after_values = SanityCheck.values(record)
@diff = diff(@before_values, @after_values)
end
def diff(a, b)
a.delete_if { |k, v| b[k] == v }
.merge!(b.dup.delete_if { |k, v| a.has_key?(k) })
end
end
enter the
sanity check
77. Lumos Labs, Inc.
enter the
sanity check
User.each do |user|
sanity_check = SanityCheck.new(user)
user.migrate!
sanity_check.check
if sanity_check.diff.any?
# sanity check failed -- log an error, rollback, etc.
raise ActiveRecord::Rollback
else
# woo hoo, success!
user.update_attributes(:migrated => true)
end
end
config/initializers/user_auth.rb
81. Lumos Labs, Inc.
Did not work in all cases
● Payment system behavior changed over
time
82. Lumos Labs, Inc.
Did not work in all cases
● Payment system behavior changed over
time
● Some concepts did not map between
systems
83. Lumos Labs, Inc.
Handling the edge cases
● Replayed history as it would play out in
the new system
● Still kept most critical information the
same (e.g. transaction timestamps)
84. Lumos Labs, Inc.
Skip ahead to today
Migrated 99.9994%* of all our users
successfully to the new system
*The remaining .0006% live on for business, not technical reasons
99. 90MUnique visitors
every month
17K
New sign-ups daily
“SurveyMonkey turns online
surveys into a hot business.”
“ Start-up companies using
‘freemium’ business models,
including SurveyMonkey,
are thriving as the cost of
computer power and storage
falls.”
One of the hottest
startups to watch.
2.4MSurvey responses
are generated daily
SurveyMonkey is the world’s largest survey company
101. Billing Globalization
In 2014...
Currencies: 39 international currencies.
Payment Methods: Credit Card, PayPal, Debit Card, Bank Transfer, iTunes, Invoice.
Pricing: each package with different price per currency, multiple prices allowed for
price testing.
Revenue: $$$$ MM
102. Auto-Renewal Optimization
Retry Logic: retry failed payments at different intervals, ~60% retry success rate
Account Updater: get users’ updated credit card data (number / expiration date)
before next renewal date, ~4% users at 96% success rate
Immediate Renewal: charge pending invoices when users update payment accounts,
~4% retry success rate improvement
104. Upcoming Billing Projects in 2014
VAT 2015: preparations for VAT rule changes in Europe in 2015
Brazil Payment Processing: set up local entity, integrate with a new Payment Service
Provider in Brazil
Continue Improving Retry Logic: increase retry frequencies