When processing sports betting data, latency matters. Content must be processed in near real-time, constantly, and in a region local to where the customer and the data are. ZeroFlucs uses ScyllaDB in order to provide optimized data-storage local to the customer. In this session, Director of Software Engineering Carly Christensen takes you through how we distribute data, and how we use our recently open-sourced package, Charybdis to facilitate this.

1. Worldwide Local
Latency with ScyllaDB
Carly Christensen, Director of Software Engineering

2. ■ Who are ZeroFlucs and What Do We Do?
■ The ZeroFlucs Process
■ Challenges
■ How ScyllaDB Helped
Agenda

3. Carly Christensen
■ 20 years experience in IT industry
■ Software Development
■ Database Systems
■ Career highlights include:
■ Lead Developer and Head of Trading Solutions at Entain Australia
■ SQL Server Consultant at Wardy IT
Your photo
goes here,
smile :)

4. What is ZeroFlucs?

5. What is ZeroFlucs?
ZeroFlucs primarily provide same-game pricing calculations to the wagering
industry.
■ This allows customers to price bets on correlated outcomes from within the
same match.
■ Our vision is to let sportsbook customers explore their theory of the game:
■ Far more engaging than just “Team to Win” bets.

6. Same-Game Example
Match Winner First Touchdown Total Score
Team A
$1.50
Team B
$2.60
T.S. Wong
$8.00
B. Bhooma
$10.00
S. Gray
$14.00
Over 45.5
$1.90
Under 45.5
$1.90
Total: $14.50

7. But It’s Not That Simple…
To calculate the probability of this outcome as closely as possible, we need to simulate
the game, play by play.
■ These are correlated outcomes, so we use a simulation-based approach to
effectively model the relationships.
■ For example, if a team wins then it’s more likely they’ll score the first, last or any
individual goal.
■ To ensure we cover as much of the probability space as possible a minimum of
50,000 simulations are run for every price change, for every event.

8. How We Do It

9. Our Architecture
content
3rd
party
data
booking transforms
model
simulations
market
generation
query
engine
api

10. Our Architecture
■ Platform has been designed to be cloud-native from the ground up.
■ Software stack runs on Kubernetes
■ We use Oracle Container Engine for Kubernetes (OKE)
■ Over 120 microservices, growing every week.
■ Much of the environment can be managed via CRDs and operators.
■ Languages and Tools
■ Golang and Python
■ GRPC for internal communications
■ Kafka (for at least once message delivery)
■ GraphQL for external-facing APIs

11. Our Challenges

12. Our Goal
■ Our ultimate goal is to be able to process and simulate events fast enough to
provide Same-Game prices for Live (in-play) events.
■ Will this corner result in a goal?
■ Will this play result in a touchdown?
■ Which player will score a touchdown from this play?

13. Our Challenges
■ Price changes must be as up to date as possible.
■ Events can each be updating dozens of times per minute and trigger thousands of re-simulations.
■ We’re processing up to 250,000 in-game events per second.
■ Simulation data can be hundreds of MB.
■ Customers can be anywhere in the world:
■ With each request passing through many microservices, even a small increase in latency between the
service and repository can have a large negative impact on the end-user experience.
■ A lot of optimisation was achieved through code changes and increased
parallelism, however the database remained an area we could improve on.

14. Databases Explored
■ MongoDB
■ Familiar – several of the team had used this in previous projects.
■ Found issues with limited concurrent connections would manifest in simple queries occasionally
taking several seconds.
■ Cassandra
■ Supported network-aware replication strategies.
■ Performance and resource usage was not where we needed it to be.
■ Cosmos DB
■ Addressed all of our performance and regional distribution requirements.
■ High cost, Azure-only – limiting our portability.

15. The Solution:
ScyllaDB

16. Why ScyllaDB?
■ We had trialled ScyllaDB for a previous project and although it didn’t suit that situation, it
was perfect for ZeroFlucs.
■ Distributed architecture – data replicas can be located local to services and customers ensuring low
latency for every request.
■ High throughput and concurrency – with ScyllaDB we have yet to see an issue we can’t scale through.
■ Ease of adoption – ScyllaDB Operator meant we needed little knowledge to start.

17. Architecture with ScyllaDB
■ ScyllaDB Open Source
■ Hosted on Flex 4 VMs
■ Currently performing well, but our throughput increases with every new customer.
■ Option to scale up and run on bare metal if needed in the future.
■ In development we used ScyllaDB Operator to manage ScyllaDB, but use
self-managed in production as operator is limited to a single Kubernetes cluster (hard
to span geographically).
■ We’re still reviewing our strategy around ScyllaDB Manager and monitoring

18. Data Local to Customers
■ Global data
■ Slow-changing data, replicated to every region.
■ Regional data
■ Data used by multiple customers in a region e.g. a sport feed.
■ If a customer in another region needs that same data, it will be
replicated separately to their region.
■ Customer specific data
■ Event configuration, model and simulation outputs, margins and pricing ladders.
■ Customers have a home region, where we store multiple replicas of their data.
■ BONUS: Additional replicas can be made to other regions for DR purposes.

19. Architecture with ScyllaDB
cell
content
books
transforms
simulations
api
UK South (London) UK West (Newport)
cell
content
books
transforms
simulations
api
Australia East (Sydney)

20. Charybdis
■ Drivers:
■ Keeping keyspace configurations up to date between regions would be a challenge.
■ Similar table structures were being repeated over and over through many different services.
■ So we created Charybdis – a Golang ScyllaDB helper library:
■ A table-manager that will automatically create keyspaces and add tables, columns and indexes as
required.
■ Simplified functions for CRUD-style operations.
■ Support for LWT and TTL.

21. Maintenance
Our Topology service controls the location of data replicas on a per-service and/or
per-customer basis - each has a dedicated keyspace per service, per data type.
service topology
CREATE KEYSPACE record_customer_a WITH replication={'class':'NetworkTopologyStrategy'...
CREATE TABLE record_customer_a.records (user_id text) PRIMARY KEY (user_id);
ALTER TABLE record_customer_a.records ADD (first_name text);
ALTER TABLE record_customer_a.records ADD (visits int);
Replication Strategy

22. More to Learn…
■ We’re still early in our journey.
■ For example, our initial attempt at dynamic schema validation caused some requests
between services to time out, if it was the first access of customer specific data for that
instance.
■ There are still many ScyllaDB configuration settings we’ve yet to test and we’re certain that
we can increase performance further.

23. Thank You
Stay in Touch
Carly Christensen
carly@zeroflucs.io
@carlyflucs
www.linkedin.com/in/carly-christensen-16441375/