Game Analytics is building the next-generation analytics platform for the gaming industry. We want to make advanced machine learning available to even the small studios with just three guys in a garage. We want to have all the analytics in real-time.
We're building a real-time stream processing analytical engine in Erlang. As data comes in from our customers games, we process it and make it immediately available to our customers. In this presentation we will present the challenges we're facing, the architecture we have come up with and some really clever implementation details. We're preparing for massive volumes of data and need to be ready when big customers start using our product. Come see the challenges we're facing and how we're using Erlang to build the next-generation analytics platform.
5. Game Analytics
‣
Startup, venture funded, ~2 years old
‣
HQ in Copenhagen, engineering in Berlin
‣
Need to move fast
‣
Willing to take on technical debt
‣
Be ready for traffic growth
‣
Big games are big, millions of DAU
17. Game process
‣
Process files sequentially
‣
Keep running results in RAM
‣
Flush to DB when window closes
‣
Answer real-time queries
‣
Put all in one node
19. HyperLogLog
‣
Estimate cardinality
‣
Clever hash tricks
‣
Millions unique with <1% error in ~40k words
‣
Unions
‣
“hyper”: Erlang HLL++ from Google paper[0]
‣
Will open source Soon (TM)
[0]: “HyperLogLog in Practice: Algorithmic Engineering of
a State of The Art Cardinality Estimation Algorithm”
20. Recordinality[0]
‣
Estimate frequency of values
‣
User session count
‣
Keeps a reservoir, clever hash tricks
‣
Will open source Soon (TM)
[0]: “Data Streams as Random Permutations: the
Distinct Element Problem”