Travelbird Lambda Architecture with not so big data Bringing realtime business intelligence into reality.

1. EventPipe-λArchitecture
w/NotsoBigData
Bahadir Cambel - Senior Data Engineer
@bahadircambel bahadir.io
TravelBird

2. There are 10^11 stars in the galaxy. That used to
be a huge number. But it's only a hundred billion.
It's less than the national deficit! We used to call
them astronomical numbers. Now we should call
them economical numbers. -RF
We only have half a billion events, yet.

3. What’sanEvent?
a thing that happens (takes place), especially one of
importance
- Page view
- Link Click
- Order Save
- Sales Flow
- Mobile App Launch..
- A/B Visit
- Newsletter View
- Scroll
- Ad View
- and many more...

4. What’saPipe?theBADBADONE
- Events flowing in through.
Once upon a time in LinkedIn

5. Alsoaeventpipe(UNIFIED)

6. logslogslogs->Kafka-Kinesis-
Jay Kreps ( Ex-LinkedIn, co-creator Apache Kafka,
Confluent.io founder)
http://bit.ly/realtime-data-logs ( MUST READ - LONG READ -
READ TWICE I’m aiming to read 3rd time)
http://bit.ly/i-love-logs-video

7. What’s λ
Nathan Marz(Storm,Cascalog) come up with the term.

8. EventPIPE-Batchpart

9. EVentPipe-FULLFORceλ

11. Howtocount?
- Counter +1
- Aggregate in batch
- What happens if you need to reprocess data ?
- Page Views, Users, Sessions
- How about Counter’s for Segments
- per page/device/domain/
- How to handle Big DATA? Is it going to perform with
- Million

12. Hyperloglog-HLL
- DVc ( almost accurate )
- Redis has it since 2.8.9 ( PFCOUNT - PFADD )
- 12 KBytes per key up to 2^64 elements per key ( 0.81%
standard-error)

13. CLOJURE
- Lazy
- Concise
- Fast
- Simple
- Perfect for Data processing, distributed computing
- the power of JVM

14. stats

15. OPensource
- https://github.com/travelbird/star-track
- https://github.com/travelbird/kinesis3
- https://github.com/travelbird/modular

16. A/BTests-ExperimentsExperimentsExperiments
It doesn't matter how beautiful your
theory is, it doesn't matter how
smart you are. If it doesn't agree
with experiment, it's wrong. - RF

17. We’reallfoolsofourown
The first principle is that you must not fool
yourself and you are the easiest person to
fool. - RF

18. ORDERinCHaos

19. Thanks
- Andrii Mishkovski (Intro to Clojure)
- Philipp Wassibauer (CTO of TravelBird)
- Egle, Brandon, Niels, Brandon, Rob (Teammates)
- Rich Hickey for Clojure
- Nathan Marz for λArchitecture
- Opensourcedevelopers-You’reamazing