Enviar pesquisa
Carregar
Matt Youill Betfair
•
1 gostou
•
1,994 visualizações
D
deimos
Seguir
Negócios
Diversão e humor
Denunciar
Compartilhar
Denunciar
Compartilhar
1 de 63
Baixar agora
Baixar para ler offline
Recomendados
VF 1999AR
VF 1999AR
finance31
Takeover panorama may 2007 2007-05-01
Takeover panorama may 2007 2007-05-01
Corporate Professionals
Investor presentation full
Investor presentation full
Multiplus
Maximizing Sales from eCommerce Sites (Forte Consultancy Group)
Maximizing Sales from eCommerce Sites (Forte Consultancy Group)
Berkin Ozmen
Daily equity-report by epic research 8 feb 2013
Daily equity-report by epic research 8 feb 2013
Epic Daily Report
Public notices for Oct. 3, 2012
Public notices for Oct. 3, 2012
Post-Bulletin Co.
Bpr blaq n_gels_final_pt
Bpr blaq n_gels_final_pt
동진 문
120815 investor presentation full
120815 investor presentation full
Multiplus
Recomendados
VF 1999AR
VF 1999AR
finance31
Takeover panorama may 2007 2007-05-01
Takeover panorama may 2007 2007-05-01
Corporate Professionals
Investor presentation full
Investor presentation full
Multiplus
Maximizing Sales from eCommerce Sites (Forte Consultancy Group)
Maximizing Sales from eCommerce Sites (Forte Consultancy Group)
Berkin Ozmen
Daily equity-report by epic research 8 feb 2013
Daily equity-report by epic research 8 feb 2013
Epic Daily Report
Public notices for Oct. 3, 2012
Public notices for Oct. 3, 2012
Post-Bulletin Co.
Bpr blaq n_gels_final_pt
Bpr blaq n_gels_final_pt
동진 문
120815 investor presentation full
120815 investor presentation full
Multiplus
Search engines, SEO and SEM
Search engines, SEO and SEM
Vinay Shriyan, Certified ScrumMaster® (CSM)
Spy v Spy - Treachery in the Dev/Ops Trenches
Spy v Spy - Treachery in the Dev/Ops Trenches
bloodredsun
Mis11e ch02
Mis11e ch02
nghoanganh
Betfair + Couchbase
Betfair + Couchbase
bloodredsun
How to Sell Social Media to the C Suite _ Shashi Bellamkonda
How to Sell Social Media to the C Suite _ Shashi Bellamkonda
Shashi Bellamkonda
TRANSICION TERRENO
TRANSICION TERRENO
guest0ea344
How2 Start Ocw
How2 Start Ocw
Terri Bays
My Presentation for PYP2013
My Presentation for PYP2013
Michael Street
Exercícios de Trigonometria Resolvidos
Exercícios de Trigonometria Resolvidos
Flaber Bertochi
Satio手机
Satio手机
Liu Xing
Usagility
Usagility
superduperfly
IA, UX and SEO
IA, UX and SEO
Ian Lurie
Joe Walker Interactivewebsites Cometand Dwr
Joe Walker Interactivewebsites Cometand Dwr
deimos
The new framework
The new framework
iansillett
Newcomers Breakfast
Newcomers Breakfast
Terri Bays
Realtime communication over a dual stack network
Realtime communication over a dual stack network
Olle E Johansson
長尾理論
長尾理論
maomao
Java Script
Java Script
Liu Xing
Imagenes Curiosas
Imagenes Curiosas
Descojonate
Introduction to Git
Introduction to Git
Rick Umali
Aspect Orientated Programming in Ruby
Aspect Orientated Programming in Ruby
deimos
Randy Shoup eBays Architectural Principles
Randy Shoup eBays Architectural Principles
deimos
Mais conteúdo relacionado
Destaque
Search engines, SEO and SEM
Search engines, SEO and SEM
Vinay Shriyan, Certified ScrumMaster® (CSM)
Spy v Spy - Treachery in the Dev/Ops Trenches
Spy v Spy - Treachery in the Dev/Ops Trenches
bloodredsun
Mis11e ch02
Mis11e ch02
nghoanganh
Betfair + Couchbase
Betfair + Couchbase
bloodredsun
How to Sell Social Media to the C Suite _ Shashi Bellamkonda
How to Sell Social Media to the C Suite _ Shashi Bellamkonda
Shashi Bellamkonda
TRANSICION TERRENO
TRANSICION TERRENO
guest0ea344
How2 Start Ocw
How2 Start Ocw
Terri Bays
My Presentation for PYP2013
My Presentation for PYP2013
Michael Street
Exercícios de Trigonometria Resolvidos
Exercícios de Trigonometria Resolvidos
Flaber Bertochi
Satio手机
Satio手机
Liu Xing
Usagility
Usagility
superduperfly
IA, UX and SEO
IA, UX and SEO
Ian Lurie
Joe Walker Interactivewebsites Cometand Dwr
Joe Walker Interactivewebsites Cometand Dwr
deimos
The new framework
The new framework
iansillett
Newcomers Breakfast
Newcomers Breakfast
Terri Bays
Realtime communication over a dual stack network
Realtime communication over a dual stack network
Olle E Johansson
長尾理論
長尾理論
maomao
Java Script
Java Script
Liu Xing
Imagenes Curiosas
Imagenes Curiosas
Descojonate
Introduction to Git
Introduction to Git
Rick Umali
Destaque
(20)
Search engines, SEO and SEM
Search engines, SEO and SEM
Spy v Spy - Treachery in the Dev/Ops Trenches
Spy v Spy - Treachery in the Dev/Ops Trenches
Mis11e ch02
Mis11e ch02
Betfair + Couchbase
Betfair + Couchbase
How to Sell Social Media to the C Suite _ Shashi Bellamkonda
How to Sell Social Media to the C Suite _ Shashi Bellamkonda
TRANSICION TERRENO
TRANSICION TERRENO
How2 Start Ocw
How2 Start Ocw
My Presentation for PYP2013
My Presentation for PYP2013
Exercícios de Trigonometria Resolvidos
Exercícios de Trigonometria Resolvidos
Satio手机
Satio手机
Usagility
Usagility
IA, UX and SEO
IA, UX and SEO
Joe Walker Interactivewebsites Cometand Dwr
Joe Walker Interactivewebsites Cometand Dwr
The new framework
The new framework
Newcomers Breakfast
Newcomers Breakfast
Realtime communication over a dual stack network
Realtime communication over a dual stack network
長尾理論
長尾理論
Java Script
Java Script
Imagenes Curiosas
Imagenes Curiosas
Introduction to Git
Introduction to Git
Mais de deimos
Aspect Orientated Programming in Ruby
Aspect Orientated Programming in Ruby
deimos
Randy Shoup eBays Architectural Principles
Randy Shoup eBays Architectural Principles
deimos
Remy Sharp The DOM scripting toolkit jQuery
Remy Sharp The DOM scripting toolkit jQuery
deimos
Ola Bini J Ruby Power On The Jvm
Ola Bini J Ruby Power On The Jvm
deimos
Aslak Hellesoy Executable User Stories R Spec Bdd
Aslak Hellesoy Executable User Stories R Spec Bdd
deimos
Venkat Subramaniam Building DSLs In Groovy
Venkat Subramaniam Building DSLs In Groovy
deimos
Venkat Subramaniam Blending Java With Dynamic Languages
Venkat Subramaniam Blending Java With Dynamic Languages
deimos
Udi Dahan Intentions And Interfaces
Udi Dahan Intentions And Interfaces
deimos
Tim Mackinnon Agile And Beyond
Tim Mackinnon Agile And Beyond
deimos
Steve Vinoski Rest And Reuse And Serendipity
Steve Vinoski Rest And Reuse And Serendipity
deimos
Stefan Tilkov Soa Rest And The Web
Stefan Tilkov Soa Rest And The Web
deimos
Stefan Tilkov Pragmatic Intro To Rest
Stefan Tilkov Pragmatic Intro To Rest
deimos
Rod Johnson Cathedral
Rod Johnson Cathedral
deimos
Mike Stolz Dramatic Scalability
Mike Stolz Dramatic Scalability
deimos
Pete Goodliffe A Tale Of Two Systems
Pete Goodliffe A Tale Of Two Systems
deimos
Paul Fremantle Restful SOA Registry
Paul Fremantle Restful SOA Registry
deimos
Ola Bini Evolving The Java Platform
Ola Bini Evolving The Java Platform
deimos
Neal Gafter Java Evolution
Neal Gafter Java Evolution
deimos
Markus Voelter Textual DSLs
Markus Voelter Textual DSLs
deimos
Marc Evers People Vs Process Beyond Agile
Marc Evers People Vs Process Beyond Agile
deimos
Mais de deimos
(20)
Aspect Orientated Programming in Ruby
Aspect Orientated Programming in Ruby
Randy Shoup eBays Architectural Principles
Randy Shoup eBays Architectural Principles
Remy Sharp The DOM scripting toolkit jQuery
Remy Sharp The DOM scripting toolkit jQuery
Ola Bini J Ruby Power On The Jvm
Ola Bini J Ruby Power On The Jvm
Aslak Hellesoy Executable User Stories R Spec Bdd
Aslak Hellesoy Executable User Stories R Spec Bdd
Venkat Subramaniam Building DSLs In Groovy
Venkat Subramaniam Building DSLs In Groovy
Venkat Subramaniam Blending Java With Dynamic Languages
Venkat Subramaniam Blending Java With Dynamic Languages
Udi Dahan Intentions And Interfaces
Udi Dahan Intentions And Interfaces
Tim Mackinnon Agile And Beyond
Tim Mackinnon Agile And Beyond
Steve Vinoski Rest And Reuse And Serendipity
Steve Vinoski Rest And Reuse And Serendipity
Stefan Tilkov Soa Rest And The Web
Stefan Tilkov Soa Rest And The Web
Stefan Tilkov Pragmatic Intro To Rest
Stefan Tilkov Pragmatic Intro To Rest
Rod Johnson Cathedral
Rod Johnson Cathedral
Mike Stolz Dramatic Scalability
Mike Stolz Dramatic Scalability
Pete Goodliffe A Tale Of Two Systems
Pete Goodliffe A Tale Of Two Systems
Paul Fremantle Restful SOA Registry
Paul Fremantle Restful SOA Registry
Ola Bini Evolving The Java Platform
Ola Bini Evolving The Java Platform
Neal Gafter Java Evolution
Neal Gafter Java Evolution
Markus Voelter Textual DSLs
Markus Voelter Textual DSLs
Marc Evers People Vs Process Beyond Agile
Marc Evers People Vs Process Beyond Agile
Último
Intro to BCG's Carbon Emissions Benchmark_vF.pdf
Intro to BCG's Carbon Emissions Benchmark_vF.pdf
pollardmorgan
8447779800, Low rate Call girls in Uttam Nagar Delhi NCR
8447779800, Low rate Call girls in Uttam Nagar Delhi NCR
ashishs7044
Kenya Coconut Production Presentation by Dr. Lalith Perera
Kenya Coconut Production Presentation by Dr. Lalith Perera
ictsugar
Pitch Deck Teardown: Geodesic.Life's $500k Pre-seed deck
Pitch Deck Teardown: Geodesic.Life's $500k Pre-seed deck
HajeJanKamps
Organizational Structure Running A Successful Business
Organizational Structure Running A Successful Business
Seta Wicaksana
Marketplace and Quality Assurance Presentation - Vincent Chirchir
Marketplace and Quality Assurance Presentation - Vincent Chirchir
ictsugar
8447779800, Low rate Call girls in Shivaji Enclave Delhi NCR
8447779800, Low rate Call girls in Shivaji Enclave Delhi NCR
ashishs7044
Keppel Ltd. 1Q 2024 Business Update Presentation Slides
Keppel Ltd. 1Q 2024 Business Update Presentation Slides
KeppelCorporation
Case study on tata clothing brand zudio in detail
Case study on tata clothing brand zudio in detail
Ariel592675
Buy gmail accounts.pdf Buy Old Gmail Accounts
Buy gmail accounts.pdf Buy Old Gmail Accounts
Buy Verified Accounts
Digital Transformation in the PLM domain - distrib.pdf
Digital Transformation in the PLM domain - distrib.pdf
Jos Voskuil
/:Call Girls In Indirapuram Ghaziabad ➥9990211544 Independent Best Escorts In...
/:Call Girls In Indirapuram Ghaziabad ➥9990211544 Independent Best Escorts In...
lizamodels9
Call Girls Miyapur 7001305949 all area service COD available Any Time
Call Girls Miyapur 7001305949 all area service COD available Any Time
delhimodelshub1
Kenya’s Coconut Value Chain by Gatsby Africa
Kenya’s Coconut Value Chain by Gatsby Africa
ictsugar
The CMO Survey - Highlights and Insights Report - Spring 2024
The CMO Survey - Highlights and Insights Report - Spring 2024
christinemoorman
Cybersecurity Awareness Training Presentation v2024.03
Cybersecurity Awareness Training Presentation v2024.03
DallasHaselhorst
Innovation Conference 5th March 2024.pdf
Innovation Conference 5th March 2024.pdf
richard876048
8447779800, Low rate Call girls in Tughlakabad Delhi NCR
8447779800, Low rate Call girls in Tughlakabad Delhi NCR
ashishs7044
Ten Organizational Design Models to align structure and operations to busines...
Ten Organizational Design Models to align structure and operations to busines...
Seta Wicaksana
Independent Call Girls Andheri Nightlaila 9967584737
Independent Call Girls Andheri Nightlaila 9967584737
Riya Pathan
Último
(20)
Intro to BCG's Carbon Emissions Benchmark_vF.pdf
Intro to BCG's Carbon Emissions Benchmark_vF.pdf
8447779800, Low rate Call girls in Uttam Nagar Delhi NCR
8447779800, Low rate Call girls in Uttam Nagar Delhi NCR
Kenya Coconut Production Presentation by Dr. Lalith Perera
Kenya Coconut Production Presentation by Dr. Lalith Perera
Pitch Deck Teardown: Geodesic.Life's $500k Pre-seed deck
Pitch Deck Teardown: Geodesic.Life's $500k Pre-seed deck
Organizational Structure Running A Successful Business
Organizational Structure Running A Successful Business
Marketplace and Quality Assurance Presentation - Vincent Chirchir
Marketplace and Quality Assurance Presentation - Vincent Chirchir
8447779800, Low rate Call girls in Shivaji Enclave Delhi NCR
8447779800, Low rate Call girls in Shivaji Enclave Delhi NCR
Keppel Ltd. 1Q 2024 Business Update Presentation Slides
Keppel Ltd. 1Q 2024 Business Update Presentation Slides
Case study on tata clothing brand zudio in detail
Case study on tata clothing brand zudio in detail
Buy gmail accounts.pdf Buy Old Gmail Accounts
Buy gmail accounts.pdf Buy Old Gmail Accounts
Digital Transformation in the PLM domain - distrib.pdf
Digital Transformation in the PLM domain - distrib.pdf
/:Call Girls In Indirapuram Ghaziabad ➥9990211544 Independent Best Escorts In...
/:Call Girls In Indirapuram Ghaziabad ➥9990211544 Independent Best Escorts In...
Call Girls Miyapur 7001305949 all area service COD available Any Time
Call Girls Miyapur 7001305949 all area service COD available Any Time
Kenya’s Coconut Value Chain by Gatsby Africa
Kenya’s Coconut Value Chain by Gatsby Africa
The CMO Survey - Highlights and Insights Report - Spring 2024
The CMO Survey - Highlights and Insights Report - Spring 2024
Cybersecurity Awareness Training Presentation v2024.03
Cybersecurity Awareness Training Presentation v2024.03
Innovation Conference 5th March 2024.pdf
Innovation Conference 5th March 2024.pdf
8447779800, Low rate Call girls in Tughlakabad Delhi NCR
8447779800, Low rate Call girls in Tughlakabad Delhi NCR
Ten Organizational Design Models to align structure and operations to busines...
Ten Organizational Design Models to align structure and operations to busines...
Independent Call Girls Andheri Nightlaila 9967584737
Independent Call Girls Andheri Nightlaila 9967584737
Matt Youill Betfair
1.
FROM BETTING TO
GAMING TO TRADEFAIR Matt Youill, Chief Technologist, Betfair Asher Glynn, Primary Initiatives Manager, Betfair QCon, London UK, March 2008 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
2.
INDUSTRY / TRADITIONAL
BOOKMAKING. • Bets are only made between the customer and the Bookmaker. • Bookmaker sets odds (prices) and factors in a margin. Traditional Bookmakers are still major players in the gaming industry. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
3.
INDUSTRY / BETFAIR
EXCHANGE. • New bookmaker Betfair appears in 2000. Defined the Betting Exchange concept. • Customer’s bets matched between themselves. • Customers set the odds (prices). • An exchange has “perfect” risk management and therefore, a lower margin. • Commission on winnings. • Prices around 20% better than Traditional Bookmakers. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
4.
INDUSTRY / BETFAIR
CASINO, POKER AND GAMES. • Complete portfolio of gaming products complements the exchange. • Strategically key for Betfair. • Share Betfair’s principle of fairness such as the “Zero edge” Casino. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
5.
INDUSTRY / BETFAIR.
• Betfair operates a betting exchange, games exchange, poker room, and casino. • Annual revenues in excess of £200 million. • Over 1,000,000 registered users. • Over 1500 employees in offices globally. • 5 billion page views/week. • Almost half of all global traffic to gambling sites comes to Betfair. • £2,000 deposited every minute. • World’s leading betting exchange. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
6.
INDUSTRY / TRADEFAIR.
• The Tradefair exchange, a subsidiary of Betfair, launched in late 2007. • Currently offering financial binaries and spread betting. • Accessible, available, transparent retail channel. • Commission on profits. • Prices very closely reflect the underlying market. • Very low cost trading direct to customers. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
7.
CHALLENGE / TRANSACTION
PROCESSING. • Numerous challenges when delivering these products directly to customers over retail channels. • Security on the wild west web. • Low capability devices - browsers, mobiles, etc. • Channel predictability. • And so on. • State management , particularly the changing of and delivery of state (a.k.a transaction processing) very difficult. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
8.
CHALLENGE / BETTING
EXCHANGE. • The exchanges have very large capacity requirements. • Currently, up to 1000 transactions per second. • Desired objective of 50,000 low cost transactions per second. • Plus increased reliability, maintainability, etc. • The 100X project formed to achieve these goals. • A transaction = a bet placement, a cancel or an edit. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
9.
CHALLENGE / EXCHANGE
TRANSACTION ENGINE. • The Betfair exchange’s “flux capacitor” is the Exchange Transaction Engine (ETE). • It provides 4 key functions: • Exposure: Validates and stores bet orders, reserves customer funds. • Matching: Matches customer’s bets and reports the result. • Bet Management: Allows customers to view the status of their bets. • Market View: Allows customers to view a summary of all the bets placed on an event. • Most have parallels in financials. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
10.
CHALLENGE / EXCHANGE
TRANSACTION ENGINE. Exposure Bet Market Management View Matching CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
11.
CHALLENGE / EXCHANGE
TRANSACTION ENGINE. • Currently implemented in PL/SQL on a single Oracle instance. • Rated as one of top 5 “hottest” Oracle databases in the world. • Since Betfair was founded it has been a constant struggle to satisfy capacity demands. • Why?... Growing pains aside, it is because Betfair’s rules of fairness present a challenge when scaling the Matching component of the ETE. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
12.
CHALLENGE / MATCHING.
• Betfair has two key business rules… Best Execution (Best customer value) Each bet placed is matched against opposing bets in order of descending odds. First come, First served (Fairness for all) The first bet placed is the first matched exclusive of others. • Means everything processed serially. • There is an unavoidable traffic jam in the business rules. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
13.
CHALLENGE / MATCHING.
• Rules only apply per event. • Means system can be scaled by processing each event in parallel. (Similar to the concurrent execution of trades on financial instruments). • But… • The nature of big events, particularly horse racing, is that they are short lived and start times rarely coincide. (Unlike the more evenly spread activity on financial exchanges). • Over 75% of betting activity at any one time is on a single Betfair event – the “hot market”. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
14.
CHALLENGE / EXPOSURE
AND BET MANAGEMENT. • Exposure and bet management are less of a challenge as they occur on a per customer (account) basis. • With activity roughly evenly spread across accounts, and peak activity on individual accounts relatively low, it’s possible to simply partition by account. • Parallelisation, partitioning and distribution provides sufficient capacity. • Except, of course, in the case of a big individual users. But none of them are that big…yet. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
15.
CHALLENGE / MARKET
VIEW. • The Market View or the view of the odds and stakes currently available on an event is challenging but in a different way. • Every transaction will potentially change the view of a market. • Hence, every transaction requires the new view to be delivered to customers. • This is a lot of information that needs to go to a lot of people many times every second. • Multicasting scales effectively but in the world of the web it is harder requiring manual polling, AJAX, Comet, etc. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
16.
APPROACH / THE
PIONEERS. • Many initiatives tried before. • Approached as optimisations or evolutions of the existing engine. • Optimizations - hardware upgrades, index tuning, schema reorganisations, etc. • Evolutions – product solutions, functional partitioning, asynchronicity, etc. • Essential yet modest gains. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
17.
APPROACH / 100X
PROGRAMME. “100 times (100X) more transactions per second” “Infinite capacity at zero cost” • At the time of initiation (500 TPS -> 50,000 TPS) • 10,000% target. • Challenge existed for many years, well defined. Solution less so ☺. Phases Investigation Had this problem or anything like it been solved before? Research & Design Sketch and elaborate various architectures. Proof of Concept Multiple Vendors including Betfair Engineering. Build, test and compare. Integration How to get into production? What changes and compromises needed? Build and Deploy Build the “Lite” version and plug it into the live system. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
18.
SOLUTION / CONCURRENCY
(ISOLATION). • Each of the major functions in the ETE require exclusive access to either an account or event (market). • Rather than client sessions acquiring and releasing locks on particular pieces of data, each instance of an entity is assigned to a single execution unit (i.e. Actor). Unit A may own Account 123, or unit B may own Market 456 for example. • With execution units tied to individual entities, access is inherently serialised and “transactions” are isolated from one another. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
19.
SOLUTION / CONCURRENCY
(ISOLATION). Exposure & Exposure & Exposure & Bet Management Bet Management Bet Management Matching & Matching & Market View Market View CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
20.
SOLUTION / CONCURRENCY
(ISOLATION). Brokering & Brokering & Brokering & Order Management Order Management Order Management Matching & Matching & Order Book Order Book CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
21.
SOLUTION / CONCURRENCY
(ISOLATION). Account Exposure & Account ID Funds Exposure Bet Management 1 £1,000 £735 Bets Bet ID Type Runner Odds Stake 1 Bet For Field Commander 12.5 £10 2 Bet For Wingate Street 6.8 £20 3 Bet For Keyton Grace 19.5 £30 4 Bet Against Foxy Boxy 14.5 £50 • Only one execution unit can work on this data. It is “owned” by the Actor. • No interleaved/inconsistent updates. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
22.
SOLUTION / COMMUNICATIONS.
• Each Actor operates on the data it owns as a result of messages (operations) sent to it. • Each Actor may in turn send more messages to further execution units. • Message passing has its benefits, but also its costs… Can distribute widely, conceptually simple But messages can be lost, arrive out of concurrency model, threads can be order, duplicated, corrupted, need detached from sessions, messages can be correlation, no system wide consistency or batched, easier to take advantage of atomicity, enqueue/dequeue costs, etc. asynchronous IO, etc. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
23.
SOLUTION / CONSISTENCY.
• No Actor in the system has a complete and consistent view of the entire system and there is no strict integrity across units. • Each only responsible for its individual view of the world. These chunks of reality define the boundaries of consistency. • Ultimately full system wide consistency is desirable but at any instance in time, global consistency may vary. That is - global consistency is weak and achieved only eventually. • When something inconsistent is found, explicit correcting actions need to be taken. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
24.
SOLUTION / CONSISTENCY.
Bets Exposure & Bet ID Type Runner Odds Stake Bet Management 1 Bet For Field Commander 12.5 £10 Matches Bet ID Matched Bet ID Matched Odds Matched Stake 1 5 12.5 £5 Unmatched Bets Matching & Bet ID Type Runner Odds Unmatched Stake Market View 1 Bet For Field Commander 12.5 £5 Example: These must add up - eventually • Data exists under two Actors control, but no explicit consistency. • Don’t become inconsistent in the first place. State and operations must not be lost, corrupted, duplicated, etc. But if something does, take explicit compensating action. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
25.
SOLUTION / ATOMICITY.
• Each operation sent to an Actor defines the scope of a transaction. The instructions that the operation performs is defined by the deterministic function that it executes. • Receipt and journaling of that operation defines the success of a transaction. The instructions need not be executed, only captured. • In most case, transactions across Actors require explicit compensating logic. • Would be nice to have one Actor and avoid spanning transactions. Not practical though. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
26.
The edit can’t
complete, the SOLUTION / ATOMICITY. edit work done so far must be “rolled back” – manually. Place Bet Place Bet Edit Bet Exposure & Bet Management Bet A Bet A Bet A Bet A Bet A (Complete, (Incomplete Place) (Complete) (Complete, Incomplete Edit) (Complete) Incomplete Edit) Match Bet Match Bet Exposure & Bet Management Bet B Bet B (Incomplete Place) (Complete) Match Bet Bet Unmatched Bet Matched Bet Already Matched Matching & Bet Matched Market View Bet A Bet A (Matched) Bet A (Unmatched) (Matched) Bet B (Matched) CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
27.
SOLUTION / WHEN
THINGS GO WRONG. • Highly distributed architecture allows for partial failures and disruptions. • Large chunks of the system can fail and the system as a whole will keep running. • Durable state and message reliability means system stabilises eventually in the event of a failure. Design isn’t overly fussy. “Simple” design important. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
28.
SOLUTION / RELIABILITY
(STATE). • Actor’s inbox turned into a journal. State persistence provided by logging all messages. • Replay of journal restores state. Simple ☺ Logger • All functions following the log must be Exposure & deterministic so under replay everything is Bet Management restored to exactly how it was. • Disk shared between nodes in the event of a full node failure. • Adding check pointing reduces replay time and prevents log exhaustion. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
29.
SOLUTION / RELIABILITY
(MESSAGING). • The log also serves as the basis for reliable messaging. • Journal replay also restores the state of message counters along with application Logger Conversations state. Exposure & • These message counters (or conversations) Bet Management are maintained for each sending and receiving pair. • Using the counters, lost messages can be detected, duplicate messages can be removed and in doubt messages can be retried safely. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
30.
SOLUTION / RELIABILITY
(MESSAGING). Conversation Conversation Receive: 23 Receive: 233 Send: 2166 Send: 84 234 24 2166 Conversations are between 84 logical entities Conversation Receive: 83 Send: 234 Conversation Receive: 2165 Send: 24 580 Conversation Receive: 153 Send: 580 Conversation Receive: 579 Send: 154 154 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
31.
SOLUTION / DEPLOYMENT.
Logger Logger Exposure & Exposure & Exposure & Bet Management Bet Management Bet Management Logger Logger Matching & Market View Matching & Matching & Market View Market View CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
32.
SOLUTION / HIGH
AVAILABILITY. • Log replay means faults are tolerated… slowly. • Need fast(er) recovery. • Two flavours of HA – catastrophic and fast. • Need a solution for both. • We need a back up ready to go, but if both “disappear” then all is not lost. • Lots of different ways to do this, but no perfect solution. • We tried various ways – about 8. Mostly well established methods. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
33.
SOLUTION / CATASTROPHIC
FAILURES (1. LOG REPLAY). Failed Actors Recovering Actors Failed Units Recovering Units • Conceptually simple. • Disk logging fast enough. • Survives most problems – eventually. • Slow recovery. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
34.
SOLUTION / CATASTROPHIC
FAILURES (2. UPSTREAM REPLAY). Upstream Actor Upstream Actor Upstream Actor Failed Actors • Potentially fast, no need for slow Recovering Actors Fed Unit Fed Unit logging on “hot market” units. • Very complex slow recovery. • Problem if upstream actor is also down. • Still need to log somewhere. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
35.
SOLUTION / CATASTROPHIC
FAILURES (3. MULTI-ORDER MIRRORING). Failed Actors Failed Actors Recovering Actors Failed Units Failed Units Recovering Units • Appears simple, but can be tricky. • Slow. • Won’t survive high multi-order failures. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
36.
SOLUTION / FAST
FAILURES (4. STATE MIRRORING). Hot Actor Periodic check point of state deltas Warm Actor • Appears simple. • Somewhat slow. • Reintroducing a failed node can be tricky. • Synergy with logging (check pointing) mechanism for catastrophic failures. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
37.
SOLUTION / FAST
FAILURES (5. LOG TAILING). Continuous “recovery” Hot Actor Warm Actor • Conceptually simple. • Synergy with logging mechanism for catastrophic failures. • Needs shared disk. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
38.
SOLUTION / FAST
FAILURES (6. SIMULTANEOUS DELIVERY). Reliable, ordered multicast Hot Actor Warm Actor • Slow and tricky. • No need for shared disk. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
39.
SOLUTION / FAST
FAILURES (7. RELAYING). Hot Actor Warm Actor /dev/null • Conceptually simple. • No need for shared disk. • Throughput not affected. • Though increases latency. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
40.
SOLUTION / FAST
FAILURES (8. LOCK STEPPING). Lock Stepped Actor • Off the shelf solution (NonStop, Stratus, Qumranet, etc). • Specialised. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
41.
SOLUTION / LOG
TAILING. • Best compromise. • Simple to understand and implement. • Catches both catastrophic and fast failure requirements. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
42.
RESULTS.
• 70K TPS (hot market throughput test at Sun Solution Centre, Manchester, England). – Single biggest cost, by far, was serialization on and off network and disk. • 70K TPS on a single market. Potentially millions per second across multiple markets (i.e. instruments). Results Load Injectors “Account Controllers” “Market Controllers” Throughput Latency 2 2 1 29,950 TPS 250 ms 3 3 1 73,370 TPS 550 ms 6 5 2 136,150 TPS 679 ms Machines Specification: 2 x Dual Core 2Ghz AMD Opteron Processor 8GB Memory Copper Gbit Network Red Hat Enterprise Linux 4 O/S Java 1.5 VM Sun 3510 FC Array Storage • “Lite” (Production) version passed testing, being integrated. – Scaled up instead of out and integrated with “classic” Oracle ETE. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
43.
FLYWHEEL LITE.
• Production version of Flywheel. • Demanding integration challenges. • Simplified to work on single node. – Advanced journaling. – Less network hops. – Optimization opportunities. • Many advances. – Algorithm refinement. – Improved parallelism. – Improved journaling technology. Slowed by bridge to “classic” ETE. Removal of bridge indicates 80K TPS at sub 100ms. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
44.
TRADEFAIR.
• Betfair already provides a number of financial bets. • Low(ish) take up. • Big potential market • Acquired an understanding of how to cope with high volume/low latency system from Sports and Games exchanges • Looking for areas to diversify into CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
45.
TRADEFAIR.
• Recognised an opportunity and launched Tradefair. • “Betfair for Financials rather than sport”. • Retail focused - compete on value to customers. • Just starting up now. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
46.
TRADEFAIR.
Betfair’s Contribution Cash. People. Infrastructure. … CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
47.
TRADEFAIR.
And a really fast exchange system. Flywheel technology the key enabler for entry into this market. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
48.
FROM BETFAIR TO
TRADEFAIR. Flywheel Lite Pipelines CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
49.
TRADEFAIR.
• Betfair – historically customers tolerant of latency. • Now expect financial markets quality of service. • Cannot launch Tradefair with relaxed latency requirements. – Low latency. – Consistent latency. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
50.
TRADEFAIR.
• Tradefair – Customers demand consistent latency. – Standard deviation down to less than 5 msec under load. – But need to preserve Flywheel level throughput. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
51.
TRADEFAIR.
• Solutions – technical – Non disk journaling. – GC tuning. – Heavy abstraction avoidance (data kept as plain bytes). – Throughput throttling. – Smarter buffering. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
52.
TRADEFAIR.
• Can buy better latency (to a degree) – Bigger servers. – Faster networks. – Better storage (avoiding spinning media). – Still need to keep costs down to avoid damaging customer value. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
53.
TRADEFAIR.
• Betfair : Customer Liability Management (Exposure). – Simple, fixed one off calculation cost. – Very easy to optimise. – Some specialisation around order type. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
54.
TRADEFAIR.
• Tradefair: Customer Liability Management (Exposure). – For binaries – same as Betfair (exactly). – For spreads – more complicated. Continuous update. Driven by market. More order types. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
55.
TRADEFAIR.
• Solution: Liability Management. – Not actually the bottleneck for throughput in Flywheel architecture. – Number of nice algorithmic workarounds. Optimise update graph. Optimise actual calculation. Avoid broad distribution of updates. Core Flywheel patterns give solution if ever does become overall bottleneck. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
56.
TRADEFAIR.
• Betfair: Bet matching. – Large number of market types. – A large number of markets. – Straight forward order types – immediate execution, persist until market suspended or order modified. – Already well optimised without getting into exotic technologies. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
57.
TRADEFAIR.
• Tradefair: Order matching – Small number of market types (in terms of business logic). – Smaller number of markets, but more consistent throughput. – Large number of different order types (time sensitive, conditional, immediate, lapsable). – Need for immediate settlement. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
58.
TRADEFAIR.
• Solution: – Same architecture, specialised business logic – No significant modifications to underlying framework CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
59.
TRADEFAIR.
• Data distribution: Betfair (historically). – Pull website. – SOAP API. – Restricted granularity of updates. – Diversification to new media like mobile phones still largely same technology/principles. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
60.
TRADEFAIR.
• Data distribution: Tradefair. – Push website. – Pushed Data. – Standard financials API (FIX). – Tick by tick granularity. – Transport reliability CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
61.
TRADEFAIR.
• Solution. – Flywheel inherently message driven. – Need series of adapters to translate internal and external messages. – Still a latency minimisation and consistency problem – core technology can be reused for this. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
62.
FURTHER WORK.
• Geographic Separation. • Regulation. • Generality. • Latency. • HA. • Fault tolerance. • Evolution. • Integration. • Complexity. • Hardware. More importantly: Deploy across Betfair and Tradefair. CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited
63.
END. CONFIDENTIAL and not
for reproduction without prior written consent. © of the Sporting Exchange Limited
Baixar agora