O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
AN ALTERNATIVE USE CASE
Analytics Technology
Rob Harty & Rob Purnell, Willis
23rd March 2015
Analytics Technology – An Alternative Use Case | 1
Who are Willis?
 The Analytical Broker.
 Global Presence.
 18,000+ e...
Analytics Technology – An Alternative Use Case | 2
Analytics Technology
 Agile development team of developers, QA, BA and...
Analytics Technology – An Alternative Use Case | 3
Our Products…
Atlas
Analytics Technology – An Alternative Use Case | 4
Our Products…
Alert:24
Analytics Technology – An Alternative Use Case | 5
Our Products…
Willis Data Mart
Analytics Technology – An Alternative Use Case | 6
Our Products…
WillPLACE Web App
Analytics Technology – An Alternative Use Case | 7
Typical Architecture
How Couchbase integrates with our applications:
Analytics Technology – An Alternative Use Case | 8
The Problem
Basic process overview
Analytics Technology – An Alternative Use Case | 9
The Problem
Stats – SQL vs. In Memory
Sample portfolio with 1 million l...
Analytics Technology – An Alternative Use Case | 10
The Problem
 Relatively low user base, very large set of data, often ...
Analytics Technology – An Alternative Use Case | 11
What can we do about it?
Researching the solution
 Of primary concern...
Analytics Technology – An Alternative Use Case | 12
Performance
 As part of evaluation, we performed serialization tests ...
Analytics Technology – An Alternative Use Case | 13
Couchbase?
• Quickest performance in our usage out of the box
• Provid...
Analytics Technology – An Alternative Use Case | 14
Offline vs Online Processing
Using Couchbase as a cache for near real-...
Analytics Technology – An Alternative Use Case | 15
Performance
 Performance still not enough for our needs, investigate ...
Analytics Technology – An Alternative Use Case | 16
Performance
Overall statistics
485 11 15 3 2
0
5
10
15
20
25
30
35
40
...
Analytics Technology – An Alternative Use Case | 17
Summary
 In our use case the Couchbase process is 44x faster than our...
Analytics Technology – An Alternative Use Case | 18
Summary
Willis
http://www.willis.com/
Robert Purnell Senior Developer ...
Próximos SlideShares
Carregando em…5
×

Couchbase Live Europe 2015: Couchbase at Willis: Some Alternative Use Cases

805 visualizações

Publicada em

Most Couchbase use cases cover the scenario of lots of users accessing a small amount of data from a big bucket. At Willis, we face a different challenge – small amounts of users needing to access a large amount of data in real time to visualise analytics. Our talk covers a couple of examples showing how we used Couchbase to augment our client products and take application performance and scalability to the next level. We’ll cover our reasons for choosing Couchbase over other products, swapping out the default serializer and using Couchbase as a "Hot Tier".

Presented by: Rob Purnell & Rob Harty, Willis Group

Publicada em: Software
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

Couchbase Live Europe 2015: Couchbase at Willis: Some Alternative Use Cases

  1. 1. AN ALTERNATIVE USE CASE Analytics Technology Rob Harty & Rob Purnell, Willis 23rd March 2015
  2. 2. Analytics Technology – An Alternative Use Case | 1 Who are Willis?  The Analytical Broker.  Global Presence.  18,000+ employees  400 offices  We are Analytics Technology  We provide analytical tools, data and knowledge both within Willis and to our clients.  Traditional specialist in GIS applications Willis is a leading global risk advisor, insurance and reinsurance broker.
  3. 3. Analytics Technology – An Alternative Use Case | 2 Analytics Technology  Agile development team of developers, QA, BA and PMs.  Based in London, Ipswich and Mumbai.  .NET development.  Primarily web based.  Application examples:  Atlas – engineering services portal that allows clients to visualize site risk information  Alert:24 – inform, prevent and respond to personal risks  Wills DataMart – store of spatial analytical data and models  WillPLACE App – broker tool to allow matching of insurers to prospective placements
  4. 4. Analytics Technology – An Alternative Use Case | 3 Our Products… Atlas
  5. 5. Analytics Technology – An Alternative Use Case | 4 Our Products… Alert:24
  6. 6. Analytics Technology – An Alternative Use Case | 5 Our Products… Willis Data Mart
  7. 7. Analytics Technology – An Alternative Use Case | 6 Our Products… WillPLACE Web App
  8. 8. Analytics Technology – An Alternative Use Case | 7 Typical Architecture How Couchbase integrates with our applications:
  9. 9. Analytics Technology – An Alternative Use Case | 8 The Problem Basic process overview
  10. 10. Analytics Technology – An Alternative Use Case | 9 The Problem Stats – SQL vs. In Memory Sample portfolio with 1 million locations 485 2 0 100 200 300 400 500 600 SQL Server In Memory Seconds
  11. 11. Analytics Technology – An Alternative Use Case | 10 The Problem  Relatively low user base, very large set of data, often all required, and quick serialization  Large client data set; typical example is 3 million locations  Flexibility required due to filtering operations on data sets  On demand SQL performance poor due to I/O, serialization and overheads building the objects up in memory  Unsustainable memory usage to persist on app tier beyond a few minutes  What we need is a way to have data available, quickly, without affecting app performance… A different use case
  12. 12. Analytics Technology – An Alternative Use Case | 11 What can we do about it? Researching the solution  Of primary concern was a high performance cache  Memcached fits use case.  Naturally, lead on to Membase (which became Couchbase)  MongoDB experience existed in team  Lead to performance comparison between systems
  13. 13. Analytics Technology – An Alternative Use Case | 12 Performance  As part of evaluation, we performed serialization tests of Couchbase and MongoDB. This is using out of the box clients. Evaluating Couchbase vs. MongoDB 485 11 15 2 0 5 10 15 20 25 30 35 40 45 50 SQL Server Couchbase MongoDB In Memory Seconds
  14. 14. Analytics Technology – An Alternative Use Case | 13 Couchbase? • Quickest performance in our usage out of the box • Provides high availability through replication and auto-failover • Can handle millions of data items • Commercially supported and managed NoSQL product • Provides a flexible data model for ease of storing unstructured data • Easily integrated into the existing Microsoft development environment • Free product enabled us to fully test and evaluate Couchbase before committing financially • Future GIS application and spatial features Reasons for choosing
  15. 15. Analytics Technology – An Alternative Use Case | 14 Offline vs Online Processing Using Couchbase as a cache for near real-time data
  16. 16. Analytics Technology – An Alternative Use Case | 15 Performance  Performance still not enough for our needs, investigate other means of improving response time  Issue with the fact we couldn’t filter the data at Couchbase – all data required on first load  Using profiler tools to identify bottle necks.  Solution for this use case, drop default serializer in favor of binary data, using Protobuf.  What is Protobuf?  Exceptionally efficient serializer  Performance highest priority  Minimal payload Still not quick enough…
  17. 17. Analytics Technology – An Alternative Use Case | 16 Performance Overall statistics 485 11 15 3 2 0 5 10 15 20 25 30 35 40 45 50 SQL Server Couchbase MongoDB Couchbase Protobuf In Memory Seconds
  18. 18. Analytics Technology – An Alternative Use Case | 17 Summary  In our use case the Couchbase process is 44x faster than our SQL equivalent  Couchbase with Protobuf as the serializer is 4x faster again  Limitation of storing in a binary format:  Lose ability to filter at source (views, etc.)  Still more to do (time allowing):  Evaluate Geospatial queries  Evaluate N1Q1 Outcomes
  19. 19. Analytics Technology – An Alternative Use Case | 18 Summary Willis http://www.willis.com/ Robert Purnell Senior Developer purnellr@willis.com Robert Harty Development Manager hartyr@willis.com References Protobuf-net https://code.google.com/p/protobuf-net/ Thank You. Questions?

×