Pinot: Realtime Distributed OLAP datastore

352.550 visualizações

Publicada em

Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally.

Publicada em: Tecnologia
96 comentários
784 gostaram
Estatísticas
Notas
  • I AM SO HAPPY DR TRUST, MY HUSBAND IS BACK. I completely trusted Dr Trust totally from the time I spoke with him during the period my husband Left me after 11 years of our marriage, He started the spell work on my husband, and gave me so much assurance and guaranteed me that he was going to bring my husband back to my feet in just 48 hours of the spell casting. I was so confident in his work and just as he said in the beginning, my husband is finally back to me again, yes he is back with all his hearts, Love, care, emotions and flowers and things are better now. I would have no hesitation to recommend this powerful spell caster to anybody who is in need of help.. E-mail Ultimatespellcast@gmail.com or Ultimatespellcast@gmail.com his website: https://utimatespellcaster.com WhatApp or call him +2348156885231 Do accept my gratitude sir.
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Welcome to the illuminati world of riches and fames Possess your riches today by joining the illuminati church of Satan created and organizing by our high priest ANTON SZANDOR LAVEY who is now in the bossom of the lord Lucifer, here is an opportunity for you today to join the illuminati and make all your heart desire come to pass by contacting us today with our contact id….. email us on Bavariaillumjnatitemple@gmail.com or call or whatsapp +2348034056542 Here is the benefit of becoming a born again child of the lord Lucifer A HOUSE IN ANY COUNTRY OF YOUR CHOICE NEW BRAND CARS OF YOUR CHOICE $50000000 AS A NEW MEMBER ILLUMINATI BIBLE ILLUMINATI RING OF POWERS ILLUMINATI GARMENT $10000000 AS YOUR MONTHLY PAYMENT $5000000 EVERY WEEK AS YOUR BLESSING FROM THE LORD LUCIFER AS A BORN AGAIN CHILD INTERNATIONAL PASSPORT FOR A VISIT TO ANY COUNTRY OF YOUR CHOICE
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Nice !! Download 100 % Free Ebooks, PPts, Study Notes, Novels, etc @ https://www.ThesisScientist.com
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Thenk Yuo. https://beautyperry.com
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Testimony On How To Get Urgent Real Love Spell Online That Work Fast..... Junia Noel by name and i am located in Usa ,I recently saw a testimony about a spell caster of some sort in a blog i visit for relationship and dating counseling problems because i had been having serious problems with my boyfriend and we had been dating for 2 year, he just suddenly changed, he wasn’t returning my calls, he started cheating, he was hurting me in so many ways i never thought possible and i just thought i should try this spell caster called Dr happy cos there are so many good comments about him on the website, but before i could read his post,i have already contacted many spell caster that cheated me out of my hard earned money without given me result.but when i meet this spell caster out of desperation of some sort.i decided to contact him to please render his help to me, i explained all the problems that i was going through to him and he told me all i needed to do to get this spell castes. at first everything felt dreamy and unbelievable, their consultations and solution was a little bit easy and strange and i was scared a little cos I had read and heard lots of stories of fake spell casters, scams and i never really believed in magic. i played along with a little hope and faith and i was sent some few stuffs after everything and it worked like a miracle, everything went to a whole new direction, it was and is amazing. i guess it was all good faith that made me read that particular post that faithful day.i hope he could help other people too like he did to me. i did a little and I got everything i wanted and wished for, my boyfriend, my family and my life back you can contact Dr happy at: happylovespell2@gmail.com because he changed my life call him or add him on whats-app +2348133873774
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
Sem downloads
Visualizações
Visualizações totais
352.550
No SlideShare
0
A partir de incorporações
0
Número de incorporações
3.469
Ações
Compartilhamentos
0
Downloads
1.805
Comentários
96
Gostaram
784
Incorporações 0
Nenhuma incorporação

Nenhuma nota no slide

Pinot: Realtime Distributed OLAP datastore

  1. Pinot Kishore Gopalakrishna Tuesday, August 18, 15
  2. Agenda • Pinot @ LinkedIn - Current • Pinot - Architecture • Pinot Operations • Pinot @ LinkedIn - Future Tuesday, August 18, 15
  3. WVMP Tuesday, August 18, 15
  4. Slice and Dice Metrics Tuesday, August 18, 15
  5. Pinot @ LinkedIn Customers Members Internal tools Tuesday, August 18, 15
  6. • 100B documents • 1B documents ingested per day • 100M queries per day • 10’s of ms latency • 30 tables in prod, 250 * 3 std app nodes Pinot @ LinkedIn Tuesday, August 18, 15
  7. Key features SQL-like interface Columnar storage and indexing Real-time data load Tuesday, August 18, 15
  8. (S)QL: Filters and Aggs SELECT count(*) FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND 'day' >= 15949 AND 'day' <= 15963 AND paid = 'y’ AND action = 'stop' Tuesday, August 18, 15
  9. (S)QL: Group By SELECT count(*) FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND 'day' >= 15949 AND 'day' <= 15963 AND paid = 'y’ GROUP BY action Tuesday, August 18, 15
  10. (S)QL: ORDER BY and LIMIT SELECT * FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND entityId = 1000 AND action = 'start' ORDER BY creationTime DESC LIMIT 1 Tuesday, August 18, 15
  11. Whats not supported • JOIN: unpredictable performance • NOT A SOURCE OF TRUTH • Mutation Tuesday, August 18, 15
  12. Pinot • Data flow • Query Execution • How to use/operate • Pinot @ LinkedIn - Future Tuesday, August 18, 15
  13. Broker Helix Real time Historical Kafka Hadoop Pinot Architecture Queries Raw Data Tuesday, August 18, 15
  14. Pinot • Pinot segments Tuesday, August 18, 15
  15. Pinot Segment layout: Columnar storage Tuesday, August 18, 15
  16. Pinot Segment layout: Sorted Forward Index Tuesday, August 18, 15
  17. Pinot Segment layout: Other techniques • Indexes: Inverted index, Bitmap, RoaringBitmap • Compression: Dictionary Encoding, P4Delta • Multi Valued columns, skip lists, • Hyperloglog for unique • T-digest for Percentile, Quantile Tuesday, August 18, 15
  18. Data aware pre-computation Star tree Index Tuesday, August 18, 15
  19. Pinot • Query Execution Tuesday, August 18, 15
  20. Pinot Query Execution: Distributed Servers S1 S3 S2 S1 S3 S2 Helix Brokers Tuesday, August 18, 15
  21. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix Brokers Tuesday, August 18, 15
  22. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers Tuesday, August 18, 15
  23. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request Tuesday, August 18, 15
  24. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response Tuesday, August 18, 15
  25. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response 5. Gather Response Tuesday, August 18, 15
  26. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response 5. Gather Response 6. Return Response Tuesday, August 18, 15
  27. Pinot Query Execution: Single Node Architecture EXECUTION ENGINE INVERTED INDEX BITMAP INDEX COLUMN FORMAT PLANNER Tuesday, August 18, 15
  28. Pinot Query Execution: Single Node Architecture SELECT campaignId, sum(clicks) FROM Table A WHERE accountId = 121011 AND 'day' >= 15949 GROUP BY campaignId account Id daycampaign Id click Filter Operator Projection Operator Aggregation Group by Operator Combine Operator Pinot Segments Data sources Matching doc ids campaignId,Click tuple Tuesday, August 18, 15
  29. Pinot • Operations Tuesday, August 18, 15
  30. Cluster Management: Deployment Helix Brokers Servers • Brokers and Servers register themselves in Helix • All servers start with no use case specific configuration Controller Tuesday, August 18, 15
  31. On boarding new use case Helix Brokers Servers XLNT XLNT XLNT Create Table command Controller XLNT XLNTTag Servers TableName Brokers 3 XLNT_T1 1 Tuesday, August 18, 15
  32. Segment Assignment Servers S3 S2 S1 Upload Segment S2 S1 S3 S2 S1 S3 Helix Brokers Copies TableName 2 XLNT_T1 Controller Tuesday, August 18, 15
  33. • AUTO recovery mode: Automatically redistribute segments on failure/addition of new nodes • Custom mode: Run in degraded mode until node is restarted/replaced. Pinot - Fault tolerance/Elasticity Tuesday, August 18, 15
  34. Pinot vs Druid Druid Pinot Architecture Realtime + Offline, Realtime only Realtime + Offline Realtime only -> consistency is hard and schema evolution/Bootstrap is hard Inverted Index Always On all columns, Fixed Configurable on per column basis Allows trade off between scanning v/s inverted index + scanning. More data can be fit in given memory size Data organization N/A Sorts data Organizing data provides speed/better compression and removes the need for inverted index Smart pre- materialization N/A star-tree Allows trade off between latency and space Query Execution Layer Fixed Plan Split into Planning and execution Smart choices can be made at runtime based on metadata/query. Tuesday, August 18, 15
  35. • Documentation & tooling • In progress - consistency among real time replicas. • Improve cost to serve - leverage SSD, partial pre materialization • ThirdEye - Business Metrics Monitoring Pinot - Future Tuesday, August 18, 15
  36. Thank You 30 Tuesday, August 18, 15

×