O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Pinot: Realtime Distributed OLAP datastore

395.617 visualizações

Publicada em

Pinot is a realtime distributed OLAP datastore, which is used at LinkedIn to deliver scalable real time analytics with low latency. It can ingest data from offline data sources (such as Hadoop and flat files) as well as online sources (such as Kafka). Pinot is designed to scale horizontally.

Publicada em: Tecnologia
  • Hello Everyone ,I have been living with my wife for some years now,she has been keeping late nights and also receiving calls late at night,i was afraid she has been cheating on me and a friend of mine introduced me to a private investigator who helped me and gave me the password to her Facebook and Gmail account and also linked all her phone conversations to me.He saved me from all her lies and infidelity,he gave me all i wanted during our divorce and helped me with evidence in court.Contact ethicalhacking618atgmaildotcom .tell him i referred you he will help you
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • ●▬▬▬▬ஜ ¯` ►★☆★● My PREVIOUS month's on-line financ-ial gain is $6500. i'm currently ready to fulfill my dreams simply and reside home with my family additionally.GFq I work just for two hours on a daily basis. everybody will use this home profit system by this link.........netcash7.com
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • How I Got My Ex Husband Back........... I am Shannon by name. Greetings to every one that is reading this testimony. I have been rejected by my husband after three(3) years of marriage just because another woman had a spell on him and he left me and the kid to suffer. one day when i was reading through the web, i saw a post on how this spell caster on this address Makospelltemple@yahoo.com , have help a woman to get back her husband and i gave him a reply to his address and he told me that a woman had a spell on my husband and he told me that he will help me and after 2 days that i will have my husband back. i believed him and today i am glad to let you all know that this spell caster have the power to bring lovers back. because i am now happy with my husband. Thanks for Dr.Mako. His email: Makospelltemple@yahoo.com OR.His WhatsApp Number:+2347054263874.
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • *I knew I was a bad wife and mother for invariably pushing my husband to the point of hating me or seeing me with the kids. But I never let my bad attitude towards him let me feel like I am the worst woman out there. So, I fought inside of me to change the bad attitude and win back my husband’s heart for over six years but could not until he threatened me with a divorce and a restraining order, not only to keep me away from him, but away from our kids. *Because I can’t stand losing them, I threw away my pride and went spiritual on myself with the aid of doctor wakina (dr.wakinalovetemple@gmail.com), I was brought to light and made to understand that “it is of a great significant to make my husband happy and I must start by making myself happy”. *I boldly acknowledge the spiritual power doctor wakina carries that cured me from myself for four days and overturned my husband’s decisions. For over four months I have been having an entertaining moment with myself and family, a lot of things has changed both physical and spiritual. I say thank you Dr. Wakina for saving me from myself and restoring my marriage. *Love… Paris Tyne
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • I am very beautifulhttp://www.dinihaber.com/diyanet-haber/diyanet-in-imamlari-aklandi-h35.html
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Pinot: Realtime Distributed OLAP datastore

  1. Pinot Kishore Gopalakrishna Tuesday, August 18, 15
  2. Agenda • Pinot @ LinkedIn - Current • Pinot - Architecture • Pinot Operations • Pinot @ LinkedIn - Future Tuesday, August 18, 15
  3. WVMP Tuesday, August 18, 15
  4. Slice and Dice Metrics Tuesday, August 18, 15
  5. Pinot @ LinkedIn Customers Members Internal tools Tuesday, August 18, 15
  6. • 100B documents • 1B documents ingested per day • 100M queries per day • 10’s of ms latency • 30 tables in prod, 250 * 3 std app nodes Pinot @ LinkedIn Tuesday, August 18, 15
  7. Key features SQL-like interface Columnar storage and indexing Real-time data load Tuesday, August 18, 15
  8. (S)QL: Filters and Aggs SELECT count(*) FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND 'day' >= 15949 AND 'day' <= 15963 AND paid = 'y’ AND action = 'stop' Tuesday, August 18, 15
  9. (S)QL: Group By SELECT count(*) FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND 'day' >= 15949 AND 'day' <= 15963 AND paid = 'y’ GROUP BY action Tuesday, August 18, 15
  10. (S)QL: ORDER BY and LIMIT SELECT * FROM companyFollowHistoricalEvents WHERE entityId = 121011 AND entityId = 1000 AND action = 'start' ORDER BY creationTime DESC LIMIT 1 Tuesday, August 18, 15
  11. Whats not supported • JOIN: unpredictable performance • NOT A SOURCE OF TRUTH • Mutation Tuesday, August 18, 15
  12. Pinot • Data flow • Query Execution • How to use/operate • Pinot @ LinkedIn - Future Tuesday, August 18, 15
  13. Broker Helix Real time Historical Kafka Hadoop Pinot Architecture Queries Raw Data Tuesday, August 18, 15
  14. Pinot • Pinot segments Tuesday, August 18, 15
  15. Pinot Segment layout: Columnar storage Tuesday, August 18, 15
  16. Pinot Segment layout: Sorted Forward Index Tuesday, August 18, 15
  17. Pinot Segment layout: Other techniques • Indexes: Inverted index, Bitmap, RoaringBitmap • Compression: Dictionary Encoding, P4Delta • Multi Valued columns, skip lists, • Hyperloglog for unique • T-digest for Percentile, Quantile Tuesday, August 18, 15
  18. Data aware pre-computation Star tree Index Tuesday, August 18, 15
  19. Pinot • Query Execution Tuesday, August 18, 15
  20. Pinot Query Execution: Distributed Servers S1 S3 S2 S1 S3 S2 Helix Brokers Tuesday, August 18, 15
  21. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix Brokers Tuesday, August 18, 15
  22. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers Tuesday, August 18, 15
  23. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request Tuesday, August 18, 15
  24. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response Tuesday, August 18, 15
  25. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response 5. Gather Response Tuesday, August 18, 15
  26. Pinot Query Execution: Distributed Servers 1.Query S1 S3 S2 S1 S3 S2 Helix 2. Fetch routing table from HelixBrokers 3. Scatter Request 4. Process Request & send response 5. Gather Response 6. Return Response Tuesday, August 18, 15
  27. Pinot Query Execution: Single Node Architecture EXECUTION ENGINE INVERTED INDEX BITMAP INDEX COLUMN FORMAT PLANNER Tuesday, August 18, 15
  28. Pinot Query Execution: Single Node Architecture SELECT campaignId, sum(clicks) FROM Table A WHERE accountId = 121011 AND 'day' >= 15949 GROUP BY campaignId account Id daycampaign Id click Filter Operator Projection Operator Aggregation Group by Operator Combine Operator Pinot Segments Data sources Matching doc ids campaignId,Click tuple Tuesday, August 18, 15
  29. Pinot • Operations Tuesday, August 18, 15
  30. Cluster Management: Deployment Helix Brokers Servers • Brokers and Servers register themselves in Helix • All servers start with no use case specific configuration Controller Tuesday, August 18, 15
  31. On boarding new use case Helix Brokers Servers XLNT XLNT XLNT Create Table command Controller XLNT XLNTTag Servers TableName Brokers 3 XLNT_T1 1 Tuesday, August 18, 15
  32. Segment Assignment Servers S3 S2 S1 Upload Segment S2 S1 S3 S2 S1 S3 Helix Brokers Copies TableName 2 XLNT_T1 Controller Tuesday, August 18, 15
  33. • AUTO recovery mode: Automatically redistribute segments on failure/addition of new nodes • Custom mode: Run in degraded mode until node is restarted/replaced. Pinot - Fault tolerance/Elasticity Tuesday, August 18, 15
  34. Pinot vs Druid Druid Pinot Architecture Realtime + Offline, Realtime only Realtime + Offline Realtime only -> consistency is hard and schema evolution/Bootstrap is hard Inverted Index Always On all columns, Fixed Configurable on per column basis Allows trade off between scanning v/s inverted index + scanning. More data can be fit in given memory size Data organization N/A Sorts data Organizing data provides speed/better compression and removes the need for inverted index Smart pre- materialization N/A star-tree Allows trade off between latency and space Query Execution Layer Fixed Plan Split into Planning and execution Smart choices can be made at runtime based on metadata/query. Tuesday, August 18, 15
  35. • Documentation & tooling • In progress - consistency among real time replicas. • Improve cost to serve - leverage SSD, partial pre materialization • ThirdEye - Business Metrics Monitoring Pinot - Future Tuesday, August 18, 15
  36. Thank You 30 Tuesday, August 18, 15

×