My other computer is a datacentre

•Transferir como PPT, PDF•

0 gostou•2,037 visualizações

Steve Loughran

Talk on the engineering aspects of building a datacentre, then a look at running MapReduce over the machines

Tecnologia

My other computer is a datacentre Steve Loughran Julio Guijarro November 2008

His other computer is a datacentre ,[object Object],[object Object],[object Object],code.google.com

His other computer is a datacentre ,[object Object],[object Object]

Their other computer is this datacentre Yahoo! 8000 nodes, 32K cores, 16 Petabytes

Economical Scale ,[object Object],[object Object],[object Object],[object Object]

Power ,[object Object],[object Object],[object Object],[object Object]

Network Fabric ,[object Object],[object Object],[object Object],[object Object],Bandwidth between racks is a bottleneck

Where? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

High Availability ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Web Front End ,[object Object],[object Object],[object Object],[object Object],[object Object]

Work engine ,[object Object],[object Object],[object Object],[object Object],[object Object]

The Cotham Bluetooth Dataset ,[object Object],[object Object],[object Object],[object Object],{lost,"00:0F:B3:92:05:D3","2008-04-17T22:11:15",1124313075} {found,"00:0F:B3:92:05:D3","2008-04-17T22:11:29",1124313089} {lost,"00:0F:B3:92:05:D3","2008-04-17T22:24:45",1124313885} {found,"00:0F:B3:92:05:D3","2008-04-17T22:25:00",1124313900} {found,"00:60:57:70:25:0F","2008-04-17T22:29:00",1124314140}

$MapReduce to Day of Week map_found_event_by_day_of_week( {event, found, Device, _, Timestamp}, Reducer) -> DayOfWeek = timestamp_to_day(Timestamp), Reducer ! {DayOfWeek, Device}. size(Key, Entries, A) -> L = length(Entries), [ {Key, L} | A]. mr_day_of_week(Source) -> mr(Source, fun traffic:map_found_event_by_day_of_week/2, fun traffic:size/3, []).$

$Results traffic:mr_day_of_week(big). [{3,3702}, {6,3076}, {2,3747}, {5,3845}, {1,3044}, {4,3850}, {7,2274}] Monday 3044 Tuesday 3747 Wednesday 3702 Thursday 3850 Friday 3845 Saturday 3076 Sunday 2274$

Filesystem ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Mais conteúdo relacionado

Semelhante a My other computer is a datacentre

My other computer_is_a_datacentreSteve Loughran

Farms, Fabrics and CloudsSteve Loughran

Google Cloud Computing on Google Developer 2008 Dayprogrammermag

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Bhupesh Bansal

Hadoop and Voldemort @ LinkedInHadoop User Group

Ethernet summit 2011_toeintilop

Resilience: the key requirement of a [big] [data] architecture - StampedeCon...StampedeCon

IOT with Drupal 8 - Webinar Hyderabad Drupal CommunityPrateek Jain

Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...Nati Shalom

Introduction To Web TechnologyArun Kumar

Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)Ontico

Design Choices for Cloud Data PlatformsAshish Mrig

Cloud Computing ...changes everythingLew Tucker

HTTP at your local BigCopgriess

Cloud Experience: Data-driven Applications Made Simple and FastDatabricks

Building scalable and reliable websitesTomasz Zen Napierala

Distributed Systems: scalability and high availabilityRenato Lucindo

History of ComputingPierre Vigneras

Mphasis Digital POV - Emerging Open Standard Protocol stack for IoTAniruddha Chakrabarti

What Have We Lost - A look at some historical techniquesLloydMoore

Semelhante a My other computer is a datacentre (20)

My other computer_is_a_datacentre

Farms, Fabrics and Clouds

Google Cloud Computing on Google Developer 2008 Day

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010

Hadoop and Voldemort @ LinkedIn

Ethernet summit 2011_toe

Resilience: the key requirement of a [big] [data] architecture - StampedeCon...

IOT with Drupal 8 - Webinar Hyderabad Drupal Community

Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...

Introduction To Web Technology

Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)

Design Choices for Cloud Data Platforms

Cloud Computing ...changes everything

HTTP at your local BigCo

Cloud Experience: Data-driven Applications Made Simple and Fast

Building scalable and reliable websites

Distributed Systems: scalability and high availability

History of Computing

Mphasis Digital POV - Emerging Open Standard Protocol stack for IoT

What Have We Lost - A look at some historical techniques

Mais de Steve Loughran

Hadoop Vectored IOSteve Loughran

The age of rename() is overSteve Loughran

What does Rename Do: (detailed version)Steve Loughran

Put is the new rename: San Jose Summit EditionSteve Loughran

@Dissidentbot: dissent will be automated!Steve Loughran

PUT is the new rename()Steve Loughran

Extreme Programming DeployedSteve Loughran

TestingSteve Loughran

I hate mockingSteve Loughran

What does rename() do?Steve Loughran

Dancing Elephants: Working with Object Storage in Apache Spark and HiveSteve Loughran

Apache Spark and Object Stores —for London Spark User GroupSteve Loughran

Spark Summit East 2017: Apache spark and object storesSteve Loughran

Hadoop, Hive, Spark and Object StoresSteve Loughran

Apache Spark and Object StoresSteve Loughran

Household INFOSEC in a Post-Sony EraSteve Loughran

Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionSteve Loughran

Hadoop and Kerberos: the Madness Beyond the GateSteve Loughran

Slider: Applications on YARNSteve Loughran

YARN ServicesSteve Loughran

Mais de Steve Loughran (20)

Hadoop Vectored IO

The age of rename() is over

What does Rename Do: (detailed version)

Put is the new rename: San Jose Summit Edition

@Dissidentbot: dissent will be automated!

PUT is the new rename()

Extreme Programming Deployed

Testing

I hate mocking

What does rename() do?

Dancing Elephants: Working with Object Storage in Apache Spark and Hive

Apache Spark and Object Stores —for London Spark User Group

Spark Summit East 2017: Apache spark and object stores

Hadoop, Hive, Spark and Object Stores

Apache Spark and Object Stores

Household INFOSEC in a Post-Sony Era

Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition

Hadoop and Kerberos: the Madness Beyond the Gate

Slider: Applications on YARN

YARN Services

Último

Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous

AXA XL - Insurer Innovation Award Americas 2024The Digital Insurer

Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services

DBX First Quarter 2024 Investor PresentationDropbox

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

Ransomware_Q4_2023. The report. [EN].pdfOverkill Security

A Year of the Servo Reboot: Where Are We Now?Igalia

2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong

Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1

Corporate and higher education May webinar.pptxRustici Software

Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2

GenAI Risks & Security Meetup 01052024.pdflior mazor

Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays

Real Time Object Detection Using Open CVKhem

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung

My other computer is a datacentre

1. My other computer is a datacentre Steve Loughran Julio Guijarro November 2008

2. Our other computer is a datacentre

5. Their other computer is this datacentre Yahoo! 8000 nodes, 32K cores, 16 Petabytes

6. Problem: scale

8. There are no free electrons

10. Hardware

11.

12.

13.

14. How?

15.

16.

17.

18.

19. MapReduce: Hadoop

20.

21. MapReduce to Day of Week map_found_event_by_day_of_week( {event, found, Device, _, Timestamp}, Reducer) -> DayOfWeek = timestamp_to_day(Timestamp), Reducer ! {DayOfWeek, Device}. size(Key, Entries, A) -> L = length(Entries), [ {Key, L} | A]. mr_day_of_week(Source) -> mr(Source, fun traffic:map_found_event_by_day_of_week/2, fun traffic:size/3, []).

22. Results traffic:mr_day_of_week(big). [{3,3702}, {6,3076}, {2,3747}, {5,3845}, {1,3044}, {4,3850}, {7,2274}] Monday 3044 Tuesday 3747 Wednesday 3702 Thursday 3850 Friday 3845 Saturday 3076 Sunday 2274

23. Hadoop running MapReduce

24.

25. What will your datacentre do?

My other computer is a datacentre

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a My other computer is a datacentre

Semelhante a My other computer is a datacentre (20)

Mais de Steve Loughran

Mais de Steve Loughran (20)

Último

Último (20)

My other computer is a datacentre