Let's Talk Operations! (Hadoop Summit 2014)

•

0 likes•1,766 views

These are the introductory slides I used (in some form or another) for the Let's Talk Operations! sessions for the 2014 Hadoop Summits. No video for this one!

Technology

Let’s Talk Operations!
Allen Wittenauer!

Twitter: @_a__w_
Email: aw @ apache.org!

How many individual
grids should I have?

One big
grid
Grid per
project
• Pros!
• Lower ops overhead!
• One location for all data!
• Cons !
• Dev and Prod on one
system
• Pros!
• Capacity planning per project!
• Cons !
• More headcount to maintain!
• Multiple copies of data!
• Data ingress is a mess

ETL
Dev Prod
Base ETL Pull
Event Feeds
Database Feeds
Base ETL Pull
Base ETL Pull
Post-Processed
Data

How do I solve some
common distcp issues?

• Common issues!
• Version incompatibilities!
• Network bandwidth consumption!
!
• Some tricks!
• Use WebHDFS!
• All modern versions support it!
• Read and write in both directions!
• Create a separate queue with hard limits!
• Pull from larger, push from smaller

Q&A
Allen
Wittenauer

Twitter:
@_a__w_
Email:
aw
@
apache.org

20 GB /, ... 200 GB task space (rest) HDFS
• root partitioning
!
!
!
!
!
• non-root partitioning
5 GB
swap
200 GB task space (rest) HDFS

What's hot

Hadoop in Practice (SDN Conference, Dec 2014)Marcel Krcah

Real-time searching of big data with Solr and HadoopRogue Wave Software

Top 10 lessons learned from deploying hadoop in a private cloudRogue Wave Software

All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...Ernie Souhrada

Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014NoSQLmatters

File ContextHadoop User Group

Gophers Riding Elephants: Writing PostgreSQL tools in GoAJ Bahnken

October 2013 HUG: HBase 0.96Yahoo Developer Network

DrupalCon 2011 HighlightSupakit Kiatrungrit

The Evolution of Hadoop at Spotify - Through Failures and PainRafał Wojdyła

Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit

NYC HUG - Application Architectures with Apache Hadoopmarkgrover

Application architectures with Hadoop – Big Data TechCon 2014hadooparchbook

Using Morphlines for On-the-Fly ETLCloudera, Inc.

How to build your query engine in sparkPeng Cheng

Fault Tolerance in Spark: Lessons Learned from Production: Spark Summit East ...Spark Summit

CoscupGiivee The

Hadoop operations-2015-hadoop-summit-san-jose-v5Chris Nauroth

Kafka and Hadoop at LinkedIn MeetupGwen (Chen) Shapira

Simplified Cluster Operation & TroubleshootingDataWorks Summit/Hadoop Summit

What's hot (20)

Hadoop in Practice (SDN Conference, Dec 2014)

Real-time searching of big data with Solr and Hadoop

Top 10 lessons learned from deploying hadoop in a private cloud

All Your IOPS Are Belong To Us - A Pinteresting Case Study in MySQL Performan...

Johnny Miller – Cassandra + Spark = Awesome- NoSQL matters Barcelona 2014

File Context

Gophers Riding Elephants: Writing PostgreSQL tools in Go

October 2013 HUG: HBase 0.96

DrupalCon 2011 Highlight

The Evolution of Hadoop at Spotify - Through Failures and Pain

Flexible and Real-Time Stream Processing with Apache Flink

NYC HUG - Application Architectures with Apache Hadoop

Application architectures with Hadoop – Big Data TechCon 2014

Using Morphlines for On-the-Fly ETL

How to build your query engine in spark

Fault Tolerance in Spark: Lessons Learned from Production: Spark Summit East ...

Coscup

Hadoop operations-2015-hadoop-summit-san-jose-v5

Kafka and Hadoop at LinkedIn Meetup

Simplified Cluster Operation & Troubleshooting

Viewers also liked

Apache Hadoop Shell RewriteAllen Wittenauer

Apache Hadoop for System AdministratorsAllen Wittenauer

Apache Yetus: Intro to Precommit for HBase ContributorsAllen Wittenauer

Apache Yetus: Helping Solve the Last Mile ProblemAllen Wittenauer

Hadoop Operations at LinkedInAllen Wittenauer

Hadoop Performance at LinkedInAllen Wittenauer

Viewers also liked (6)

Apache Hadoop Shell Rewrite

Apache Hadoop for System Administrators

Apache Yetus: Intro to Precommit for HBase Contributors

Apache Yetus: Helping Solve the Last Mile Problem

Hadoop Operations at LinkedIn

Hadoop Performance at LinkedIn

Similar to Let's Talk Operations! (Hadoop Summit 2014)

VoltDB and Erlang - Tech planet 2012Eonblast

Introduction to Cassandra and CQL for Java developersJulien Anguenot

Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...Ceph Community

Fixing twitterRoger Xia

Fixing_Twitterliujianrong

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...smallerror

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...xlight

Big Data Developers Moscow Meetup 1 - sql on hadoopbddmoscow

Liferay & Big Data Dev Con 2014Miguel Pastor

What ya gonna do?CQD

PhegData X - High Performance EBSHanson Dong

A Mayo Clinic Big Data ImplementationBDPA Education and Technology Foundation

Inside Wordnik's ArchitectureTony Tam

Five Years of EC2 DistilledGrig Gheorghiu

SQL Server Konferenz 2014 - SSIS & HDInsightTillmann Eitelberg

Elixir Programming Language 101Around25

Petabyte scale on commodity infrastructureelliando dias

1. SQL Server forSharePoint geeksA gentle introductionThomas Vochten • Septem...BIWUG

Why ruby and railsReuven Lerner

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)Bob Pusateri

Similar to Let's Talk Operations! (Hadoop Summit 2014) (20)

VoltDB and Erlang - Tech planet 2012

Introduction to Cassandra and CQL for Java developers

Ceph Day London 2014 - Best Practices for Ceph-powered Implementations of Sto...

Fixing twitter

Fixing_Twitter

Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...

Big Data Developers Moscow Meetup 1 - sql on hadoop

Liferay & Big Data Dev Con 2014

What ya gonna do?

PhegData X - High Performance EBS

A Mayo Clinic Big Data Implementation

Inside Wordnik's Architecture

Five Years of EC2 Distilled

SQL Server Konferenz 2014 - SSIS & HDInsight

Elixir Programming Language 101

Petabyte scale on commodity infrastructure

1. SQL Server forSharePoint geeksA gentle introductionThomas Vochten • Septem...

Why ruby and rails

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)

Recently uploaded

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos

WordPress Websites for Engineers: Elevate Your Brandgvaughan

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

CloudStudio User manual (basic edition):comworks

Commit 2024 - Secret Management made easyAlfredo García Lavilla

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey

Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University

Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm

Take control of your SAP testing with UiPath Test SuiteDianaGray10

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxnull - The Open Security Community

Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB

Gen AI in Business - Global Trends Report 2024.pdfAddepto

"ML in Production",Oleksandr BaganFwdays

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfPrecisely

Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge

Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren

DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Recently uploaded (20)

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)

WordPress Websites for Engineers: Elevate Your Brand

Human Factors of XR: Using Human Factors to Design XR Systems

CloudStudio User manual (basic edition):

Commit 2024 - Secret Management made easy

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

TeamStation AI System Report LATAM IT Salaries 2024

Nell’iperspazio con Rocket: il Framework Web di Rust!

Streamlining Python Development: A Guide to a Modern Project Setup

Take control of your SAP testing with UiPath Test Suite

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx

Developer Data Modeling Mistakes: From Postgres to NoSQL

Gen AI in Business - Global Trends Report 2024.pdf

"ML in Production",Oleksandr Bagan

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf

Designing IA for AI - Information Architecture Conference 2024

Advanced Test Driven-Development @ php[tek] 2024

DevoxxFR 2024 Reproducible Builds with Apache Maven

Scanning the Internet for External Cloud Exposures via SSL Certs

Let's Talk Operations! (Hadoop Summit 2014)

1. Let’s Talk Operations! Allen Wittenauer!

3. Twitter: @_a__w_ Email: aw @ apache.org!

4. How many individual grids should I have?

5. One big grid Grid per project • Pros! • Lower ops overhead! • One location for all data! • Cons ! • Dev and Prod on one system • Pros! • Capacity planning per project! • Cons ! • More headcount to maintain! • Multiple copies of data! • Data ingress is a mess

6. Data Center Production ETL Development

7. ETL Dev Prod Base ETL Pull Event Feeds Database Feeds Base ETL Pull Base ETL Pull Post-Processed Data

8. DC2DC1 Production ETL Development

9. How do I solve some common distcp issues?

10. • Common issues! • Version incompatibilities! • Network bandwidth consumption! ! • Some tricks! • Use WebHDFS! • All modern versions support it! • Read and write in both directions! • Create a separate queue with hard limits! • Pull from larger, push from smaller

11. Q&A Allen Wittenauer Twitter: @_a__w_ Email: aw @ apache.org

12. Bonus Slide!

13. 20 GB /, ... 200 GB task space (rest) HDFS • root partitioning ! ! ! ! ! • non-root partitioning 5 GB swap 200 GB task space (rest) HDFS

Let's Talk Operations! (Hadoop Summit 2014)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Let's Talk Operations! (Hadoop Summit 2014)

Similar to Let's Talk Operations! (Hadoop Summit 2014) (20)

Recently uploaded

Recently uploaded (20)

Let's Talk Operations! (Hadoop Summit 2014)