Architecting the Future of Big Data and Search

Architecting the Future of Big Data and Search Eric Baldeschwieler, Hortonworks e14@hortonworks.com, 19 October 2011

What I Will Cover ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Apache Hadoop is… ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

More Apache Hadoop Projects Programming Languages Computation Object Storage Zookeeper (Coordination) Core Apache Hadoop Related Apache Projects HDFS (Hadoop Distributed File System) MapReduce (Distributed Programing Framework) Ambari (Management) Table Storage Hive (SQL) Pig (Data Flow) HBase (Columnar Storage) HCatalog (Meta Data)

Example Hardware & Network ,[object Object],[object Object],[object Object],2 * 10GigE 2 * 10GigE 2 * 10GigE 2 * 10GigE ,[object Object],[object Object],[object Object],[object Object],[object Object],… Rack Switch 1-2U server … Rack Switch 1-2U server … Rack Switch 1-2U server … Rack Switch 1-2U server …

MapReduce ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HDFS: Scalable, Reliable, Managable ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Switch … Switch … Switch … Core Switch Core Switch …

HBase ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

A Brief History 2006 – present ,[object Object],[object Object],Apache Hadoop ,[object Object],[object Object],Nascent / 2011 ,[object Object],[object Object],2008 – present … ,[object Object],[object Object],2010 – present … Cloudera, MapR Microsoft IBM, EMC, Oracle

Early Adopters & Uses advertising optimization mail anti-spam video & audio processing ad selection web search user interest prediction customer trend analysis analyzing web logs content optimization data analytics machine learning data mining text mining social media

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CASE STUDY YAHOO! WEBMAP © Yahoo 2011

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],CASE STUDY WEBMAP PROJECT RESULTS © Yahoo 2011

[object Object],CASE STUDY YAHOO SEARCH ASSIST™ © Yahoo 2011 ,[object Object],[object Object],[object Object],Before Hadoop After Hadoop Time 26 days 20 minutes Language C++ Python Development Time 2-3 weeks 2-3 days

HADOOP @ YAHOO! TODAY 40K+ Servers 170 PB Storage 5M+ Monthly Jobs 1000+ Active users © Yahoo 2011

[object Object],CASE STUDY YAHOO! HOMEPAGE ,[object Object],[object Object],[object Object],[object Object],© Yahoo 2011 +160% clicks vs. one size fits all +79% clicks vs. randomly selected +43% clicks vs. editor selected Recommended links News Interests Top Searches

CASE STUDY YAHOO! HOMEPAGE ,[object Object],[object Object],[object Object],[object Object],USER BEHAVIOR CATEGORIZATION MODELS (weekly) SERVING MAPS (every 5 minutes) USER BEHAVIOR ,[object Object],[object Object],[object Object],© Yahoo 2011 SCIENCE HADOOP CLUSTER SERVING SYSTEMS PRODUCTION HADOOP CLUSTER ENGAGED USERS

[object Object],CASE STUDY YAHOO! MAIL ,[object Object],[object Object],[object Object],[object Object],© Yahoo 2011 ,[object Object],[object Object],[object Object],SCIENCE PRODUCTION

Adoption Drivers ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Gartner predicts 800% data growth over next 5 years 80-90% of data produced today is unstructured

Key Success Factors ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Our Roadmap ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2011 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2012 (Alphas in Q4 2011)

Investigating Apache Hadoop and Lucene

Developer Questions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Business Questions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Thank You ,[object Object],[object Object]

Architecting the Future of Big Data and Search

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Architecting the Future of Big Data and Search

Semelhante a Architecting the Future of Big Data and Search (20)

Mais de Hortonworks

Mais de Hortonworks (20)

Último

Último (20)

Architecting the Future of Big Data and Search

Notas do Editor