Hbase: Introduction to column oriented databases

•

14 gostaram•20,272 visualizações

Big Data is getting more attention each day, followed by new storage paradigms. This presentation shows a fast intro to HBase, a column oriented database used by Facebook and other big players to store and extract knowledge of high volume of data.

Tecnologia

HBase
Introduction to
column oriented
databases

Luís Cipriani
@lfcipriani (twitter, linkedin, github, ...)
22o. GURU (2012-02-25) - Sao Paulo/Brazil

sexta-feira, 24 de fevereiro de 12

intro

“A BigTable HBase is a sparse,
distributed, persistent
multidimensional sorted map”

http://research.google.com/archive/bigtable.html
sexta-feira, 24 de fevereiro de 12

intro > data model
{ <-- table
// ...
"aaaaa" : { <-- row
"A" : { <-- column family
"foo" : { <-- column (qualifier)
15: "y", <-- timestamp, value
4: "m"
}
"bar" : {...}
},
"B" : {
"" : {...}
}
},
"aaaab" : {
"A" : {
"foo" : {...},
"bar" : {...},
"joe" : {...}
},
"B" : {
"" : {...}
}
},
// ...
}
sexta-feira, 24 de fevereiro de 12

intro > data model

(Table, RowKey, Family, Column, Timestamp) → Value

sexta-feira, 24 de fevereiro de 12

intro > hadoop stack

• hadoop HDFS (or not)
• hadoop MapReduce
• hadoop ZooKeeper
• hadoop HBase
• hadoop Hue, Whirr, etc...

sexta-feira, 24 de fevereiro de 12

architecture

sexta-feira, 24 de fevereiro de 12

key design > read/write model

• randon reads (get)
• sequential reads (scan)
• partial key scans
• writes (put = update)

sexta-feira, 24 de fevereiro de 12

key design > storage model

http://ofps.oreilly.com/titles/9781449396107/advanced.html
sexta-feira, 24 de fevereiro de 12

key design > strategies

• tall-narrow vs ﬂat-wide
• partial key scans
• pagination
• time series
• salting
• ﬁeld swap
• randomization
• secondary indexes

sexta-feira, 24 de fevereiro de 12

key design > example

sexta-feira, 24 de fevereiro de 12

development

• installation modes
• standalone, pseudo-distributed, distributed
• JRuby console
• Access
• java/jruby API (more features)
• entrypoints REST, Thrift, Avro, Protobuffers
• there several other libs

sexta-feira, 24 de fevereiro de 12

cons

• complex conﬁg and maintenance
• hot regions
• no secondary index built-in
• no transactions built-in
• complex schema design

sexta-feira, 24 de fevereiro de 12

pros

• distributed
• scalable (auto-sharding)
• built on Hadoop stack
• handles Big Data
• high performance for write and read
• no SPOF
• fault tolerant, no data loss
• active community

sexta-feira, 24 de fevereiro de 12

Reformulação Box de Login Abril ID
http://engineering.abril.com.br/
http://abr.io/hbase-intro
https://pinboard.in/u:lfcipriani/t:hbase/
http://hbase.apache.org/

? http://shop.oreilly.com/product/0636920014348.do

sexta-feira, 24 de fevereiro de 12

Mais conteúdo relacionado

Semelhante a Hbase: Introduction to column oriented databases

Hive introduction 介绍

ablozhou

Intro to HBase - Lars George

JAX London

Hadoop Overview & Architecture

EMC

Brust hadoopecosystem

Andrew Brust

Generalized framework for using NoSQL Databases

KIRAN V

Hadoop/Mahout/HBaseでテキスト分類器を作ったよ

Naoki Yanai

20100128ebay

Jeff Hammerbacher

Jayesh Thakrar, Senior Systems Engineer, Conversant The venerable HBase shell is often regarded as a simple utility to perform basic DDL and maintenance activities. However, it is in fact a powerful, interactive programming environment, primarily due to the JRuby engine under the covers. In this presentation, I'll describe its JRuby heritage and show some of the things that can be done with the "ird" (interactive ruby shell), as well as show how to exploit JRuby and Java integration via concrete working examples. In addition, I will demonstrate how the "shell" can be used in Hadoop streaming to quickly perform complex and large volume batch jobs.

Big Data Everywhere Chicago: Unleash the Power of HBase Shell (Conversant)

BigDataEverywhere

HBase, no trouble

LINE Corporation

Hadoop - Introduction to Hadoop

Vibrant Technologies & Computers

HBaseConEast2016: HBase and Spark, State of the Art

Michael Stack

HBase and Impala Notes - Munich HUG - 20131017

larsgeorge

MongoDB at GUL

Israel Gutiérrez

Hadoop @ eBay: Past, Present, and Future

Ryan Hennig

Semelhante a Hbase: Introduction to column oriented databases (14)

Hive introduction 介绍

Intro to HBase - Lars George

Hadoop Overview & Architecture

Brust hadoopecosystem

Generalized framework for using NoSQL Databases

Hadoop/Mahout/HBaseでテキスト分類器を作ったよ

20100128ebay

Big Data Everywhere Chicago: Unleash the Power of HBase Shell (Conversant)

HBase, no trouble

Hadoop - Introduction to Hadoop

HBaseConEast2016: HBase and Spark, State of the Art

HBase and Impala Notes - Munich HUG - 20131017

MongoDB at GUL

Hadoop @ eBay: Past, Present, and Future

Mais de Luis Cipriani

Adventures with Raspberry Pi and Twitter API

Luis Cipriani

500 milhões de tweets são criados todos os dias pelos usuários do Twitter. A sua plataforma permite que qualquer desenvolvedor tenha acesso para analisar o grande volume de informação gerado todos os dias via a sua API de streaming. Essa palestra apresentará como acessar esse laboratório de dados para criar aplicações que monitoram protestos, permitem usuários interagir com a TV, identificam padrões de comportamento ou que sua imaginação mandar. Discutiremos também quais arquiteturas de software são as ideais para consumir esse tipo de fluxo de dados.

Capturando o pulso do planeta com as APIs de Streaming do Twitter

Luis Cipriani

Twitter e suas APIs de Streaming - Campus Party Brasil 7

Luis Cipriani

Liberar uma API HTTP para seus usuários é a melhor forma de estimular ideias inovadoras baseadas no seu produto, porém para ser livre é necessário responsabilidade. Você já pensou qual o melhor método para garantir a segurança dos dados de seus usuários? Será que só basta implementar o melhor método de autenticação do mercado? Como balancear segurança com a queda de performance percebida pelo usuário, devido ao overhead do processo de autenticação? Devo me preocupar com rate-limiting? Com foco nesses desafios, a palestra apresentará os métodos de autenticação de APIs, suas vantagens e desvantagens, quando utilizar cada uma e liberar os dados com cabeça tranquila.

Segurança de APIs HTTP, um guia sensato para desenvolvedores preocupados

Luis Cipriani

The best HTTP request made to your server is that one that never reaches it. Do you know the life cycle time of your resources? How to be sure that the user never reaches an expired response without the need to open the connection door with the origin server? What kinds of caches do exist and when do I need to use each one of them? Why can I not be afraid to read the RFCs? This talk will present good practices on the usage of HTTP cache for APIs and web applications, turning your digital products to optimize the usage of machines and save money.

API Caching, why your server needs some rest

Luis Cipriani

This talk was given in a Marster's Mobile Computing course (the idea is that each student present to the class the contribution a respective paper gives to the scientific community). So all the credit from parts taken from the article are owned by the article authors, here is the reference: S. Sehic, F. Li, S. Nastic, S Dustdar,“A Programming Model for Context-Aware Applications in Large-Scale Pervasive Systems”, IEEE 8th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), pp 142-149, 2012.

Explaining A Programming Model for Context-Aware Applications in Large-Scale ...

Luis Cipriani

A palestra irá apresentar os desafios e as lições aprendidas em 3 anos de desenvolvimento de uma plataforma para gerência e publicação de conteúdo digital na Abril Mídia que aplica os conceitos de arquiteturas REST e APIs Hipermídia para conseguir entregar um grande volume de informação em diversos tipos de dispositivos. A plataforma é constituída de inúmeros domínios de informação, modulares e independentes, que se integram para permitir a criação de produtos digitais de forma mais ágil. Atualmente a plataforma suporta 14 sites e mais de 12 milhões de pageviews/mês e possui uma arquitetura poliglota, ou seja, com uma grande variedade de linguagens e tecnologias utilizadas.

Alexandria: um Sistema de Sistemas para Publicação de Conteúdo Digital utiliz...

Luis Cipriani

A palestra irá compartilhar a experiência e lições aprendidas no desenvolvimento da plataforma de publicação da Abril, um sistema distribuído com vários nós independentes que se comunicam usando REST e hypermidia. Também introduziremos alguns conceitos avançados de HTTP que podem fazer com que sistemas REST executem com melhor performance, evitando os problemas comuns de se manter uma plataforma em larga escala, com uma grande diversidade de usuários.

Como um verdadeiro sistema REST funciona: arquitetura e performance na Abril

Luis Cipriani

The web is revolutionary when the subject is speed, scale and availbility of information, but not in information classification and organization. It's very common to be in trouble when trying to manipulate public data or even private internal company data. The goal of this presentation is to show a way to understand Semantic Web and how we can use this tool to generate knowledge from raw data, and introduce the technology and specifications involved such as RDF, ontologies and triplestores.

Explaining Semantic Web

Luis Cipriani

Fearless HTTP requests abuse

Luis Cipriani

Mais de Luis Cipriani (10)

Adventures with Raspberry Pi and Twitter API

Capturando o pulso do planeta com as APIs de Streaming do Twitter

Twitter e suas APIs de Streaming - Campus Party Brasil 7

Segurança de APIs HTTP, um guia sensato para desenvolvedores preocupados

API Caching, why your server needs some rest

Explaining A Programming Model for Context-Aware Applications in Large-Scale ...

Alexandria: um Sistema de Sistemas para Publicação de Conteúdo Digital utiliz...

Como um verdadeiro sistema REST funciona: arquitetura e performance na Abril

Explaining Semantic Web

Fearless HTTP requests abuse

Último

A Principled Technologies deployment guide Conclusion Deploying VMware Cloud Foundation 5.1 on next gen Dell PowerEdge servers brings together critical virtualization capabilities and high-performing hardware infrastructure. Relying on our hands-on experience, this deployment guide offers a comprehensive roadmap that can guide your organization through the seamless integration of advanced VMware cloud solutions with the performance and reliability of Dell PowerEdge servers. In addition to the deployment efficiency, the Cloud Foundation 5.1 and PowerEdge solution delivered strong performance while running a MySQL database workload. By leveraging VMware Cloud Foundation 5.1 and PowerEdge servers, you could help your organization embrace cloud computing with confidence, potentially unlocking a new level of agility, scalability, and efficiency in your data center operations.

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Principled Technologies

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Manulife - Insurer Innovation Award 2024

The Digital Insurer

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

This presentation explores the impact of HTML injection attacks on web applications, detailing how attackers exploit vulnerabilities to inject malicious code into web pages. Learn about the potential consequences of such attacks and discover effective mitigation strategies to protect your web applications from HTML injection vulnerabilities. for more information visit https://bostoninstituteofanalytics.org/category/cyber-security-ethical-hacking/

HTML Injection Attacks: Impact and Mitigation Strategies

Boston Institute of Analytics

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Scaling API-first – The story of a global engineering organization

Radu Cotescu

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Hbase: Introduction to column oriented databases

1. HBase Introduction to column oriented databases Luís Cipriani @lfcipriani (twitter, linkedin, github, ...) 22o. GURU (2012-02-25) - Sao Paulo/Brazil sexta-feira, 24 de fevereiro de 12

2. ME sexta-feira, 24 de fevereiro de 12

3. intro “A BigTable HBase is a sparse, distributed, persistent multidimensional sorted map” http://research.google.com/archive/bigtable.html sexta-feira, 24 de fevereiro de 12

4. intro > data model { <-- table // ... "aaaaa" : { <-- row "A" : { <-- column family "foo" : { <-- column (qualifier) 15: "y", <-- timestamp, value 4: "m" } "bar" : {...} }, "B" : { "" : {...} } }, "aaaab" : { "A" : { "foo" : {...}, "bar" : {...}, "joe" : {...} }, "B" : { "" : {...} } }, // ... } sexta-feira, 24 de fevereiro de 12

5. intro > data model (Table, RowKey, Family, Column, Timestamp) → Value sexta-feira, 24 de fevereiro de 12

6. intro > hadoop stack • hadoop HDFS (or not) • hadoop MapReduce • hadoop ZooKeeper • hadoop HBase • hadoop Hue, Whirr, etc... sexta-feira, 24 de fevereiro de 12

7. architecture sexta-feira, 24 de fevereiro de 12

8. key design > read/write model • randon reads (get) • sequential reads (scan) • partial key scans • writes (put = update) sexta-feira, 24 de fevereiro de 12

9. key design > storage model http://ofps.oreilly.com/titles/9781449396107/advanced.html sexta-feira, 24 de fevereiro de 12

10. key design > strategies • tall-narrow vs ﬂat-wide • partial key scans • pagination • time series • salting • ﬁeld swap • randomization • secondary indexes sexta-feira, 24 de fevereiro de 12

11. key design > example sexta-feira, 24 de fevereiro de 12

12. development • installation modes • standalone, pseudo-distributed, distributed • JRuby console • Access • java/jruby API (more features) • entrypoints REST, Thrift, Avro, Protobuffers • there several other libs sexta-feira, 24 de fevereiro de 12

13. cons • complex conﬁg and maintenance • hot regions • no secondary index built-in • no transactions built-in • complex schema design sexta-feira, 24 de fevereiro de 12

14. pros • distributed • scalable (auto-sharding) • built on Hadoop stack • handles Big Data • high performance for write and read • no SPOF • fault tolerant, no data loss • active community sexta-feira, 24 de fevereiro de 12

15. Reformulação Box de Login Abril ID http://engineering.abril.com.br/ http://abr.io/hbase-intro https://pinboard.in/u:lfcipriani/t:hbase/ http://hbase.apache.org/ ? http://shop.oreilly.com/product/0636920014348.do sexta-feira, 24 de fevereiro de 12

Hbase: Introduction to column oriented databases

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Hbase: Introduction to column oriented databases

Semelhante a Hbase: Introduction to column oriented databases (14)

Mais de Luis Cipriani

Mais de Luis Cipriani (10)

Último

Último (20)

Hbase: Introduction to column oriented databases