Baixar para ler offline
![Como armazenar os
dados?How do we store data?
5/11/13 Bill Howe, UW 3
###query length COG hit #1 e-value #1 identity #1 score #1 hit length #1 description #1
chr_4[480001-580000].287 4500
chr_4[560001-660000].1 3556
chr_9[400001-500000].503 4211 COG4547 2.00E-04 19 44.6 620 Cobalamin biosynthesis protein Co
chr_9[320001-420000].548 2833 COG5406 2.00E-04 38 43.9 1001 Nucleosome binding factor SPN, S
chr_27[320001-404298].20 3991 COG4547 5.00E-05 18 46.2 620 Cobalamin biosynthesis protein Co
chr_26[320001-420000].378 3963 COG5099 5.00E-05 17 46.2 777 RNA-binding protein of the Puf fam
chr_26[400001-441226].196 2949 COG5099 2.00E-04 17 43.9 777 RNA-binding protein of the Puf fam
chr_24[160001-260000].65 3542
chr_5[720001-820000].339 3141 COG5099 4.00E-09 20 59.3 777 RNA-binding protein of the Puf fam
chr_9[160001-260000].243 3002 COG5077 1.00E-25 26 114 1089 Ubiquitin carboxyl-terminal hydrol
chr_12[720001-820000].86 2895 COG5032 2.00E-09 30 60.5 2105 Phosphatidylinositol kinase and pr
chr_12[800001-900000].109 1463 COG5032 1.00E-09 30 60.1 2105 Phosphatidylinositol kinase and pr
chr_11[1-100000].70 2886
chr_11[80001-180000].100 1523
ANNOTATIONSUMMARY-COMBINEDORFANNOTATION16_Phaeo_genome
What is the data model?
How do we store data?
###query length COG hit #1 e-value #1 identity #1 score #1 hit length #1 description #1
chr_4[480001-580000].287 4500
chr_4[560001-660000].1 3556
chr_9[400001-500000].503 4211 COG4547 2.00E-04 19 44.6 620 Cobalamin biosynthesis protein CobT (nicotinate-
chr_9[320001-420000].548 2833 COG5406 2.00E-04 38 43.9 1001 Nucleosome binding factor SPN, SPT16 subunit
chr_27[320001-404298].20 3991 COG4547 5.00E-05 18 46.2 620 Cobalamin biosynthesis protein CobT (nicotinate-
chr_26[320001-420000].378 3963 COG5099 5.00E-05 17 46.2 777 RNA-binding protein of the Puf family, translation
chr_26[400001-441226].196 2949 COG5099 2.00E-04 17 43.9 777 RNA-binding protein of the Puf family, translation
chr_24[160001-260000].65 3542
chr_5[720001-820000].339 3141 COG5099 4.00E-09 20 59.3 777 RNA-binding protein of the Puf family, translation
chr_9[160001-260000].243 3002 COG5077 1.00E-25 26 114 1089 Ubiquitin carboxyl-terminal hydrolase
chr_12[720001-820000].86 2895 COG5032 2.00E-09 30 60.5 2105 Phosphatidylinositol kinase and protein kinases o
chr_12[800001-900000].109 1463 COG5032 1.00E-09 30 60.1 2105 Phosphatidylinositol kinase and protein kinases o
chr_11[1-100000].70 2886
chr_11[80001-180000].100 1523
ANNOTATIONSUMMARY-COMBINEDORFANNOTATION16_Phaeo_genome
What is the data model?
w do we store data?
###query length COG hit #1 e-value #1 identity #1 score #1 hit length #1 description #1
chr_4[480001-580000].287 4500
chr_4[560001-660000].1 3556
chr_9[400001-500000].503 4211 COG4547 2.00E-04 19 44.6 620 Cobalamin biosynthesis protein CobT (nicotinate-mononucleotide:5, 6
chr_9[320001-420000].548 2833 COG5406 2.00E-04 38 43.9 1001 Nucleosome binding factor SPN, SPT16 subunit
chr_27[320001-404298].20 3991 COG4547 5.00E-05 18 46.2 620 Cobalamin biosynthesis protein CobT (nicotinate-mononucleotide:5, 6
chr_26[320001-420000].378 3963 COG5099 5.00E-05 17 46.2 777 RNA-binding protein of the Puf family, translational repressor
chr_26[400001-441226].196 2949 COG5099 2.00E-04 17 43.9 777 RNA-binding protein of the Puf family, translational repressor
chr_24[160001-260000].65 3542
chr_5[720001-820000].339 3141 COG5099 4.00E-09 20 59.3 777 RNA-binding protein of the Puf family, translational repressor
chr_9[160001-260000].243 3002 COG5077 1.00E-25 26 114 1089 Ubiquitin carboxyl-terminal hydrolase
chr_12[720001-820000].86 2895 COG5032 2.00E-09 30 60.5 2105 Phosphatidylinositol kinase and protein kinases of the PI-3 kinase fam
chr_12[800001-900000].109 1463 COG5032 1.00E-09 30 60.1 2105 Phosphatidylinositol kinase and protein kinases of the PI-3 kinase fam
chr_11[1-100000].70 2886
chr_11[80001-180000].100 1523
ANNOTATIONSUMMARY-COMBINEDORFANNOTATION16_Phaeo_genome
hat is the data model?](https://image.slidesharecdn.com/126713b0-f677-491a-9aa3-25c90e4044b4-150316201714-conversion-gate01/85/WorkshopMaxtera_RevDados_11mar15-15-320.jpg)
![Joins: Fase da Redução
chave=11111, valor= [(Empregado, Adriano, 11111),
(Setor, 11111, Tecnologia)]
chave=2222, valor= [(Empregado, José Rodrigo, 22222),
(Setor, 22222, Vendas),
(Setor, 22222, Marketing)]
Adriano, 11111, 11111, Tecnologia
José Rodrigo, 22222, 22222,Vendas,
José Rodrigo, 22222, 22222,Marketing](https://image.slidesharecdn.com/126713b0-f677-491a-9aa3-25c90e4044b4-150316201714-conversion-gate01/85/WorkshopMaxtera_RevDados_11mar15-90-320.jpg)
![Arquiteturas
n of
sec-
een-
re-
iled
cial
s of
con-
red-
the
hal-
uter
the
hing
ntly
their architecture can be done at different levels. The most
widely used approach of Michael Stonebraker [18] builds the
basis for further architectural views and will be extended.
Basically, we distinguish between three main approaches.
The shared-everything (SE), the shared-disk (SD) and the
shared-nothing (SN) architecture are one of the basic con-
cepts in a shared database environment. While SE-systems
are sharing the processors (P) / memory resources (M) and
thus constitute a closed circuit, require the SD / SN variants
a communication network (N) to integrate their components.
Figure 1: Stonebraker Architecture with
shared-everything, shared-disk, shared-nothing
A survey of Shared-Nothing Parallel Database
Management Systems
[Comparison between Teradata, Greenplum and Netezza implementations]
Thomas Müseler
University of Applied Science Darmstadt
Haardtring 100
64295 Darmstadt, Germany
Thomas.Mueseler@gmail.com
ABSTRACT
Distributed database systems can be implemented in a many
different ways. Mostly, they are customized for a special
environment to handle big data problems. The data ware-
house sector relies on these amounts, but has changed from
a data storage to a real time management support during
the last years [3]. The resulting increase of compution and
storage capacity poses new requirements to the database
systems. Previous approaches of a parallel database envi-
ronment tried to solve this problem with shared disk and
memory approaches.
The main contribution of this paper is the presentation of
the current technology in the shared-nothing database sec-
tor. The concepts of the manufacturers Teradata, Green-
plum and Netezza will be discussed for data warehouse re-
quirements. Based on an architectural overview is a detailed
insight of the index functionality given which is a crucial
performance factor. Also data distribution algorithms of
the manufacturers are analysed under data warehouse con-
ditions.
At the end is a comparison to other shared concepts (shared-
disk, shared-everything) given and the question raised, if the
actual approach can be fulfilled by the manufacturers.
distribution to each node. Based on the application range
in the data warehouse environment (chapter 5), a compari-
son is made to other architectural models and the scaling of
these kinds of networks. Furthermore is the question raised,
if the shared-nothing can be adapted to other application
fields and is therefore a good opportunity for future imple-
mentations.
2. ARCHITECTURE
The classification of distributed database systems in terms of
their architecture can be done at different levels. The most
widely used approach of Michael Stonebraker [18] builds the
basis for further architectural views and will be extended.
Basically, we distinguish between three main approaches.
The shared-everything (SE), the shared-disk (SD) and the
shared-nothing (SN) architecture are one of the basic con-
cepts in a shared database environment. While SE-systems
are sharing the processors (P) / memory resources (M) and
thus constitute a closed circuit, require the SD / SN variants
a communication network (N) to integrate their components.
A survey of Shared-Nothing Parallel Database
Management Systems
[Comparison between Teradata, Greenplum and Netezza implementations]
Thomas Müseler
University of Applied Science Darmstadt
Haardtring 100
64295 Darmstadt, Germany
Thomas.Mueseler@gmail.com
ABSTRACT
Distributed database systems can be implemented in a many
ifferent ways. Mostly, they are customized for a special
nvironment to handle big data problems. The data ware-
ouse sector relies on these amounts, but has changed from
data storage to a real time management support during
he last years [3]. The resulting increase of compution and
torage capacity poses new requirements to the database
ystems. Previous approaches of a parallel database envi-
onment tried to solve this problem with shared disk and
memory approaches.
The main contribution of this paper is the presentation of
he current technology in the shared-nothing database sec-
or. The concepts of the manufacturers Teradata, Green-
lum and Netezza will be discussed for data warehouse re-
uirements. Based on an architectural overview is a detailed
nsight of the index functionality given which is a crucial
erformance factor. Also data distribution algorithms of
he manufacturers are analysed under data warehouse con-
itions.
At the end is a comparison to other shared concepts (shared-
isk, shared-everything) given and the question raised, if the
ctual approach can be fulfilled by the manufacturers.
1. INTRODUCTION
distribution to each node. Based on the application range
in the data warehouse environment (chapter 5), a compari-
son is made to other architectural models and the scaling of
these kinds of networks. Furthermore is the question raised,
if the shared-nothing can be adapted to other application
fields and is therefore a good opportunity for future imple-
mentations.
2. ARCHITECTURE
The classification of distributed database systems in terms of
their architecture can be done at different levels. The most
widely used approach of Michael Stonebraker [18] builds the
basis for further architectural views and will be extended.
Basically, we distinguish between three main approaches.
The shared-everything (SE), the shared-disk (SD) and the
shared-nothing (SN) architecture are one of the basic con-
cepts in a shared database environment. While SE-systems
are sharing the processors (P) / memory resources (M) and
thus constitute a closed circuit, require the SD / SN variants
a communication network (N) to integrate their components.](https://image.slidesharecdn.com/126713b0-f677-491a-9aa3-25c90e4044b4-150316201714-conversion-gate01/85/WorkshopMaxtera_RevDados_11mar15-95-320.jpg)
Carregado porAdriano Amaral, MSEE
WorkshopMaxtera_RevDados_11mar15
O documento discute a revolução dos dados e como transformar dados em informações valiosas através da modelagem, armazenamento e análise de dados. Ele aborda tópicos como a evolução dos modelos de banco de dados, desafios da era Big Data e como dados abertos podem promover crescimento econômico.
WorkshopMaxtera_RevDados_11mar15
- 1. Revolução dos Dados Adriano Amaral DiretorTecnologia e Soluções about.me/adriano_amaral
- 2.
- 3. Objetivos Teste Prototipagem Idéias Tendências Aprofundamento Tecnológico NovasTecnologias Inovação
- 4. • Explanações comintervenções…tipo bate-papo; • Não tem como objetivo ser o "dono da verdade"; • Dinâmica de Design Thinking, para solução de problemas; • Definir temas e prioridades futuras; Dinâmica
- 5. Os alquimistas estãochegando….
- 8. Teradata Unified DataArchitecture™ AUDIO & VIDEO images text Web & social Machine logs crm scm erp Dual systems Data marts Test/ dev ANALYTICAL ARCHIVE Languages Math & stats Data Mining BUSINESS INTELLIGENCE ApplicationsVIEWPOINT SUPPORT INDEPENDENT DATA MART Discovery platform INTEGRATED DATA WAREHOUSE Data lab Capture | Store | Refine Engineers Data Scientists Business Analysts Marketing Front-Line Workers Operational SystemsCustomers / Partners Executives
- 9. Como transformar dadosem ouro?
- 10. Liberte os dados… DadosInformação Inteligência Imperativo Decisões Resultados Conhecimento Necessidade Utilidade
- 11. Realizando Valor Maxtera Parceiros StreamData Fast Data Big Data Tempo Real Apps Data Sensores (IoT) Não Estruturados Estruturados ENGINE DE DADOS Open Data Decisões Análises Previsões Fraudes Valor DataScientist Consulting Team
- 13.
- 14. Como armazenar os dados?Howdo we store data? 5/11/13 Bill Howe, UW 2
- 15. Como armazenar os dados?Howdo we store data? 5/11/13 Bill Howe, UW 3 ###query length COG hit #1 e-value #1 identity #1 score #1 hit length #1 description #1 chr_4[480001-580000].287 4500 chr_4[560001-660000].1 3556 chr_9[400001-500000].503 4211 COG4547 2.00E-04 19 44.6 620 Cobalamin biosynthesis protein Co chr_9[320001-420000].548 2833 COG5406 2.00E-04 38 43.9 1001 Nucleosome binding factor SPN, S chr_27[320001-404298].20 3991 COG4547 5.00E-05 18 46.2 620 Cobalamin biosynthesis protein Co chr_26[320001-420000].378 3963 COG5099 5.00E-05 17 46.2 777 RNA-binding protein of the Puf fam chr_26[400001-441226].196 2949 COG5099 2.00E-04 17 43.9 777 RNA-binding protein of the Puf fam chr_24[160001-260000].65 3542 chr_5[720001-820000].339 3141 COG5099 4.00E-09 20 59.3 777 RNA-binding protein of the Puf fam chr_9[160001-260000].243 3002 COG5077 1.00E-25 26 114 1089 Ubiquitin carboxyl-terminal hydrol chr_12[720001-820000].86 2895 COG5032 2.00E-09 30 60.5 2105 Phosphatidylinositol kinase and pr chr_12[800001-900000].109 1463 COG5032 1.00E-09 30 60.1 2105 Phosphatidylinositol kinase and pr chr_11[1-100000].70 2886 chr_11[80001-180000].100 1523 ANNOTATIONSUMMARY-COMBINEDORFANNOTATION16_Phaeo_genome What is the data model? How do we store data? ###query length COG hit #1 e-value #1 identity #1 score #1 hit length #1 description #1 chr_4[480001-580000].287 4500 chr_4[560001-660000].1 3556 chr_9[400001-500000].503 4211 COG4547 2.00E-04 19 44.6 620 Cobalamin biosynthesis protein CobT (nicotinate- chr_9[320001-420000].548 2833 COG5406 2.00E-04 38 43.9 1001 Nucleosome binding factor SPN, SPT16 subunit chr_27[320001-404298].20 3991 COG4547 5.00E-05 18 46.2 620 Cobalamin biosynthesis protein CobT (nicotinate- chr_26[320001-420000].378 3963 COG5099 5.00E-05 17 46.2 777 RNA-binding protein of the Puf family, translation chr_26[400001-441226].196 2949 COG5099 2.00E-04 17 43.9 777 RNA-binding protein of the Puf family, translation chr_24[160001-260000].65 3542 chr_5[720001-820000].339 3141 COG5099 4.00E-09 20 59.3 777 RNA-binding protein of the Puf family, translation chr_9[160001-260000].243 3002 COG5077 1.00E-25 26 114 1089 Ubiquitin carboxyl-terminal hydrolase chr_12[720001-820000].86 2895 COG5032 2.00E-09 30 60.5 2105 Phosphatidylinositol kinase and protein kinases o chr_12[800001-900000].109 1463 COG5032 1.00E-09 30 60.1 2105 Phosphatidylinositol kinase and protein kinases o chr_11[1-100000].70 2886 chr_11[80001-180000].100 1523 ANNOTATIONSUMMARY-COMBINEDORFANNOTATION16_Phaeo_genome What is the data model? w do we store data? ###query length COG hit #1 e-value #1 identity #1 score #1 hit length #1 description #1 chr_4[480001-580000].287 4500 chr_4[560001-660000].1 3556 chr_9[400001-500000].503 4211 COG4547 2.00E-04 19 44.6 620 Cobalamin biosynthesis protein CobT (nicotinate-mononucleotide:5, 6 chr_9[320001-420000].548 2833 COG5406 2.00E-04 38 43.9 1001 Nucleosome binding factor SPN, SPT16 subunit chr_27[320001-404298].20 3991 COG4547 5.00E-05 18 46.2 620 Cobalamin biosynthesis protein CobT (nicotinate-mononucleotide:5, 6 chr_26[320001-420000].378 3963 COG5099 5.00E-05 17 46.2 777 RNA-binding protein of the Puf family, translational repressor chr_26[400001-441226].196 2949 COG5099 2.00E-04 17 43.9 777 RNA-binding protein of the Puf family, translational repressor chr_24[160001-260000].65 3542 chr_5[720001-820000].339 3141 COG5099 4.00E-09 20 59.3 777 RNA-binding protein of the Puf family, translational repressor chr_9[160001-260000].243 3002 COG5077 1.00E-25 26 114 1089 Ubiquitin carboxyl-terminal hydrolase chr_12[720001-820000].86 2895 COG5032 2.00E-09 30 60.5 2105 Phosphatidylinositol kinase and protein kinases of the PI-3 kinase fam chr_12[800001-900000].109 1463 COG5032 1.00E-09 30 60.1 2105 Phosphatidylinositol kinase and protein kinases of the PI-3 kinase fam chr_11[1-100000].70 2886 chr_11[80001-180000].100 1523 ANNOTATIONSUMMARY-COMBINEDORFANNOTATION16_Phaeo_genome hat is the data model?
- 17. • Simplificar arealidade; • Definir espaço da amostra; • Traduzir de forma ao entendimento da máquina; • Possibilitar a manipulação; Modelagem!
- 21. • 3 Componentes 1.Estruturas; 2.Constrições; 3.Operações; Oque é um modelo de dados?
- 22. 1.Estruturas; • Linhas oucolunas? • nós ou elementos? • valores chaves? • sequência de Bytes? 2.Constrições; • todas as linhas tem o mesmo número de colunas? • os valores da cada coluna devem ter o mesmo valor? • um filho não pode ter 2 pais? 3.Operações; • Ache o valor da váriavel X • Ache a linha onde a coluna “sobrenome” tem o valor “Oliveira" • Pegue os próximos N bytes O que é um modelo de dados?
- 23. "Uma coleção deinformações organizadas para facilitar a recuperação da mesma" O que é um banco de dados? http://www.usg.edu/galileo/skills/unit04/primer04_01.phtml
- 24. • Que problemaso Banco de Dados resolve? • Compartilhamento • Permite o acesso de vários leitores e escritores simultaneamente; • Forçar a modelagem de dados • Garante que todas as aplicações acessem mesmo formato e organização de dados; • Escala • Trabalha com datasets muito grandes para caber na memória; • Flexibilidade • Usar os dados de um jeito novo, e não imaginados ainda!!! O que esperar do banco de dados?
- 25. • Como essedado é organizado fisicamente no disco? • Que tipos de consulta são eficientemente suportadas por esse modelo e quais não? • Quão complexo é adicionar um dado ou atualizá- lo? • O que acontece quando surgem novas consultas que não havia previsto? Preciso reorganizar os dados? Quão complicado é isso? Questões importantes!!
- 26. • Bando deDados em Rede: Historico dos Bancos de Dados Historical Example: Network Databases 5/11/13 Bill Howe, UW 2 Database: A collection of information organized to afford efficient retrieval Orderer% Customer% Screw% Nut% Washer% Contact%Rep%
- 27. • Banco deDados Hierárquico Historico dos Bancos de Dados Historical Example: Hierarchical Databases Orderer% Customer% Screw% Nut% Nail% Contact%Rep% Orderer% Screw% Nut% Washer% master detail detail Works great if you want to find all orders for a particular customer. But what if you want to find all Customers who ordered a Nail?
- 28. "RDBMS - SistemasGerenciamento de Banco de Dados relacionais, foram inventados para permitir que você use o dado de múltiplas formas, incluindo caminhos que não haviam sido determinados quando o banco foi criado e sua primeira aplicação desenhada” Banco de Dados Relacionais Codd, 1970
- 29. Promover independencia física dosdados… 5/11/13 Bill Howe, eScience Institute Key Idea: “Physical Data Indepen physical data independence files and pointers relations SELECT seq FROM ncbi_seque WHERE seq = GATT f = fopen( table_ fseek(10030440); while (True) { fread(&buf, 1, if (buf == GATT . . .
- 30. Promover uma álgebra dosregistros Key Idea: An Algebra of Tables select project join join Other operators: aggregate, union, difference, cross product
- 31. Relacional X Analítico RelacionalX Analitico Equivalent logical expressions; different costs 1 σp=knows(R) o=s (σp=holdsAccount(R) o=s σp=accountHomepage(R)) (σp=knows(R) o=s σp=holdsAccount(R)) o=s σp=accountHomepage(R) σp1=knows & p2=holdsAccount & p3=accountHomepage (R x R x R) right associative left associative cross product Mesma operação, custos diferentes!
- 32. Complexidade Sofisticação do Dado AtualizaçãoContínua & Sensível ao tempo, consultas mais importantes OPERACIONALIZANDO O QUE está acontecendo? Comandos baseado em eventos assumem o ambiente ATIVANDO FAZENDO acontecer! Atualização Continua e Consultas Rápidas Ações baseada em eventos Cresce os modelos analíticos PREVENDO O QUE IRÁ acontecer? Batch Ad Hoc Analytics Aumento de análises Ad Hoc ANALISANDO PORQUE isso aconteceu? Batches & Relatórios Ad Hoc REPORTANDO O QUE aconteceu? Evolução de uso Ambiente Analítico
- 33. “Big Data équalquer dado que é caro demais para gerenciar e extrair valor” Bigdata: Definição… Michael Franklin Thomas M. Siebel Professor of Computer Science Director of the Algorithms, Machines and People Lab University of Berkeley
- 34. • Velocidade • latênciado dado mediante diversidade de demandas e o crescimento da interatividade; • Variedade • diversidade de formatos, qualidade, fontes e estruturas; • Volume • Tamanho dos dados; Bigdata: Desafios…
- 35. Cadê a pedra filosofalentão?
- 36. Cidadãos/Consumidores Demanda de novosserviços Governos/Empresas Adaptação para atender as necessidades Slide, Arturo Muente-Kunigami @n0wh3r3m4n
- 37.
- 38. 1.Governo/Empresas como Plataforma 2.Cidadãossabem mais e melhor que o governo (sensores) 3.Sistemas pequenos, baixos acoplamentos 4.Para ganhar a confiança, entregue! 5.Reutilize sistemas e políticas, agregue com inovação 6.Tecnologia como facilitador Proposta…
- 39.
- 41.
- 42. J U NE 2 0 1 4 Open for Business: How Open Data Can Help Achieve the G20 Growth Target A Lateral Economics report commissioned by Omidyar Network
- 43. DadosAbertos e o desenvolvimento 13trilhõesde dólares nos próximos 5 anos Crescimento de 1.1% do PIB do G20, dentre os 2% previstos nos 5 anos U$14,5 Billhões por ano, e provavelmente esse valor esta subestimado…(caso australiano)
- 44. • Reduz ocustos dos serviços do Governo e da Iniciativa Privada; • Possibilidade de novos serviços e aumento da qualidade dos serviços existentes; • Aumento da confiança no Governo devido o aumento da governança, transparência e engajamento dos cidadãos; Valor dos dados abertos…
- 45. Dados que gerammais valor… Educação Fazenda Transporte Varejo Energia Saúde Agricultura Emprego Fonte: Open for Business
- 46. Iniciativa Privada Empresas como Catalizadores!!! Governo SociedadeCivil Sociedade Civil Sociedade Civil Sociedade Civil Sociedade Civil $$$
- 47. 200M libras em prescriçõesno SUS Britânico (NHS) http://www.economist.com/news/britain/21567980-how-scrutiny-freely-available-data-might-save-nhs-money-beggar-thy-neighbour
- 59.
- 60.
- 61.
- 62.
- 63. Resolvendo problemas públicos comTecnologia
- 65.
- 67.
- 68. • Operacionalmente • NoPassado: Funcionava, mesmo se o dado não coubesse na memória; • Agora: Posso utilizar vários pequenos computadores (barato) • "Algoritmamente" • No Passado: Para uma determinada quantidade de dados (N), tenho finitas operações; (Nm) - Polinomial • Agora: Para um montante crescente de dados, preciso realizar um volume maior de operações (Nm/k) - Polinomial Paralelizado • Em breve: Dados fluem em um fluxo contínuo de diversa fontes, consultas realizadas continuamente (N*log(N)) - (StreamData) • Ex: Telescópios de Varredura (30TB/noite) Escalando…
- 69. • Imagine procuraruma seqüência de DNA • Todas as seqüências iguais a: • GATTACGATATTA Explorando possibilidades GATTACGATATTATACCTGCCGTAA
- 70. GATTACGATATTA TACCTGCCGTAA = GATTACGATATTA? Ciclo = 0 GATTACGATATTA CCCCCAATGAC = GATTACGATATTA ? Ciclo = 1
- 71. GATTACGATATTA GATTACGATATTA = GATTACGATATTA? Ciclo = 40 GATTACGATATTA 40 Registros = 40 Comparações
- 72.
- 73. GATTACGATATTA CTGTACACAACCT CTGTACACAACCT < GATTACGATATTA? 0% 100% Começamos dividindo ao meio!
- 74.
- 75. GATTACGATATTA 0% 100% 40 Registros= 4 Comparações N registros = log (N) Comparações
- 76.
- 77. ……. 40 Registros /6 Trabalhadores (N/k)
- 78.
- 79.
- 80. map map mapmap map map reduce reduce reduce reduce 3 5 4
- 81.
- 82. Hadoop in One Slide src:Huy Vo, NYU Poly
- 84. • Nigredo: ouOperação Negra, é o estágio em que a matéria é dissolvida e putrefacta (associada ao calor e ao fogo); • Albedo: ou Operação Branca, é o estágio em que a substância é purificada (associada à ablução com Aquae Vitae, à luz da lua, feminina e à prata); • Citrinitas: ou Operação Amarela, é o estágio em que se opera a transmutação dos metais, da prata em ouro, ou da luz da lua, passiva, em luz solar, ativa; Processo Alquimico http://pt.wikipedia.org/wiki/Alquimia
- 85. Map • Input =(inputkey, value) • Output = (intermediatekey, value) - distribuidos Reduce • Input = (intermediatekey, value) • Output = (outputkey, value) - reagrupados Simplificação do Modelo de Dados Dados = Arquivo = saco de pares (key, value)
- 86. Simplificação do Modelo deDados…outro…. RDF Resource Description Framework
- 87. Implementação de Joins usandoMap-Reduce Nome ID Adriano 11111 José Rodrigo 22222 Empregados EmpID Setor 11111 Tecnologia 2222 Vendas 2222 Marketing Setor Associado Empregados ⋈ SetorAssociado Nome ID EmpID Setor Adriano 11111 11111 Tecnologia José Rodrigo 22222 2222 Vendas José Rodrigo 22222 2222 Marketing
- 88. Joins: Antes do Mapeamento NomeID Adriano 11111 José Rodrigo 22222 Empregados EmpID Setor 11111 Tecnologia 2222 Vendas 2222 Marketing Setor Associado Empregado, Adriano, 11111 Empregado, José Rodrigo, 22222 Setor, 11111, Tecnologia Setor, 22222, Vendas Setor, 22222, Marketing Juntar os dados em um grande bloco de dados
- 89. Joins: Função de Mapeamento… Empregado,Adriano, 11111 Empregado, José Rodrigo, 22222 Setor, 11111, Tecnologia Setor, 22222, Vendas Setor, 22222, Marketing Pares: (chave, valor) chave=11111, valor= (Empregado, Adriano, 11111) chave=2222, valor= (Empregado, José Rodrigo, 22222) chave=11111, valor= (Setor, 11111, Tecnologia) chave=22222, valor= (Setor, 22222, Vendas) chave=22222, valor= (Setor, 22222, Marketing)
- 90. Joins: Fase daRedução chave=11111, valor= [(Empregado, Adriano, 11111), (Setor, 11111, Tecnologia)] chave=2222, valor= [(Empregado, José Rodrigo, 22222), (Setor, 22222, Vendas), (Setor, 22222, Marketing)] Adriano, 11111, 11111, Tecnologia José Rodrigo, 22222, 22222,Vendas, José Rodrigo, 22222, 22222,Marketing
- 91. • DFS -Distributed File System • Processamento Paralelo Massivo - MPP • Tolerância a falha pela duplicação de “chunks” em nós paralelos • Nó Master e Trabalhadores se dividem nas fases de Mapping e Reducing Implementações Map Reduce
- 92.
- 93. MPP - MassiveParallel Processing
- 94. MPP - MassiveParallel Processing
- 95. Arquiteturas n of sec- een- re- iled cial s of con- red- the hal- uter the hing ntly theirarchitecture can be done at different levels. The most widely used approach of Michael Stonebraker [18] builds the basis for further architectural views and will be extended. Basically, we distinguish between three main approaches. The shared-everything (SE), the shared-disk (SD) and the shared-nothing (SN) architecture are one of the basic con- cepts in a shared database environment. While SE-systems are sharing the processors (P) / memory resources (M) and thus constitute a closed circuit, require the SD / SN variants a communication network (N) to integrate their components. Figure 1: Stonebraker Architecture with shared-everything, shared-disk, shared-nothing A survey of Shared-Nothing Parallel Database Management Systems [Comparison between Teradata, Greenplum and Netezza implementations] Thomas Müseler University of Applied Science Darmstadt Haardtring 100 64295 Darmstadt, Germany Thomas.Mueseler@gmail.com ABSTRACT Distributed database systems can be implemented in a many different ways. Mostly, they are customized for a special environment to handle big data problems. The data ware- house sector relies on these amounts, but has changed from a data storage to a real time management support during the last years [3]. The resulting increase of compution and storage capacity poses new requirements to the database systems. Previous approaches of a parallel database envi- ronment tried to solve this problem with shared disk and memory approaches. The main contribution of this paper is the presentation of the current technology in the shared-nothing database sec- tor. The concepts of the manufacturers Teradata, Green- plum and Netezza will be discussed for data warehouse re- quirements. Based on an architectural overview is a detailed insight of the index functionality given which is a crucial performance factor. Also data distribution algorithms of the manufacturers are analysed under data warehouse con- ditions. At the end is a comparison to other shared concepts (shared- disk, shared-everything) given and the question raised, if the actual approach can be fulfilled by the manufacturers. distribution to each node. Based on the application range in the data warehouse environment (chapter 5), a compari- son is made to other architectural models and the scaling of these kinds of networks. Furthermore is the question raised, if the shared-nothing can be adapted to other application fields and is therefore a good opportunity for future imple- mentations. 2. ARCHITECTURE The classification of distributed database systems in terms of their architecture can be done at different levels. The most widely used approach of Michael Stonebraker [18] builds the basis for further architectural views and will be extended. Basically, we distinguish between three main approaches. The shared-everything (SE), the shared-disk (SD) and the shared-nothing (SN) architecture are one of the basic con- cepts in a shared database environment. While SE-systems are sharing the processors (P) / memory resources (M) and thus constitute a closed circuit, require the SD / SN variants a communication network (N) to integrate their components. A survey of Shared-Nothing Parallel Database Management Systems [Comparison between Teradata, Greenplum and Netezza implementations] Thomas Müseler University of Applied Science Darmstadt Haardtring 100 64295 Darmstadt, Germany Thomas.Mueseler@gmail.com ABSTRACT Distributed database systems can be implemented in a many ifferent ways. Mostly, they are customized for a special nvironment to handle big data problems. The data ware- ouse sector relies on these amounts, but has changed from data storage to a real time management support during he last years [3]. The resulting increase of compution and torage capacity poses new requirements to the database ystems. Previous approaches of a parallel database envi- onment tried to solve this problem with shared disk and memory approaches. The main contribution of this paper is the presentation of he current technology in the shared-nothing database sec- or. The concepts of the manufacturers Teradata, Green- lum and Netezza will be discussed for data warehouse re- uirements. Based on an architectural overview is a detailed nsight of the index functionality given which is a crucial erformance factor. Also data distribution algorithms of he manufacturers are analysed under data warehouse con- itions. At the end is a comparison to other shared concepts (shared- isk, shared-everything) given and the question raised, if the ctual approach can be fulfilled by the manufacturers. 1. INTRODUCTION distribution to each node. Based on the application range in the data warehouse environment (chapter 5), a compari- son is made to other architectural models and the scaling of these kinds of networks. Furthermore is the question raised, if the shared-nothing can be adapted to other application fields and is therefore a good opportunity for future imple- mentations. 2. ARCHITECTURE The classification of distributed database systems in terms of their architecture can be done at different levels. The most widely used approach of Michael Stonebraker [18] builds the basis for further architectural views and will be extended. Basically, we distinguish between three main approaches. The shared-everything (SE), the shared-disk (SD) and the shared-nothing (SN) architecture are one of the basic con- cepts in a shared database environment. While SE-systems are sharing the processors (P) / memory resources (M) and thus constitute a closed circuit, require the SD / SN variants a communication network (N) to integrate their components.
- 96.
- 97. Distribuindo… Parallel Query Example:Teradata AMP = unit of parallelism
- 98. Copyright © 2013by Teradata Corporation and MPP Systems ware that makes a MPP system appear to be a single system nistrators. BYNET 0 BYNET 1 Node 0 PEPE AMP AMP AMP AMP : : AMP AMP PDE O.S. PEPE AMP AMP AMP AMP : : AMP AMP PDE O.S. PEPE AMP AMP AMP AMP : : AMP AMP PDE O.S. PEPE AMP AMP AMP AMP : : AMP AMP PDE O.S. Node 1 Node 2 Node 3 ts of e are al PE) Layer Layer E and des MPP - Massive Parallel Processing AMP Access Module Processor
- 99. 1 5/15/2013 Copyright© 2013 by Teradata Corporation The Parsing Engine The Parsing Engine is responsible for: • Managing individual sessions (up to 120) • Parsing and Optimizing your SQL requests • Dispatching the optimized plan to the AMPs • Input conversion (EBCDIC / ASCII) - if necessary • Sending the answer set response back to the requesting client Answer Set Response Parsing Engine SQL Request Parser Optimizer Dispatcher Message Passing Layer AMP AMP AMP AMP Núcleo de Inteligência Parser Engines
- 100. Hardware: Achando dado… 1 5/16/2013Copyright © 2013 by Teradata Corporation Which AMP has the Row? Hashing Algorithm RH Data Table ID Row Hash PI values HBN and data PARSER Data Table Message Passing Layer (Hash Maps) AMP 1 AMP n - 1AMP x... ...AMP 0 AMP n PI value = 197190 Hashing Algorithm 000A1F4A SQL with primary index values and data. For example: Assume PI value is 197190 Summary The MPL accesses the Hash Map using Hash Bucket Number (HBN) of 000A1. Bucket # 000A1 contains the AMP number that has this hash value – effectively the AMP with this row. HBN – Hash Bucket Number HBN Hash Maps AMP # Row ID Row Data Row Hash Uniq Value x '00000000' x'000A1F4A' 0000 0001 38 x 'FFFFFFFF'
- 101. BIG DATA WEB Petabytes CRM Terabytes Gigabytes ERP Exabytes INCREASING DataVariety and Complexity User Generated Content Mobile Web SMS/MMS Sentiment External Demographics HD Video Speech to Text Product/ Service Logs Social Network Business Data Feeds User Click Stream Web Logs Offer History A/B Testing Dynamic Pricing Affiliate Networks Search Marketing Behavioral Targeting Dynamic Funnels Payment Record Support Contacts Customer Touches Purchase Detail Purchase Record Offer Details Segmentation Big Data: de transações para interações Análise de Comportamento ALL DATA Como extrair valor de negócio?
- 102. 5/15/13 Bill Howe,eScience Institute 31 Design Space 31" Throughput"Latency" Internet" Private" data" center" Data&" parallel" Shared" memory" The area we’re discussing inspired by a slide by Michael Isard at Microsoft Research In a few weeks
- 103.
- 104. Graph vs. SQLand SQL- MR B has high betweenness. You get that from a graph Caller Recipient # of calls made A B 10 A C 25 A D 32 A E 3 B I 7 C D 5 A B DC E GF H K J L M I SQL or SQL-MR will tell you A makes a lot of phone calls
- 105. ?Adriano Amaral Diretor Tecnologiae Soluções about.me/adriano_amaral adriano.amaral@maxtera.com.br