Seu banco de dados na nuvem: Opções de bancos de dados na AWS e padrões de arquitetura
•Transferir como PPTX, PDF•
2 gostaram•576 visualizações
Seu banco de dados na nuvem: Opções de bancos de dados na AWS e padrões de arquitetura - AWS Cloud Experience - Começando na AWS
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Seu banco de dados na nuvem: Opções de bancos de dados na AWS e padrões de arquitetura
Semelhante a Seu banco de dados na nuvem: Opções de bancos de dados na AWS e padrões de arquitetura (20)
Mais de Amazon Web Services LATAM
Mais de Amazon Web Services LATAM (20)
Seu banco de dados na nuvem: Opções de bancos de dados na AWS e padrões de arquitetura
- 1. Seu banco de dados na nuvem: Opções de Bancos de Dados na AWS e padrões de arquitetura Hugo Rozestraten – Arquiteto de Soluções
- 2. Agenda • Banco de Dados relacionais; • Migração de dados para a AWS; • Bancos de Dados NoSQL; • Outras opções de Armazenamento e Busca;
- 3. Bancos de Dados Relacionais Amazon EC2 Amazon RDS Muitos DBs suportados em Linux ou Windows
- 4. Banco de Dados na AWS: RDS ou EC2? Serviços PerformanceVolume Disponibilidade SegurançaFeatures
- 5. Seu Data Center Serviços Energia,HVAC,rede Rack e Cabeamento Manuten. Servidor Patches SO DB software patches Database backups Escalabilidade Alta Disponibilidade DB software installs Instalação SO Otimização Apps
- 6. AWS CloudFormation Amazon EC2 – Install or Clone Serviços Patches SO DB software patches Database backups Escalabilidade Alta Disponibilidade DB software installs Otimização Apps Energia,HVAC,rede Rack e Cabeamento Manuten. Servidor Instalação SO Amazon EC2 AMI
- 7. Amazon RDS – Relational Database Services Serviços Energia,HVAC,rede Rack e Cabeamento Manuten. Servidor Instalação SO Otimização Apps Patches SO DB software patches Database backups Escalabilidade Alta Disponibilidade DB software installs Amazon RDS
- 8. Database Size Volume Amazon EC2 Amazon RDS 6 TeraBytes Oracle 4 Terabytes SQLServer +20 (soft limit) TeraBytes
- 9. Performance Capacidade Computacional 1 vCPU a 128vCPUs 1 vCPU a 40vCPUs Memória GB of RAM 1 GB a 1.952 GB 1 GB a 244 GB Redes (Throughput) Low a 20 Gbps Low a 10 Gbps Storage I/O Throughput 48.000 IOPS 30.000 IOPS R3, R4 instance support Instance Families: T2, M3, M4 Amazon EC2 Amazon RDS
- 10. Disponibilidade Node 1 Node 2 Storage 1 Storage 2 Storage 3 Node 3 Node 4 Mesmo Rack Mesmo appliance Mesmo Data Center Mesma entrada de energia Mesma localização geográfica
- 12. Disponibilidade SQL Server Amazon EC2Amazon RDS
- 14. Criptografia de Dados EBS AWS KMS volume Oracle Instance RDS TDE CloudHSM TDE_HSM Segurança Segurança
- 15. Banco de Dados Relacional, compatível com MySQL Entregando Performance e disponibilidade dos Bancos de dados Comerciais Simplicidade e eficiência de custo de um Banco de dados open-source O que é Amazon Aurora?
- 16. Aurora cluster Amazon S3 AZ 1 AZ 2 AZ 3 Aurora primary instance Cluster volume spans 3 AZs
- 17. Aurora cluster com réplicas Amazon S3 AZ 1 AZ 2 AZ 3 Aurora primary instance Cluster volume spans 3 AZs Aurora Replica Aurora Replica
- 18. Tráfego de I/O com MySQL BINLOG DATA DOUBLE-WRITELOG FRM FILES T Y P E O F W R I T E MYSQL WITH STANDBY EBS mirrorEBS mirror AZ 1 AZ 2 Amazon S3 EBS Amazon Elastic Block Store (EBS) Primary Instance Standby Instance 1 2 3 4 5 Fluxo de IO Complexo 1, 3 e 5 sequenciais e síncronos Aumenta a latência Muitas operações de IO para uma única escrita do usuário Observações 780 K transactions 7,388 K I/Os per million transactions (excludes mirroring, standby) Average 7.4 I/Os per transaction PERFORMANCE 30 minute SysBench write-only workload, 100 GB dataset, RDS Single AZ, 30 K PIOPS
- 19. Tráfego de I/O no Aurora (database) AZ 1 AZ 3 Primary Instance Amazon S3 AZ 2 Replica Instance AMAZON AURORA ASYNC 4/6 QUORUM DISTRIBUTED WRITES BINLOG DATA DOUBLE-WRITELOG FRM FILES T Y P E O F WR IT ES 30 minute SysBench writeonly workload, 100GB dataset I/O FLOW Só escreve logs; Outros passos assíncronos Não escreve blocos (checkpoint, cache replacement) Mais 6x log writes, mas 9x menos tráfego de rede Tolerante à latência de redes e storage Observações 27,378 K transactions 35x MORE 950K I/Os per 1M transactions (6x amplification) 7.7x LESS PERFORMANCE Fluxo de redo log records— ordenados por LSN Enviados para storage nodes para operações de escrita
- 20. Rápido, simples, petabyte-scale data warehousing por menos de $1,000/TB/year Amazon Redshift
- 21. N A S D A Q L I S T S3 , 6 0 0 G L O B A L C O M P A N I E S IN MARKET CAP REPRESENTING WORTH $9.6TRILLION DIVERSE INDUSTRIES AND MANY OF THE WORLD’S MOST WELL-KNOWN AND INNOVATIVE BRANDSMORE THAN U.S. 1 TRILLIONNATIONAL VALUE IS TIED TO OUR LIBRARY OF MORE THAN 41,000 GLOBAL INDEXES N A S D A Q T E C H N O L O G Y IS USED TO POWER MORE THAN IN 50 COUNTRIES 100 MARKETPLACES OUR GLOBAL PLATFORM CAN HANDLE MORE THAN 1 MILLION M ES SAG ES / S EC O N D AT SUB-40 MICROSECONDS AV E R A G E S P E E D S 1 C L E A R I N G H O U S E WE OWN AND OPERATE 26 MARKETS 5 CENTRAL SECURITIES DEPOSITORIES INCLUDING A C R O S S A S S E T CL A S SE S & GEOGRAPHIES
- 22. Amazon Redshift Entrega Performance “Redshift é vinte vezes mais rápido que Hive” (5x – 20x redução no tempo das queries) link “Queries que costumavam rodar em horas, retornam em segundos. Nossos analistas estão visivelmente mais produtivos.” (20x – 40x redução em tempo de execução) link …[Redshift] performance deixou todos impressionados (geralmente vemos 50-100x de melhoria comparando com Hive). link “O Time brincou com Redshift hoje concluiu é ****** incrível. Queries complexas sem índices retornando em < 10s.” “Eu falei ridiculamente rápido? Nós vamos usar imediatamente para prover alternativa ao Hadoop para os analistas.” “Nós vimos…queries 2x mais rápidas” Channel Nós regularmente processamos conjuntos de vários bilhões de linhas e fazemos em questões de horas. link
- 23. Amazon Redshift Arquitetura 10 GigE (HPC) Ingestão Backup Restore JDBC/ODBC
- 24. Arquitetura dos nós de computação Leader Node Dense Compute Nodes Large • 2 slices/cores • 15GB RAM • 160GB SSD 8XL • 32 slices/cores • 244 GB RAM • 2.56TB SSD Dense Storage Nodes X-Large • 2 slices/ 4 cores • 31GB RAM • 2TB HDD 8XL • 16 slices/ 36 cores • 244 GB RAM • 16TB HDD
- 25. Use múltiplos arquivos de carga para Maximizar Throughput • Comando COPY • Você precisa de pelo menos a quantidade de arquivos = ”Slices” • Com 16 input files, todas as ”slices” estão trabalhando para maximizar throughput • Tenha 100 MB/s por nó; escalabilidade linear !!! 16 Input Files DW1.8XL Compute Node
- 26. Migração de Bancos de Dados Relacionais para a AWS – Amazon EC2 Amazon EC2 Data Center On premises AWS Internet VPN Amazon EC2 Backup Lógico/Físico Sincronismo ReplicaçãoAmazon S3
- 27. Migração de Bancos de Dados Relacionais para a AWS – Amazon RDS Amazon RDS Customer premises AWS Internet VPN Backup Lógico/Físico Sincronismo ReplicaçãoAmazon RDS Amazon S3 Amazon EC2
- 28. Agenda • Banco de Dados relacionais; • Migração de dados para a AWS; • Bancos de Dados NoSQL; • Outras opções de Armazenamento e Busca;
- 29. Comece a migração em poucos minutos Mantenha a aplicação rodando enquanto migra Replicação entre, para e de Amazon EC2 ou Amazon RDS Movimenta dados para o mesmo motor de DB ou outro AWS Database Migration Service (AWS DMS) Amazon Aurora
- 30. AWS Schema Conversion Tool • Features • Conversão Oracle e Microsoft SQL Server para MySQL, Amazon Aurora, MariaDB, ou PostgreSQL • Ou converter seu schema entre PostgreSQL e qualquer MySQL engine • Relatório de Assessment de Database Migration para escolher o motor de banco de dados e tratar as diferenças • Varredura de código evidenciando os locais aonde serão necessárias edições manuais • Conexão segura com SSL • Código otimizado para Cloud O AWS Schema Conversion Tool ajuda a automatizar a conversão de schema de banco de dados e códigos, para migrações entre motores de bancos de dados ou data warehouses
- 31. Origens e Destinos com AWS DMS Origens: On-premises and Amazon EC2 instance databases: • Oracle Database 10g – 12c • Microsoft SQL Server 2005 – 2014 • MySQL 5.5 – 5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.4 – 9.5 • SAP ASE 15.7+ RDS instance databases: • Oracle Database 11g – 12c • Microsoft SQL Server 2008R2 - 2014. CDC operations are not supported yet. • MySQL versions 5.5 – 5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.4 – 9.5. CDC operations are not supported yet. • Amazon Aurora (MySQL-compatible data source) Destinos: On-premises and EC2 instance databases: • Oracle Database 10g – 12c • Microsoft SQL Server 2005 – 2014 • MySQL 5.5 – 5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.3 – 9.5 • SAP ASE 15.7+ RDS instance databases: • Oracle Database 11g – 12c • Microsoft SQL Server 2008 R2 - 2014 • MySQL 5.5 – 5.7 • MariaDB (MySQL-compatible data source) • PostgreSQL 9.3 – 9.5 • Amazon Aurora (MySQL-compatible data source) Amazon Redshift
- 32. SCT ajuda a converter tabelas, códigos e views Sequences User-defined types Synonyms Packages Stored procedures Functions Triggers Schemas Tables Indexes Views Sort and distribution keys
- 33. Demonstração
- 34. Agenda • Banco de Dados relacionais; • Migração de dados para a AWS; • Bancos de Dados NoSQL; • Outras opções de Armazenamento e Busca;
- 35. Bancos de Dados NoSQL
- 36. Bancos de Dados NoSQL
- 37. Bancos de Dados NoSQL
- 38. Amazon DynamoDB Documento ou Chave-Valor Escala qualquer WorkloadNoSQL 100% Gerenciado Controle de Acesso Programação baseada em evento Rápido e Consistente
- 39. Escalabilidade • Throughput Qualquer tamanho ou throughput • Tamanho Adiciona qualquer número de itens a uma tabela • Escalabilidade Atingida por particionamento
- 40. Throughput • Provisionado na tabela • Write capacity units (WCUs) medidos em 1 KB por second • Read capacity units (RCUs) medidos em 4 KB por second • Consistência eventual é 1/2 da consistência forte • Limites independentes para Read e write WCU RCU
- 43. Agenda • Banco de Dados relacionais; • Migração de dados para a AWS; • Bancos de Dados NoSQL; • Outras opções de Armazenamento e Busca;
- 44. EMR com Amazon S3 é seu novo Data Warehouse Hive, Pig, Cascading Spark Presto HBase Amazon S3
- 45. Amazon Athena Amazon Athena é um serviço de queries interativo que facilita a análise de dados diretamente do Amazon S3, com SQL padrão ANSI
- 46. Athena é Serverless • Sem Infraestrutura, zero administração • Não existem tempo de provisionamento • Upgrades são transparentes
- 47. Motor de busca distribuído Serviço gerenciado usando Elasticsearch e Kibana Totalmente gerenciado - zero administração Totalmente disponível e confiável Totalmente integrado com outros serviços AWS Amazon Elasticsearch Service
- 48. Demo
- 49. Resumo Bancos de Dados - AWS Amazon EC2 Amazon RDS Vários DBs suportados em Linux ou Windows Amazon S3 EMR Amazon Elasticsearch Service
- 50. Obrigado ! Amazon RDS https://aws.amazon.com/pt/rds/ Amazon DMS https://aws.amazon.com/pt/dms/ Amazon Aurora https://aws.amazon.com/pt/rds/aurora/details/ Amazon Redshift https://aws.amazon.com/pt/redshift/ Amazon Athena https://aws.amazon.com/pt/athena/ Amazon EMR https://aws.amazon.com/pt/emr/ Amazon DynamoDB https://aws.amazon.com/pt/dynamodb/ MongoDB na AWS http://docs.aws.amazon.com/quickstart/latest/mongodb/deployment.html Apache Cassandra https://d0.awsstatic.com/whitepapers/Cassandra_on_AWS.pdf Oracle na AWS https://d0.awsstatic.com/whitepapers/best-practices-for-running-oracle-database-on-aws.pdf MS SQL Server na AWS https://aws.amazon.com/windows/products/sql/ Documentação
Notas do Editor
- Picture from: http://www.amazon.com/Black-Aluminum-Control-Amplifier-Wheel/dp/B005HU1ZHA Scan and Query Cumulative size of processed items – ceiling (Ʃ(item sizes)/4KB) Batch GetItem ceiling [(Ʃ(item1 size)/4KB) + …ceiling (Ʃ(itemN size)/4KB)] Consumed throughput is measured per operation Provisioned throughput is divided between all partitions Expand on what are WCU and RCU?
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine types.
- The first thing you want to do is to tightly control access to your data resources. You should start with database-level access restrictions. Create a user that can only read, write, but can’t delete tables. Don’t allow users to Drop Table or Truncate. Next, secure your network. Security Groups will allow you to define which hosts can access your databases. You should define separate Security Groups for your Database hosts and Application hosts. Use the Application Security Group names in the egress/ingress rules for your Database instances instead of CIDRs. The next layer of protection are the VPCs (Virtual Private Cloud). VPCs separate Amazon into logically separate parts. You can further subdivide VPCs into subnets. You should define separate subnets for your databases, application and web servers. The first one will be a private subnet not accessible to the internet. You should deploy your database and application server there. The second one, where you deploy web servers, will be public and accessible to internet hosts. You can use the Network Access Lists (ACLs for short) to define egress/ingress rules for subnets. Lastly, use the Identity and Access Management (IAM for short) to manage permissions for your users.
- After you secured the access to your database, you have additional mechanisms to further securing your databases. You can encrypt data in transit using NNE – native network encryption. You can audit access to your database using the native Orcale audit logs that are accessible to you through the API and the AWS Management Console. And you can also encrypt your data at rest. I want to dive deep into this topic. Volume Encryption: - Works with all editions of Oracle - Use your own AWS KMS keys for more control Use the default key managed by RDS for ease of use Using TDE with Enterprise Edition TDE_HSM and TDE database options TDE_HSM integrates with AWS CloudHSM Share CloudHSM partition between databases and accounts
- Amazon Aurora is a MySQL-compatible, ACID compliant enterprise-grade relational database engine built for the cloud. In many ways, Aurora is a game changer and helps overcome the limitations of traditional relational database engines. Most traditional databases have been built and optimized for on-premises environments, and the fundamentals of their architecture have not changed much in the last few decades. One of the key objectives of Aurora is to overcome the performance, scalability, and availability limitations of traditional databases in a cost-effective manner similar to open-source databases. Durable, fault tolerant, self healing, storage auto-scaling, automatic backups, performance, read replicas, instant crash recovery, survivable buffer cache, security.
- Here we have a single Aurora instance shown with its storage layer. This is called an Aurora cluster. The cluster consists of one or more Aurora instances and the underlying shared storage layer which spans 3 azs. Here we are showing Aurora with one primary instance only, but you can have multiple replicas – up to 15 – in these 3 Azs.. Let’s take a look at the storage layer. Aurora maintains 6 copies of your data across three Availability Zones – by default – no matter if you have chosen to have additional Aurora instances – the storage is on a virtual disk that spans 3 availability zones. Because Aurora maintains multiple copies of your data in three Availability Zones, the chance of losing data as a result of a disk failure is minimized. Aurora automatically detects failures in the disk volumes that make up the cluster volume. When a segment of a disk volume fails, Aurora immediately repairs the segment. When Aurora repairs the disk segment, it uses the data in the other volumes that make up the cluster volume to make sure that the data in the repaired segment is current. As a result, Aurora avoids data loss and reduces the need to perform a point-in-time restore to recover from a disk failure. And by the way, at the storage layer, you are only billed for one of those copies, not six.( You are charged based on the storage your database consumes at the database layer, not the storage consumed in this virtualized storage layer.)
- For comparison, this is what your cluster would look like with additional Aurora replicas provisioned, here one in each AZ
- Lets compare and contrast : So here is io traffic in RDS MySQL with a standby instance. You issue a write, it goes to EBS, EBS issues to a mirror asynchronously in parallel. Once both writes have happened we synch are going to go to another AZ to do another write with DRBD. Then you go to the standby, it has to issue a write, and then it proprages and comes back. The interesting thing to point out here is steps 1,3 and 5 are sequential and synchronous. So if there is any network “weather” going on it is oing to amplify both the latency and the jitter – minimally ou are going to be looking at 3x and possibly much more than that. The other thing you see is there are a lot of diferent types of write operations: log, binlob, data, double write to avoid the torn write, and the frm files for metadata And the thing that is particularly difficult int his is the data blocs are a good deal larger than the log, about 40x, and you are writing them twice so that is 80x. So that is a big deal. But like any solid db mysql is going to be doing a lot of batching under the covers to try to mitigate that overhead. So what we did here is we ran a 30 minute write only SysBench workload, with 100GB data set, SingleAZ, 30K PIOPS –and we ran 780 k transactions. We saw over 7.4 million and that’s pretty oggd –that is the work the primary instance is seeing. that is steps 1. Steps 2,3,4,5 are not reflected. You want to focus on the network throughtput on that single instance. Ets compare it to aurora. Aurora – 27,378,519 transactions, 35x more. 158,323 operations – 46x fewer. Latency aggregates Jitter amplified Write both log and data. Data IO flow Observations Multiple sequential steps Latency and jitter amplify Double-writes because overwriting page Performance
- So let’s compare it to Aurora. So in Aurora what we are doing is we are only shipping log records. So if we see a bunch of log records come to us, we boxcar them together SysBench writeonly workload 100GB SingleAZ 30K PIOPS – 30 minutes 779,664 txns. 7,387,798 operations Aurora – 27,378,519 transactions, 35x more. 158,323 operations – 46x fewer. 949938
- If there is one thing you take away from today’s presentation, I want it to be this. Redshift is a “Fast, Simple, Scalable and Inexpensive Data Warehouse solution” and it is successfully deployed for a variety of use cases.
- Common themes that customers focus on when they describe their experience with Redshift are its high performance, low price, and easy of use. Here are some quotes form customers about performance.
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine types.
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine types.
- Picture from: http://www.amazon.com/Black-Aluminum-Control-Amplifier-Wheel/dp/B005HU1ZHA Scan and Query Cumulative size of processed items – ceiling (Ʃ(item sizes)/4KB) Batch GetItem ceiling [(Ʃ(item1 size)/4KB) + …ceiling (Ʃ(itemN size)/4KB)] Consumed throughput is measured per operation Provisioned throughput is divided between all partitions Expand on what are WCU and RCU?
- Mention why no older engines as sources +Sybase v15+
- Picture from: http://www.amazon.com/Black-Aluminum-Control-Amplifier-Wheel/dp/B005HU1ZHA Scan and Query Cumulative size of processed items – ceiling (Ʃ(item sizes)/4KB) Batch GetItem ceiling [(Ʃ(item1 size)/4KB) + …ceiling (Ʃ(itemN size)/4KB)] Consumed throughput is measured per operation Provisioned throughput is divided between all partitions Expand on what are WCU and RCU?
- Make more splashy, add Icons from detail page
- There is a limit in place to avoid run-away apps. But you can request a limit increase. http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html For every distinct hash key value, the total sizes of all table and index items cannot exceed 10 GB This request would have caused the ReadCapacityUnits limit to be exceeded for the account in us-west-2. Current ReadCapacityUnits reserved by the account: 237. Limit: 2000. Requested: 5000. Refer to the Amazon DynamoDB Developer Guide for current limits and how to request higher limits. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: LimitExceededException; Request ID: 0U3G4FLA19BQMELKS4G7RSTH83VV4KQNSO5AEMVJF66Q9ASUAAJG)
- Picture from: http://www.amazon.com/Black-Aluminum-Control-Amplifier-Wheel/dp/B005HU1ZHA Scan and Query Cumulative size of processed items – ceiling (Ʃ(item sizes)/4KB) Batch GetItem ceiling [(Ʃ(item1 size)/4KB) + …ceiling (Ʃ(itemN size)/4KB)] Consumed throughput is measured per operation Provisioned throughput is divided between all partitions Expand on what are WCU and RCU?
- Picture from: http://www.amazon.com/Black-Aluminum-Control-Amplifier-Wheel/dp/B005HU1ZHA Scan and Query Cumulative size of processed items – ceiling (Ʃ(item sizes)/4KB) Batch GetItem ceiling [(Ʃ(item1 size)/4KB) + …ceiling (Ʃ(itemN size)/4KB)] Consumed throughput is measured per operation Provisioned throughput is divided between all partitions Expand on what are WCU and RCU?
- Soring data in S3 allows users to spin up as many clusters as needed to test out new ideas and to terminate clusters when they are no longer in use. This can help speed up innovation and lower the cost of experimentation. And you can optimize each cluster for a particular application and use case. With EMR you can run custom MapReduce code or use a variety of powerful applications and frameworks such as Hive, Pig, Hbase, Accumulo, Impala, Cascading, ElasticSearch, Kibana, Spark, and Presto. You can use a variety of different programming languages and we provide code samples and tutorials that help get you up and running quickly.
- You simply put your Data in S3 and submit SQL against it
- Amazon Elasticsearch Service makes it easy to deploy, operate, and scale Elasticsearch for log analytics, application monitoring, interactive search, and more. It is a fully managed service that delivers the easy-to-use API’s and real-time capabilities of Elasticsearch along with the availability, scalability, and security required by real-world applications. It offers built-in integrations with other AWS services including Amazon Kinesis, AWS Lambda, and Amazon CloudWatch, and 3rd party tools like Logstash and Kibana so that you can go from raw data to actionable insights quickly.
- So it was with all these factors in mind that we developed the database migration service. We designed it to be simple - you can get started in less than ten minutes. We designed it to enable near-zero-downtime migration. And we designed it to be a kind of replication Swiss army knife. To replicate data between on-premises systems, RDS, EC2, and across database engine types.