Data Management in a Microservices World
•
2 gostaram•1,711 visualizações
This talk was first given at Code Mash 2019. It covers some of the aspects of how data management changes in the microservices world but also how data management has changed over time and where we have reinvented the wheel with failed attempts to change how we store and organize data. Parts of this presentation is about to remind us that certain ways of doing things we had before and we are about to repeat the same mistakes. Parts of this presentation is about common sense applied when dealing with data. The data tier cannot just be ignored or neglected when building new systems, data is the only thing that will still be there long when the applications or system on top has evolved. It's therefore crucial to make smart choices, based on actual requirements and business asks how to store and manage our data, rather than basing the decision on hyped technologies ranking high on Hackernews the last week. Parts of the presentation is about capabilities that we already have since a long time in databases and that you might not know about. These are pointed out as potentially complimentary to your application architectures. It is again up to you to apply _your_ business requirements and common sense whether these capabilities solve your problem or not. Note that this presentation is intended as visual aid for a talk. Parts of this presentation may be meaningless or wrongly intended without having the context that was given by me.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Data Management in a Microservices World
Semelhante a Data Management in a Microservices World (20)
Último
Último (20)
Data Management in a Microservices World
- 3. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | 3 Gerald Venzl • Oracle Database Product Manager • Previous: – Java Developer – Performance Engineer – Enterprise Architect • Twitter: @GeraldVenzl • Blog: https://geraldonit.com
- 21. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Different applications/ µServices have different needs User Sessions Financial Transactions Shopping Cart Recommend. Engine Product Catalog Reporting Analytics Activity Logs Processing Heavy Writes √ Heavy Reads √ √ √ √ Fast Read/Write √ Data Consistency √ Data Durability √ Analytic √ √ Graph √ Spatial Geo Distribution √ √ √ Data Relational √ √ √ Key/Value √ √ √ Document/JSON √ √ √ Graph √ √
- 30. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Multi-model prevails over time C o m p l e x it y 1970s Relational SQL 1980s Object DBs Multi-Model SQL XML XPath Multi-Model SQL JSON Multi-Model SQL 1990s 2000s 2010s 20?? Cobol ISAM Multi-Model SQL Next “big” thing
- 31. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Multi-model capabilities • The relational model is a logical model • Databases store data not in rows and columns, but in blocks – Blocks are aligned to block storage, i.e. disks and the way they store data – Data is logically represented as rows and columns • Multi-model databases contain logical models to other data formats
- 32. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Polyglot Persistence Market Trends • Single-model architectures are most pervasive for ‘edge’ applications – New business & workload requirements • Business applications naturally converge to multi-model architectures – Today’s ‘edge’ applications are tomorrow’s mainstream business applications – Efficiencies of multi-model architecture override advantages of special-purpose systems over time • There will always be single-model polyglot architectures – Because there are always new ‘edge’ applications
- 34. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Considerations for Polyglot Persistence Multi-model Polyglot: • Benefits of consolidation and standardization • Standardized administration • Consistent data security policies • Simple integration across multiple data formats • Transactions and data consistency Single-model Polyglot: • Benefits of specialization • Specialized APIs • Specialized data formats • Specialized access methods and indexes
- 48. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | The data reasoning problem • Much harder to understand the structure from other domains – Now you have to know how other applications structure their data • Schema-on-read brings an inherent data cleansing problem – As no constraints are enforced at write no guarantees can be assumed • The more data you have, the more complex it gets – Need to hire Data Scientists where SQL used to be sufficient
- 53. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Query (SELECT AS OF) “Retrieve risk calculations as of 90 days ago” SELECT * FROM risk_calculations AS OF (SYSDATE – 90);
- 54. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Query (SELECT AS OF) “Retrieve risk calculations as of 90 days ago” SELECT * FROM risk_calculations AS OF (SYSDATE – 90);
- 55. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Version Query (SELECT VERSIONS BETWEEN start AND end) “Retrieve all risk calculations changes from two weeks ago” SELECT * FROM risk_calculations VERSIONS BETWEEN SYSDATE-14 AND SYSDATE-7; Flashback Query (SELECT AS OF) “Retrieve risk calculations as of 90 days ago” SELECT * FROM risk_calculations AS OF (SYSDATE – 90);
- 56. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Version Query (SELECT VERSIONS BETWEEN start AND end) “Retrieve all risk calculations changes from two weeks ago” SELECT * FROM risk_calculations VERSIONS BETWEEN SYSDATE-14 AND SYSDATE-7; Flashback Query (SELECT AS OF) “Retrieve risk calculations as of 90 days ago” SELECT * FROM risk_calculations AS OF (SYSDATE – 90);
- 57. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Version Query “Pull entire change history for factor = ’Major’ ” SELECT * FROM risk_calculations VERSIONS BETWEEN SYSDATE AND MAXVALUE WHERE factor = 'Major’; UNDO
- 58. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history Flashback Version Query “Pull entire change history for factor = ’Major’ ” SELECT * FROM risk_calculations VERSIONS BETWEEN SYSDATE AND MAXVALUE WHERE factor = 'Major’; UNDO
- 59. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Change tracking for all DMLs Analysis over entire history of changes via SQL Risk calculations Risk calculation history • Fully secure – Encryption at transport and rest – Fine grain access control – Fully automatic, never loosing a single change.
- 69. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Recommend engine Cluster EMEA Financial data Recommend engine • PDB Hot Clone – Faster container instantiation • PDB Relocate – Relocate with no downtime Cluster US Product catalog
- 70. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Recommend engine Cluster EMEA Financial data Recommend engine • PDB Hot Clone – Faster container instantiation • PDB Relocate – Relocate with no downtime Cluster US Product catalog Product catalog
- 71. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Recommend engine Cluster EMEA Financial data Recommend engine • PDB Hot Clone – Faster container instantiation • PDB Relocate – Relocate with no downtime Cluster US Product catalog Product catalog Product catalog
- 72. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Recommend engine Cluster EMEA Financial data Recommend engine • PDB Hot Clone – Faster container instantiation • PDB Relocate – Relocate with no downtime Cluster US Product catalog Product catalog Product catalog
- 73. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Financial data Product catalog Cluster B Product catalog Recommend engine • PDB Hot Clone – Faster container instantiation • PDB Relocate – Relocate with no downtime Cluster A Financial data
- 74. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Cluster B Product catalog Recommend engine Financial data • PDB Hot Clone – Faster container instantiation • PDB Relocate – Relocate with no downtime Cluster A Financial data
- 75. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Containerized databases Deployment transparency Product catalog Cluster B Product catalog Recommend engine Financial data • PDB Hot Clone – Faster container instantiation • PDB Relocate – Relocate with no downtime Cluster A Financial data
- 78. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Oracle Database Sharding Oracle Database for web-scale applications One giant database partitioned into many small databases (shards) Customers Americas Customers Customers Europe Customers Asia • RAC and Data Guard meet needs of over 99% of applications while preserving application transparency • Some Global-Scale OLTP applications prefer to shard massive databases into a farm of smaller databases – Avoid scalability or availability edge cases of a single large database – Willing to customize data model and applications to enable transactions to be automatically routed to the right shard • Native SQL for shardingtables across up to 1000 Shards - Routing of SQL based on shard key, and cross shard queries - Online addition and reorganization of shards - Linear scalability of data, workload, users with isolation
- 79. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Oracle Sharding Automated Distribution Enhanced SQL syntax for Sharding … CREATE SHARDED TABLE Customers ( CustId VARCHAR2(60) NOT NULL, FirstName VARCHAR2(60), LastName VARCHAR2(60), … PRIMARY KEY(CustId), ) PARTITION BY CONSISTENT HASH (CustId) … • SQL syntax for creating sharded tables • Not proprietary APIs as with NoSQL • Creation of a sharded table automatically partitions data across shards • Transparent resharding as data grows • Choice of sharding methods: • System managed - consistent hash • User defined - range, list • Composite - range-hash, list-hash • Common reference data (e.g. Price List) is automatically duplicated on all shards • Supports shard placement in specific geographies to satisfy government data privacy … …
- 91. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Communication Financial data API Shopping cart API JSON Async ? • Was the payment processed or not? • Was the data persisted or not? • If not, how do we replay the payment? • If yes, how do we guarantee that we don’t bill the customer twice? • … Reliable communication without tight coupling
- 93. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Communication Financial data API Shopping cart API JSON Sync • Possible solution, but… • Services will be tightly coupled • Outage of one system will impact the other system • Prevents service independent scaling Reliable communication without tight coupling
- 99. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. | Financial data API Oracle Advanced Queueing Shopping cart API JSON Async AQ Reliable communication without tight coupling • Allows reliable communication without coupling • Further increases scalability – REST payloads are never lost and put into one or multiple queues to work off from • Provides transparency – Works for any data – Works for PDBs and non-CDBs AQ