Presented at JAX London 2013
The need to operate terabyte-size databases becomes very common these days. Unless you have implemented architectures that use NoSQL databases and frameworks that support data-intensive distributed applications, then many technology options available are probably a slight enigma. This session focuses on real-world successful attempts to benchmark four of the most popular NoSQL databases side by side. The base tool selected for the purpose of this research is Yahoo Cloud Serving Benchmark and benchmarking is performed on Amazon Elastic Compute Cloud instances.
Often referred to as NoSQL, non-relational databases feature elasticity and scalability. In addition, they can store big data and work with cloud computing systems. All of these factors make them extremely popular.
Why did NoSQL data stores appear? Mostly because relational databases (RDBMS) have a number of disadvantages, if you have to work with large datasets.For example, RDBMS are hard to scale and their architecture is designed to work on a single machine. - Scaling write operations is either hard, expensive, or impossible.- Vertical scaling (or upgrading equipment) is either limited or very expensive. Unfortunately, this is often the only possible way you can scale.- Horizontal scaling (or adding new nodes to the cluster) is either unavailable or you can only implement it partially. There are some solutions from Oracle and Microsoft that make it possible to have computing instances on several servers. Still, the database itself remains in shared storage.In addition to poor scalability, RDBMS have strict data models. The schema is created together with the database and you will need a lot of time and effort to change this structure. In most cases it is an extremely complex task. Apart from that, RDBMS have difficulties with semistructured data.
NoSQL solutions address many of these problems.POINT 1: In 2013, the number of NoSQL products reached 150+ and the figure is still growing. That variety makes it difficult to select the best tool for a particular case.POINT 2: They come in many types--key-value, columnar, document-oriented, and graph.POINT 3: There is one thing in common for all NoSQL databases. They don't use the relational data model. This means they do not use the SQL query language.POINT 4: NoSQL data management systems are inherently schema-free (with no obsessive complexity and a flexible data model) and eventually consistent (complying with BASE rather than ACID)POINT 5: They provide APIs to perform various operations. Some of NoSQL data stores support query language operations, for example, Cassandra and Hbase. However, there is no standard. This is another difference between NoSQL databases and traditional RDBMS.POINT 6: RDBMS usually have strong data consistency. In contrast to that, NoSQL data stores operate with eventual consistency. When you add data to the system, it becomes consistent after some time.POINT 7: NoSQL architectures are designed to run in cluster that consist of several nodes. This makes it possible to scale them horizontally by increasing the number of nodes.In addition, NoSQL data stores serve huge amounts of data and provide high throughput.
POINT 1: NoSQL databases differ from RDBMS in their data models. These systems can be divided into 4 groups:A. Key Value StoresKey value stores are similar to maps or dictionaries where data is addressed by a unique key.B. Document StoresDocument Stores encapsulate key value pairs in JSON or JSON like documents. Within documents, keys have to be unique. In contrast to key-value stores, values are not opaque to the system and can be queried as well.C. Column Family StoresColumn Family Stores are also known as column oriented stores, extensible record stores and wide columnar stores.D. Graph databasesKey-value stores, document stores, and column family stores have a common feature. They do store denormalized data in order to gain advantages in distribution.In contrast to relational databases and the already introduced key oriented NoSQL databases, graph databases are specialized on efficient management of heavily linked data.POINT 2: All NoSQL data stores have an API to work with data. Some DBs use certain SQL operations. Others support MapReduce aggregation.POINT 3: Multiversion concurrency control (MVCC) relaxes strict consistency in favor of performance. In order to support transactions without reserving multiple datasets for exclusive access, optimistic locking is provided by many stores. Before changed data is committed, each transaction checks, whether another transactions made any conflicting modifications to the same datasets.POINT 4: NoSQL databases differ in the way they distribute data on multiple machines. Since data models of key-value stores, document stores and column family stores are key oriented, the two common partition strategies are based on keys, too.The first strategy distributes datasets by the range of their keys. A routing server splits the whole keyset into blocks and allocates these blocks to different nodes. Afterwards, one node is responsible for storage and request handling of his specific key ranges. In order to find a certain key, clients have to contact the routing server for getting the partition table.Higher availability and much simpler cluster architecture can be achieved with the second distributionstrategy called consistent hashing. In contrast to range based partitioning, keys are distributed by using hash functions. Since every server is responsible for a certain hash region, addresses of certain keys within the cluster can be calculated very fast.In addition to better read performance through load balancing, replication also brings better availability and durability, because failing nodes can be replaced by other servers. If all replicas of a master server were updated synchronously, the system would not be available until all slaves had committed a write operation. Ifmessages got lost due to network problems, the system would not be available for a longer period of time. This solution is not suitable for platforms that rely on high availability, because even a few milliseconds of latency can have a big influence on user behavior.POINT 5: (PERFORMANCE: TYPICAL WORKLOADS)Obviously, performance is a very important factor. Performance of data storage solutions can be evaluated using typical scenarios. These scenarios simulate the most common operations performed by applications that use the data store, also known as typical workloads. The tests that we performed to compare performance of several NoSQL data stores also used typical workloads.
Database vendors usually measure productivity of their products with custom hardware and software settings designed to demonstrate the advantages of their solutions. In our tests we tried to see how NoSQL data stores perform under the same conditions.POINT 1: For benchmarking, we used the Yahoo Cloud Serving Benchmark (YCSB)The kernel of YCSB has a a framework with a workload generator that creates test workload and a set of workload scenarios.POINT 2: Developers need to describe the scenario of the workload by operation type: what operations are performed on what types of records. POINT 3: Supported operations include: insert, update (change one of the fields), read (one random field or all the field of one record), and scan (read the records in the order of the key starting from a randomly selected record).We can define the workload by the data that will be loaded into the database during the loading phase and the operations that will be executed against the data set during the transaction phase.
Each workload was targeted at a table of 100,000,000 records; each record was 1,000 bytes in size and contained 10 fields. A primary key identified each record, which was a string, such as “user234123”. Each field was named field0, field1, and so on. The values in each field were random strings of ASCII characters, 100 bytes each. Database performance was defined by the speed at which a database computed basic operations. A basic operation is an action performed by the workload executor, which drives multiple client threads. Each thread executes a sequential series of operations by making calls to the database interface layer both to load the database (the load phase) and to execute the workload (the transaction phase). The threads throttle the rate at which they generate requests, so that we may directly control the offered load against the database. In addition, the threads measure the latency and achieved throughput of their operations and report these measurements to the statistics module.-- The tests:We defined the following values for the workload executor:the number of threadsthe types of operations in the workload and the desired number of operations per second (target throughput)Then we measured the time it took to perform these transactions (latency).
This is a component diagram of the YCSB framework. It consists of several modules.Workload executor applies the workload to the data store. For each session, when the client accesses the DB, a client thread is initiated. Each thread performs a set of operations from the workload. The results in the form of statistics are then sent to the statistics module, which prints the output of the test to console where benchmark is started. These tests are consequently repeated for all the selected solutions.The YCSB framework has connectors for a wide range of DBs. For each database tested with YCSB, a developer needs to determine the type of database, target throughput, the number of concurrent threads on the client side, and how many operations we want to perform. This is necessary to create and start a test.
Now let's take a look at the NoSQL data stores that we tested:Cassandra 2.0: This is a column-value data store. We ran it on a virtual machines with Java 1.7.40 installed. The transactions were performed with the non-default configuration. In particular, we used a random partitioner to section data by nodes. The amount of data cash for the keys was 1 GB. The size of row cash was 6 GB. The size of JVM heap was 6 GB. Data was not replicated (there were no copies). This approach was intentional, we wanted to test performance, not failure tolerance of the cluster. MongoDB: This is a document-oriented DB. Here, we didn't do much additional configuration or tuning. As I have mentioned before, for Mongo, we added two VMs that served as routers because according to official documentation, the mongo router process should run on a separate machine. However, if you need to simplify the model, mongo router may run on the same machine, where the YCSB client is. In one of our earlier tests, we discovered that it uses a lot of CPU power. This is why it should be placed on a separate machine. Data sharding for MongoDB was based on document key.
We used the following workloads: Workload A: Update-heavily mode. Workload A is an update-heavily scenario that simulates how a database works, when recording typical actions of an e-commerce solution user.Settings for the workload: Read/update ratio: 50/50Zipfian request distributionWorkload BWorkload B is a read-mostly workload that has a 95/5 (ninety five to five percent) read/update ratio. It recaps content tagging, when adding a tag is an update, but most operations include reading tags.Workload CWorkload C is a read-only workload that simulates a data caching layer, for example a user profile cache.Workload D Workload D has 95/5 read/insert ratio. The workload simulates access to the latest data, such as user status updates or working with inbox messages first.Workload EWorkload E is a scan-short-ranges workload with a scan/insert percentile proportion of 95/5. It corresponds to threaded conversations that are clustered by a thread ID. Each scan is performed for the posts of a given thread.Workload F Workload F has read-modify-write/read ops in a proportion of 50/50. It simulates access to user database, where user records are read and modified by the user. User activity is also recorded to this database.Workload G Workload G has a 10/90 read/insert ratio. It simulates data migration process or highly intensive data creation.
For this benchmark, we generated a number of workloads based on a target throughputs and then measured the number of operations per second or the actual throughput of the each database. We also measured their latencies, or how long it took to perform each operation.During the first stage of the test, we uploaded 100 mln records of 1 kb each to each data store. YCSB measured average throughput in operations per second and latency in milliseconds.Hbase demonstrated the lowest throughput, probably because we turned on the auto-flash mode. This mode ensures that each operation that creates a record will be sent from the client to the server and then persisted to the database. Hbase also supports an alternative mode that generates additional cash on the client side. When the client is out of storage it sends this cash to the server. In this alternative mode, Hbase saves data to disk in batches.As we expected, Couchbase and Cassandra demonstrated excellent results. Cassandra simultaneously updates data in memory and writes it to the transaction journal on disk. Couchbase writes data to memory and then asynchronously persists it to disc. The result of the transaction returns after everything has been saved to memory. In this particular test, all data was loaded in a single iteration. But insert, update, and read operations that I will describe later were performed in several iterations.
The last workload, G, mostly consists of insert operations. It simulates the process of data migration or inserting a lot of data. The results are similar to what we saw on the previous graph. Hbase, Cassandra, and Couchbase demonstrate low latencies and high throughput. MongoDB’s performance starts to decline at about 4000 ops per second with an average latency of up to 5 times greater than in other databases.
This is the last diagram that shows the results of read operations, which make up 10% of workload G. Here we can see that latencies vary for different solutions. This might be because the data is in network storage on the cloud. Couchbase and Cassandra show a maximum throughput of up to 6000 operations per second.
1. What you choose depends on your needs. Before making the decision, answer the following questions: determine what your data sets and your data model will be like; the type data model will depend on the data sets and typical operations that your app will performdetermine your requirements to transaction support; decide whether you need transactionschoose whether you need replication; decide on your requirements to data consistencydetermine your performance requirements (how fast your DB should be)If your project is based on an existing solution, you should see if it is possible to migrate your data2. Then, taking into account all these factors, evaluate different solutions and test their performance (that’s what this presentation is all about). It is very useful to build a prototype and perform proof of concept. Then based on this prototype, you can select the best solution for your system. 3. Prototyping makes it possible to see how the solution will work in a real-life project. If it doesn’t work well enough, you need to review the architecture, the components, and build a new prototype. 4. There are no perfect solutions and there are no bad NoSQL or RDBMS data stores. The database and its implementation depends on a particular use case. The tests we have performed show that in different scenarios, different solutions have very different results. Your final choice might be a compromise. The main determinant will be what you want to achieve and what properties you need most. The solution may use several different solutions, including relational and NoSQL databases.
