NoSQLMark is a research project and benchmarking framework developed at the University of Hamburg. Based on YCSB as the most popular and widely accepted benchmarking framework in the field of NoSQL OLTP benchmarking, NoSQLMark takes its best features (e.g. support for many different databases or a generic workload generator) and additionally provides:
NoSQLMark is built on and and thus features scalable workloads compatible with YCSB. In addition, nodes can easily join the benchmarking cluster and the workload can define how many nodes should be used to set the database under load. You no longer have to start different YCSB nodes manually nor do you have to aggregate the different measurement results by your own.
is our homegrown single-node inconsistent key-value store designed to validate different measurement methods. Originally developed to validate staleness measurement approaches (cf. Who Watches the Watchmen?), it can also simulate various aspects of system behavior and corresponding anomalies to validate NoSQLMarks new distinctive measurement methods.
NoSQLMark avoids the Coordinated Omission Problem by measuring asynchronously by default. The Coordinated Omission Problem became very popular due to a talk from Gil Tene, CTO and co-founder at Azul Systems, who also coined the name. In short, it describes a situation where benchmarking tools drop potential samples in the result set. In fact, most benchmarking tools are broken in this sense. This also applies to YCSB, but since 2015 some efforts have been made to correct the poblem and additionally report corrected latencies alongside the actual measurements. In an upcoming publication, we will show that this correction can produce worse results than the uncorrected measurements.
There are several academic papers about staleness-based consistency measurement methods. However, these rely on either system clocks whose clock drift distort the measured values in the distributed context, or the measurement method itself is faulty, as in the case of YCSB++ (see our paper Who Watches the Watchmen?). Akka's actor model enables us to implement our approach as already proposed in our Survey on NoSQL OLTP Benchmarking to provide clearly defined lower and upper bounds for the reported staleness values in a given database setting.
work in progress
work in progress
Download the latest release: ...
cd nosqlmark-1.0.1 bin/backbench [2015-04-22 15:01:37,495] INFO ...To start NoSQLMark in a cluster environment, you need a change the configuration file config/nosqklmark.conf.