Planet Cassandra

All your NoSQL Apache Cassandra resources in one place.

Cassandra as a time series database

When I think of Time series databases, my first thoughts go to my own experiences using a relational database to store time series data. The mapping tends to be fairly straight forward. Each row stores data for a single event for a specific entity, whether it be a sensor, server metric, financial transaction etc. There are many, many kinds of time based data. But, it often turns out that really small rows in a relational database aren’t all that efficient and the kinds of aggregate operations you may want to perform are just not available in your relational database of choice. Additionally, traditional B-Tree architectures are not necessarily well suited for writing time series data. B-Trees are great for reading and writing random or semi-random data. They are optimized for fast random reads with writes, updates and deletes being expensive. When writing time series data often writes are much more frequent than reads. There are no updates, the data is written once, and deletes are often over large ranges of data as it “Expires”.

So the search is on for an alternative. We go to Wikipedia here: For a topic of major interest to many this is one of the poorest Wikipedia articles I have read in quite a while. Hmmm… Maybe I should finally sign up as an editor and fix it. Well that’s for later. The article leaves out a few time series databases I know of.

The Wikipedia article lists seven databases or variants of databases designed or tuned for time series data. They are: Graphite, InfluxDB, Informix TimeSeries, RRDtool, Riak-TS, ExtremeDB and QuasarDB. In addition, I know of a few more, there is OpenTSDB, CeresDB, and Prometheus. OpenTSDB doesn’t even have a page in Wikipedia but it’s a thriving Apache project. Hmmm… I should start contributing to Wikipedia. In addition, Splunk and Logstash are, in effect, time series databases specifically designed around storing and querying log data which is a special form of sparse time series data. That is the last I am going to discuss them in this blog entry.

Graphite/Whisper, Ceres, RRDtool and Prometheus are very specialized. Designed to work with, or part of a monitoring and metrics system designed to collect server data produced on a regular basis for display on a Graphical user interface. The others are more general purpose designed to support many kinds of time series data.

So why, with all those special built or at least specially modified databases dedicated to time series data did I title this blog “Time series databases and Cassandra”? It turns out that even with all those existing Time series databases specially designed to support time series data, a whole lot of people are using Cassandra to store their time series data instead.

Which brings me to the obvious next question, why?…..

Right off the top Cassandra does not use B-Trees to store data. Instead it uses Log Structured Merge Trees (LSM-Trees) to store its data. This data structure is very good for high write volumes, turning updates and deletes into new writes. Reads are often an order of magnitude slower than writes which is exactly the opposite of B-Trees.  The underlying storage mechanism would appear to be compatible with Time Series Data.

Beyond that the reasons are not so simple. There are a number of reasons, including purpose, cost, scalability, availability, ease of use and my favorite, my competitor or buddy or just the guy down the street is doing it.

So, let’s take a look at each of those reasons:



Whisper (Graphite), RRDtool, Ceres and Prometheus are special purpose databases designed  for, or part of existing monitoring and metrics systems and are not general purpose. If you are building a monitoring and metrics system to keep track of the performance of your server farm they are going to be obvious candidates.  If you are tracking freeway sensor or fitness tracker data probably not so much.


Some of the databases in the list are proprietary licensed products. Nothing wrong with that, but it does mean a potentially significant upfront cost and to be honest in the modern open source popular world this becomes a major negative to many organizations. Informix, InfluxDB enterprise, and DataStax Enterprise Cassandra (DSE) loose on this score.


If you are tracking the processor, memory and disk utilization of a couple dozen servers you don’t need much disk space to store many years’ worth of your time series data and you don’t much care about scalability. In that case Graphite and the community version of InfluxDB will probably work just fine. But…., if you want to track server metrics from one hundred thousand virtual servers with one-minute granularity you probably need a really scalable platform. Then you are going to need InfluxDB enterprise, Riak TS OpenTSDB or can I say it now? Cassandra.


If you are monitoring a large number of somethings you probably really want to monitor them all the time and outages of your metrics database is going to be a BAD thing. So you want a highly available database. In the highly available world the best highly available systems are those which are designed from the ground up to be highly available where you can write or read to/from any node in the system and get meaningful results. Master Slave architectures no matter how good are just not as effective in handling a continuously incoming stream of data as multi master databases are. With that in mind you are going to be limited to Riak TS, OpenTSDB (sort of) and Cassandra. All three are based on some similar concepts involving a cluster of distributed nodes. although the implementations vary quite a bit.

Ease of Use

Graphite and open source InfluxDB are incredibly easy to set up and use although they have their own API languages for accessing the data that you have to learn they provide powerful aggregates very useful for Time series data. The Influx query language is SQL like. Graphite doesn’t really support a query language outside of the graphical UI beyond a basic csv export function. Cassandra and Riak TS are fairly easy to set up (more difficult primarily in that they are distributed in nature) and they offer an SQL like language to access the data, although they do not have some of the very nice aggregates available to Graphite and InfluxDB. They are also a bit more general purpose and can be used for other things. OpenTSDB is based on Hadoop and Hbase requiring that you set up a Hadoop cluster then install Hbase and zookeeper. Finally, you add the OpenTSDB Daemons. This is a lot of work to get your time series database up and running. OpenTSDB also uses its own proprietary Query language which although very powerful for Time Series data is not anything like SQL.

Everyone is using it

Now we get to the more subjective warm and fuzzy (pun intended) logic for making a decision which database to use. If you are a big data user with Hadoop, Zookeeper and Hbase already running OpenTSDB is the obvious choice. If you are a DevOps person with experience building monitoring systems, you are probably going to lean in the direction of Graphite, InfluxDB or Prometheus. At least until you run into scaling issues. If you know Erlang and Riak you might try Riak TS (there are not so very many of us in that group and if you haven’t heard the company that developed Riak went Bankrupt last year). If you don’t have DevOps experience or the scale of your project eliminates InfluxDB or Graphite and you don’t have a Hadoop cluster ready to go for you a quick search of the internet or chatting with your buddy over at xyz organization, you are going to hear about the wonders of Cassandra for storing Time Series data.



Cassandra is not a purpose built for time series data. In fact, its lack of aggregates can make the choice an odd one at times but it accepts rapid writes (in fact writes are usually an order of magnitude faster than reads), scales to really, large clusters with lots of disk space and its multi master write and read anywhere even in geographically separate locations with lots of other people doing it makes a lot of sense.


The Mandatory Chart



Find out about Pythian’s services for Cassandra.

Running sysbench based benchmarks against Cassandra

I was recently discussing benchmarking options for Cassandra with some colleagues and given my background with MySQL, sysbench was the first tool I thought of.

Sysbench is a high performance and flexible benchmark tool that can be used to run both Database and Operating System based experiments. In case you’re not familiar with it, there’s an excellent introduction to it by my colleague Martin Arrieta here.

One interesting thing about this tool is that while it includes out of the box support for MySQL and PostgreSQL, it uses Lua for scripting so other databases can be supported provided there’s a Lua driver for it.

Naturally, that means my quest for a way to run sysbench against Cassandra started with the search for a Cassandra Lua drive, which led me to

The next task was looking for a simple way to deploy benchmark clients without the need to install dependencies on the host OS. These days my answer to that involves Docker and what I found for this was a handy image from Severalnines:

This image was a good starting point but did not fully support my use case as I need to install custom Lua modules on the container, which requires installing some additional packages on it.

Given my full stack for this is Open Source I went ahead and modified the Dockerfile for this image to add what I needed.
I felt this was a change that could benefit others too as I’m probably not the only one using sysbench to run experiments against databases that don’t have a driver bundled with it, so I submitted the following PR which has been merged already:

Putting it all together, I can now launch sysbench against a Cassandra cluster to test the performance of different schemas and workloads.

To give a simple example let’s consider the following Lua script:

#!/usr/bin/env sysbench

function event()
   local cassandra = require "cassandra"

   local peer= assert( {
    host = "",
    port = 9042,
    keyspace = "test"


  assert(peer:execute("select * from user"))


A couple of comments about the script:

  • is the IP address of the Cassandra node I want to connect to. In my case, this is another container, but be sure to change this as needed if you want to reproduce this test (or refer to this gist to see how I ran mine).
  • For the script to work, the Cluster must have a ‘test’ keyspace with a ‘user’ table on it (as you can see from the query, the table structure does not matter here).

We can use sysbench to execute it via docker like so:

telecaster:sysbench-docker fipar$ docker run -v ~/src/:/src/ --name=sb -it severalnines/sysbench bash -c 'luarocks install lua-cassandra --local; luarocks install luasocket --local; /src/tmp/benchmark1.lua run'
Warning: The directory '/root/.cache/luarocks' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing /usr/local/lib/luarocks/rocks/luarocks/2.4.3-1/bin/luarocks with sudo, you may want sudo's -H flag.
sysbench 1.0.13 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 1
Initializing random number generator from current time

Initializing worker threads...

Threads started!

General statistics:
    total time:                          10.0017s
    total number of events:              2792

Latency (ms):
         min:                                    2.17
         avg:                                    3.57
         max:                                   48.12
         95th percentile:                        4.65
         sum:                                 9958.60

Threads fairness:
    events (avg/stddev):           2792.0000/0.00
    execution time (avg/stddev):   9.9586/0.00

You can see I am installing the required Lua modules when starting the container. If a benchmark will be executed several times (which is usually the case) a better approach would be to further customize the Dockerfile to include the necessary modules. I have not done that in the PR though because I think that would bloat the existing image. You can also see that I’m making my machine’s src directory available on the container via the -v ~/src/:/src/. That’s why I can then execute the script from /src/ on the container. Be sure to adjust this as needed to point to a directory tree where the lua script can be found on your machine.

In conclusion, if you have benchmarking needs and have not considered sysbench, don’t be put off if your database of choice is not listed as supported: as long as there’s a Lua driver for it there’s a good chance that you will be able to use sysbench for the task!

Cassandra Use Cases: When To Use and When Not to Use Cassandra


I have a database server that has these features:

  • Highly available by design.
  • Can be Globally distributed.
  • Allows applications write to any node anywhere, anytime .
  • Linearly scalable by simply adding more nodes to the cluster.
  • Automatic workload and data balancing.
  • A query language that looks a lot like SQL.

With the above list of features, why don’t we all use Cassandra for all our database needs? That is the hype I hear at conferences and from some commercial entities pushing their version of Cassandra. Unfortunately, some people believe it. Especially now when many users of proprietary database technologies like Oracle and SQL Server are looking to get out of massive license fees. The low apparent cost of open-source in combination with the list of features above make Cassandra very attractive to many corporate CTOs and CFOs. What they are missing is the core features they assume a database has but are missing from Cassandra.

I am a database architect and consultant. I have been working with Cassandra since version 0.7. in 2010.

I like, and often promote Cassandra to my customers—for the right use cases.

Unfortunately, I often find myself being asked to help after the choice was already made and it ended up either being a poor use case for Cassandra or they made some poor choices in their data modeling for Cassandra.

In this blog post I am going to discuss some of the pitfalls to avoid, suggest a few good use cases for Cassandra and offer just a bit of data modeling advice.

Where Cassandra users go wrong

Cassandra projects tend to as a result of one or more of these reasons:

  1. They used the wrong Cassandra features.
  2. The use case is totally wrong for Cassandra.
  3. They do the data modeling wrong.

Wrong Features

To be honest it doesn’t help that Cassandra has a bunch of features that probably shouldn’t be there. Features leading one to believe you can do some of the things everyone expects a relational database to do:

  • Secondary indexes: They have their uses but not as an alternative access path into a table.
  • Counters: They work mostly, but they are very expensive and should not be used very often. Again, they only work mostly.
  • Light weight transactions: They are not transactions nor are they light weight.
  • Batches: Sending a bunch of operations to the server at one time is usually good, saves network time, right? Well in the case of Cassandra not so much.
  • Materialized views: I got taken in on this one. It looked like it made so very much sense. Because of course it does. But then you look at how it has to work, and you go…Oh No!
  • CQL: Looks like SQL which confuses people into thinking it is SQL.

Using any of the above features the way you would expect them to work in a traditional database is certain to result in serious performance problems and in some cases a broken database.

Get your data model right

Another major mistake developers make in building a Cassandra database is making a poor choice for partition keys.

Cassandra is distributed. This means you need to have a way to distribute the data across multiple nodes. Cassandra does this by hashing a part of every table’s primary key called the partition key and assigning the hashed values (called tokens) to specific nodes in the cluster. It is important to consider the following rules when choosing you partition keys:

  • There should be enough partition key values to spread the data for each table evenly across all the nodes in the cluster.
  • Keep data you want to retrieve in single read within a single partition
  • Don’t let partitions get too big. Cassandra can handle large partitions >100 Megabytes but its not very efficient. Besides if you are getting partitions that large its unlikely your data distribution will be even.
  • In the ideal case all partitions would be roughly the same size. It almost never happens.

Typical real-world partition keys are user id, device id, account number etc. To manage partition size, often a time modifier like year and month or year are added to the partition key.

If you get this wrong, you will suffer greatly. I should probably point out that this is true in one way or another of all distributed databases. The key word here is distributed.

Wrong Use Cases for Cassandra

If you have a database where you are depending on any of the following things Cassandra is wrong for your use case. Please don’t think about Cassandra for it. You will be unhappy.

  • Tables have multiple access paths. Example: lots of secondary indexes.
  • The application depends on identifying rows with sequential values. MySQL autoincrement or Oracle sequences.
  • Cassandra does not do ACID. LSD, Sulphuric or any other kind. If you think you need it go elsewhere. Many times people think they do need it when they don’t though to think about this one.
  • Aggregates: Cassandra does not support aggregates if you need to do a lot of them think another database.
  • Joins: You many be able to data model your self out of this one but take care.
  • Locks: Honestly Cassandra does not support locking. There is a good reason for this don’t try to implement them yourself. I have seen the results of people trying to do locks using Cassandra the results were not pretty.
  • Updates: Cassandra is very good at writes, okay with reads. Updates and deletes are implemented as special cases of writes and that has consequences not immediately obvious.
  • Transactions: CQL has no begin/commit transaction syntax. If you think you need it then Cassandra is a poor choice for you. Don’t try to simulate it. The results won’t be pretty.

If you are thinking about using Cassandra with any of the above requirements, you likely don’t have an appropriate use case. Please think about using another database technology that might better meet your needs.

When you should think about using Cassandra

Every database server ever designed was built to meet specific design criteria. Those design criteria define the use cases where the database will fit well and the use cases where it will not.

Cassandra’s design criteria are the following:

  • Distributed: Runs on more than one server node.
  • Scale linearly: By adding nodes, not more hardware on existing nodes.
  • Work globally: A cluster may be geographically distributed.
  • Favor writes over reads: Writes are an order of magnitude faster than reads.
  • Democratic peer to peer architecture: No master/slave.
  • Favor partition tolerance and availability over consistency: Eventually consistent (see the CAP theorem:
  • Support fast targeted reads by primary key: Focus on primary key reads alternative paths are very sub-optimal.
  • Support data with a defined lifetime: All data in a Cassandra database has a defined lifetime no need to delete it after the lifetime expires the data goes away.

There is nothing in the list about ACID, support for relational operations or aggregates. At this point you might well say, “what is it going to be good for?” ACID, relational and aggregates are critical to the use of all databases. No ACID means no Atomic and without Atomic operations how do you insure anything ever happens correctly meaning consistently. The answer is you don’t. If you were thinking of using Cassandra to keep track of account balances at a bank, you probably should look at alternatives.

Ideal Cassandra Use Cases

It turns out that Cassandra is really very good for some applications.

The ideal Cassandra application has the following characteristics:

  • Writes exceed reads by a large margin.
  • Data is rarely updated and when updates are made they are idempotent.
  • Read Access is by a known primary key.
  • Data can be partitioned via a key that allows the database to be spread evenly across multiple nodes.
  • There is no need for joins or aggregates.

Some of my favorite examples of good use cases for Cassandra are:

  • Transaction logging: Purchases, test scores, movies watched and movie latest location.
  • Storing time series data (as long as you do your own aggregates.)
  • Tracking pretty much anything including order status, packages etc.
  • Storing health tracker data.
  • Weather service history.
  • Internet of things status and event history.
  • Telematics: IOT for cars and trucks
  • Email envelopes—not the contents.


Frequently executives and developers look at the feature set of a technology without understanding the underlying design criteria and the methods used to implement those features. When dealing with distributed databases it’s also very important to recognize how the data and workload will be distributed. Without understanding the design criteria, implementation and distribution plan, any attempt to use a distributed database like Cassandra is going to fail. Usually in a spectacular fashion.


Whether you’re considering an open source or commercial Cassandra deployment, planning to implement it, or already have it in production, Pythian’s certified experts can work with your team to ensure the success of your project at every phase. Learn more about Pythian Services for Cassandra.

KairosDB and Scylla: A Time Series Solution for Performance and Scalability

time series

A highly available time-series solution requires an efficient tailored front-end framework and a backend database with a fast ingestion rate. KairosDB, a time-series database, provides a simple and reliable tooling to ingest and retrieve chronologically created data, such as sensors’ information or metrics. Scylla provides a large-scale, highly reliable and available backend to store large quantities of time-series data.

KairosDB v1.2.0 introduced native CQL support and we tasked KairosDB and Scylla for a performance test. We conducted two tests of KairosDB consisting of single and multiple nodes to demonstrate the performance and scalability of this solution.

What is KairosDB?

KairosDB is a fast distributed scalable time-series database. Initially a rewrite of the original OpenTSDB project, it evolved to a different system where data management, data processing, and visualization are fully separated. To learn how to build a highly available time-series solution with KairosDB, register for our upcoming webinar. More information about the Scylla integration with KairosDB can be found in the Scylla docs. For more information about KairosDB, please visit their website.

Why KairosDB and Scylla?

The usage of Scylla as a highly available, distributed data store for KairosDB allows users to store a large number of metrics while easing the concern of data loss. Scylla provides high throughput, low latency write characteristics to support millions of data point ingestions per second. KairosDB provides users with a simple API to ingest data create aggregations and rollups for fast and reliable metering and reporting. With the KairosDB 1.2 release using the native CQL language, integration of Scylla and KarioDB becomes an easy task. Compared to other solutions, Scylla and KairosDB integration provides unlimited scalability, high availability, and performance. Scylla’s open-source version and KairosDB can scale to an unlimited number of instances.

Testing conf and Setups

  • Scylla v2.1 – 3 nodes cluster running on AWS EC2 i3.8xlarge instances
  • KairosDB (v1.2.0-0.3beta) and Loaders run on AWS EC2 m5.2xlarge instances
  • A dedicated Python code is used to emulate 60K / 180K sensors. Each sensor is emitting 2000 data points at a rate of one data point per second. In other words, 60K / 180K sensors are sending their data to KairosDB every second.
  • In the multiple KairosDB nodes test, we shard the clients and distribute the ingesting clients among the KairosDB nodes
  • Latency measurements were taken by $ nodetool proxyhistograms before and at the end of each run.
  • The KairosDB schema and configuration are available in Appendix-A at the end of this post.

KairosDB Single Instance | 60K Sensors

  • Single KairosDB node with a 200GB disk for KairosDB data files queue | KairosDB batch size = 50
  • Load: 1 loader (2 python scripts) emulating 60K sensors.
  • Runtime: ~40 min
  • Total of 120M partitions (data_points) | Total used storage (RF=3): ~12GB

KairosDB Cluster of Instances | 180K Sensors

  • 3 KairosDB nodes with 200GB disk for KairosDB data files queue | KairosDB batch size = 15
  • In /etc/sysconfig/scylla-server set SCYLLA_ARGS --background-writer-scheduling-quota = 0.3
  • Load: 3 loaders (6 python scripts) emulating 180K sensors. 2 Python scripts per loader.
  • Runtime: ~40 min
  • Total of 360M partitions (data_points) | Total used storage (RF=3): ~36GB

Batch Usage Notes:

  • Batching helps on data aggregation transfers and atomicity of multi-table insertions. For more about batches, read here.
  • See Appendix-A below for more details on the values we used



Scylla Avg. CPU-reactor Load

  • Single KairosDB Instance: ~30%
  • KairosDB Cluster: ~85%

KairosDB Single Instance – nodetool proxyhistogram

KairosDB Cluster of Instances – Screenshots

The “humps” we see in the cpu-reactor load are memtables flush to disk and/or cash merges.


In this post, we demonstrated the performance of Scylla as Backend datastore for KairosDB. Scylla provides low-tail latencies and in this experiment it was no different, keeping both the 95% and 99% latencies at single digits, even when handling writes from 180K sensors per second

We also proved that Scylla’s throughput scales linearly when more KairosDB nodes are added and with little impact on tail latencies. This is all done without making any changes to either the size or available resources of the Scylla Cluster.

Scylla is a great choice for time-series workload.

Next Steps

  • Learn more about Scylla from our product page.
  • See what our users are saying about Scylla.
  • Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
  • Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

Appendix A


KairosDB conf changes:

kairosdb.datastore.cassandra.write_consistency_level=ONE (assuming CL=ONE is sufficient for time series writes)
kairosdb.datastore.cassandra.connections_per_host.local.core=30 (number of cores/shards)
kairosdb.datastore.cassandra.connections_per_host.local.max=30 (same)
kairosdb.datastore.cassandra.connections_per_host.remote.core=30 (same)
kairosdb.datastore.cassandra.connections_per_host.remote.max=30 (same)
kairosdb.datastore.cassandra.max_requests_per_connection.local=8 (default=128, Brian H. recommended using a lower value) kairosdb.datastore.cassandra.max_requests_per_connection.remote=8
kairosdb.ingest_executor.thread_count=20 (default was 10)
kairosdb.queue_processor.batch_size=50 (single KairosDB node) / 15 (three KairosDB nodes) (default: 200)
kairosdb.queue_processor.min_batch_size=50 / 15 (default: 100)


  • Scylla is using 30 CPU threads (out of 32) as 1 physical core is dedicated for interrupts handling
  • Testing found that it is necessary to use a smaller value than the default setting. This was because one of Scylla’s shard handling batches can spike to 100% CPU when handling a heavy load from KairosDB. This leads to write timeouts and poor latency results. In the example, we found the best performance when set to 50. If you are using 3 nodes, you need to divide the batch size evenly (20 per node for example) per node.

KairosDB default schema (RF = 3)

The post KairosDB and Scylla: A Time Series Solution for Performance and Scalability appeared first on ScyllaDB.

Cassandra and Vault – Securing C* Secrets

Hello all! This is the second blog post in this mini-series (You can read the first one here.) This time I will go through the steps and demonstrate how to setup Vault to manage Cassandra credentials! Including generating, leasing, revoking, etc.

Vault, as explained on the previous blog post, is a secret store with a lot of control over those secrets. So let’s setup Vault to do this for our Cassandra cluster.
Note, that a Secret Engine, as explained in this blog post, is not ideal for Super User credentials, as those credentials would expire and regenerate. Making impossible, for example, for Vault to communicate with the Cassandra Cluster. Use the Vault K/V store for those kind of secrets.

Setup Secret Engine

Vault needs to be setup to allow a Secret Engine. Secret Engines allow Vault to manage secrets, via an API, of a given engine (ex: Cassandra, AWS, SSH, etc…). So we are going to now setup our Vault installation to use the Cassandra Secret Engine (official documentation).

I will explain in detail what each configuration does, so you can adapt for your specific needs. Let’s start!

Enabling Vault Secret Engine

Assuming your Vault is running, enabling the secret engine with the following command:

$ ./vault secrets enable database

Now, we need to setup the Cassandra Secret Engine:
<code language='bash'>
$ ./vault write database/config/my-cassandra-database \
>     plugin_name="cassandra-database-plugin" \
>     hosts= \
>     protocol_version=4 \
>     username=cassandra \
>     password=cassandra \
>     tls=false \
>     allowed_roles=read-role,cassandra-all
WARNING! The following warnings were returned from Vault:

  * Read access to this endpoint should be controlled via ACLs as it will
  return the connection details as is, including passwords, if any.

The breakdown of this command is the following:

  • vault write database/config/my-cassandra-database: Write a database configuration called “my-cassandra-database”
  • plugin_name=”cassandra-database-plugin”: Use the Cassandra plugin
  • protocol_version=4: Protocol version to connect to Cassandra
  • username=cassandra: Cassandra super user
  • password=cassandra: Cassandra super user password
  • tls=false: Use TLS or not
  • allowed_roles=read-role,cassandra-all: What roles will be allowed to use this plugin

At this point, the Cassandra secret engine is running and configured. The next step is the role setup.

Configuring Cassandra Roles

The secret engine is useless if it doesn’t know how and what credentials to create. So now we are going to setup 2 roles for our Secret Engine.
One role read-role will give full READ only to all keyspaces. Another role, cassandra-all will create a role
that has ALL access on all keyspaces. For this part, is important to know how to create a user and granting access to resources in Cassandra.
For more information regarding that, consult the user documentation and the grant documentation for Cassandra.
NOTE: CREATE USER is currently deprecated, being replaced by CREATE ROLE, but for backward compatibility and Vault plugin internals, CREATE USER is used.

I will start creating a READ only role.

$ ./vault write database/roles/read-role \
    db_name=my-cassandra-database \
    creation_statements="CREATE USER '{{username}}' WITH PASSWORD '{{password}}' NOSUPERUSER; \
         GRANT SELECT ON ALL KEYSPACES TO {{username}};" \
    default_ttl="1h" \

And now a ALL role:

$ ./vault write database/roles/cassandra-all \
    db_name=my-cassandra-database \
    creation_statements="CREATE USER '{{username}}' WITH PASSWORD '{{password}}' NOSUPERUSER; \
         GRANT ALL ON ALL KEYSPACES TO {{username}};" \
    default_ttl="1h" \

Breakdown of the commands:

  • vault write database/roles/cassandra-all: Write a database role called “cassandra-all”
  • db_name=my-cassandra-database: Use the database called “my-cassandra-database”
  • creation_statements=”…”: How to create this user
  • default_ttl=”1h”: If a lease is not Renewed, how long it will last
  • max_ttl=”24h”: The lease will expire after this time, regardless of renewal

So, at this moment, we do have the secret engine created, and 2 roles. The next and final step, usage and testing!

Using the Secret Engine

To use the secret generation, we just call Vault commands to generate credentials. Lets create both Roles:

$ ./vault read database/creds/read-role
Key                Value
---                -----
lease_id           database/creds/read-role/d671e0ca-9c45-6408-d0ee-7973f81285fb
lease_duration     1h
lease_renewable    true
password           A1a-8p8sp7w99u8v6p62
username           v_root_read_role_5754r5vy9uwrww1qs4pq_1520264273

./vault read database/creds/cassandra-all
Key                Value
---                -----
lease_id           database/creds/cassandra-all/15c576fa-3156-7f61-3293-e570831f1279
lease_duration     1h
lease_renewable    true
password           A1a-rs7uqwwswp5z71w2
username           v_root_cassandra_all_63p22r2s60vxsswwq7xt_1520520093

The commands are explanatory, you tell Vault that you want to READ, vault read, a credential for a given role, database/creds/read-role. Vault will generate a credential each time you read.
Then you can use the displayed username and password to connect to Cassandra.
As I said, each time you read, a credential is generated!! So if you need to re-use the same credential, you renew the lease. For that, you take note
of the lease_id (ex: database/creds/my-role/d671e0ca-9c45-6408-d0ee-7973f81285fb) , and issue a renew command:

$ ./vault lease renew database/creds/read-role/d671e0ca-9c45-6408-d0ee-7973f81285fb

But on the contrary, you need to revoke a lease, issue a revoke command:

$ ./vault lease revoke database/creds/read-role/d671e0ca-9c45-6408-d0ee-7973f81285fb

Now, that we know how to manage and generate dynamic secrets, let’s test!


Testing is the easy part, it’s a matter of logging in and test! Since we revoked the read role, I will create a new one:

$ ./vault read database/creds/read-role
Key                Value
---                -----
lease_id           database/creds/read-role/71af19de-9038-6aab-fa0e-6e587108e94e
lease_duration     1h
lease_renewable    true
password           A1a-25wyp295q476u848
username           v_root_read_role_wptz6yrw7q3649ss13yt_1520265497

Now using the Cassandra super user, lets see if all the users are there:

$ ccm node3 cqlsh -u cassandra -p cassandra
cassandra@cqlsh> select * from system_auth.roles;

 role                                                 | can_login | is_superuser | member_of | salted_hash
 v_root_cassandra_all_63p22r2s60vxsswwq7xt_1520520093 |      True |        False |      null | $2a$10$xfq5Mx8a6iAJSmKisC8m4eVn24dIfcV4cYFnMX5j5InbFNTJaVuh2
     v_root_read_role_w6xur1p369rrzv5xvv9r_1520520122 |      True |        False |      null | $2a$10$FId4t.N./ep8YND3vN9zsuXIjXyF.qxlhndHwhcrHk4UPJvoCuirC
                                            cassandra |      True |         True |      null | $2a$10$Nt/bO/pcyZ1dkKriipC.meL4Pdrke2w2MG2Q6uhb5fjlB4p9Qc8sy

(3 rows)
cassandra@cqlsh> exit

All good! Now lets test the GRANT permissions:

$ ccm node1 cqlsh -u v_root_read_role_wptz6yrw7q3649ss13yt_1520265497 -p A1a-25wyp295q476u848
Connected to triple3111 at
[cqlsh 5.0.1 | Cassandra 3.11.1 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
v_root_read_role_wptz6yrw7q3649ss13yt_1520265497@cqlsh> create KEYSPACE vault_test with replication = {'class': 'NetworkTopologyStrategy', 'datacenter1':3};
Unauthorized: Error from server: code=2100 [Unauthorized] message="User v_root_read_role_wptz6yrw7q3649ss13yt_1520265497 has no CREATE permission on <all keyspaces> or any of its parents"
v_root_read_role_wptz6yrw7q3649ss13yt_1520265497@cqlsh> exit;

Again, all good! Now the other user:

$ ccm node1 cqlsh -u v_root_cassandra_all_63p22r2s60vxsswwq7xt_1520520093 -p A1a-rs7uqwwswp5z71w2
Connected to triple3111 at
[cqlsh 5.0.1 | Cassandra 3.11.1 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
v_root_cassandra_all_63p22r2s60vxsswwq7xt_1520520093@cqlsh> create KEYSPACE vault_test with replication = {'class': 'NetworkTopologyStrategy', 'datacenter1':3};

All good!
It seems all is working, and I hope that this is useful for anyone that needs to manage their Cassandra secrets!


Learn more about Pythian’s services for Cassandra.

Scylla on Samsung NVMe Z-SSDs

Organizations are continuing to adopt Solid State Drives (SSD) in their data centers for optimal performance and lower latencies. With that in mind, it only makes sense to use them with a database solution like Scylla to get the most bang for your buck. One of the popular SSD’s that organizations are adopting now in their data centers is the Samsung Z-SSD drive. In this post, we will go over the Z-SSD and see how Scylla users can benefit from the drives.

The Samsung Z-SSD is a powerful drive that can deliver optimal performance and comes with generous storage capacities up to 800GB. Looking at the numbers presented by Samsung at Scylla Summit 2017, we can see that under sequential traffic, the drive can support up to 3.2 GB/s read and write operations. For random traffic, it can do up to 750K IOPS for reads and 170K IOPs for writes. It is nice that performance is good, but what about latency?


The image below compares latency between the Z-SSD and a PM1725a, a top of the line SSD produced by Samsung. The throughput of the two devices was close, but in terms of latencies, the Z-SSD was 6 times faster. It is also very interesting to see how consistently low the latencies with the Z-SSD were over time from the performance test. Now that we understand how the Z-SSD delivers excellent performance and lower latency, let’s see what this means for Scylla users.


When Samsung tested Scylla, their goal was to compare Scylla in-memory versus Scylla on the Z-SSD using max throughput under given tail-latency constraints as the metric. Testing was done with cassandra-stress with a varying read/write ratio (100%/0%-75/25%) and varying percent of request served from memory (25% to 75%). The database was populated with 300 GB of data with a payload of 1 kb. Let’s see the results!



The images above show a read and write that that was done. You can see additional testing results in the video or slides below. In all of the tests, Running Scylla on the Z-SSD helped reduce the performance gap between memory and storage and delivers 44-23% of RAM performance at a lower cost. Also, 50-75% of requests were served from memory and future software improvements could further shrink the memory/storage gap.

I encourage you to watch the video above or the slides below to learn more about getting the best performance out of Scylla with Samsung Z-SSD’s.

Next Steps

  • Learn more about Scylla from our product page.
  • See what our users are saying about Scylla.
  • Download Scylla. Check out our download page to run Scylla on AWS, install it locally in a Virtual Machine, or run it in Docker.
  • Take Scylla for a Test drive. Our Test Drive lets you quickly spin-up a running cluster of Scylla so you can see for yourself how it performs.

The post Scylla on Samsung NVMe Z-SSDs appeared first on ScyllaDB.

ScyllaDB at Linux SCaLE 16x

Linux Scylla Love graphic

We’re at Linux SCaLE 16x this week

Stop by booth #219 at Linux SCaLE 16x at the Pasadena Convention Center where we’ll be running live demos of Scylla running different workloads. We’d love to discuss your database projects and use cases, and you can pick up a Scylla t-shirt or sea monster stuffie!

Linux SCALE crowd at ScyllaDB booth

ScyllaDB presenting “Internal Architecture of a Modern NoSQL Datastore”

On Sunday, March 11th we will present “Internal Architecture of a Modern NoSQL Datastore.” In this session, we will discuss the make-up of the Scylla datastore—a C++ from-scratch reimplementation of the popular Apache Cassandra database with a performance baseline not previously seen in the NoSQL world. You’ll get a deep dive into Scylla’s unique components—including a userspace CPU Scheduler, a userspace disk I/O Scheduler, a tailored memory allocator and changes to the standard C++ exception handling. We’ll explain why we developed them and the advantages they provide.

We’ll also cover the areas where Linux helps us the most and where we were better off moving functionality that traditionally lives in the Kernel to userspace, which Linux interfaces are employed in modern datastores and why, and the trade-offs that usually come with them.

What: Internal Architecture of a Modern NoSQL Datastore
Where: Ballroom C
When: Sunday, March 11, 2018 – 11:30 to 12:30

Hope to see you then!

The post ScyllaDB at Linux SCaLE 16x appeared first on ScyllaDB.

Cassandra and Vault – Distributed Secret Store

Hello all, I’m doing a mini-series of blog posts on using Cassandra and Hashicorp Vault.

For the first blog in this series, I’ll look at using Cassandra as a Datastore for Vault secrets, then how to generate Cassandra secrets using Vault, and I’ll also demonstrate how to manage Cassandra credentials from Vault.

Attention: This is not intended to be followed straight on for production systems as several key security features are not done properly (ex: Disabling Cassandra superuser).

The definition of Vault is:

Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. Vault handles leasing, key revocation, key rolling, and auditing. Through a unified API, users can access an encrypted Key/Value store and network encryption-as-a-service, or generate AWS IAM/STS credentials, SQL/NoSQL databases, X.509 certificates, SSH credentials, and more.

There are several ways to deploy Vault. And several storage engines, Cassandra being one of them. Cassandra gives the advantage of a distributed, highly-available, geo-replicated storage. Since we want to manage secrets, this can be an interesting point.

From here, I’m going to guide you how to use Cassandra as a Vault datastore.

1. Setup Cassandra

First, we need to setup Cassandra for storing the data. In this guide, I’m using CCM to achieve this.

Create a 3.11.1, 3 node cassandra cluster

$ ccm create triple3111 -v3.11.2 -n 3 -s
Check if the cluster is running
$ ccm status
Cluster: 'triple3111'
node3: UP
node2: UP
node1: UP

$ ccm node1 nodetool status
Datacenter: datacenter1
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
UN  166.89 KiB  1            100.0%            6c160c38-9041-4109-a98f-c9100f0f297a  rack1
UN  153.52 KiB  1            100.0%            900c1007-b3e9-402b-8ab1-9f1f093a0720  rack1
UN  174.13 KiB  1            100.0%            5f54cf00-533e-4400-a898-ab2bd3fe8c17  rack1
Create the Vault keyspaces (You can find more information here)
$ ccm node1 cqlsh -e "CREATE KEYSPACE \"vault\" WITH REPLICATION = {'class': 'NetworkTopologyStrategy','datacenter1': 3};"
$ ccm node1 cqlsh -e "CREATE TABLE \"vault\".\"entries\" (bucket text,key text,value blob,PRIMARY KEY (bucket, key)) WITH CLUSTERING ORDER BY (key ASC);"
Now, we need to enable Authentication and Authorization, for that, we stop the cluster, edit the configs, and start again
$ ccm stop
$ # Edit cassandra.yaml on the 3 nodes
$ ccm start
Confirm that this is working as expected:
$ ccm node1 cqlsh -u cassandra -p cassandra
cassandra@cqlsh> list users;
name | super
cassandra | True
Everything Cassandra is done at this point! Let’s move to Vault!

2. Install and Configure Vault

Now that we have Cassandra set and running, let’s get vault working!
Create a directory to install vault, and download vault.
$ mkdir vault
$ cd vault
$ wget ""
$ unzip
Let’s test vault:
$ ./vault -h
Usage: vault <command> [args]

Common commands:
    read        Read data and retrieves secrets
    write       Write data, configuration, and secrets
    delete      Delete secrets and configuration
    list        List data or secrets
    login       Authenticate locally
    server      Start a Vault server
    status      Print seal and HA status
    unwrap      Unwrap a wrapped secret

Other commands:
    audit          Interact with audit devices
    auth           Interact with auth methods
    lease          Interact with leases
    operator       Perform operator-specific tasks
    path-help      Retrieve API help for paths
    policy         Interact with policies
    secrets        Interact with secrets engines
    ssh            Initiate an SSH session
    token          Interact with tokens
At this moment, Vault is not yet running, and we just confirmed that the vault binary is in working condition. For this blog, I’m running vault in dev mode (-dev), but since there is a storage engine present, the second time you would start vault with the storage engine and in dev mode, it will fail to start. To avoid this, either TRUNCATE the Cassandra table, or use Vault in non-dev mode.
First, create a vault config file (documentation), setting Cassandra as the data engine:
$ vim config.json
storage "cassandra" {
  hosts            = "localhost"
  consistency      = "LOCAL_QUORUM"
  protocol_version = 3
  username = "cassandra"
  password = "cassandra"

listener "tcp" {
  address     = ""
  tls_disable = 1
Now let’s start vault:
$ ./vault server -dev -config config.json
==> Vault server configuration:

                     Cgo: disabled
         Cluster Address:
              Listener 1: tcp (addr: "", cluster address: "", tls: "disabled")
              Listener 2: tcp (addr: "", cluster address: "", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
        Redirect Address:
                 Storage: cassandra
                 Version: Vault v0.9.5
             Version Sha: 36edb4d42380d89a897e7f633046423240b710d9

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

    $ export VAULT_ADDR=''

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: irNaBhZFlHjbx/2y76/+Hd0u41XxxaE6HbuCGWzc0ME=
Root Token: f467dc15-4381-061a-7fe3-2561baaaa6ce

Development mode should NOT be used in production installations!

==> Vault server started! Log data will stream in below:

At this point we have vault running. Let’s export the variable and insert some secrets

$ export VAULT_ADDR=''
$ vault write secret/hello value=world
Success! Data written to: secret/hello
Now we can check if Cassandra is getting data:
$ ccm node1 cqlsh -u cassandra -p cassandra
cassandra@cqlsh> SELECT count(*) FROM "vault"."entries";

It is working! In my next blog post, I will demonstrate how to configure and use the Cassandra database engine to generate Cassandra secrets!


Six Plus Years (and counting) with No Downtime? Seriously?

I’ve been at DataStax for over six years now (seven-year anniversary coming up in July) and I’ve been privileged to witness customers do some amazing things with DataStax Enterprise (DSE) and Apache Cassandra™. But at this point with DataStax, one thing stands out to me over everything else:

We have customers who have been with DataStax as long as I have who have not experienced a single second of data platform downtime.

As someone who’s worked in database tech for decades now, that fact really blows me away. Customers like Walmart, eBay, Netflix, and others who use DSE and its Cassandra foundation may have seen hiccups and outages in other areas of their infrastructure, but not their data layer. It’s literally been on and alive for over six years, across version upgrades and even hardware migrations.

How is that possible?

The Simple Explanation

When I tell non-tech people what I do for a living, they usually need some kind of example for it to make sense. So, I typically pull out my phone, talk about the digital apps that everyone has, and say: “We deliver the information these apps and you need anywhere, anytime, all the time.” That’s the simple but powerful value that DataStax solutions deliver.

Now, to be able to provide these types of applications, there’s a simple—yet very difficult—requirement: you have to be able to not just read data from anywhere 24x7x365, but write changes to it as well, and then make sure it’s up to date and available in multiple locations. The concepts of write-anywhere and location independence ensure not only consistent performance for distributed applications but also 100% uptime, at least as far as the data layer is concerned.

Given that this requirement is pretty much table stakes in our digital world, you’d think a lot of database vendors would be able supply this capability. But you’d be wrong.

The Simple Foundation

What does it take to pull off read-write anywhere and location independence? It’s a matter of architecture.

Almost all major database vendors fall into two different architectures where distributing data is concerned: (1) master-slave; (2) multi-master. While fine for centralized or semi-distributed apps, these designs are fundamentally ill equipped to handle widely distributed systems, such as most of the digital applications we use today.

A third architecture—masterless—is used by DataStax/Cassandra and is currently fairly unique to the database scene, at least where major vendors are concerned. A masterless design allows users to read and write their data to any node in a cluster and have it automatically replicated and synced with other copies of that data in multiple locations. It’s this masterless architecture that gives DSE/Cassandra the ability to deliver true constant uptime over other database management system providers.

It probably comes as no surprise that all of the relational database management system  vendors are master-slave and/or multi-master. But you may be surprised to learn that almost all NoSQL vendors have the same design.

MongoDB? Replica sets and master-slave (primary/secondary).

Couchbase and Redis? Master-slave.

What about some of the big cloud vendor’s NoSQL platforms, like Microsoft Cosmos DB? Master-slave and multi-master. Amazon DynamoDB? Even with its global tables option, DynamoDB is still master-slave and multi-master. Also, don’t forget that both Cosmos DB and DynamoDB are proprietary products specific to their cloud owners, so you won’t get the data autonomy (e.g. the ability to run both on premise and on multiple clouds) with them that you get with DSE.

Getting back to the architecture differences, DSE’s masterless architecture exceeds both master-slave and multi-master designs, which require much more work to achieve even a semblance of high availability and have no chance of reaching continuous availability, which is what today’s digital applications require.

For example, as reported by Amazon, the downtime experienced by DynamoDB speaks to the difference I’m highlighting. Don’t get me wrong—this isn’t to point fingers at Amazon’s or Microsoft’s infrastructure, which is probably better than average overall. The point is that you need a database that can tolerate outages with no loss of availability.

How About You?

If you’d like the comfort of knowing your applications’ data layer won’t ever go down and want to experience the same uptime as many DataStax customers, check out our resources page to learn more or visit DataStax Academy for free, go-at-your-own-pace online education that shows you how to deploy always-on apps that keep your enterprise not just open for business but thriving in the Right-Now Economy. 

Mutant Monitoring Systems (MMS) Day 5 – Visualizing Data with Zeppelin


This is part 5 of a series of blog posts that provides a story arc for Scylla Training.

In the last post, we simulated that Division 3 was under attack and learned how to recover from a node failure, consistency levels, how to repair the Scylla cluster. Now that operations are back in a good operating state, we can get back to the important part of the Mutant Monitoring System, analyzing data. In this post, we will learn how to visualize the data in MMS with Apache Zeppelin.

Apache Zeppelin is a Java Web-based solution that allows users to interact with a variety of data sources like MySQL, Spark, Hadoop, and Scylla. Once in Zeppelin, you can run CQL queries and view the output in a table format with the ability to save the results. Also, the query can be visualized in an array of different graphs. To get started with Zeppelin, let’s build and run the container.

Building and Running the Scylla Cluster and Zeppelin

First, the Scylla Cluster should be up and running with the data imported from the previous blog posts. The MMS Git repository has been updated to provide the ability to automatically import the keyspaces and data. If you have the Git repository cloned already, you can simply do a “git pull” in the scylla-code-samples directory.

git clone
cd scylla-code-samples/mms

Modify docker-compose.yml and add the following line under the environment: section of scylla-node1:


Now the container can be built and run:

docker-compose build
docker-compose up -d

Roughly after 60, the existing MMS data will be automatically imported.

To build and run Zeppelin in Docker, run the following commands:

cd scylla-code-samples/mms/zeppelin
docker build -t zeppelin .
docker run --name zeppelin --network mms_web -p 9080:8080 -d zeppelin

Using Zeppelin

Shortly after running the above commands, you should be able to access the Zeppelin Web Interface in your browser at

Now that Zeppelin is loaded, we can get started with creating a Notebook. A Notebook in Zeppelin is basically a collection of data from a source like Scylla. Click Notebook at the top of the web console and click “Create new note”. Type MMS for the Note Name and choose Cassandra for the Default Interpreter. Finally, click Create Note.

We can use Zeppelin to run queries just like cqlsh. Let’s view all of the tracking data like we did before by typing the following in the query box:

The data can be visualized in a variety of graphs by choosing one from the toolbar:


Let’s visualize the heat readings from the Tracking system for each mutant by clicking on the Area Chart button. Once the Area Chart has been chosen, click on the settings link and adjust the values as shown:

We can see that Jim Jeffries had the highest heat signature. If you recall from Day 3, Jim has been known to have increased heat levels around 20 or higher when he receives negative feedback on his comedy performance. Also, he will likely injure them out of anger. Because of this, we alerted the authorities.

Let’s use the information available to see which mutant is likely the fastest with a Line Chart with the following settings:

Here we can see that Bob Loblaw was the fastest mutant. Also, Jim Jeffries should probably hit the gym and practice cardiovascular exercises a few times a week.


Luckily there was no havoc done to Division 3 or the general public today. With this downtime, we were able to focus on a new way to analyze data from the Mutant Monitoring System. With Apache Zeppelin, we can easily visualize our queries and turn them into useful graphs. Please be safe out there as we continue to track the mutants and evolve our Monitoring System.

The post Mutant Monitoring Systems (MMS) Day 5 – Visualizing Data with Zeppelin appeared first on ScyllaDB.