AppMonet: Painting the Digital Landscape with Scylla

29 September 2020, 5:42 pm by ScyllaDB

AppMonet, based in the heart of New York City, connects mobile app developers with the largest brand advertisers and agencies for higher revenue through video ads. With one unified SDK, AppMonet supports all major video brands while maximizing and diversifying apps’ advertising demand sources.

In addition, they use AI to automatically fit the content better to screen with a strong call to action (CTA) button that increases ad effectiveness. By offering unique brand demand and rich media formats, AppMonet has become a popular choice for hundreds of global applications.

How AppMonet Works

AppMonet enables publishers to use multiple ad bidding services on the backend with a single small software development kit (SDK). Almost all of the top 100 brand advertising demand sources work with AppMonet.

Beneath every ad served there is a process known as Real Time Bidding (RTB). This standard, maintained by the Internet Advertising Bureau (IAB), defines the different aspects of what sort of ads can be placed, such as their size and page placement, acceptable media formats, and how auctions are conducted. For in-app video ads, AppMonet specifically supports both Video Ad Serving Technology (VAST) and Video Player Ad-Serving Interface Definition (VPAID) pre-caching and playing of ads. This dramatically increases effective video fill rate and revenue.

The issue for mobile developers has been that each ad network supports their own methods to embed ads in mobile apps. AppMonet, through its unified container technology for video and display content, supports all of the Supply Side Platforms (SSPs): Smaato, AOL, Smart Adserver, InMobi, AerServ, LoopMe. PubNative, MobileFuse, Fyber, Xandr, OpenX, Rubicon, TappX, Mobfox, SpotX, RhythmOne, AdColony, and so on.

AppMonet’s SDK is efficient, both in terms of memory and bandwidth. For example, it watches your device’s memory usage to ensure optimal video playing. It also avoids using various hacks found elsewhere in the industry, such as a Javascript video-to-image converter, which can dramatically balloon bandwidth usage.

AppMonet’s Path to Scylla

AppMonet’s path to Scylla Cloud was a migration from their first implementation. Their infrastructure was originally developed in Erlang mnesia. Written by Ericsson for highly distributed telecom work, mnesia seemed like a good fit for their purposes. However, growth and success proved their back-end was not capable of scaling to meet their production demand. AppMonet needed an alternative that would let them scale by an order of magnitude, and fast.

They benchmarked a number of leading solutions: Redis, Aerospike, Cassandra and Scylla. While they had experience with Redis, they felt it had architectural limitations for their use case. Aerospike was popular in AdTech, but seemed proprietary. They knew of Cassandra, but while it was famous for fast writes, it didn’t seem a good fit for their read-heavy workload.

That’s when they read a blog post about AdGear’s use of Scylla, and learned about their Scylla Summit presentation. According to AppMonet co-founder Nick Jacob, “It was an inspiration. It seemed like a perfect fit.”

Their one-table schema was pretty simple. They use a compound primary key, but it was facile to model it across different databases.

“It was pretty clear, pretty quickly, that Scylla gave us the most flexibility, but also the best performance. For our use case it was pretty obvious that Scylla was the best option.”

How AppMonet Uses Scylla

AppMonet turned to Scylla Cloud to run their backend on their own AWS account. They need to support both real-time writes to the application, as well as batch bulk updates daily and weekly from Apache Spark. They also export data to Presto or Athena to perform critical analytics. To ensure their bulk updates or analytics do not impact production transactions they utilize Scylla’s unique workload prioritization feature.

Another advantage of Scylla Cloud was offloading database management. The Scylla Cloud team handles all of the administration, which AppMonet relies upon for handling incidents and isolating errors. Even if it was not the database but their own app code which required some tuning.

“From maintaining the database, we really don’t need to do anything,” Nick noted. “We use the advanced monitoring via the Grafana dashboard. It has been helpful to identify issues.”

AppMonet are deployed on a cluster of i3.4xlarge Amazon Web Services EC2 servers, managing 600 GB currently, with plenty of room to expand. Nick observed that with Scylla they continue to grow linearly without any dropoff in performance. Currently they are running around 20,000 queries per second — 52 billion requests a month — but the load was only at 18%. “We’re confident we can handle multiple times the workload.”

To serve customers with the right ads in the right location, the team at AppMonet perfected the bidding process. In order to make the bidding efficient, latency plays a crucial part. The time it takes to retrieve data from the user interests and match it to the advertisers define the efficiency of the ad. Low latency data retrieval makes sure data presented to the user is contextual to the information she is watching.

Most critically of all, Scylla delivers latencies matching AppMonet’s needs.

“Having it be in our AWS account we can do sub-millisecond average latencies on the reads.” Nick confirmed average latencies were 200 µseconds (microseconds). “It’s awesome.”

AppMonet’s Next Steps

AppMonet is a small company trying to grow quickly. With Scylla their future is secure. They wanted a database that handled multi-datacenter replication — a built-in feature of Scylla — to deploy to multiple global datacenters. Nick was bullish. “The product can grow with us.”

Your Next Steps

Now it’s your turn. Speaking of effective calls-to-action, here’s ours:

TRY SCYLLA CLOUD TODAY!

The post AppMonet: Painting the Digital Landscape with Scylla appeared first on ScyllaDB.

Scylla University: Coding with Scala, Part 1

24 September 2020, 6:41 pm by ScyllaDB

This post is based on the Scylla Specific Drivers, Overview, Paging and Shard Awareness lesson in Scylla University, ScyllaDB’s free resource to learn NoSQL database development and Scylla administration. You can find the lesson here in Scylla University.

In a previous lesson, we explained how a Scylla Administrator restores and backs-up a cluster. As the number of mutants is on the rise, Division 3 decided that we must use more applications to connect to the mutant catalog and decided to hire Java developers to create powerful applications that can monitor the mutants. This lesson will explore how to connect to a Scylla cluster using the Phantom library for Scala: a Scala-idiomatic wrapper over the standard Java driver.

When creating applications that communicate with a database such as Scylla, it is crucial that the programming language being used includes support for database connectivity. Since Scylla is compatible with Cassandra, we can use any of the available Cassandra libraries. For example, in Go, there is GoCQL and GoCQLX. In Node.js, there is the cassandra-driver. For the JVM, we have the standard Java driver available. Scala applications on JVM can use it, but a library tailored for Scala’s features offers a more enjoyable and type-safe development experience. Since Division 3 wants to start investing in Scala, let’s begin by writing a sample Scala application.

Creating a Sample Scala Application

The sample application that we will create connects to a Scylla cluster, displays the contents of the Mutant Catalog table, inserts and deletes data, and shows the table’s contents after each action. First, we will go through each section of the code used and then explain how to run the code in a Docker container that accesses the Scylla Mutant Monitoring cluster. You can see the file here: scylla-code-samples/mms/scala/scala-app/src/main/scala/com/scylla/mms/App.scala.

As mentioned, we’re using the Phantom library for working with Scylla. The library’s recommended usage pattern involves modeling the database, table, and data service as classes. We’ll delve into those in a future lesson.

First, let’s focus on how we set up the connection itself. We start by importing the library:

This import brings all the necessary types and implicits into scope. The main entry point for our application is the App object’s main method. In it, we set up the connection to the cluster:

Next, in a list, we specify the DNS names through which the application contacts Scylla and the keyspace to use.

Finally, we instantiate the MutantsDatabase and the MutantsService classes. These classes represent, respectively, the collection of tables in use by our application and a domain-specific interface to interact with those tables.

We’re not operating on low-level types like ResultSet or Row objects with the Phantom library, but rather on strongly-typed domain objects. Specifically, for this lesson, we will be working with a Mutant data type defined as follows:

You can find the file here: scylla-code-samples/mms/scala/scala-app/src/main/scala/com/scylla/mms/model/Mutant.scala

The MutantsService class contains implementations of the functionality required for this lesson. You can find it here:

scylla-code-samples/mms/scala/scala-app/src/main/scala/com/scylla/mms/service/MutantService.scala.

Let’s consider how we would fetch all the mutant definitions present in the table:

Phantom provides a domain-specific language for interacting with Scylla. Instead of threading CQL strings through our code, we’re working with a strongly-typed interface that verifies, at compile-time, that the queries we are generating are correct (according to the data type definitions).

In this case, we are using the select method to fetch all Mutant records from Scylla. Note that by default, Phantom uses Scala’s Future data type for representing all the Scylla operations. Behind the scenes, Phantom uses a non-blocking network I/O to efficiently perform all of the queries. If you’d like to know more about Future, this is a good tutorial.

Moving forward, the following method allows us to store an instance of the Mutant data type into Scylla:

Everything is strongly typed, and as such, we’re using actual instances of our data types to interact with Scylla. This gives us increased maintainability of our codebase. If we refactor the Mutant data type, this code will no longer compile, and we’ll be forced to fix it.

And lastly, here’s a method that deletes a mutant with a specific name:

We can see that Phantom’s DSL can represent predicates in a type-safe fashion as well. We’re forced to use a matching type for comparing the firstName and lastName.

We can sequence the calls to these methods in our main entry point using a for comprehension on the Future values returned. The code below is from the following file:

scylla-code-samples/mms/scala/scala-app/src/main/scala/com/scylla/mms/App.scala

In this example, we execute all the futures sequentially (with some additional callbacks registered on them for printing out the information).

With the coding part done, let’s set up a Scylla Cluster and then run the sample application in Docker.

Set-up a Scylla Cluster

The example requires a single DC cluster. Follow this procedure to remove previous clusters and set up a new cluster.

Once the cluster is up, we’ll create the catalog keyspace and populate it with data.

The first task is to create the keyspace named catalog for the mutants’ catalog.

docker exec -it mms_scylla-node1_1 cqlsh

CREATE KEYSPACE catalog WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy','DC1' : 3};

Now that the keyspace is created, it is time to create a table to hold the mutant data.

use catalog;

CREATE TABLE mutant_data (
    first_name text,
    last_name text,
    address text,
    picture_location text,
    PRIMARY KEY((first_name, last_name)));

Now let’s add a few mutants to the catalog with the following statements:

insert into mutant_data ("first_name","last_name","address","picture_location") VALUES ('Bob','Loblaw','1313 Mockingbird Lane', 'http://www.facebook.com/bobloblaw');
insert into mutant_data ("first_name","last_name","address","picture_location") VALUES ('Bob','Zemuda','1202 Coffman Lane', 'http://www.facebook.com/bzemuda');
insert into mutant_data ("first_name","last_name","address","picture_location") VALUES ('Jim','Jeffries','1211 Hollywood Lane', 'http://www.facebook.com/jeffries');

Building the Scala Example

If you previously built the Scala Docker, you can skip directly to the section Running the Scala Example below.

Otherwise, to build the application in Docker, change into the scala subdirectory in scylla-code-samples:

cd scylla-code-samples/mms/scala

Now we can build and run the container:

docker build -t scala-app .

docker run -d --net=mms_web --name some-scala-app scala-app

To connect to the shell of the container, run the following command:

docker exec -it some-scala-app sh

Running the Scala Example

Finally, the sample Scala application can be run:

java -jar scala-app/target/scala-2.13/mms-scala-app-assembly-0.1.0-SNAPSHOT.jar

The output of the application will be:

Conclusion

In this lesson, we explained how to create a sample Scala application that executes a few basic CQL operations on a Scylla cluster using the Phantom library. These were only the basics, and there are more exciting topics that Division 3 wants developers to explore. In the next lesson, we will review the structure of an application using the Phantom library.

In the meantime, please be safe out there and continue to monitor the mutants!

*This lesson was written with the help of Itamar Ravid. Itamar is a Distributed systems engineer with a decade of experience in a wide range of technologies. He focuses on JVM-based microservices in Scala using functional programming. Thank you, Itamar!

TAKE THIS LESSON IN SCYLLA UNIVERSITY

The post Scylla University: Coding with Scala, Part 1 appeared first on ScyllaDB.

[Webcast] Leave it to Astra: Cassandra-as-a-Service on Google Cloud

23 September 2020, 6:59 pm by Datastax Technical HowTo's

Scylla Summit 2021 Call for Speakers

23 September 2020, 5:16 pm by ScyllaDB

Scylla Summit 2021 will be held as an online event this year, January 12 – 14, 2021. Our own sessions will showcase the latest Scylla features and capabilities, roadmap plans and the view from the top from our executives. We also want to extend this Call for Speakers, inviting you to tell your own story to the global community of your peers. Users’ voices and real-world perspectives are vital for the success of any software industry conference.

What Our Attendees Most Want to Hear

Our attendees appreciate hearing real-world examples from practitioners — developers and devops, architects, technical leaders, and product owners. We’re looking for compelling technical sessions: case studies, operational best practices, and more. Here’s a list of suggested topics, or feel free to recommend others.

Real-world Use Cases

Everything from design and architectural considerations to POCs, to production deployment and operational lessons learned, including…

• Practical examples for vertical markets — IoT, AI/ML, security, e-commerce, fraud detection, adtech/martech, customer/user data, media/multimedia, social apps, B2B or B2C apps, and more.
• Migrations — How did you get from where you were in the past to where you are today? What were the limitations of other systems you needed to overcome? How did Scylla address those issues?
• Hard numbers — Clusters, nodes, CPUs, RAM and disk, data size, growth, throughput, latencies, benchmark and stress test results (think graphs, charts and tables). Your cluster can be any size — large or small. What attendees appreciate hearing most is how you got the best results out of your available resources.
• Integrations — Scylla is just one of many big data systems running in user environments. What feeds it from upstream, or what does it populate downstream? Kafka? Spark? Parquet? Pandas? Other SQL or NoSQL systems? JanusGraph? KairosDB? Our users love block diagrams and want to understand data flows.
• War stories — What were the hardest big data challenges you faced? How did you solve them? What lessons can you pass along?

Tips & Tricks

From getting started to performance optimization to disaster management, tell us your DevOps secrets, or unleash your chaos monkey!

• Computer languages and dev tools — What’s your favorite languages and tools? Are you a Pythonista? Doing something interesting in Golang or Rust?
• Working on Open Source? — Got a Github repo to share? Our attendees would love to walk your code.
• Seastar — The Seastar infrastructure at the heart of Scylla can be used for other projects as well. What systems architecture challenges are you tackling with Seastar?

Schedule

• CFP opens: Mon, Sep 21st, 2020
• CFP closes (deadline for submissions: Thursday, Oct 22th, 2020, 5:00 PM Pacific
• Scylla Summit (event dates): Tue, Jan 12th – Thu, Jan 14th, 2021

Required Information

You’ll be asked to include the following information in your proposal:

• Proposed title
• Description
• Suggested main topic
• Audience information
• Who is this presentation for?
• What will the audience take away?
• What prerequisite knowledge would they need?
• Videos or live demos included in presentation?
• Length of the presentation (10 or 15 minute slots or longer)

Ten Tips for a Successful Speaker Proposal

Please keep in mind this event is made by and for deeply technical professionals. All presentations and supporting materials must be respectful and inclusive.

• Be authentic — Your peers want your personal experiences and examples drawn from real-world scenarios
• Be catchy — Give your proposal a simple and straightforward but catchy title
• Be interesting — Make sure the subject will be of interest to others; explain why people will want to attend and what they’ll take away from it
• Be complete — Include as much detail about the presentation as possible
• Don’t be “pitchy” — Keep proposals free of marketing and sales. We tend to ignore proposals submitted by PR agencies and require that we can reach the suggested participant directly.
• Be understandable — While you can certainly cite industry terms, try to write a jargon-free proposal that contains clear value for attendees
• Be deliverable — Sessions have a fixed length, and you will not be able to cover everything. The best sessions are concise and focused.
• Be specific — Overviews aren’t great in this format; the narrower and more specific your topic is, the deeper you can dive into it, giving the audience more to take home
• Be cautious — Live demos sometimes don’t go as planned, so we don’t recommend them
• Be rememberable — Leave your audience with take-aways they’ll be able to take back to their own organizations and work. Give them something they’ll remember for a good long time.

SUBMIT A SPEAKER PROPOSAL

The post Scylla Summit 2021 Call for Speakers appeared first on ScyllaDB.

Seedless NoSQL: Getting Rid of Seed Nodes in Scylla

22 September 2020, 6:09 pm by ScyllaDB

Nodes in a Scylla cluster are symmetric, which means any node in the cluster can serve user read or write requests, and no special roles are assigned to a particular node. For instance, no primary nodes vs. secondary nodes or read nodes vs. write nodes.

This is a nice architecture property. However, there is one small thing that breaks the symmetry: that is the seed concept. Seed nodes are nodes that help the discovery of the cluster and propagation of gossip information. Users must assign a special role to some of the nodes in the cluster to serve as seed nodes.

In addition to breaking the symmetric architecture property, do seed nodes introduce any real issues? In this blog post, we will walk through the pains of seed nodes and how we get rid of them to make Scylla easier and more robust to operate than ever before.

What are the real pains of the seed nodes?

First, the seed node does not bootstrap. Bootstrap is a process that a new joining node streams data from existing nodes. However, seed nodes skip the bootstrap process and stream no data. It is a popular source of confusion for our users since users are surprised to see no data streamed on the new node after adding a node.

Second, the user needs to decide which nodes will be assigned as the seed nodes. Scylla recommends 3 seed nodes per datacenter (DC) if the DC has more than 6 nodes, or 2 seed nodes per DC if the DC has less than 6 nodes. If the number of nodes grows, the user needs to update the seed nodes configuration and modify scylla.yaml on all nodes.

Third, it is quite complicated to add a new seed node or replace a dead seed node. When a user wants to add a node as a seed node, the node can not be added into the cluster as a seed node directly. The correct way to do this is to add the node as a non-seed node, then promote it as a seed node. This is because a seed node does not bootstrap which means it does not stream data from existing nodes. When a seed node is dead and the users want to replace it, they can not use the regular replacing node procedure to replace it directly. Users need to first promote a non-seed node to act as a seed node and update the configuration on all nodes, then perform the replacing node procedure to replace it.

Those special treatments for seed nodes complicate the administration and operation of a Scylla cluster. Scylla needs to carefully document those differences between seed and no-seed nodes. Users can make mistakes easily even with the documents.

Can we get rid of those seed concepts to simplify things and make it more robust?

Let’s first take a closer look into what is the special role of seed nodes.

Seed nodes help in two ways:

1) Define the target nodes to talk within gossip shadow round

But what is a gossip shadow round? It is a routine used for a variety of purposes when a cluster boots up:

• to get gossip information from existing nodes before normal gossip service starts
• to get tokens and host IDs of the node to be replaced in the replacing operation
• to get tokens and status of existing nodes for bootstrap operation
• to get features known by the cluster to prevent an old version of a Scylla node that does not know any of those features from joining the cluster

2) Help to converge gossip information faster

Seed nodes help converge gossip information faster because, in each normal gossip round, a seed node is contacted once per second. As a result, the seed nodes are supposed to have more recent gossip information. Any nodes communicating with seed nodes will obtain the more recent information. This speeds up the gossip convergence.

How do we get rid of the seed concept?

To get rid of seen nodes entirely, you will have to solve for each of the functions seed nodes currently provide:

1) Configuration changes

• The seeds option

The parameter --seed-provider-parameters seeds= is now used only once when a new node joins for the first time. The only purpose is to find the existing nodes.

• The auto-bootstrap option

In Scylla, the --auto-bootstrap option defaults to true and is not present in the scyllal.yaml. This was intentionally done to avoid misuse. It is designed to make the bootstrap process faster when initializing a fresh cluster, by skipping the process to stream data from existing nodes.

In contrast, the Scylla AMI sets --auto-bootstrap to false because most of the time when AMI nodes start they are forming a fresh cluster. When adding a new AMI node to an existing cluster, users must set the –auto-bootstrap option to true. It is easy to forget setting the --auto-bootstrap option to true for AMIs and end up with a new node without any data streamed, which is annoying.

The new solution is that the --auto-bootstrap option will now be ignored so that we have less dangerous options and fewer errors to make. With this change, all new nodes — both seed and non-seed nodes — must bootstrap when joining the cluster for the first time.

One small exception is that the first node in a fresh cluster is not bootstrapped. This is because it is the first node and there are no other nodes to bootstrap from. The node with the smallest IP address is selected as the first node automatically, e.g., with seeds = “192.168.0.1,192.168.0.2,192.168.0.3”, node 192.168.0.1 will be selected as the first node and skips the bootstrap procedure. So when starting nodes to form a fresh cluster, always use the same list of nodes in the seeds config option.

2) Gossip shadow round target nodes selection

Before this change, only the seed nodes communicated within the gossip shadow round. The shadow round finishes immediately after any of the seed nodes have responded.

After this change, if the node is a new node that only knows the nodes listed in the seed config option, it talks to all the nodes listed in the seed option and waits for each of them to respond to finish the shadow round.

If the node is an existing node, it will talk to all nodes saved in the system.peers, without consulting the seed config option.

3) Gossip normal round target nodes selection

Currently, in each gossip normal round, 3 types of nodes are selected to talk with:

• Select 10% of live nodes randomly
• Select a seed node if the seed node is not selected above
• Select an unreachable node

After this change, the selection has been changed to below:

• Select 10% of live nodes in a random shuffle + group fashion

For example, there are 20 nodes in the cluster [n1, n2, .., n20]. The nodes are first being shuffled randomly. Then they are divided into 10 groups so that each group has 10% of live nodes. In each round, nodes in one of the groups are selected to talk with. When all groups have talked once, the nodes are shuffled randomly again and divided into 10 groups. This procedure repeats. Using a random shuffle method to select target nodes is also used in other gossip implementations, e.g., SWIM.

This method helps converge gossip information in a more deterministic way so we can drop the selection of one seed node in each gossip round.

• Select an unreachable node

Unreachable node selection is not changed. It is communicated in order to bring back dead nodes into the cluster.

4) Improved message communication for shadow round

Currently, Scylla reuses the gossip SYN and ACK messages for the shadow round communication. Those messages are one way and are asynchronous which means the sender will not wait for the response from the receiver. As a result, there are many special cases in gossip message handlers in order to support the shadow round.

After this change, a new two way and synchronous RPC message has been introduced for communications in the gossip shadow round. This dedicated message makes it much easier to track which nodes have responded to the gossip shadow round and waits until all the nodes have responded. It also allows requesting the gossip application states that the sender is really interested in, which reduces the message size transferred on the network. Thanks to the new RPC message, the special handling of the regular gossip SYN and ACK message can be eliminated.

In a mixed cluster, the node will fall back to the old shadow round method in case the new message is not supported for compatibility reasons. We cannot introduce a gossip feature bit to decide if the new shadow round method can be used because the gossip service is not even started when we conduct a shadow round.

Summary

Getting rid of the seed concept simplifies Scylla cluster configuration and administration, makes Scylla nodes fully symmetric, and prevents unnecessary operation errors.

The work is merged in Scylla master and will be released in the upcoming Scylla 4.3 release.

Farewell seeds and hello seedless!

LEARN MORE ABOUT SCYLLA OPEN SOURCE

The post Seedless NoSQL: Getting Rid of Seed Nodes in Scylla appeared first on ScyllaDB.

Apache Cassandra Usage Report 2020

17 September 2020, 11:00 am by Apache Cassandra Blog

Apache Cassandra is the open source NoSQL database for mission critical data. Today the community announced findings from a comprehensive global survey of 901 practitioners on Cassandra usage. It’s the first of what will become an annual survey that provides a baseline understanding of who, how, and why organizations use Cassandra.

“I saw zero downtime at global scale with Apache Cassandra. That’s a powerful statement to make. For our business that’s quite crucial.” - Practitioner, London

Key Themes

Cassandra adoption is correlated with organizations in a more advanced stage of digital transformation.

People from organizations that self-identified as being in a “highly advanced” stage of digital transformation were more likely to be using Cassandra (26%) compared with those in an “advanced” stage (10%) or “in process” (5%).

Optionality, security, and scalability are among the key reasons Cassandra is selected by practitioners.

The top reasons practitioners use Cassandra for mission critical apps are “good hybrid solutions” (62%), “very secure” (60%), “highly scalable” (57%), “fast” (57%), and “easy to build apps with” (55%).

A lack of skilled staff and the challenge of migration deters adoption of Cassandra.

Thirty-six percent of practitioners currently using Cassandra for mission critical apps say that a lack of Cassandra-skilled team members may deter adoption. When asked what it would take for practitioners to use Cassandra for more applications and features in production, they said “easier to migrate” and “easier to integrate.”

Methodology

Sample. The survey consisted of 1,404 interviews of IT professionals and executives, including 901 practitioners which is the focus of this usage report, from April 13-23, 2020. Respondents came from 13 geographies (China, India, Japan, South Korea, Germany, United Kingdom, France, the Netherlands, Ireland, Brazil, Mexico, Argentina, and the U.S.) and the survey was offered in seven languages corresponding to those geographies. While margin of sampling error cannot technically be calculated for online panel populations where the relationship between sample and universe is unknown, the margin of sampling error for equivalent representative samples would be +/- 2.6% for the total sample, +/- 3.3% for the practitioner sample, and +/- 4.4% for the executive sample.

To ensure the highest quality respondents, surveys include enhanced screening beyond title and activities of company size (no companies under 100 employees), cloud IT knowledge, and years of IT experience.

Rounding and multi-response. Figures may not add to 100 due to rounding or multi-response questions.

Demographics

Practitioner respondents represent a variety of roles as follows: Dev/DevOps (52%), Ops/Architect (29%), Data Scientists and Engineers (11%), and Database Administrators (8%) in the Americas (43%), Europe (32%), and Asia Pacific (12%).

Respondents include both enterprise (65% from companies with 1k+ employees) and SMEs (35% from companies with at least 100 employees). Industries include IT (45%), financial services (11%), manufacturing (8%), health care (4%), retail (3%), government (5%), education (4%), telco (3%), and 17% were listed as “other.”

Cassandra Adoption

Twenty-two percent of practitioners are currently using or evaluating Cassandra with an additional 11% planning to use it in the next 12 months.

Of those currently using Cassandra, 89% are using open source Cassandra, including both self-managed (72%) and third-party managed (48%).

Practitioners using Cassandra today are more likely to use it for more projects tomorrow. Overall, 15% of practitioners say they are extremely likely (10 on a 10-pt scale) to use it for their next project. Of those, 71% are currently using or have used it before.

Cassandra Usage

People from organizations that self-identified as being in a “highly advanced” stage of digital transformation were more likely to be using Cassandra (26%) compared with those in an “advanced” stage (10%) in “in process” (5%).

Cassandra predominates in very important or mission critical apps. Among practitioners, 31% use Cassandra for their mission critical applications, 55% for their very important applications, 38% for their somewhat important applications, and 20% for their least important applications.

“We’re scheduling 100s of millions of messages to be sent. Per day. If it’s two weeks, we’re talking about a couple billion. So for this, we use Cassandra.” - Practitioner, Amsterdam

Why Cassandra?

The top reasons practitioners use Cassandra for mission critical apps are “good hybrid solutions” (62%), “very secure” (60%), “highly scalable” (57%), “fast” (57%), and “easy to build apps with” (55%).

“High traffic, high data environments where really you’re just looking for very simplistic key value persistence of your data. It’s going to be a great fit for you, I can promise that.” - Global SVP Engineering

For companies in a highly advanced stage of digital transformation, 58% cite “won’t lose data” as the top reason, followed by “gives me confidence” (56%), “cloud native” (56%), and “very secure” (56%).

“It can’t lose anything, it has to be able to capture everything. It can’t have any security defects. It needs to be somewhat compatible with the environment. If we adopt a new database, it can’t be a duplicate of the data we already have.… So: Cassandra.” - Practitioner, San Francisco

However, 36% of practitioners currently using Cassandra for mission critical apps say that a lack of Cassandra-skilled team members may deter adoption.

“We don’t have time to train a ton of developers, so that time to deploy, time to onboard, that’s really key. All the other stuff, scalability, that all sounds fine.” – Practitioner, London

When asked what it would take for practitioners to use Cassandra for more applications and features in production, they said “easier to migrate” and “easier to integrate.”

“If I can get started and be productive in 30 minutes, it’s a no brainer.” - Practitioner, London

Conclusion

We invite anyone who is curious about Cassandra to test the 4.0 beta release. There will be no new features or breaking API changes in future Beta or GA builds, so you can expect the time you put into the beta to translate into transitioning your production workloads to 4.0.

We also invite you to participate in a short survey about Kubernetes and Cassandra that is open through September 24, 2020. Details will be shared with the Cassandra Kubernetes SIG after it closes.

Survey Credits

A volunteer from the community helped analyze the report, which was conducted by ClearPath Strategies, a strategic consulting and research firm, and donated to the community by DataStax. It is available for use under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0).

AWS Outposts: Run Fully Managed NoSQL Workloads On-Premises Using Scylla

15 September 2020, 2:00 pm by ScyllaDB

AWS Outposts are a fully managed on-premises extension of Amazon Web Services. It puts all of your familiar infrastructure and APIs available right in your own local environment. Ideal for low-latency use cases, AWS Outposts allow you to run EC2 instances, Elastic Block Storage (EBS), and container-based services like Amazon’s EKS for Kubernetes.

Order what you need through your AWS Console, and a team of experts shows up on-site for the installation. Your local on-premises environment shows up as your own private zone and you store all of the physical assets in a traditional 80 x 48 x 24 inch 42U rack.

Scylla as an On-Premises Alternative to DynamoDB

Not all Amazon services are available in AWS Outposts, however. Notably, Amazon DynamoDB is currently unable to run in an on-premise environment.

This is where Scylla serves a role. With Scylla’s DynamoDB-compatible API, Alternator, you can run your DynamoDB workloads in your local AWS Outpost, using Scylla Cloud, our fully managed database-as-a-service (DBaaS) running in your own AWS account. Want to use the same data on AWS Outposts and AWS public region? Run Scylla Cloud in a multi-datacenter, hybrid, setup: one DC on-prem, one DC on a public region.

By the way, if you are more familiar with Scylla’s CQL interface, the same Cassandra Query Language used in Apache Cassandra, that’s fine too. You can also use Scylla Cloud on AWS Outposts as an on-premises managed Cassandra-compatible service. (Learn more about the differences between the DynamoDB API and the Cassandra CQL interface here.)

Scylla Cloud has been tested and certified by Amazon as “AWS Outposts ready.” As a member of the AWS Partner Network you can rely on the experts at ScyllaDB to manage an enterprise-grade NoSQL database on-premises for your critical business needs.

You can provision standard EC2 instance types such as the storage-intensive I3en series, which can store up to 60 terabytes of data. Administrators can also divide larger physical servers into smaller virtual servers for separate workloads and use cases.

Afterwards, you can then deploy Scylla Cloud on your AWS account. Once you have, you should be ready to run!

If you need to learn more in detail about how the Scylla DynamoDB-compatible API works, check out our lesson on using the DynamoDB API in Scylla from Scylla University. It’s completely free.

As a last point, you can also use Scylla Enterprise or Scylla Open Source on AWS Outposts if you want to manage your own database.

Next Steps

Contact us to learn more about using Scylla in AWS Outposts, or sign up for our Slack channel to discuss running on Outposts with our engineers and your big data industry peers.

ASK US ABOUT RUNNING SCYLLA ON AWS OUTPOSTS

The post AWS Outposts: Run Fully Managed NoSQL Workloads On-Premises Using Scylla appeared first on ScyllaDB.

Better Cassandra Indexes for a Better Data Model: Introducing Storage-Attached Indexing

9 September 2020, 11:20 pm by Datastax Technical HowTo's

Improving Apache Cassandra’s Front Door and Backpressure

3 September 2020, 11:00 am by Apache Cassandra Blog

As part of CASSANDRA-15013, we have improved Cassandra’s ability to handle high throughput workloads, while having enough safeguards in place to protect itself from potentially going out of memory. In order to better explain the change we have made, let us understand at a high level, on how an incoming request is processed by Cassandra before the fix, followed by what we changed, and the new relevant configuration knobs available.

How inbound requests were handled before

Let us take the scenario of a client application sending requests to C* cluster. For the purpose of this blog, let us focus on one of the C* coordinator nodes.

Below is the microscopic view of client-server interaction at the C* coordinator node. Each client connection to Cassandra node happens over a netty channel, and for efficiency purposes, each Netty eventloop thread is responsible for more than one netty channel.

The eventloop threads read requests coming off of netty channels and enqueue them into a bounded inbound queue in the Cassandra node.

A thread pool dequeues requests from the inbound queue, processes them asynchronously and enqueues the response into an outbound queue. There exist multiple outbound queues, one for each eventloop thread to avoid races.

The same eventloop threads that are responsible for enqueuing incoming requests into the inbound queue, are also responsible for dequeuing responses off from the outbound queue and shipping responses back to the client.

Issue with this workflow

Let us take a scenario where there is a spike in operations from the client. The eventloop threads are now enqueuing requests at a much higher rate than the rate at which the requests are being processed by the native transport thread pool. Eventually, the inbound queue reaches its limit and says it cannot store any more requests in the queue.

Consequently, the eventloop threads get into a blocked state as they try to enqueue more requests into an already full inbound queue. They wait until they can successfully enqueue the request in hand, into the queue.

As noted earlier, these blocked eventloop threads are also supposed to dequeue responses from the outbound queue. Given they are in blocked state, the outbound queue (which is unbounded) grows endlessly, with all the responses, eventually resulting in C* going out of memory. This is a vicious cycle because, since the eventloop threads are blocked, there is no one to ship responses back to the client; eventually client side timeout triggers, and clients may send more requests due to retries. This is an unfortunate situation to be in, since Cassandra is doing all the work of processing these requests as fast as it can, but there is no one to ship the produced responses back to the client.

So far, we have built a fair understanding of how the front door of C* works with regard to handling client requests, and how blocked eventloop threads can affect Cassandra.

What we changed

Backpressure

The essential root cause of the issue is that eventloop threads are getting blocked. Let us not block them by making the bounded inbound queue unbounded. If we are not careful here though, we could have an out of memory situation, this time because of the unbounded inbound queue. So we defined an overloaded state for the node based on the memory usage of the inbound queue.

We introduced two levels of thresholds, one at the node level, and the other more granular, at client IP. The one at client IP helps to isolate rogue client IPs, while not affecting other good clients, if there is such a situation.

These thresholds can be set using cassandra yaml file.

native_transport_max_concurrent_requests_in_bytes_per_ip
native_transport_max_concurrent_requests_in_bytes

These thresholds can be further changed at runtime (CASSANDRA-15519).

Configurable server response to the client as part of backpressure

If C* happens to be in overloaded state (as defined by the thresholds mentioned above), C* can react in one of the following ways:

• Apply backpressure by setting “Autoread” to false on the netty channel in question (default behavior).
• Respond back to the client with Overloaded Exception (if client sets “THROW_ON_OVERLOAD” connection startup option to “true.”

Let us look at the client request-response workflow again, in both these cases.

THROW_ON_OVERLOAD = false (default)

If the inbound queue is full (i.e. the thresholds are met).

C* sets autoread to false on the netty channel, which means it will stop reading bytes off of the netty channel.

Consequently, the kernel socket inbound buffer becomes full since no bytes are being read off of it by netty eventloop.

Once the Kernel Socket Inbound Buffer is full on the server side, things start getting piled up in the Kernel Socket Outbound Buffer on the client side, and once this buffer gets full, client will start experiencing backpressure.

THROW_ON_OVERLOAD = true

If the inbound queue is full (i.e. the thresholds are met), eventloop threads do not enqueue the request into the Inbound Queue. Instead, the eventloop thread creates an OverloadedException response message and enqueues it into the flusher queue, which will then be shipped back to the client.

This way, Cassandra is able to serve very large throughput, while protecting itself from getting into memory starvation issues. This patch has been vetted through thorough performance benchmarking. Detailed performance analysis can be found here.

Apache Cassandra vs DynamoDB

24 August 2020, 8:15 pm by Cassandra – Instaclustr

In this post, we’ll look at some of the key differences between Apache Cassandra (hereafter just Cassandra) and DynamoDB.

Both are distributed databases and have similar architecture, and both offer incredible scalability, reliability, and resilience. However, there are also differences,  and understanding the differences and cost benefits can help you determine the right solution for your application.

Apache Cassandra is an open source database available at no cost from the Apache Foundation. Installing and configuring Cassandra can be challenging and there is more than one pitfall along the way. However, Cassandra can be installed on any cloud service or at a physical location you choose.

The typical Cassandra installation is a cluster which is a collection of nodes (a node is a single instance of Cassandra installed on a computer or in a Docker container). Nodes can then be grouped in racks and data centers which can be in different locations (cloud zones and regions or physical collocations). You must scale Cassandra as your demand grows and are responsible for the ongoing management tasks such as backups, replacing bad nodes, or adding new nodes to meet demand.

Amazon DynamoDB is a fully managed database as a service. All implementation details are hidden and from the user viewpoint DynamoDB is serverless. DynamoDB automatically scales throughput capacity to meet workload demands, and partitions and repartitions your data as your table size grows, and distributes data across multiple availability zones. However, the service is available only through Amazon Web Services (AWS).

Replica Configuration and Placement

NoSQL data stores like Cassandra and DynamoDB use multiple replicas (copies of data) to ensure high availability and durability. The number of replicas and their placement determines the availability of your data.

With Cassandra, the number of replicas to have per cluster—the replication factor—and their placement is configurable. A cluster can be subdivided into two or more data centers which can be located in different cloud regions or physical collocations. The nodes in a data center can be assigned to different racks that can be assigned to different zones or to different physical racks.

In contrast, with DynamoDB, Amazon makes these decisions for you. By default, data is located in a single region and is replicated to three (3) availability zones in that region. Replication to different AWS regions is available as an option. Amazon streams must be enabled for multi-region replication.

Data Model

The top level data structure in Cassandra is the keyspace which is analogous to a relational database. The keyspace is the container for the tables and it is where you configure the replica count and placement. Keyspaces contain tables (formerly called column families) composed of rows and columns. A table schema must be defined at the time of table creation.

The top level structure for DynamoDB is the table which has the same functionality as the Cassandra table. Rows are items, and cells are attributes. In DynamoDB, it’s possible to define a schema for each item, rather than for the whole table.

Both tables store data in sparse rows—for a given row, they store only the columns present in that row. Each table must have a primary key that uniquely identifies rows or items. Every table must have a primary key which has two components: 

• A partition key that determines the placement of the data by subsetting the table rows into partitions. This key is required.
• A key that sorts the rows within a partition. In Cassandra, this is called the clustering key while DynamoDB calls it the sort key. This key is optional.

Taken together, the primary key ensures that each row in a table is unique. 

 Differences 

• DynamoDB limits the number of tables in an AWS region to 256.  If you need more tables, you must contact AWS support. There are no hard limits in Cassandra. The practical limit is around 500 tables.
• DynamoDB is schemaless. Only the primary key attributes need to be defined at table creation. 
• DynamoDB charges for read and write throughput and requires you to manage capacity for each table. Read and write throughput and associated costs must be considered when designing tables and applications. 
• The maximum size of an item in DynamoDB is 400KB. With Cassandra, the hard limit is 2GB; the practical limit is a few megabytes.
• In DynamoDB, the primary key can have only one attribute as the primary key and one attribute as the sort key. Cassandra allows composite partition keys and multiple clustering columns.
• Cassandra supports counter, time, timestamp, uuid, and timeuuid data types not found in DynamoDB.

Allocating Table Capacity

Both Cassandra and DynamoDB require capacity planning before setting up a cluster. However, the approaches are different. 

To create a performant Cassandra cluster, you must first make reasonably accurate estimates of your future workloads. Capacity is allocated by creating a good data model, choosing the right hardware, and properly sizing the cluster. Increasing workloads are met by adding nodes.

With DynamoDB, capacity planning is determined by the type of the read/write capacity modes you choose. On demand capacity requires no capacity planning other than setting an upper limit on each table. You pay only for the read and write requests on the table. Capacity is measured in Read Resource Units and Write Resource Units. On demand mode is best when you have an unknown workload, unpredictable application traffic, or you prefer the ease of paying for only what you use.

With provisioned capacity, you must specify the number of reads and write throughput limits for each table at the time of creation. If you exceed these limits for a table or tables, DynamoDB will throttle queries until usage is below defined capacity. Auto-scaling will adjust your table’s provisioned capacity automatically in response to traffic changes although there is a lag between the time throttling starts and increased capacity is applied.   

The throughput limits are provisioned in units called Read Capacity Units (RCU) and Write Capacity Units (WCU); queries are throttled whenever these limits are exceeded. One read capacity unit represents one strongly consistent read per second, or two eventually consistent reads per second, for an item up to 4 KB in size. Transactional read requests require two read capacity units to perform one read per second for items up to 4 KB. If you need to read an item that is larger than 4 KB, DynamoDB must consume additional read capacity units. One write capacity unit represents one write per second for an item up to 1 KB in size. If you need to write an item that is larger than 1 KB, DynamoDB must consume additional write capacity units. Transactional write requests require 2 write capacity units to perform one write per second for items up to 1 KB. For more information, see Managing Settings on DynamoDB Provisioned Capacity Tables.

Provisioned mode is a good option if any of the following are true: 

• You have predictable application traffic.
• Application traffic is consistent or ramps gradually.
• You can forecast capacity requirements to control costs.

Partitions

Both Cassandra and DynamoDB group and distribute data based on the hashed value of the partition key. Both call these grouping partitions but they have very different definitions.

In Dynamo the partition is a storage unit that has a maximum size of 10 GB. When a partition fills, DynamoDB creates a new partition and the user has no control over the process. A partition can contain items with different partition key values. When a sort key is used, all the items with the same partition key value physically close together, ordered by sort key value.

DynamoDB partitions have capacity limits of 3,000 RCU or 1,000 WCU even for on-demand tables. Furthermore, these limits cannot be increased. If you exceed the partition limits, your queries will be throttled even if you have not exceeded the capacity of the table. See Throttling and Hot Keys (below) for more information.

A Cassandra partition is a set of rows that share the same hashed partition key value.  Rows with the same partition key are stored on the same node. Rows within the partition are sorted by the clustering columns. If no clustering column was specified, the partition holds a single row. While it would not be desirable, it would be possible for an application to drive tens of thousands of reads/writes to a single partition. 

See Cassandra Data Partitioning.

Query Language 

Cassandra provides a SQL-like language called Cassandra Query Language (CQL) to access data. DynamoDB uses JSON syntax. The following table shows the syntax for the query “return all information from the Music table for the song title ‘Lullaby of Broadway’ and the artist ‘Tommy Dorsey’”

Secondary Indexes

By default, Cassandra and DynamoDB queries can use only the primary key columns in the search condition which must include all partition key columns. Non-key columns can be used in a search by creating an index on that column.

Cassandra supports creating an index on most columns including a clustering column of a compound primary key or on the partition key itself. Creating an index on a collection or the key of a collection map is also supported. However, when used incorrectly a secondary index can hurt performance. A general rule of thumb is to index a column with low cardinality of few values and to use only with the partition key in the search clause. Because the index table is stored on each node in a cluster, a query using a secondary index can degrade performance if multiple nodes are accessed. 

DynamoDB has local secondary indexes. This index uses the same partition key as the base table but has a different sort key. Scoped to the base table partition that has the same partition key value. Local secondary indexes must be created at the same time the table is created. A maximum of 5 local secondary indexes may be created per table. 

Materialized Views versus Global Secondary Indexes 

In Cassandra, a Materialized View (MV) is a table built from the results of a query from another table but with a new primary key and new properties. Queries are optimized by the primary key definition. The purpose of a materialized view is to provide multiple queries for a single table. It is an alternative to the standard practice of creating a new table with the same data if a different query is needed. Data in the materialized view is updated automatically by changes to the source table. However, the materialized view is an experimental feature and should not be used in production.

A similar object is DynamoDB is the Global Secondary Index (GSI) which creates an eventually consistent replica of a table. The GSI are created at table creation time and each table has a limit of 20. The GSI must be provisioned for reads and writes; there are storage costs as well.

Time To Live

Time To Live (TTL) is a feature that automatically removes items from your table after a period of time has elapsed.

• Cassandra specifies TTL as the number of seconds from the time of creating or updating a row, after which the row expires.
• In DynamoDB, TTL is a timestamp value representing the date and time at which the item expires.
• DynamoDB applies TTL at item level. Cassandra applies it to the column.

Consistency

Both Cassandra and DynamoDB are distributed data stores.  In a distributed system there is a tradeoff between consistency—every read receives the most recent write or an error, and availability—every request receives a (non-error) response but without the guarantee that it contains the most recent write. In such a system there are two levels possible levels of consistency:

• Eventual consistency. This implies that all updates reach all replicas eventually. A read with eventual consistency may return stale data until all replicas are reconciled to a consistent state.
• Strong consistency returns up-to-date data for all prior successful writes but at the cost of slower response time and decreased availability.

DynamoDB supports eventually consistent and strongly consistent reads on a per query basis. The default is eventual consistency. How it is done is hidden from the user.

 Strongly consistent reads in DynamoDB have the following issues: 

• The data might not be available if there is a network delay or outage.
• The operation may have higher latency.
• Strongly consistent reads are not supported on global secondary indexes.
• Strongly consistent reads use more throughput capacity than eventually consistent reads and therefore is more expensive. See Throttling and Hot Keys (below).

Cassandra offers tunable consistency for any given read or write operation that is configurable by adjusting consistency levels. The consistency level is defined as the minimum number of Cassandra nodes that must acknowledge a read or write operation before the operation can be considered successful. You are able to configure strong consistency for both reads and writes at a tradeoff of increasing latency.

Conflicts Resolution

In a distributed system, there is the possibility that a query may return inconsistent data from the replicas. Both Cassandra and DynamoDB resolve any inconsistencies with a “last write wins” solution but with Cassandra, every time a piece of data is written to the cluster, a timestamp is attached. Then, when Cassandra has to deal with conflicting data, it simply chooses the data with the most recent timestamp.

For DynamoDB, “last write wins” applies only to global tables and strongly consistent reads.

Security Features

Cassandra and DynamoDB provide methods for user authentication and authorization and data access permissions  Both use encryption for client and inter-node communication. DynamoDB also offers encryption at rest. Instaclustr offers encryption at rest for its Cassandra Managed Services.

Performance Issues

Consistency and Read Speed

Choosing strong consistency requires more nodes to respond to a request which increases latency.

Scans

Scans are expensive for both systems. Scans in Cassandra are slow because Cassandra has to scan all nodes in the cluster. Scans are faster in DynamoDB but are expensive because resource use is based on the amount of data read not returned to the client. If the scan exceeds your provisioned read capacity, DynamoDB will generate errors.

DynamoDB’s Issues

Auto Scaling

Amazon DynamoDB auto scaling uses the AWS Application Auto Scaling Service to dynamically adjust provisioned throughput capacity on your behalf, in response to actual traffic patterns. This enables a table or a global secondary index to increase its provisioned read and write capacity to handle sudden increases in traffic, without throttling. When the workload decreases, application auto scaling decreases the throughput so that you don’t pay for unused provisioned capacity.

• Auto scaling does not work well with varying and bursty workloads. The table will scale up only based on consumption, triggering these alarms time after time until it reaches the desired level.
• There can be a lag (5-15 minutes) between the time capacity is exceeded and autoscaling takes effect.
• Capacity decreases are limited. The maximum is 27 per day (A day is defined according to the GMT time zone).
• Tables scale based on the number of requests made, not the number of successful requests.
• Autoscaling cannot exceed hard I/O limits for tables.

Throttling and Hot Keys

In DynamoDB, the provisioned capacity of a table is shared evenly across all the partitions in a table with the consequence that each partition has a capacity less than the table capacity. For example, a table that has been provisioned with 400 WCU and 100 RCU and had 4 partitions, each partition would have a write capacity of 100 WCU and 25 RCU.  

So if you had a query send most read/write requests to a single partition—a hot partition— and that throughput exceeded the shared capacity of that partition, your queries would be throttled even though you had unused capacity in the table. If your application creates a hot partition, your only recourse would be either to redesign the data model or to over allocate capacity to the table.

Adaptive capacity can provide up to 5 minutes of grace time by allocating unused capacity from other partitions to the “hot” one provided unused capacity is available and hard limits are not reached. The hard limits on a partition are 3,000 RCU or 1,000 WCU.

Cross-Region Replication

If you want a DynamoDB table to span regions, you’ll need to use global tables that require careful planning and implementation. The tables in all regions must use on-demand allocation or have auto scaling enabled. The tables in each region must be the same: time to live, the number of global, and local secondary indexes, and so on.

Global tables support only eventual consistency that can lead to data inconsistency. In such a conflict, DynamoDB applies a last writer wins (lww) policy. The data with the most recent timestamp is used.

Migrating an existing local table to a global table requires additional steps. First, DynamoDB Streams is enabled on the original local table to create a changelog. Then an AWS Lambda function is configured to copy the corresponding change from the original table to the new global table.

Cost Explosion 

As highlighted in Managed Cassandra Versus DynamoDB, DynamoDB’s pricing model can easily make it expensive for a fast growing company:

• The pricing model is based on throughput, therefore costs increase as IO increases. 
• A hot partition may require you to overprovision a table.
• Small reads and writes are penalized because measured throughput is rounded up to the nearest 1 KB boundary for writes and 4 KB boundary for reads.
• Writes are four times more expensive than reads.
• Strongly consistent reads are twice the cost of eventually consistent reads. 
• Workloads performing scans or queries can be costly because the read capacity units are calculated on the number of bytes read rather than the amount of data returned.
• Read heavy workloads may require DynamoDB Accelerator (DAX) which is priced per node-hour consumed. A partial node-hour consumed is billed as a full hour.
• Cross region replication incurs additional charges for the amount of data replicated by the Global Secondary Index and the data read from DynamoDB Streams.
• DynamoDB does not distinguish between a customer-facing, production table versus tables used for development, testing, and staging.

Summary 

DynamoDB’s advantages are an easy start; absence of the database management burden; sufficient flexibility, availability, and auto scaling; in-built metrics for monitoring; encryption of data at rest.

Cassandra’s main advantages are: fast speed of writes and reads; constant availability; a SQL-like Cassandra Query Language instead of DynamoDB’s complex API; reliable cross data center replication; linear scalability and high performance. 

However, the mere technical details of the two databases shouldn’t be the only aspect to analyze before making a choice.

Cassandra versus DynamoDB costs must be compared carefully. If you have an application that mostly writes or one that relies on strong read consistency, Cassandra may be a better choice.

You will also need to know which technologies you need to supplement the database? If you need the open source like ElasticSearch, Apache Spark, or Apache Kafka, Cassandra is your choice. If you plan to make extensive use of AWS products, then it’s DynamoDB.

If you use a cloud provider other than AWS or run your own data centers, then you would need to choose Cassandra.

The post Apache Cassandra vs DynamoDB appeared first on Instaclustr.

Python scylla-driver: how we unleashed the Scylla monster’s performance

24 August 2020, 12:21 pm by scylla – Ultrabug

At Scylla summit 2019 I had the chance to meet Israel Fruchter and we dreamed of working on adding shard-awareness support to the Python cassandra-driver which would be known as scylla-driver.

A few months later, when Israel reached out to me on the Scylla-Users #pythonistas Slack channel with the first draft PR I was so excited that I jumped in the wagon to help!

The efforts we put into the scylla-driver paid off and significantly improved the performance of the production applications that I had the chance to switch to using it by 15 to 25%!

Before we reached those numbers and even released the scylla-driver to PyPi, EuroPython 2020 RFP was open and I submitted a talk proposal which was luckily accepted by the community.

So I had the chance to deep-dive into Cassandra vs Scylla architecture differences, explain the rationale behind creating the scylla-driver and give Python code details on how we implemented it and the challenges we faced doing so. Check my talk spage for

I also explained that I wrote an RFC on the scylla-dev mailing list which lead the developers of Scylla to implement a new connection-to-shard algorithm that will allow clients connecting to a new listening port to select the actual shard they want a connection to.

This is an expected major optimization from the current mostly random way of connecting to all shards and I’m happy to say that it’s been implemented and is ready to be put to use by all the scylla drivers.

I’ve recently been contacted by PyCon India and other Python related conferences organizers for a talk so I’ve submitted one to PyCon India where I hope I’ll be able to showcase even better numbers thanks to the new algorithm!

After my Europython talk we also had very interesting discussions with Pau Freixes about his work on a fully asynchronous Python driver that wraps the C++ driver to get the best possible performance. First results are absolutely amazing so if you’re interested in this, make sure to give it a try and contribute to the driver!

Stay tuned for more amazing query latencies

Benchmarking Cassandra with Local Storage on Azure

19 August 2020, 8:17 pm by Cassandra – Instaclustr

Continuing our efforts in adding support for the latest generation of Azure Virtual Machines, Instaclustr is pleased to announce support for the D15v2 and L8v2 types with local storage support for Apache Cassandra clusters on the Instaclustr Managed Platform. So let’s begin by introducing the two forms of storage.

Local storage is a non-persistent (i.e. volatile) storage that is internal to the physical Hyper-V host that a VM is running on at any given time. As soon as the VM is moved to another physical host as a result of a deallocate (stop) and start command, hardware or VM crash, or Azure maintenance, the local storage is wiped clean and any data stored on it is lost.

Remote storage is persistent and comes in the form of managed (or unmanaged) disks.  These disks can have different performance and redundancy characteristics. Remote storage is generally used for any VM’s OS disk or data disks that are intended to permanently store important data. Remote storage can be detached from the VM from one physical host to another in order to preserve the data. It is stored on Azure’s storage account which VMs use to attach it via the network. The physical Hyper-V host running the VM is independent of the remote storage account, which means that any Hyper-V host or VM can mount a remotely stored disk on more than one occasion.

Remote storage has the critical advantage of being persistent, while local storage has the advantage of being faster (because it’s local to the VM – on the same physical host) and included in the VM cost. When starting an Azure VM you only pay for its OS and disks. Local storage on a VM is included in the price of the VM and often provides a much cheaper storage alternative to remote storage.

The Azure Node Types

D15_v2s

D15_v2s are the latest addition to Azure’s Dv2 series, featuring more powerful CPU and optimal CPU-to-memory configuration making them suitable for most production workloads. The Dv2-series is about 35% faster than the D-series. D15_v2s are a memory optimized VM size offering a high memory-to-CPU ratio that is great for distributed database servers like Cassandra, medium to large caches, and in-memory data stores such as Redis.

Instaclustr customers can now leverage these benefits with the release of the D15_v2 VMs, which provide 20 virtual CPU cores and 140 GB RAM backed with 1000 GB of local storage.

L8_v2s

L8_v2s are a part of the Lsv2-series featuring high throughput, low latency, and directly mapped local NVMe storage. L8s like the rest of the series were designed to cater for high throughput and high IOPS workloads including big data applications, SQL and NoSQL databases, data warehousing, and large transactional databases. Examples include Cassandra, Elasticsearch and Redis. In general, applications that can benefit from large in-memory databases are a good fit for these VMs.

L8s offer 8 virtual CPU cores and 64 GB in RAM backed by local storage space of 1388 GB. L8s provide roughly equivalent compute capacity (cores and memory) to a D13v2 at about ¾ of the price. This offers speed and efficiency at the cost of forgoing persistent storage.

Benchmarking

Prior to the public release of these new instances on Instaclustr, we conducted Cassandra benchmarking for:

All tests were conducted using Cassandra 3.11.6

The results are designed to outline the performance of utilising local storage and the benefits it has over remote storage. Testing is split into two groups, fixed and variable testing. Each group runs two different sets of payload:

1. Small – Smaller more frequent payloads which are quick to execute but are requested in large numbers.
2. Medium – Larger Payloads which take longer to execute and are potentially capable of slowing Cassandra’s ability to process requests

The fixed testing procedure is:

1. Insert data to fill disks to ~30% full.
2. Wait for compactions to complete.
3. Run a fixed rate of operations
4. Run a sequence of tests consisting of read, write and mixed read/write operations.
5. Run the tests and measure the operational latency for a fixed rate of operations. 

The variable testing procedure is:

1. Insert data to fill disks to ~30% full.
2. Wait for compactions to complete.
3. Target a median latency of 10ms.
4. Run a sequence of tests comprising read, write and mixed read/write operations. These tests were broken down into multiple types.
5. Run the tests and measure the results including operations per second and median operation latency. 
6. Quorum consistency for all operations.
7. We incrementally increase the thread count and re-run the tests until we have hit the target median latency and are under the allowable pending compactions.
8. Wait for compactions to complete after each test.

As with any generic benchmarking results the performance may vary from the benchmark for different data models or application workloads. However, we have found this to be a reliable test for comparison of relative performance that will match many practical use cases.

Comparison

When comparing performance between instance types with local and remote storage it is important to take note of the latency of the operations. The latency of read operations indicate how long a request takes to retrieve information from the disk and return it back to the client.

In the fixed small and medium read tests, local storage offered better latency results in comparison to instances with remote storage. This is more noticeable in the medium read tests, where the tests had a larger payload and required more interaction with the locally available disks. Both L8s (with local storage) and D15_v2s offered a lower latency to match the operation rate given to the cluster.

When running variable-small-read tests under a certain latency, the operation rate for D15v2 reached nearly 3 times the number of ops/s for D13v2s with remote storage. Likewise L8s with local storage out performed L8s (with remote storage). L8s (with local storage) had twice the performance and half the latency of the L8s (with remote storage). 

In the medium read tests, instances with local storage outperformed instances with remote storage. Both L8s (with local storage) and D15s had a far better ops/s and latency result, even with a significantly higher operation rate. This makes a very convincing argument for local storage over remote storage when seeking optimal performance.

Looking at the writes on the other hand, D15s outperformed due to their large pool of CPU cores. While differences were less obvious in the small tests, results were more obvious in the medium tests. Further investigation will be conducted to determine why this is the case.

Based on the graph for variable medium testing, D15s outperformed in all categories. D15v2s have both a larger number of cores to outpace competition with heavy loads of writes and local storage offering faster disk intensive reads. This was additionally supported by strong performance in the mixed medium testing results.

L8s with local storage took second place, performing better than DS13v2s in the read and mixed tests. Whilst DS13v2 nodes slightly edged out L8s in writes for larger payloads. The mixed results showed a substantial difference in performance between the two with L8s taking a lead thanks to local storage providing faster disk intensive reads. 

Conclusion

Based on the results from this comparison, we find that local storage offers amazing performance results for disk intensive operations such as reads. D15v2 nodes, with their large number of cores to perform CPU intensive writes and local storage to help with disk intensive reads, offer top tier performance for any production environment.

Furthermore, L8s with local storage offer a great cost efficient solution at around ¾ of the price of D13v2s and offer a better price-performance gain notably in read operations. This is especially beneficial for a production environment which prioritizes reads over writes.

In order to assist customers in upgrading their Cassandra clusters from currently used VM types to D15v2s or L8 VM type nodes, Instaclustr technical operations team has built several tried and tested node replacement strategies to provide zero-downtime, non-disruptive migrations for our customers. Reach out to our support team if you are interested in these new instance types for your Cassandra cluster.

You can access pricing details through the Instaclustr Console when you log in, or contact our Support team.

The post Benchmarking Cassandra with Local Storage on Azure appeared first on Instaclustr.

Cassandra and Kubernetes: SIG Update and Survey

14 August 2020, 5:00 am by Apache Cassandra Blog

Five operators for Apache Cassandra have been created that have made it easier to run containerized Cassandra on Kubernetes. Recently the major contributors to these operators came together to discuss the creation of a community-based operator with the intent of making one that makes it easy to run C* on K8s. One of the project’s organizational goals is that the end result will eventually become part of the Apache Software Foundation or the Apache Cassandra project.

The community created a special interest group (SIG) to set goals for what the operator should do at different levels to find a path for creating a standard community-based operator. The Operator Framework suggests five maturity levels for operator capabilities starting from basic installation to auto-pilot.

(Source: OperatorFramework.io)

The five Cassandra Kubernetes operators all come from different backgrounds, so the first major goal is to develop a common understanding as to what an operator needs to do and at which level. This first step involves collaborating on a Custom Resource Definition (CRD) that will set the syntax / schema which will be used to create Cassandra clusters on Kubernetes. Once this is done, a software extension can be developed in a variety of languages including Go, Java, or using the Operator SDK in Helm or Ansible without making changes to Kubernetes.

We’re not starting from zero, as the creators of the five operators are actively participating in the SIG. Hopefully much of the decided upon CRD will have code fragments that can be leveraged from the other projects. The major operators out publicly today are those by Sky UK, Orange Telecom, Instaclustr, Elassandra, and DataStax (list sourced from the awesome-cassandra project):

• CassKop - Cassandra Kubernetes Operator - This Kubernetes operator by Orange automates Cassandra operations such as deploying a new rack aware cluster, adding/removing nodes, configuring the C and JVM parameters, upgrading JVM and C versions. Written in Go. This one was also one of the first ones out and is the only one that can support multiple Kubernetes clusters using Multi-CassKop
• Cassandra Operator - A Kubernetes operator by SkyUK that manages Cassandra clusters inside Kubernetes. Well designed and organized. This was among the first operators to be released.
• Instaclustr - Kubernetes Operator for Cassandra operator - The Cassandra operator by Instaclustr manages Cassandra clusters deployed to Kubernetes and automates tasks related to operating an Cassandra cluster.
• Cass Operator - DataStax’s Kubernetes Operator supports Apache Cassandra as well as DSE containers on Kubernetes. Cassandra configuration is managed directly in the CRD, and Cassandra nodes are managed via a RESTful management API.
• Elassandra Operator - The Elassandra Kubernetes Operator automates the deployment and management of Elassandra clusters deployed in multiple Kubernetes clusters.

If you’re interested in catching up on what the SIG has been talking about, you can watch the YouTube videos of the sessions and read up on the working documents:

As with any Kubernetes operator, the goal is to create a robot which takes the manual work of setting up complex configurations of containers in Kubernetes easier. An operator can also be seen as a translator between the logical concepts of the software and the concrete Kubernetes resources such as nodes, pods, services. Combined with controllers, operators can abstract out operations such that the human operators can focus on problems related to their industry or domain. As mentioned above, the different operator capability levels offer a roadmap to creating a robust operator for Cassandra users that is easy to use, set up, maintain and upgrade, and expand a cluster.

When a platform needs Cassandra, it’s probably exhausted the other potential datastores available because it needs high availability and fault tolerance, at high speeds, around the world. Kubernetes is a technology that can match well with Cassandra’s capabilities because it shares the features of being linearly scalable, vendor neutral, and cloud agnostic. There is a healthy debate about whether Cassandra belongs in Kubernetes — and whether databases belong in Kubernetes at all — because other orchestration tools are good enough, though the growing user base of Kubernetes in hobby and commercial realms suggests that we need to provide an operator that can keep up with the demand.

Most likely if someone is thinking about moving Cassandra workloads from public cloud, on-premises VMs, or even on-premises bare metal servers to either a public or private cloud hosted K8s, they’ll want to evaluate whether or not the existing architecture could run and be performant.

As part of the SIG, we’re also coming up with reference architectures on which to test the operator. Here are some of the common and most basic reference architectures that are likely candidates.

• Single Workload in Single Region
  • 1 DCs in 1 region, with 3 nodes (3 total)
  • DC expands to 6 (6 total)
  • DC contracts to 3 ( 3 total)

• Multi-Workload in Single Region
  • 2 DCs, both in the same region, with 3 nodes in each DC (6 total)
  • Both DCs expand to 6 each (12 total)
  • Both DCs contract to 3 each ( 6 total)
  • Add a third DC in the same region with 3 nodes (9 nodes)
  • Remove third DC

• Single Workload in Multi-Regions
  • 2 DCs, 1 in each region, with 3 nodes in each DC (6 total)
  • Both DCs expand to 6 each (12 total)
  • Both DCs contract to 3 each ( 6 total)
  • Add a third DC in a 3rd region with 3 nodes (9 total)
  • Remove third DC

Although each organization is different, these scenarios or combinations of these scenarios account for 80% of most pure Apache Cassandra use cases. The SIG would love to know more about Cassandra users’ use cases for Kubernetes. Please take this short survey, which will remain open through September 17, 2020.

Join the biweekly meetings to stay informed.

Dial C* for Operator - Cass Operator Meet Reaper

14 August 2020, 2:00 am by The Last Pickle

Reaper is a critical tool for managing Apache Cassandra. Kubernetes-based deployments of Cassandra are no exception to this. Automation is the name of the game with Kubernetes operators. It therefore makes sense that Cass Operator should have tight integration with Reaper. Fortunately, Cass Operator v1.3.0 introduced support for Reaper. This post will take a look at what that means in practice.

Note: If you want to try the examples in this post, install Cass Operator using the instructions in the project’s README.

Pods

Before we dive into the details, let’s take a moment to talk about Kubernetes pods. If you think a pod refers to a container, you are mostly right. A pod actually consists of one or more containers that are deployed together as a single unit. The containers are always scheduled together on the same Kubernetes worker node.

Containers within a pod share network resources and can communicate with each other over localhost. This lends itself very nicely to the proxy pattern. You will find plenty of great examples of the proxy pattern implemented in service meshes.

Containers within a pod also share storage resources. The same volume can be mounted within multiple containers in a pod. This facilitates the sidecar pattern, which is used extensively for logging, among other things.

The Cassandra Pod

Now we are going to look at the pods that are ultimately deployed by Cass Operator. I will refer to them as Cassandra pods since their primary purpose is running Cassandra.

Consinder the following CassandraDatacenter:

# example-cassdc.yaml

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: example
spec:
  clusterName: example
  serverType: cassandra
  serverVersion: 3.11.6
  managementApiAuth:
    insecure: {}
  size: 3
  allowMultipleNodesPerWorker: true
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: server-storage
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi

Create the CassandraDatacenter as follows:

$ kubectl apply -f example-cassdc.yaml

Note: This example as well as the later one specify serverVersion: 3.11.6 for the Cassandra version. Cassandra 3.11.7 was recently released, but Cass Operator does not yet support it. See this ticket for details.

Note: Remember to create the server-storage StorageClass.

It might take a few minutes for the Cassandra cluster to fully initialize. The cluster is ready when the Ready condition in the CassandraDatacenter status reports True, e.g.,

$ kubectl -n cass-operator get cassdc example -o yaml
...
status:
  cassandraOperatorProgress: Ready
  conditions:
  - lastTransitionTime: "2020-08-10T15:17:59Z"
    status: "False"
    type: ScalingUp
  - lastTransitionTime: "2020-08-10T15:17:59Z"
    status: "True"
    type: Initialized
  - lastTransitionTime: "2020-08-10T15:17:59Z"
    status: "True"
    type: Ready
...   

Three (3) pods are created and deployed, one per Cassandra node.

$ kubectl -n get pods -l cassandra.datastax.com/cluster=example
NAME                            READY   STATUS    RESTARTS   AGE
example-example-default-sts-0   2/2     Running   0          4h18m
example-example-default-sts-1   2/2     Running   0          4h18m
example-example-default-sts-2   2/2     Running   0          133m

Each row in the output has 2/2 in the Ready column. What exactly does that mean? It means that there are two application containers in the pod, and both are ready. Here is a diagram showing the containers deployed in a single Cassandra pod:

This shows three containers, the first of which labled as an init container. Init containers have to run to successful completion before any of the main application containers are started.

We can use a JSONPath query with kubectl to verify the names of the application containers:

$ kubectl get pod example-example-default-sts-0 -o jsonpath={.spec.containers[*].name} | tr -s '[[:space:]]' '\n'
cassandra
server-system-logger

The cassandra container runs the Management API for Apache Cassandra, which manages the lifecycle of the Cassandra instance.

server-system-logger is a logging sidecar container that exposes Cassandra’s system.log. We can conveniently access Cassandra’s system.log using the kubectl log command as follows:

$ kubectl logs example-example-default-sts-0 -c server-system-logger

The Cassandra Pod with Reaper

Here is another CassandraDatacenter specifying that Reaper should be deployed:

# example-reaper-cassdc.yaml

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: example-reaper
spec:
  clusterName: example-reaper
  serverType: cassandra
  serverVersion: 3.11.6
  managementApiAuth:
    insecure: {}
  size: 3
  reaper:
    enabled: true
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: server-storage
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi

The only difference from the first CassandraDatacenter are these two lines:

  reaper:
    enabled: true

This informs Cass Operator to deploy Reaper in sidecar mode. One of the main benefits of deploying Reaper in sidecar mode is security. Reaper only needs local JMX access to perform repairs. There is no need for remote JMX access or JMX authentication to be enabled.

Once again three pods are created and deployed, one per Cassandra node.

$ kubectl -n cass-operator get pods -l cassandra.datastax.com/cluster=example-reaper
NAME                                          READY   STATUS    RESTARTS   AGE
example-reaper-example-reaper-default-sts-0   3/3     Running   1          6m5s
example-reaper-example-reaper-default-sts-1   3/3     Running   1          6m5s
example-reaper-example-reaper-default-sts-2   3/3     Running   1          6m4s

Now, each pod reports 3/3 in the Ready column. Here is another diagram to illustrate which containers are deployed in a single Cassandra pod:

Now we have the reaper application container in addition to the cassandra and server-system-logger containers.

Reaper Schema Initialization

In sidecar mode, Reaper automatically uses the Cassandra cluster as its storage backend. Running Reaper with a Cassandra backend requires first creating the reaper_db keyspace before deploying Reaper. Cass Operator takes care of this for us with a Kubernetes Job. The following kubectl get jobs command lists the Job that gets deployed:

$ kubectl get jobs -l cassandra.datastax.com/cluster=example-reaper
NAME                                COMPLETIONS   DURATION   AGE
example-reaper-reaper-init-schema   1/1           12s        45m

Cass Operator deploys a Job whose name is of the form <cassandradatacenter-name>-init-schema. The Job runs a small Python script named init_keyspace.py.

The output from kubectl -n cass-operator get pods -l cassandra.datastax.com/cluster=example-reaper showed one restart for each pod. Those restarts were for the reaper containers. This happened because the reaper_db keyspace had not yet been initialized.

We can see this in the log output:

$ kubectl -n cass-operator logs example-reaper-example-reaper-default-sts-1 -c reaper | grep ERROR -A 1
ERROR  [2020-08-10 20:28:19,965] [main] i.c.ReaperApplication - Storage is not ready yet, trying again to connect shortly...
com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'reaper_db' does not exist

The restarts are perfectly fine as there are no ordering guarantees with the start of application containers in a pod.

Accessing the Reaper UI

Reaper provides a rich UI that allows you to do several things including:

• Monitor Cassandra clusters
• Schedule repairs
• Manager and monitor repairs

Cass Operator deploys a Service to expose the UI. Here are the Services that Cass Operator deploys.

$ kubectl -n cass-operator get svc
NAME                                             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
cass-operator-metrics                            ClusterIP   10.0.37.211   <none>        8383/TCP,8686/TCP   8h
cassandradatacenter-webhook-service              ClusterIP   10.0.33.233   <none>        443/TCP             8h
example-reaper-example-reaper-all-pods-service   ClusterIP   None          <none>        <none>              14m
example-reaper-example-reaper-service            ClusterIP   None          <none>        9042/TCP,8080/TCP   14m
example-reaper-reaper-service                    ClusterIP   10.0.47.8     <none>        7080/TCP            10m
example-reaper-seed-service                      ClusterIP   None          <none>        <none>              14m

The Service we are interested in has a name of the form <clusterName>-reaper-service which is example-reaper-reaper-service. It exposes the port 7080.

One of the easiest ways to access the UI is with port forwarding.

$ kubectl -n cass-operator port-forward svc/example-reaper-reaper-service 7080:7080
Forwarding from 127.0.0.1:7080 -> 7080
Forwarding from [::1]:7080 -> 7080
Handling connection for 7080

Here is a screenshot of the UI:

Our example-reaper cluster shows up in the cluster list because it gets automatically registered when Reaper runs in sidecar mode.

Accessing the Reaper REST API

Reaper also provides a REST API in addition to the UI for managing clusters and repair schedules. It listens for requests on the ui port which means it is accessible as well through example-reaper-reaper-service. Here is an example of listing registered clusters via curl:

$ curl -H "Content-Type: application/json" http://localhost:7080/cluster
["example-reaper"]

Wrap Up

Reaper is an essential tool for managing Cassandra. Future releases of Cass Operator may make some settings such as resource requirements (i.e., CPU, memory) and authentication/authorization configurable. It might also support deploying Reaper with a different topology. For example, instead of using sidecar mode, Cass Operator might provide the option to deploy a single Reaper instance. This integration is a big improvement in making it easier to run and manage Cassandra in Kubernetes.

Apache Cassandra Benchmarking: 4.0 Brings the Heat with New Garbage Collectors ZGC and Shenandoah

28 July 2020, 2:08 pm by Datastax Technical HowTo's

[Webcast] 10 Clever Astra Demos: Build Cloud-Native Apps with the Astra Cassandra DBaaS

27 July 2020, 4:24 pm by Datastax Technical HowTo's