Planet Cassandra

Better “Goodput” Performance through C++ Exception Handling

14 May 2024, 12:01 pm by ScyllaDB

Some very strange error reporting performance issues we uncovered while working on per-partition query rate limiting — and how we addressed it through C++ exception handling In Retaining Goodput with Query Rate Limiting, I introduced why and how ScyllaDB implemented per-partition rate limiting to maintain stable performance under stress by rejecting excess requests to maintain stable goodput. The implementation works by estimating request rates for each partition and making decisions based on those estimates. And benchmarks confirmed how rate limiting restored goodput, even under stressful conditions, by rejecting problematic requests. However, there was actually an interesting behind-the-scenes twist in that story. After we first coded the solution, we discovered that the performance wasn’t as good as expected. In fact, we were achieving the opposite effect of what we expected. Rejected operations consumed more CPU than the successful ones. This was very strange because tracking per-partition hit counts shouldn’t have been compute-intensive. It turned out that the problem was related to how ScyllaDB reports failures: namely, via C++ exceptions. In this article, I will share more details about the error reporting performance issues that we uncovered while working on per-partition query rate limiting and explain how we addressed them. On C++ (and Seastar) Exceptions Exceptions are notorious in the C++ community. There are many problems with their design, of course. But the most important and relevant one here is the unpredictable and potentially degraded performance that occurs when an exception is actually thrown. Some C++ projects disable exceptions altogether for those reasons. However, it’s hard to avoid exceptions because they’re thrown by the standard library and because they interact with other language features. Seastar, an open-source C++ framework for I/O intensive asynchronous computing, embraces exceptions and provides facilities for handling them correctly. Since ScyllaDB is built on Seastar, the ScyllaDB engineering team is rather accustomed to reporting failures via exceptions. They work fine, provided that errors aren’t very frequent. Under overload, it’s a different story though. We noticed that throwing exceptions in large volumes can introduce a performance bottleneck. This problem affects both existing errors such as timeouts and the new “rate limit exception” that we tried to introduce. This wasn’t the first time that we encountered performance issues with exceptions. In libstdc++, the standard library for GCC, throwing an exception involves acquiring a global mutex. That mutex protects some information important to the runtime that can be modified when a dynamic library is being loaded or unloaded. ScyllaDB doesn’t use dynamic libraries, so we were able to disable the mutex with some clever workarounds. As an unfortunate side effect, we disabled some caching functionality that speeds up further exception throws. However, avoiding scalability bottlenecks was more important to us. Exceptions are usually propagated up the call stack until a try..catch block is encountered. However, in programs with non-standard control flow, it sometimes makes sense to capture the exception and rethrow it elsewhere. For this purpose, std::exception_ptr can be used. Our Seastar framework is a good example of code that uses it. Seastar allows running concurrent, asynchronous tasks that can wait for the results of each other via objects called futures. If a task results in an exception, it is stored in a future as std::exception_ptr and can be later inspected by a different task.

future<> do_thing() { return
really_do_thing().finally([] { std::cout << "Did the
thing\n"; }); } future<> really_do_thing() { if (fail_flag) {
return make_exception_future<>( std::runtime_error("oh
no!")); } else { return make_ready_future<>(); } }

Seastar nicely reduces the time spent in the exception handling runtime. In most cases, Seastar is not interested in the exception itself; it only cares about whether a given future contains an exception. Because of that, the common control flow primitives such as then, finally etc. do not have to rethrow and inspect the exception. They only check whether an exception is present. Additionally, it’s possible to construct an “exceptional future” directly, without throwing the exception. Unfortunately, the standard library doesn’t make it possible to inspect an std::exception_ptr without rethrowing it. Because each exception must be inspected and handled appropriately at some point (sometimes more than once), it’s impossible for us to eliminate throws if we want to use exceptions and have portable code. We had to provide a cheaper way to return timeouts and “rate limit exceptions.” We tried two approaches. Approach 1: Avoid The Exceptions The first approach we explored was quite heavy-handed. It involved plowing through the code and changing all the important functions to return a boost::result. That’s a type from the boost library that holds either a successful result or an error. We also introduced a custom container for holding exceptions that allows inspecting the exception inside it without having to throw it.

future<result<>> do_thing() { return
really_do_thing().then( [] (result<> res) ->
result<> { if (res) { // handle success } else { // handle
failure } } ); }

This approach worked, but had numerous drawbacks. First, it required a lot of work to adapt the existing code: return types had to be changed, and the boost::result returned by the functions had to be explicitly checked to see whether they held an exception or not. Moreover, those checks introduced a slight overhead on the happy path. We applied this approach to the coordinator logic. Approach 2: Implement the Missing Parts Another approach: create our own implementation for inspecting the value of std::exception_ptr. We encountered a proposal to extend the std::exception_ptr’s interface that included adding the possibility to inspect its value without rethrowing it. The proposal includes an experimental implementation that uses standard-library specific and ABI-specific constructs to implement it for some of the major compilers. Inspired by it, we implemented a small utility function that allows us to match on the exception’s type and inspect its value, and replaced a bunch of try..catch blocks with it.

std::exception_ptr ep =
get_exception(); if (auto* ex = try_catch(ep)) { // ... } else if
(auto* ex = try_catch(ep)) { // ... } else { // ... }

This is how we eliminated throws in the most important parts of the replica logic. It was much less work than the previous approach since we only had to get rid of try-catch blocks and make sure that exceptions were propagated using the existing, non-throwing primitives. While the solution is not portable, it works for us since we use and support only a single toolchain at a time to build ScyllaDB. However, it can be easy to accidentally reintroduce performance problems if you’re not careful and forget to use the non-throwing primitives, including the new try_catch. In contrast, the first approach does not have this problem. Results: Better Exception Handling, Better Throughput (and Goodput) To measure the results of these optimizations on the exception handling path, we ran a benchmark. We set up a cluster of 3 i3.4xlarge AWS nodes with ScyllaDB 5.1.0-rc1. We pre-populated it with a small data set that fits in memory. Then, we ran a write workload to a single partition. The data included in each request is very small, so the workload was CPU bound, not I/O bound. Each request had a small server-side timeout of 10ms. The benchmarking application gradually increases its write rate from 50k writes per second to 100k. The chart shows what happens when ScyllaDB is no longer able to handle the throughput. There are two runs of the same benchmark, superimposed on the same charts – a yellow line for ScyllaDB 5.0.3 which handles timeouts in the old, slow way, and a green line for ScyllaDB 5.1.0-rc1 which has exception handling improvements. The older version of ScyllaDB became overwhelmed somewhere around the 60k writes per second mark. We were pushing the shard to its limit, and some requests inevitably failed because we set a short timeout. Since processing timed out requests is more costly than processing a regular request, the shard very quickly entered a state where all requests that it processed resulted in timeouts. The new version, which has better exception handling performance, was able to sustain a larger throughput. Some timeouts occurred, but they did not overwhelm the shard completely. Timeouts don’t start to be more prominent until the 100k requests per second mark. Summary We discovered unexpected performance issues while implementing a special case of ScyllaDB’s per-partition query rate limiting. Despite Seastar’s support for handling exceptions efficiently, throwing exceptions in large volumes during overload situations became a bottleneck, affecting both existing errors like timeouts and new exceptions introduced for rate limiting. We solved the issue by using two different approaches in different parts of the code: Change the code to pass information about timeouts by returning boost::result instead of throwing exceptions. Implement a utility function that allows catching exceptions cheaply without throwing them. Benchmarks with ScyllaDB versions 5.0.3 and 5.1.0-rc1 demonstrated significant improvements in handling throughput with better exception handling. While the older version struggled and became overwhelmed around 60k writes per second, the newer version sustained larger throughput levels with fewer timeouts, even at 100k requests per second. Ultimately, these optimizations not only improved exception handling performance but also contributed to better throughput and “goodput” under high load scenarios.

ScyllaDB University Updates, High Availability Lab & NoSQL Training Events

9 May 2024, 12:21 pm by ScyllaDB

There are many paths to adopting and mastering ScyllaDB, but the vast majority all share a visit to ScyllaDB University. I want to update the ScyllaDB community about recent changes to ScyllaDB University, introduce the High Availability lab, and share some future events and plans. What is ScyllaDB University? For those new to ScyllaDB University, it’s an online learning and training center for ScyllaDB. The material is self-paced and includes theory, quizzes, and hands-on labs that you can run yourself. Access is free; to get started, just create an account. Start Learning ScyllaDB University has six courses and covers most ScyllaDB-related material. As the product evolves, the training material is updated and enhanced. In addition to the self-paced, on-demand material available on ScyllaDB University, we host live training events. These virtual and highly interactive events offer special opportunities for users to learn from the leading ScyllaDB engineers and experts, ask questions, and connect with their peers. The last ScyllaDB University LIVE event took place in March. It had two tracks: an Essentials track covering basic content, as well as an Advanced track covering the latest features and more complex use cases. The next event in the US/EMEA time zones will be on June 18th. Join us at ScyllaDB University LIVE Also, if you’re in India, we’re hosting a special event for you! On May 29, 9:30am-11:30am IST, join us for an interactive workshop where we’ll go hands-on to build and interact with high-performance apps using ScyllaDB. This is a great way to discover the NoSQL strategies used by top teams and apply them in a guided, supportive environment. Register for ScyllaDB Labs India Example Lab: High Availability Here’s a taste of the type of content you will find on ScyllaDB University. The High Availability lab is one of the most popular labs on the platform. It demonstrates what happens when we read and write data while simulating a situation where some of the replica nodes are unavailable, with different consistency levels. ScyllaDB, at its core, is designed to be highly available. With the right settings, it will remain up and available at all times. The database features a peer-to-peer architecture that is leaderless and uses the Gossip protocol for internode communication. This enhances resiliency and scalability. Upon startup and during topology changes such as node addition or removal, nodes automatically discover and communicate with each other, streaming data between them as needed. Multiple copies of the data (replicas) are kept. This is determined by setting a replication factor (RF) to ensure data is stored on multiple nodes, thereby allowing for data redundancy and high availability. Even if a node fails, the data is still available on other nodes, reducing downtime risks. Topology awareness is part of the design. ScyllaDB enhances its fault tolerance through rack and data center awareness, allowing distribution across multiple locations. It supports multi-datacenter replication with customizable replication factors for each site, ensuring data resiliency and availability even in case of significant failures of entire data centers. First, we’ll bring up a 3-node ScyllaDB Docker cluster. Set up a Docker Container with one node, called Node_X:

docker run --name Node_X -d
scylladb/scylla:5.2.0 --overprovisioned 1 --smp 1

Create two more nodes, Node_Y and Node_Z, and add them to the cluster of Node_X. The command “$(docker inspect –format='{{ .NetworkSettings.IPAddress }}’ Node_X)” translates to the IP address of Node-X:

docker run --name Node_Y -d
scylladb/scylla:5.2.0 --seeds="$(docker inspect --format='{{
.NetworkSettings.IPAddress }}' Node_X)" --overprovisioned 1 --smp
1

docker run --name Node_Z -d
scylladb/scylla:5.2.0 --seeds="$(docker inspect --format='{{
.NetworkSettings.IPAddress }}' Node_X)" --overprovisioned 1 --smp
1

Wait a minute or so and check the node status: docker exec -it Node_Z nodetool status You’ll see that eventually, all the nodes have UN for status. U means up, and N means normal. Read more about Nodetool Status Here. Once the nodes are up, and the cluster is set, we can use the CQL shell to create a table. Run a CQL shell: docker exec -it Node_Z cqlsh Create a keyspace called “mykeyspace”, with a Replication Factor of three:

CREATE KEYSPACE mykeyspace WITH REPLICATION = { 'class' :
'NetworkTopologyStrategy', 'replication_factor' : 3};

Next, create a table with three columns: user id, first name, and last name, and insert some data:

use
mykeyspace;

CREATE TABLE users ( user_id int,
fname text, lname text, PRIMARY KEY((user_id)));

Insert into the newly created table two rows:

insert into
users(user_id, fname, lname) values (1, 'rick', 'sanchez');
insert into users(user_id, fname, lname) values (4, 'rust',
'cohle');

Read the table contents:

select *
from users;

Now we will see what happens when we try to read and write data to our table, with varying Consistency Levels and when some of the nodes are down. Let’s make sure all our nodes are still up:

docker exec -it Node_Z nodetool
status

Now, set the Consistency Level to QUORUM and perform a write: docker exec -it Node_Z cqlsh use mykeyspace; CONSISTENCY QUORUM

insert into users (user_id, fname, lname) values (7,
'eric', 'cartman');

Read the data to see if the insert was successful: select * from users; The read and write operations were successful. What do you think would happen if we did the same thing with Consistency Level ALL? CONSISTENCY ALL

insert into users
(user_id, fname, lname) values (8, 'lorne', 'malvo');

select * from users; The operations were successful again. CL=ALL means that we have to read/write to the number of nodes according to the Replication Factor, 3 in our case. Since all nodes are up, this works. Next, we’ll take one node down and check read and write operations with a Consistency Level of Quorum and ALL. Take down Node_Y and check the status (it might take some time until the node is actually down): exit docker stop Node_Y

docker exec -it Node_Z
nodetool status

Now, set the Consistency Level to QUORUM and perform a write: docker exec -it Node_Z cqlsh CONSISTENCY QUORUM use mykeyspace;

insert into users (user_id, fname, lname) values (9, 'avon',
'barksdale');

Read the data to see if the insert was successful: select * from users; With CL = QUORUM, the read and write were successful. What will happen with CL = ALL? CONSISTENCY ALL

insert into users (user_id,
fname, lname) values (10, 'vm', 'varga');

select *
from users;

Both read and write fail. CL = ALL requires that we read/write to three nodes (based on RF = 3), but only two nodes are up. What happens if another node is down? Take down Node_Z and check the status: exit docker stop Node_Z docker exec -it Node_X nodetool status Now, set the Consistency Level to QUORUM and perform a read and a write: docker exec -it Node_X cqlsh

CONSISTENCY
QUORUM

use mykeyspace;

insert into users
(user_id, fname, lname) values (11, 'morty', 'smith');

select * from users; With CL = QUORUM, the read and write fail. Since RF=3, QUORUM means at least two nodes need to be up. In our case, just one is. What will happen with CL = ONE? CONSISTENCY ONE

insert into users (user_id,
fname, lname) values (12, 'marlo', 'stanfield');

select * from users; With CL = QUORUM, the read and write fail. Since RF=3, QUORUM means at least two nodes need to be up. In our case, just one is. What do you think will happen with CL = ONE? Check out the full High Availability lesson and lab on ScyllaDB University, with expected outputs, full explanation of the theory, and an option for seamlessly running the lab within a browser, regardless of the platform you’re using. Next Steps and Upcoming NoSQL Events A good starting point on ScyllaDB University is ScyllaDB Essentials – Overview of ScyllaDB and NoSQL Basics course. There, you can find more hands-on labs like the one above, and learn about distributed databases, NoSQL, and ScyllaDB. ScyllaDB is backed by a vibrant community of users and developers. If you have any questions about this blog post or about a specific lesson, or want to discuss it, you can write on the ScyllaDB community forum. Say hello here. The next live training events are taking place in: An Asia-friendly time zone event will take place on the 29th of May US/Europe, Middle East, and Africa time zones on the 18th of June. Stay tuned for more details!

ScyllaDB Internal Engineering Summit in Cyprus

3 May 2024, 11:20 am by ScyllaDB

ScyllaDB is a remote-first company with associates working in distributed locations around the globe. We’re good at it, and we love it. Still, each year we look forward to in-person company summits to spend time together, share knowledge, and have fun. When the time comes, we say goodbye to our kids and cats and fly from all corners of the world to convene in one place. This April, we met in Limassol, a historical city on the southern coast of Cyprus. Many People, One Goal Over 120 attendees gathered at this year’s ScyllaDB Engineering Summit. With the company rapidly growing, it was a great opportunity for dozens of new associates to put faces to names, while ScyllaDB veterans could welcome the new people on board and reunite with their long-time workmates. Peyia, Paphos area with the Edro III shipwreck in the background In our everyday work life, we work in different teams, we have different roles and responsibilities. The event gathered software engineers, QA engineers, product managers, technical support engineers, developer advocates, field engineers, and more. Some of us write the code, and others make sure it’s defect-free. Some take care of the ScyllaDB roadmap, while others provide assistance to ScyllaDB customers. If we were a Git project, we would live on different branches – but there, at the Summit, they would be merged into one. Sharing knowledge, brainstorming ideas, or discussing the things we’d like to improve, we had a profound sense of a common goal – to make a cool product we’re happy with and make the users happy, too. Who-is-who team building activity On the Road to Success The goal is clear, but how do we get there? That question was the central theme of the event. The summit agenda was filled with technical sessions, each with ample time for Q&A. They were a great platform for engaging in discussions and brainstorming, as well as an opportunity to get an insight into other teams’ areas of work. The speakers covered a wide range of topics, including: ScyllaDB roadmap Security features Data distribution with tablets Strong data consistency with Raft Observability and metrics Improving customer experience Streamlining the management of our GitHub projects And many more Dor Laor kicking off the Summit Let’s Get The Party Started The sessions were inspiring and fruitful, but hey, we were in Cyprus! And what is Cyprus famous for if not for its excellent food and wine, beautiful beaches, and holiday vibes? We could enjoy all those things at company dinners in beautifully located Cypriot restaurants – and at amazing beaches or quiet coves. One of the many amazing beachside meals Then came the party. Live music and a DJ putting on all-time hit songs got the crowd going. Rumor has it that engineers don’t know how to party. Please don’t believe it 🙂 Party time! Off-Road Adventure Mid-week, we took a break to recharge the batteries and went on an off-road jeep safari. Fifteen 4x4s took us on a scenic tour along the sea-coast of Paphos, where we could admire beautiful views. Setting off on the jeep safari adventure While enjoying a bumpy ride, we learned a lot about Cyprus’ history and culture from the jeep drivers, who willingly answered all our questions. From time to time, we stopped to see masterpieces of nature, such as a stunning gorge, waterfalls, and the Blue Lagoon, our favorite, where we swam in crystal-clear water. A view over the lagoons, Akamas Peninsula Together as One It was an eventful Summit filled with many memorable moments. It inspired us, fed our spirits, and gave us some food for thought. But most importantly, it brought us together and let us enjoy a shared sense of purpose. When the Summit was coming to a close, we could genuinely feel that we were one team. When we all got safely home, our CEO emailed us to say, “All of our previous summits were great, but this felt even better. The level of teamwork is outstanding, and many companies can just envy what we have here.” I couldn’t agree more. Want to Join our Team? ScyllaDB has a number of open positions worldwide, including many remote positions across Engineering, QA & Automation, Product, and more. Check out the complete list of open positions on our Careers page. Join the ScyllaDB Team!

Node throughput Benchmarking for Apache Cassandra® on AWS, Azure and GCP

3 May 2024, 6:19 am by Apache Cassandra - Instaclustr

At NetApp, we are often asked for comparative benchmarking of our Instaclustr Managed Platform across various cloud providers and the different node sizes we offer.

For Apache Cassandra® on the Instaclustr Managed Platform, we have recently completed an extensive benchmarking exercise that will help our customers evaluate the node types to use and the differing performance between cloud providers.

Each cloud service provider is continually introducing new and improved nodes, which we carefully select and curate to provide our customers with a range of options to suit their specific workloads. The results of these benchmarks can assist you in evaluating and selecting the best node sizes that provide optimal performance at the lowest cost.

How Should Instaclustr Customers Use the Node Benchmark Data?

Instaclustr node benchmarks provide throughput for each node performed under the same test conditions, serving as valuable performance metrics for comparison.

As with any generic benchmarking results, for different data models or application workload, the performance may vary from the benchmark. Instaclustr node benchmarks do not account for specific configurations of the customer environment and should only be used to compare the relative performance of node sizes.

Instaclustr recommends customers complete testing and data modelling in their own environment across a selection of potential nodes.

Test Objective

Cassandra Node Benchmarking creates a Performance Metric value for each General Availability (GA) node type that is available for Apache Cassandra across AWS, GCP, and Azure AZ. Each node is benchmarked before being released to General Availability. The tests ensure that all nodes will deliver sufficient performance based on their configuration.

All Cassandra production nodes Generally Available on the Instaclustr platform are stress-tested to find the operational capacity of the nodes. This is accomplished by measuring throughput to find the highest ops rate the test cluster can achieve without performance degradation.

Methodology

We used the following configuration to conduct this benchmarking:

Cassandra-stress tool
QUORUM consistency level for data operations
3-node cluster using Cassandra 4.0.10 with each node in a separate rack within a single datacenter

On each node size, we run the following testing procedure:

Fill Cassandra with enough records to approximate 5x the system memory of the node using the “small” size writes. This is done to ensure that the entire data model is not sitting in memory, and to give a better representation of a production cluster.
Allow the cluster to finish all compactions. This is done to ensure that all clusters are given the same starting point for each test run and to make test runs comparable and verifiable.
Allow compactions to finish between each step.
Perform multiple stress tests on the cluster, increasing the number of operations per second for each test, until we overload it. The cluster will be considered overloaded if one of three conditions is met:
1. The latency of the operation is above the provided threshold. This tells us that the cluster cannot keep up with the current load and is unable to respond quickly.
2. If a node has more than 20 pending compactions a minute after the test completes. This tells us that the cluster is not keeping up with compactions, and this load is not sustainable in the long term.
3. The CPU load is above the provided threshold. This is the reason that we pass the number of cores into the Cassandra Stress Tool.
Using this definition of “overloaded”, the following tests are run to measure maximum throughput:
1. Perform a combination of reads and write operations in 30-minute test runs. The combination of writes, simple reads, and range reads is at a rate of 10:10:1, increasing threads until we reach a read median latency of 20ms indicating if a mixed workload throughput has changed.

Results

Following this approach, the tables below show the results we measured on the different providers and different node sizes we offer.

AWS
Node Type (Node Size Used)	Ram / Cores	Result (MixedOPS)
t4g.small (CAS-DEV-t4g.small-30)	2 GiB / 2	728*
t4g.medium (CAS-DEV-t4g.medium-80)	4 GiB / 2	3,582
r6g.medium (CAS-PRD-r6g.medium-120)	8 GiB / 1	1,849
m6g.large (CAS-PRD-m6g.large-120)	8 GiB / 2	2,144
r6g.large (CAS-PRD-r6g.large-250)	16 GiB / 2	3,117
r6g.xlarge (CAS-PRD-r6g.xlarge-400)	32 GiB / 4	16,653
r6g.2xlarge (CAS-PRD-r6g.2xlarge-800)	64 GiB / 8	36,494
r6g.4xlarge (CAS-PRD-r6g.4xlarge-1600)	128 GiB / 16	62,195
r7g.medium (CAS-PRD-r7g.medium-120)	8 GiB / 1	2,527
m7g.large (CAS-PRD-m7g.large-120)	8 GiB / 2	2,271
r7g.large (CAS-PRD-r7g.large-120)	16 GiB / 2	3,039
r7g.xlarge (CAS-PRD-r7g.xlarge-400)	32 GiB / 4	20,897
r7g.2xlarge (CAS-PRD-r7g.2xlarge-800)	64 GiB / 8	39,801
r7g.4xlarge (CAS-PRD-r7g.4xlarge-800)	128 GiB / 16	70,880
c5d.2xlarge (c5d.2xlarge-v2)	16 GiB / 8	26,494
c6gd.2xlarge (CAS-PRD-c6gd.2xlarge-441)	16 GiB / 8	27,066
is4gen.xlarge (CAS-PRD-is4gen.xlarge-3492)	24 GiB / 4	19,066
is4gen.2xlarge (CAS-PRD-is4gen.2xlarge-6984)	48 GiB / 8	34,437
im4gn.2xlarge (CAS-PRD-im4gn.2xlarge-3492)	32 GiB / 8	31,090
im4gn.4xlarge (CAS-PRD-im4gn.4xlarge-6984)	64 GiB / 16	59,410
i3en.xlarge (i3en.xlarge)	32 GiB / 4	19,895
i3en.2xlarge (CAS-PRD-i3en.2xlarge-4656)	64 GiB / 8	40,796
i3.2xlarge (i3.2xlarge-v2)	61 GiB / 8	24,184
i3.4xlarge (CAS-PRD-i3.4xlarge-3538)	122 GiB / 16	42,234

* Nodes overloaded with the stress test started dropping out

Azure
Node Type (Node Size Used)	Ram / Cores	Result (MixedOPS)
Standard_DS12_v2 (Standard_DS12_v2-512-an)	28 GiB / 4	23,878
Standard_DS2_v2 (Standard_DS2_v2-256-an)	7 GiB / 2	1,535
L8s_v2 (L8s_v2-an)	64 GiB / 8	24,188
Standard_L8s_v3 (CAS-PRD-Standard_L8s_v3-1788)	64 GiB / 8	33,990
Standard_DS13_v2 (Standard_DS13_v2-2046-an)	56 GiB / 8	37,908
D15_v2 (D15_v2-an)	140 GiB / 20	68,226
Standard_L16s_v2 (CAS-PRD-Standard_L16s_v2-3576-an)	128 GiB / 16	33,969

GCP
Node Type (Node Size Used)	Ram / Cores	Result (MixedOPS)
n1-standard-1 (CAS-DEV-n1-standard-1-5)	3.75 GiB / 1	No data*
n1-standard-2 (CAS-DEV-n1-standard-2-80)	7.5 GiB / 2	3,159
t2d-standard-2 (CAS-PRD-t2d-standard-2-80)	8 GiB / 2	6,045
n2-standard-2 (CAS-PRD-n2-standard-2-120)	8 GiB / 2	4,199
n2-highmem-2 (CAS-PRD-n2-highmem-2-250)	16 GiB / 2	5,520
n2-highmem-4 (CAS-PRD-n2-highmem-4-400)	32 GiB / 8	19,893
n2-standard-8 (cassandra-production-n2-standard-8-375)	32 GiB / 8	37,437
n2-highmem-8 (CAS-PRD-n2-highmem-8-800)	64 GiB / 8	38,139
n2-highmem-16 (CAS-PRD-n2-highmem-16-800)	128 GiB / 16	73,918

* Cannot be tested due to the fill data requirement being greater than the available disk space

Conclusion: What We Discovered

In general, we see that clusters with more processing power (CPUs and RAM) produce higher throughput as expected. Some of the key takeaways include:

When it comes to the price-to-performance ratio of Cassandra, there is a sweet spot around the XL/2XL node size (eg r6g.xlarge or r6g.2xlarge).
- Moving from L to XL nodes, doubles performance, and moving from XL to 2XL, almost always doubles performance again.
- However, moving from 2XL to 4XL, the increase in performance is less than double. This is expected as you move from a memory-constrained state to a state with more resources than can be utilized. These findings are specific to Cassandra.
Different node families are tailored for workloads, significantly impacting their performance. Hence, nodes should be selected based on workload requirements and use cases.

Overall, these benchmarks provide an indication of potential throughput for different environments and instance sizes. Performance can vary significantly depending on individual use cases, and we always recommend benchmarking with your own specific use case prior to production deployment.

The easiest way to see all the new node offerings for your provider is to log into our Console. We ensure that you have access to the latest instance types by supporting different node types from each cloud provider on our platform to help you get high performance at reduced costs.

Do you know that R8g instances from AWS deliver up to 30% better performance than Graviton3-based R7g instances? Keep an eye out, as we will make these new instances available on our platform as soon as they are released for general availability.

If you are interested in migrating your existing Cassandra clusters to the Instaclustr Managed Platform, our highly experienced Technical Operations team can provide all the assistance you need. We have built several tried-and-tested node replacement strategies to provide zero-downtime, non-disruptive migrations. Read our Advanced Node Replacement blog for more details on one such strategy.

If you want to know more about this benchmarking or need clarification on when to use which instance type for Cassandra, reach out to our Support team (if you are an existing customer) or contact our Sales team. Our support team can assist you in resizing your existing clusters. Alternatively, you can use our in-place data-center resizing feature to do it on your own.

The post Node throughput Benchmarking for Apache Cassandra® on AWS, Azure and GCP appeared first on Instaclustr.

How to Build a Real-Time CDC pipeline with ScyllaDB & Redpanda

30 April 2024, 12:01 pm by ScyllaDB

A step-by-step tutorial to capturing changed data for high-throughput, low-latency applications, using Redpanda and ScyllaDB This blog is a guest post by Kovid Rathee. It was first published on the Redpanda blog. CDC is a methodology for capturing change events from a source database. Once captured, the change events can be used for many downstream use cases, such as triggering events based on specific conditions, auditing, and analytics. ScyllaDB is a distributed, wide-column NoSQL database known for its ultra-fast performance as it’s written in C++ to make the best use of the low-level Linux primitives, which helps attain much better I/O. It’s also an excellent alternative to DynamoDB as well as a popular alternative to Apache Cassandra. ScyllaDB’s modern approach to NoSQL and its architecture supporting data-intensive and low-latency applications make it an excellent choice for many applications. ScyllaDB shines when you need it to respond and scale super fast. Redpanda is a source available (BSL), Apache Kafka®-compatible, streaming data platform designed from the ground up to be lighter, faster, and simpler to operate. It employs a single binary architecture, free from ZooKeeper™ and JVMs, with a built-in Schema Registry and HTTP Proxy. Since both ScyllaDB and Redpanda are purpose-built for data-intensive applications, it makes perfect sense to connect the two. This post will take you through setting up a CDC integration between ScyllaDB and Redpanda, using a Debezium CDC connector that’s compatible with Kafka Connect. Let’s dig in. Note: Want an in-depth look at using Rust + Redpanda + ScyllaDB to build high-performance systems for real-time data streaming? Join our interactive Masterclass on May 15. You will learn how to: Understand core concepts for working with event-driven architectures Build a high performance streaming app with Rust Avoid common mistakes that surface in real-world systems at scale Evaluate the pros, cons, and best use cases for different event streaming options How to build a real-time CDC pipeline between ScyllaDB and Redpanda This tutorial will first guide you through the process of setting up ScyllaDB and Redpanda. You’ll create a standalone Kafka Connect cluster and incorporate the ScyllaDB CDC Connector JAR files as plugins. This will allow you to establish a connection between ScyllaDB and Kafka Connect. This tutorial uses ScyllaDB’s CDC quickstart guide, which takes an arbitrary orders table for an e-commerce business and streams data from that to a sink using the Kafka Connect-compatible Debezium connector. The following image depicts the simplified architecture of the setup: Connecting ScyllaDB and Redpanda using Kafka Connect The orders table receives new orders and updates on previous orders. In this example, you’ll insert a few simple orders with an order_id, a customer_id, and a product. You’ll first insert a few records in the orders table and then perform a change on one of the records. All the data, including new records, changes, and deletes, will be available as change events on the Redpanda topic you’ve tied to the ScyllaDB orders table. Prerequisites To complete this tutorial, you’ll need the following: Docker Engine and Docker Compose A Redpanda instance on Docker A ScyllaDB instance running on Docker ScyllaDB Kafka Connect drivers The jq CLI tool The rpk CLI tool

Note: The operating system used in this tutorial is macOS.

1. Run and configure ScyllaDB After installing and starting Docker Engine on your machine, execute the following Docker command to get ScyllaDB up and running:

docker run --rm -ti \ -p
127.0.0.1:9042:9042 scylladb/scylla \ --smp 1 --listen-address
0.0.0.0 \ --broadcast-rpc-address 127.0.0.1

This command will spin up a container with ScyllaDB, accessible on 127.0.0.1 on port 9042. To check ScyllaDB’s status, run the following command that uses the ScyllaDB nodetool utility:

docker exec -it 225a2369a71f nodetool
status

This command should give you an output that looks something like the following:

Datacenter: datacenter1
======================= Status=Up/Down |/
State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
Host ID Rack UN 0.0.0.0 256 KB 256 ?
2597950d-9cc6-47eb-b3d6-a54076860321 rack1

The UN at the beginning of the table output means Up and Normal. These two represent the cluster’s status and state, respectively. 2. Configure CDC on ScyllaDB To set up CDC on ScyllaDB, you first need to log in to the cluster using the cqlsh CLI. You can do that using the following command: docker exec -it 225a2369a71f cqlsh If you’re able to log in successfully, you’ll see the following message:

Connected to at 0.0.0.0:9042. [cqlsh 5.0.1 |
Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4] Use HELP for
help. cqlsh>

Keyspaces are high-level containers for all the data in ScyllaDB. A keyspace in ScyllaDB is conceptually similar to a schema or database in MySQL. Use the following command to create a new keyspace called quickstart_keyspace with SimpleStrategy replication and a replication_factor of 1:

CREATE KEYSPACE
quickstart_keyspace WITH REPLICATION = {'class': 'SimpleStrategy',
'replication_factor': 1};

Now, use this keyspace for further CQL commands: USE quickstart_keyspace; Please note that the SimpleStrategy replication class is not recommended for production use. Instead, you should use the NetworkTopologyStrategy replication class. You can learn more about replication methodologies from ScyllaDB University. 3. Create a table Use the following CQL statement to create the orders table described at the beginning of the tutorial:

CREATE TABLE orders( customer_id
int, order_id int, product text, PRIMARY KEY(customer_id,
order_id)) WITH cdc = {'enabled': true};

This statement creates the orders table with a composite primary key consisting of customer_id and order_id. The orders table will be CDC-enabled. ScyllaDB stores table data in a base table, but when you enable CDC on that table, an additional table that captures all the changed data is created as well. That table is called a log table. 4. Insert a few initial records Populate a few records for testing purposes:

INSERT INTO
quickstart_keyspace.orders(customer_id, order_id, product) VALUES
(1, 1, 'pizza'); INSERT INTO
quickstart_keyspace.orders(customer_id, order_id, product) VALUES
(1, 2, 'cookies'); INSERT INTO
quickstart_keyspace.orders(customer_id, order_id, product) VALUES
(1, 3, 'tea');

During the course of the tutorial, you’ll insert three more records and perform an update on one record, totaling seven events in total, four of which are change events after the initial setup. 5. Set up Redpanda Use the docker-compose.yaml file in the Redpanda Docker quickstart tutorial to run the following command: docker compose up -d A Redpanda cluster and a Redpanda console will be up and running after a brief wait. You can check the status of the Redpanda cluster using the following command: docker exec -it redpanda-0 rpk cluster info The output of the command should look something like the following:

CLUSTER ======= redpanda.3fdc3646-9c9d-4eff-b5d6-854093a25b67

BROKERS ======= ID HOST PORT 0* redpanda-0 9092

Before setting up an integration between ScyllaDB and Redpanda, check if all the Docker containers you have spawned are running by using the following command:

docker ps --format "table
{{.Image}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}"

Look for the STATUS column in the table output:

IMAGE
NAMES STATUS PORTS scylladb/scylla gifted_hertz Up 8 hours 22/tcp,
7000-7001/tcp, 9160/tcp, 9180/tcp, 10000/tcp,
127.0.0.1:9042->9042/tcp
docker.redpanda.com/vectorized/console:v2.2.4 redpanda-console Up 9
hours 0.0.0.0:8080->8080/tcp
docker.redpanda.com/redpandadata/redpanda:v23.1.8 redpanda-0 Up 9
hours 8081-8082/tcp, 0.0.0.0:18081-18082->18081-18082/tcp,
9092/tcp, 0.0.0.0:19092->19092/tcp,
0.0.0.0:19644->9644/tcp

If these containers are up and running, you can start setting up the integration. 6. Set up Kafka Connect Download and install Kafka using the following sequence of commands:

# Extract the binaries wget
https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
&& tar -xzf kafka_2.13-3.4.0.tgz && cd
kafka_2.13-3.4.0

#start Kafka connect in standalone mode bin/connect-standalone.sh config/connect-standalone.properties The .properties files will be kept in the kafka_2.13-3.4.0/config folder in your download location. To use Kafka Connect, you must configure and validate two .properties files. You can name these files anything. In this tutorial, the two files are called connect-standalone.properties and connector.properties. The first file contains the properties for the standalone Kafka Connect instance. For this file, you will only change the default values of two variables: The default value for the bootstrap.servers variable is localhost:9092. As you’re using Redpanda and its broker is running on port 19092, you’ll replace the default value with localhost:19092. Find the plugin path directory for your Kafka installation and set the plugin.path variable to that path. In this case, it’s /usr/local/share/kafka/plugins. This is where your ScyllaDB CDC Connector JAR file will be copied. The comment-stripped version of the connect-standalone.properties file should look like the following:

bootstrap.servers=localhost:19092
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/usr/local/share/kafka/plugins

The second .properties file will contain settings specific to the ScyllaDB connector. You need to change the scylla.cluster.ip.addresses variable to 127.0.0.1:9042. The connector.properties file should then look like the following:

name =
QuickstartConnector connector.class =
com.scylladb.cdc.debezium.connector.ScyllaConnector key.converter =
org.apache.kafka.connect.json.JsonConverter value.converter =
org.apache.kafka.connect.json.JsonConverter
scylla.cluster.ip.addresses = 127.0.0.1:9042 scylla.name =
QuickstartConnectorNamespace scylla.table.names =
quickstart_keyspace.orders

Using both the .properties files, go to the Kafka installation directory and run the connect-standalone.sh script with the following command:

bin/connect-standalone.sh
config/connect-standalone.properties
config/connector.properties

When you created the orders table, you enabled CDC, which means that there’s a log table with all the records and changes. If the Kafka Connect setup is successful, you should now be able to consume these events using the rpk CLI tool. To do so, use the following command:

rpk topic consume --brokers
'localhost:19092'
QuickstartConnectorNamespace.quickstart_keyspace.orders | jq
.

The output should result in the following three records, as shown below:

{ "topic":
"QuickstartConnectorNamespace.quickstart_keyspace.orders", "key":
"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"int32\",\"optional\":true,\"field\":\"customer_id\"},{\"type\":\"int32\",\"optional\":true,\"field\":\"order_id\"}],\"optional\":false,\"name\":\"QuickstartConnectorNamespace.quickstart_keyspace.orders.Key\"},\"payload\":{\"customer_id\":1,\"order_id\":1}}",
"value": …output omitted… "timestamp": 1683357426891, "partition":
0, "offset": 0 } { "topic":
"QuickstartConnectorNamespace.quickstart_keyspace.orders", "key":
"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"int32\",\"optional\":true,\"field\":\"customer_id\"},{\"type\":\"int32\",\"optional\":true,\"field\":\"order_id\"}],\"optional\":false,\"name\":\"QuickstartConnectorNamespace.quickstart_keyspace.orders.Key\"},\"payload\":{\"customer_id\":1,\"order_id\":2}}",
"value": …output omitted… "timestamp": 1683357426898, "partition":
0, "offset": 1 } { "topic":
"QuickstartConnectorNamespace.quickstart_keyspace.orders", "key":
"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"int32\",\"optional\":true,\"field\":\"customer_id\"},{\"type\":\"int32\",\"optional\":true,\"field\":\"order_id\"}],\"optional\":false,\"name\":\"QuickstartConnectorNamespace.quickstart_keyspace.orders.Key\"},\"payload\":{\"customer_id\":1,\"order_id\":3}}",
"value": …output omitted… "timestamp": 1683357426898, "partition":
0, "offset": 2 }

If you get a similar output, you’ve successfully integrated ScyllaDB with Redpanda using Kafka Connect. Alternatively, you can go to the Redpanda console hosted on localhost:8080 and see if the topic corresponding to the ScyllaDB orders table is available: You can now test whether data changes to the orders table can trigger CDC. 7. Capture Change Data (CDC) from ScyllaDB To test CDC for new records, insert the following two records in the orders table using the cqlsh CLI:

INSERT INTO quickstart_keyspace.orders(customer_id, order_id,
product) VALUES (1, 4, 'chips'); INSERT INTO
quickstart_keyspace.orders(customer_id, order_id, product) VALUES
(1, 5, 'lollies');

If the insert is successful, the rpk topic consume command will give you the following additional records:

{ "topic":
"QuickstartConnectorNamespace.quickstart_keyspace.orders", "key":
"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"int32\",\"optional\":true,\"field\":\"customer_id\"},{\"type\":\"int32\",\"optional\":true,\"field\":\"order_id\"}],\"optional\":false,\"name\":\"QuickstartConnectorNamespace.quickstart_keyspace.orders.Key\"},\"payload\":{\"customer_id\":1,\"order_id\":4}}",
"value": …output omitted… "timestamp": 1683357768358, "partition":
0, "offset": 3 } { "topic":
"QuickstartConnectorNamespace.quickstart_keyspace.orders", "key":
"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"int32\",\"optional\":true,\"field\":\"customer_id\"},{\"type\":\"int32\",\"optional\":true,\"field\":\"order_id\"}],\"optional\":false,\"name\":\"QuickstartConnectorNamespace.quickstart_keyspace.orders.Key\"},\"payload\":{\"customer_id\":1,\"order_id\":5}}",
"value": …output omitted… "timestamp": 1683358068355, "partition":
0, "offset": 4 }

You’ll now insert one more record with the product value of pasta:

INSERT INTO
quickstart_keyspace.orders(customer_id, order_id, product) VALUES
(1, 5, 'pasta');

Later on, you’ll change this value with an UPDATE statement to spaghetti and trigger a CDC update event. The newly inserted record should be visible with your rpk topic consume command:

{ "topic":
"QuickstartConnectorNamespace.quickstart_keyspace.orders", "key":
"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"int32\",\"optional\":true,\"field\":\"customer_id\"},{\"type\":\"int32\",\"optional\":true,\"field\":\"order_id\"}],\"optional\":false,\"name\":\"QuickstartConnectorNamespace.quickstart_keyspace.orders.Key\"},\"payload\":{\"customer_id\":1,\"order_id\":6}}",
"value": …output omitted… "timestamp": 1683361158363, "partition":
0, "offset": 5 }

Now, execute the following UPDATE statement and see if a CDC update event is triggered:

UPDATE quickstart_keyspace.orders SET product =
'spaghetti' WHERE order_id = 6 and customer_id = 1;

After running this command, you’ll need to run the

rpk topic
consume

command to verify the latest addition to the QuickstartConnectorNamespace.quickstart_keyspace.orders topic. The change event record should look like the following:

{ "topic":
"QuickstartConnectorNamespace.quickstart_keyspace.orders", "key":
"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"int32\",\"optional\":true,\"field\":\"customer_id\"},{\"type\":\"int32\",\"optional\":true,\"field\":\"order_id\"}],\"optional\":false,\"name\":\"QuickstartConnectorNamespace.quickstart_keyspace.orders.Key\"},\"payload\":{\"customer_id\":1,\"order_id\":6}}",
"value": …output omitted… "timestamp": 1683362868372, "partition":
0, "offset": 6 }

8. Access the captured data using Redpanda Console When you spin up Redpanda using Docker Compose, it creates a Redpanda cluster and a Redpanda Console. Using the console, you can visually inspect topics and messages. After all the change events on the orders table are captured, you can go to the Redpanda Console and select the QuickstartConnectorNamespace.quickstart_keyspace.orders topic to see the messages, as shown below: Redpanda console showing all events in the topic You can also see the schema details, the payload, and other event metadata in the event message, as shown below: Redpanda Console showing details of one of the events You’ve now successfully set up CDC between ScyllaDB and Redpanda, consumed change events from the CLI, and accessed the same messages using the Redpanda Console! Conclusion This tutorial introduced you to ScyllaDB and provided step-by-step instructions to set up a connection between ScyllaDB and Redpanda for capturing changed data. Messaging between systems is one of the most prominent ways to create asynchronous, decoupled applications. Bringing CDC events to Redpanda allows you to integrate various systems. Using this CDC connection, you can consume data from Redpanda for various purposes, such as triggering workflows based on change events, integrating with other data sources, and creating a data lake. By following the tutorial and using the supporting GitHub repository, you can easily get started with CDC. To explore Redpanda, check the documentation and browse the Redpanda blog for more tutorials. If you have any questions or want to chat with the team, join the Redpanda Community on Slack. Likewise, if you want to learn more about ScyllaDB, see our documentation, visit ScyllaDB University (free), and feel free to reach out to the team.

Storage Attached Indexing: A Key New Feature in Apache Cassandra® 5.0

30 April 2024, 4:16 am by Apache Cassandra - Instaclustr

Apache Cassandra® version 5.0 Beta 1.0 is now available in public preview on the Instaclustr Managed Platform!

Here at NetApp, we’re excited to see the new Apache Cassandra version 5.0 release and the extended functionality that has been introduced. We have been testing the newest features that come along with this version, and in particular, getting to understand the latest secondary indexing architecture with Storage Attached Indexing (SAI).

Cassandra has always offered a secondary index, which can extend your read options by querying the database with values that are not the primary key. However, the overhead of servicing these queries across a distributed system has always been a concern due to the time and resource costs involved.

Storage Attached Indexes addresses these challenges by reducing storage overhead and enhancing query performance through optimizations on index storage and access paths. So, you can see why we’re keen to work with Storage Attached Indexing and help our customers get the most out of what is already a pretty amazing big data database.

What is Secondary Indexing and why does it matter to my Cassandra cluster?

Cassandra is designed to manage writes and reads at a large scale, but read operations– when querying for data on non-primary keys–are resource–intensive and more complex than what you find when querying in a ‘traditional’ relational database. Indexing is a technique used to quickly access and retrieve entries in a database and is typically adopted by developers to optimize read performance.

With Cassandra’s secondary index, developers have the option to configure query data on specific values over and above the primary key. Secondary Indexes solve the problem of querying columns that are not part of the primary key.

Essentially, developers choose to use a secondary index with Cassandra clusters to expand the flexibility of queries that they want to run across their data. Without a secondary index, Cassandra is very limited to what can be queried beyond the primary key. Instaclustr offers customers reliable and scalable systems data layer technologies to manage data at scale with the least hassle and lowest cost.

Having improved data access and retrieval comes with additional considerations. Secondary indexes rely on local indexing, which means index data is co-located with the source data on the same nodes. When retrieving data using only an indexed column, Cassandra doesn’t have a way to determine which nodes have the necessary data. Consequently, it ends up querying all the nodes in the cluster.

This process can cause increased data transfer, high latency, and a potential increase in costs, especially if the cluster has many nodes. As with adopting any new feature, it’s recommended to test secondary indexes with your application to ensure cluster performance is acceptable. Read here to understand more about the cost and benefits of Cassandra Filtering and Partition Keys.

What’s Different for Secondary Indexing in Cassandra 5.0?

Over the years, the Apache Cassandra community has been evolving functionality to get the most efficient and effective method for reading the vast quantities of data that the database can manage.

The Cassandra 5.0 Storage Attached Index (SAI) functionality has evolved from previous Secondary Indexes (2i) and SSTable Attached Secondary Index (SASI); the SAI architecture has been designed to operate as a native database operation, reducing query latency and resource usage, while maintaining the scalability, fault tolerance and performance that Cassandra is known for.

Design improvements with SAI include:

Enables vector search through the new storage-attached index (SAI) called “VectorMemtableIndex”
Lower disk usage: compared to SASI, SAI doesn’t create ngrams (contiguous subset of a longer sequence) for every term, saving a significant amount of disk space. Additionally, SAI eliminates the need to duplicate an entire table with a new partition key.
Simplified queries: newly introduced query syntax eliminates the need to create an SSTable for each query pattern. This means multiple indexes on one table, rather than multiple tables for multiple query patterns.

Storage Attached Indexing and Vector Search

Another exciting and significant enhancement delivered by Cassandra 5.0 is support for a Vector CQL data type. Support for this new data type means that Cassandra will handle and store embedding vectors as arrays of floating-point numbers. When coupled with Storage–Attached Indexing, the vector CQL data type introduced in Cassandra 5.0 enables Cassandra to support AI/ML workloads.

Cassandra natively manages scalability for large datasets and is the obvious choice to handle large volumes and accelerated growth of vector data as organization’s explore and establish AI/ML capabilities.

We’re excited to work with our Instaclustr customers and support them as they get started on the AI/ML journey, making the right choices at the beginning of that journey to get the best value for their business.

Getting Started with Storage Attached Indexing

At Instaclustr, we make it easy to provision and manage the operations of your data infrastructure so you can focus on accelerating development of your business applications. Trial the new Storage Attached Index feature with Apache Cassandra today and set up a non-production cluster using the Instaclustr console. Click here for a guide on how to provision an Apache Cassandra 5.0 Beta 1.0 cluster.

If you have any issues or questions about provisioning your cluster, please contact Instaclustr Support at any time.

The post Storage Attached Indexing: A Key New Feature in Apache Cassandra® 5.0 appeared first on Instaclustr.

ScyllaDB User Tech Talks: Dor Laor’s Takeaways

24 April 2024, 11:33 am by ScyllaDB

Somehow two months have passed since ScyllaDB Summit 24. Thinking back on the event, so much has changed since our inaugural ScyllaDB Summit, when ~100 people gathered in San Jose, CA in 2016. The ScyllaDB database, company, and community have all developed quite substantially. But one thing remains the same: hearing about what our users are achieving with ScyllaDB is still my favorite part of the conference. I want to share some of the highlights from these great tech talks by our users and encourage you to watch the ones that are most relevant to your situation. As you read/watch, you will notice that some of the same themes appear across multiple talks: Moving to ScyllaDB to simplify the tech stack and operations Growing demand for a database that’s not tied to a specific cloud vendor Users unwilling to compromise between low latency, high throughput, and cost-effectiveness ReversingLabs The Strategy Behind ReversingLabs’ Massive Key-Value Migration Martina Alilović Rojnić, Software Architect, ReversingLabs About: ReversingLabs offers a complete software supply chain security and malware analysis platform. ReversingLabs data is used by more than 65 of the world’s most advanced security vendors and their tens of thousands of security professionals. Takeaways: Martina is a fantastic storyteller, and she had a rather impressive story to tell. ReversingLabs collects a massive amount of file samples daily and analyzes them both statically and dynamically. They currently have around 20B samples in their data set, mostly malware. These samples generate over 300TB of metadata, which is then exposed to around 400 services and various feeds. Not surprisingly, the system architecture they designed when they were a scrappy startup well over a decade ago doesn’t suit their 2024 scale. They recognized that their initial Postgres-based architecture was not sustainable back in 2011, and they built their own simple key-value store that met their specific needs (being able to handle a ridiculous amount of data extremely efficiently). Ten years later, the company had grown significantly and they recognized they were reaching the limits of that custom key-value implementation. They selected ScyllaDB, as you can guess. But what’s most interesting is how they pulled off this massive migration flawlessly, and with zero downtime. As I said at the start, Martina is a great storyteller – so I strongly encourage you to learn about their strategy directly from her. Watch ReversingLabs Supercell Real-Time Persisted Events at Supercell Edvard Fagerholm, Senior Server Engineer, Supercell About: Supercell is the brand behind the popular global games Hay Day, Clash of Clans, Boom Beach, Clash Royale, and Brawl Stars. Supercell offers free-to-play games that yield profits through in-game microtransactions. Takeaways: Supercell recently started on an application that goes along with wildly popular video games which each have generated over $1B in revenue. Basically, they transformed an existing Supercell ID implementation to support an in-game social network that lets players communicate with each other and see what their friends are currently doing within the games. Every player update generates an event, which is immediately sent to an API endpoint, broadcasted out to the appropriate parties, and saved to ScyllaDB. In his tech talk, Edvard shared the story behind this social network’s implementation. I think we were all shocked to learn that the social network component of this system, used by over 200M players per month, is developed and run by a single backend engineer. That’s why the team wanted the implementation to be as efficient, flexible, and simple as possible, with a high level of abstraction. And to store the many events generated by all the players changing levels, updating avatars, etc., they needed a database that handled many small writes, easily supported a hierarchical data model, had low latency, and relieved that one backend engineer from having to deal with all the cluster management. We’re honored that they selected ScyllaDB Cloud for this task. See the video for examples of how they modeled data for ScyllaDB as well as a detailed look at how they built the broader system architecture that ScyllaDB plugs into. Watch Supercell Tractian MongoDB vs ScyllaDB: Tractian’s Experience with Real-Time ML JP Voltani, Director of Engineering, Tractian About: Tractian is an industrial asset monitoring company that uses AI to predict mechanical failures. They use sensors, edge computing hardware, and AI models to monitor industrial machines and identify potential failures based on vibrations and frequency patterns. Takeaways: The team at Tractian does a first-class job across the board: database POCs, benchmarking, migrations, and even award-worthy video production. Kudos to them for amazing skill and initiative, all around! With over 6 languages and 8 databases in the young and growing organization’s tech stack, they are clearly committed to adopting whatever technology is the best match for each of their use cases. And that’s also why they moved away from MongoDB when their AI model workloads increased over 2X in a single year and latencies also spiked beyond the point where scaling and tuning could help. They evaluated Postgres and ScyllaDB, with extensive benchmarking across all three options. Ultimately, they moved to ScyllaDB Cloud because it met their needs for latency at scale. Interestingly, they saw the data modeling rework required for moving from MongoDB to ScyllaDB as an opportunity, not a burden. They gained 10x better throughput with 10x better latency, together. Watch the session to get all the details on their MongoDB to ScyllaDB migration blueprint Watch TRACTIAN Discord So You’ve Lost Quorum: Lessons From Accidental Downtime Bo Ingram, Staff Software Engineer, Persistence Infrastructure, Discord About: Discord is “a voice, video, and text app that helps friends and communities come together to hang out and explore their interests — from artists and activists, to study groups, sneakerheads, plant parents, and more. With 150 million monthly users across 19 million active communities, called servers, Discord has grown to become one of the most popular communications services in the world.” Takeaways: Last year, Bo Ingram shared why and how Discord’s persistence team recently completed their most ambitious migration yet: moving their massive set of trillions of messages from Cassandra to ScyllaDB. This year, we were honored to have him back to share lessons from a very stressful Monday at Discord. Bo shared the inside perspective on an outage with one of their ScyllaDB clusters, showing how a stressed ScyllaDB cluster looks and behaves during an incident. If you watch this session, you will learn about how to diagnose issues in your clusters, see how external failure modes manifest in ScyllaDB, and how you can avoid making a fault too big to tolerate. Bonus: Bo just wrote a great book on using ScyllaDB! If you plan to run ScyllaDB at scale, make sure you read this book before going to production. Bo captured years of high scalability practices, wrapped in friendly and fun packaging. You can get early access to a preview (free) at https://lp.scylladb.com/scylladb-in-action-book-offer. That includes a 45% discount on the complete book, which is being released quite soon. Watch Discord Zee CTO Insights: Steering a High-Stakes Database Migration Kishore Krishnamurthy, CTO, ZEE5 Tracking Millions of Heartbeats on Zee’s OTT Platform Srinivas Shanmugam, Principal Architect, ZEE5 Jivesh Threja, Senior Software Engineer, ZEE5 About: Zee is a 30-year-old publicly-listed Indian media and entertainment company. ZEE5 is their premier “over the top” (OTT) streaming service, available in over 190 countries with 150M monthly active users. Takeaways: It’s honestly hard to believe that we just started working with Zee last year. They fast became power users, especially after navigating two major migrations – from DynamoDB, RDS (PostgreSQL), Redis, and Solr to ScyllaDB as well as a parallel migration to Google Cloud. The fact that these migrations went so smoothly is a great testament to the leadership of CTO Kishore Krishnamurthy. His keynote provides rapid-fire insight into what was going on through his head before, during, and after those migration processes. Zee engineers also provided an insightful tech talk on Zee’s heartbeat API, which was the primary use case driving their consideration of ScyllaDB. Heartbeat lies at the core of Zee’s playback experience, security, and recommendation systems. They currently process a whopping 100B+ heartbeats per day! Scaling up their previous heartbeats architecture to suit their rapid growth was cost-prohibitive with their previous databases. Jivesh and Srinivas outlined the technical requirements for the replacement (cloud-neutrality, multi-tenant readiness, simplicity of onboarding new use cases, and high throughput and low latency at optimal costs) and how that led to ScyllaDB. Then, they explained how they achieved their goals through a new stream processing pipeline, new API layer, and data (re)modeling that ultimately added up to 5X cost savings (from 744K to 144K annually) and single-digit millisecond P99 read latency. It all wraps up with an important set of lessons learned that anyone using or considering ScyllaDB should take a look at. Watch Kishore Watch the Zee5 Tech Talk Digital Turbine (with SADA) Radically Outperforming DynamoDB @ Digital Turbine with SADA and Google Cloud Joseph Shorter, VP of Platform Architecture, Digital Turbine Miles Ward, CTO, SADA About: Digital Turbine is a mobile advertising, app delivery, and monetization ecosystem adopted by mobile carriers and OEMs. Their technology is deployed on ~100M mobile phones and drives 5B app installs. Takeaways: First off, I think Miles and Joe deserve their own talk show. These are two great technologists and really cool people; it’s fun to watch them having so much fun talking about serious business-critical technology challenges. After some fun intros, Miles and Joe walked through why DynamoDB was no longer meeting their needs. Basically, Digital Turbine’s write performance wasn’t where they hoped it would be and getting the performance they really wanted would have required even higher costs. On top of that, Digital Turbine had already decided to move to GCP. They approached SADA, looking for a way to get better performance and migrate without radically refactoring their platform. Joe and team were sold on the fact that ScyllaDB offered Alternator, a DynamoDB-compatible API that let them move with minimal code changes (less than one sprint). They were pleasantly surprised to find that it not only improved their performance, but also came at a much lower cost due to ScyllaDB’s inherent efficiency. While DynamoDB was throttling them at 1,400 OPS, ScyllaDB wasn’t even breaking a sweat. That’s great to hear! Watch Digital Turbine and SADA JioCinema Discover the Unseen: Tailored Recommendation of Unwatched Content Charan Kamal, Back End Developer, JioCinema Harshit Jain, Software Engineer, JioCinema About: JioCinema is an Indian over-the-top media streaming service owned by Viacom18, a joint venture of Reliance Industries and Paramount Global. They feature top Indian and international entertainment, including live sports (including IPL cricket and NBA basketball), new movies, live sports, HBO originals, and more. Takeaways: It was interesting to see how JioCinema tackled the challenge of ensuring that the “prime real estate” of recommended content lists don’t include content that the viewer has already watched. This is a common concern across media streamers – or at least should be – and I’m impressed by the innovative way that this team approached it. Specifically, they used Bloom filters. They explored different options, but discovered that the most common ones required significant tradeoffs. Building their own in-memory Bloom filters offered low latency, but implementing it at the scale required was going to require too much work given their core focus. Redis was ruled out due to the deployment complexity required to maintain high availability and because Redis’ cost structure (pay for every operation) was going to be prohibitive given their 10M daily users and big events with 20M concurrent users. ScyllaDB also worked well, and with a cost structure that worked better for this rapidly growing service. In the video, Charan shares some good technical nuggets about how they implemented the Bloom filter solution with ScyllaDB. He also covers other features they liked in ScyllaDB, such as multi-region, local quorum, TTL for auto expiration, and high cardinality of key partitions. Watch JioCinema Expedia Inside Expedia’s Migration to ScyllaDB for Change Data Capture Jean Carlo Rivera Ura, NoSQL Database Engineer III, Expedia Mani Rangu, Database Administrator III, Expedia About: Expedia is one of the world’s leading full-service online travel brands helping travelers easily plan and book their whole trip with a wide selection of vacation packages, flights, hotels, vacation rentals, rental cars, cruises, activities, attractions, and services. Takeaways: Expedia shared their first ScyllaDB use case back in 2021. That was for an application that aggregates data from multiple systems, like hotel location info, third-party data, etc. with the goal of single-digit millisecond P99 read response time. The new use case presented at this year’s ScyllaDB Summit was related to their identity cluster, which numerous Expedia applications rely on for user authentication. If it goes down, users can’t log in. They were previously running on Cassandra and using Cassandra’s CDC to capture changes and stream them to a Kafka cluster using Debezium Connector. They grew frustrated by the fact that writes to the CDC-enabled tables were rejected whenever the Debezium Connector stopped consuming events (which happened all too often for their liking). Upon learning that ScyllaDB offered a straightforward and uncomplicated CDC implementation, they planned their move. Watch their talk to hear their breakdown of the two migration options they considered (SSTableLoader and the Spark-based ScyllaDB Migrator), get a step-by-step look at how they migrated, and learn what they’ll do differently when they migrate additional use cases over from Cassandra to ScyllaDB. Watch JioCinema ShareChat Getting the Most Out of ScyllaDB Monitoring: ShareChat’s Tips Andrei Manakov, Staff Software Engineer, ShareChat About: ShareChat is the leading social media platform in India. Between the ShareChat app and Moj (short-form video), they serve a user base of 400M users – and rapidly growing! Takeaways: ShareChat is now starting their second year as a ScyllaDB user, which is pretty amazing given that their best practices are among the most sophisticated of all of our users. ShareChat has been a generous contributor to our community across ScyllaDB Summit as well as P99 CONF; I personally appreciate that and I know our community is always eager to learn what they’re sharing next. The latest addition to the ShareChat library of best practices is Andrei Manakov’s talk on how they’ve really customized monitoring to their specific needs and preferences for Moj, their TikTok-like short video app with 20M daily active users and 100M monthly active users. Here, ScyllaDB is used for the ML feature store that powers users’ feeds. Andrei walks us through how they’ve used the ScyllaDB Monitoring Stack to optimize driver usage, measure the impact of changing compaction strategies, monitor cluster capacity, and more. Watch ShareChat’s Tips Bonus: In case anyone wants background on why and how ShareChat moved to ScyllaDB, we’ve also added the great introductory talk by Geetish Nayak to the on-demand library. He covers the pressure driving their database migration, why and how they implemented ScyllaDB for a still-growing number of use cases, their migration strategy, and the best practices they developed for working with ScyllaDB. See the ShareChat Story Bonus Session for Live Attendees Only Speaking of bonuses – live attendees were also treated to an additional user tech talk by a major media streaming provider. We don’t have permission to share it following that live broadcast, but here are some high-level takeaways: The team previously used Cassandra for their “Continue Watching” use case, but went looking for alternatives due to GC spikes, complex CDC, and its inability to take advantage of the modern hardware they had invested in They considered DynamoDB because of its similar data structure, but based on their data size (>10TB) and throughput (170k+ WPS and 78k+ RPS), they felt DynamoDB was “like just burning money” They conducted a POC of ScyllaDB, led by the ScyllaDB team using 6x i4i.4xlarge nodes with RF 3, 3B records preloaded They were able to hit combined load with no errors, P99 read latency of 9ms, and P99 write latency < 1ms Watch All Sessions On Demand