Optimizing Distributed System Performance: Roundtable, Videos, and Certification
First the bad news: you missed a great opportunity to pick the brains of three rather distinguished performance experts. But the good news is that you can now watch the masterclass sessions – and also take the certification exam – online for free!
To deliver a free community-focused masterclass, Henrik Rexed (Dynatrace), Leandro Melendez (Grafana/k6), and Felipe Cardeneti Mendes (ScyllaDB) came together online from Marseilles, Mexico City, and Sao Paulo – sharing their expertise with hundreds of engineers joining virtually from around the globe. The focus was on how to optimize distributed system performance, from the front-end to the back-end, with SLO-driven strategies.
- Leandro started off by providing a grounding in current expectations of what performance engineering and load testing entail. His session defined the modern challenges developers face, including continuous performance principles, Service Level Objectives (SLOs), and Service Level Indicators (SLIs). It delineated best practices and provides hands-on examples using Grafana k6, an open source modern load testing tool.
- Henrik walked through how to measure, validate and visualize these SLOs using Prometheus, an open observability platform, to provide concrete examples. Next, he demonstrated how to automate your deployment using Keptn, a cloud-native event-based life-cycle orchestration framework. Attendees learned how it could be used for multi-stage delivery, remediation scenarios, and automating production tasks.
- Felipe then dived into distributed database observability, showing several real-world situations that may affect workload performance and how to diagnose them. Simply having access to key metrics is not enough, he explained. You need a deeper understanding of them to know what to watch for. That’s why he showed how to triage problems by priority and quickly gather live insights to troubleshoot application performance.
- All three then came together for a lively chat led by cordial host Peter Corless – tying together key threads from the sessions, tackling top performance engineering questions, and sharing thoughts on the latest performance engineering news and trends.
ACCESS THE MASTERCLASS ON-DEMAND
If you’re curious about what to expect, take a look at this panel video, as well as a couple of excerpts we’ve transcribed below for easy scanning.
Key takeaways from the other instructors’ sessions
Henrik: I would say the first – from Leandro – is that I need a wider green screen to be able to record that type of presentation! [Chuckles] We are always in connection, Leandro and myself, so I knew quite a lot about his topics. Felipe’s was really interesting. I was doing analysis of databases when I was working as a performance engineer, and it was interesting to see the KPIs you walked through. It will be interesting to look at them a bit more so then I can utilize them for Keptn. [Keptn in an open source enterprise-grade control plane for cloud-native continuous delivery and automated operations.]
Felipe: That’s exactly what I wanted to say. I wasn’t aware of Keptn, so it was very insightful for me. Now I’m really excited to see how I can integrate ScyllaDB Monitoring Stack with Keptn to make things even better. Of course, Leandro, you have to teach me how you record videos because that was awesome man!
Leandro: I have to say, Felipe, some of the things that you showed I wish were available at large scale when I was trying to catch performance bottlenecks – bottlenecks that very often came from the database side. And for you, Henrik, I can’t wait to implement and try some of the new Keptn functionalities and to mix it with what we’re doing at k6; it sounds like they would go together really, really well.
Where to start when performance KPIs are trending in the wrong direction
Henrik: I would say first of all, when you are taking the mission of validating a given component, you need to ask the question, “Okay, so if I do that request, what would happen in the backend?” Because if you don’t know it, then you will basically go everywhere in the architecture and chase things that don’t make sense. By knowing your environment, you know what layer of your application will be involved when it is utilized. Then you start with, say, your latency metric. You know that after that you will have some queues or some connections. You can get those connections, and see if the memory starts to go up. If so, maybe there’s a queueing problem. There is no “state of the art” in terms of which metrics you should look at. It really depends on your own stack. To be able to pick the right metrics, you have to understand precisely what’s going on.
Leandro: I will expand a little bit on what Henrik mentioned. Yes, on one hand, it’s very important to know what you have to observe and to track and to diagnose. Speaking of diagnosing – let’s say you see your blood pressure deviating. There is no best or first thing that you always look at. Are you under stress? Are you eating and drinking well? Do you have kidney issues? How you are going to be investigating it is highly contextual. Adding on top of what Henrik is saying, do you have an MRI machine or other scanner at hand, like a good observability or monitoring platform? Maybe your environment reveals some good reason for that deviation to be happening. There are no silver bullets with performance testing, observability, or monitoring as Henrik said. If you have the MRI scan, you may have deep insight into what is happening – and even there, it will depend a lot. If you don’t have it, and you have only automations, black-box or front-end measurements, it will be just like having a blood pressure monitor and trying to figure it out from there.
Felipe: I agree with both of my peers; what they have said, it’s certainly right. There is no really specific metric that is going to mean anything to everyone. For example, I typically like to look at latency first. For example, when I’m analyzing a problem, often I go back to ask, “What is your SLA requirement?” Many times I see a latency of 10 ms, 15 ms, but if I don’t know what the SLOs or SLAs are, it’s hard for me to make sense of what I’m seeing. So a general understanding of what we are expecting is certainly what is necessary. In addition to that, one thing that I would like to add is how important it is for us to have historical metrics. You can then analyze the whole history of how your application has been performing and then you can quickly compare how it has been performing over time. There are some scenarios when, for example, you may have a bad access pattern at a specific time of the day.
Access the Performance Engineering Masterclass Now
Intrigued? You can attend the complete masterclass – and even take the certification exam – whenever you have a few hours to spare. When you complete it, you’ll know best practices on how to:
- Optimize your team’s pipelines and processes for continuous performance
- Define, validate, and visualize SLOs across your distributed systems
- Diagnose and fix subpar distributed database performance using observability methods
Avi Kivity: A Window Into Database Operations from the Disk’s Perspective
Understanding what goes on
inside a fast NVMe SSD is key for maximizing its performance,
especially with IO-heavy workloads. But how do you peer into this
traditionally black box?
That’s what ScyllaDB CTO and co-founder Avi Kivity recently solved for. Specifically, he wanted a new level of insight into what ScyllaDB, a NoSQL database with a unique close-to-the-hardware architecture, looked like from the perspective of the NVMe-based Nitro SSDs featured in new AWS EC2 I4i instances.
Avi set out to visualize this and will share his approach in a P99 CONF 2022 keynote.
JOIN AVI’S KEYNOTE AT P99 CONF – FREE AND VIRTUAL
With Avi as your guide, you will watch real-time visualizations of IO: a dynamic display of color-coded reads and writes, random IO, sequential IO, and the disk bandwidth consumed by each. After explaining how he created a window into the disk, he will narrate and interpret a real-time visualization, including:
- How ScyllaDB’s shard-per-core architecture is reflected in disk behavior
- How sequential SSTable writes differ from commitlog writes
- How compaction ramps up and down in response to read and write workloads
- How the database works with the disk to avoid lock contention
Next, Avi will explore a different perspective: how the same database operations manifest themselves in monitoring dashboards. Metric by metric, he sheds light on the database internals behind each graph’s interesting trends.
Join Avi’s P99 CONF keynote to experience this fascinating journey in system observability – and learn how you can gain this level of insight into your own IO.
A Full Lineup of Low-Latency Engineering Feats at P99 CONF
If you’re enticed by engineering feats like this, you will be excited to see what else we have planned for P99 CONF 2022. Here’s a look at 4 out of the 50+ sessions to give you an idea of what to expect:
Analyze Virtual Machine Overhead Compared to Bare Metal with Tracing
Steven Rostedt -Software Engineer, Google
Running a virtual machine will obviously add some overhead over running on bare metal. This is expected. But there are some cases where the overhead is much higher than expected. This talk discusses using tracing to analyze this overhead from a Linux host running KVM. Ideally, the guest would also be running Linux to get a more detailed explanation of the events, but analysis can still be done when the guest is something else.
Why User-Mode Threads Are Often the Right Answer
Ron Pressler – Project Loom Technical Lead, Java Platform Group, Oracle
Concurrency is the problem of scheduling simultaneous, largely-independent tasks, competing for resources in order to increase application throughput. Multiple approaches to scalable concurrency, which is so important for high-throughput servers, are used in various programming languages: using OS threads, asynchronous programming styles (“reactive”), syntactic stackless coroutines (async/await), and user-mode threads. This talk will explore the problem, explain why Java has chosen user-mode threads to tackle it, and compare the various approaches and the tradeoffs they entail. We will also cover the main source of the performance of user-mode threads and asynchronous I/O, which is so often misunderstood (it’s not about context-switching).
Overcoming Variable Payloads to Optimize for Performance
Armin Ronacher – Creator of Flask; Architect, Sentry.io
When you have a significant amount of events coming in from individual customers, but do not want to spend the majority of your time on latency issues, how do you optimize for performance? This becomes increasingly difficult when you are dealing with payload sizes that are multiple orders of magnitude difference, have complex data that impacts processing, and the stream of data is impossible to predict. In this session, you’ll hear from Armin Ronacher, Principal Architect at Sentry and creator of the Flask web framework for Python on how to build ingestion and processing pipelines to accommodate for complex events, helping to ensure your teams are reaching a throughput of hundreds of thousands of events per second.
Keeping Latency Low for User-Defined Functions with WebAssembly
Piotr Sarna – Principal Software Engineer, ScyllaDB
WebAssembly (WASM) is a great choice for user-defined functions, due to the fact that it was designed to be easily embeddable, with a focus on security and speed. Still, executing functions provided by users should not cause latency spikes – it’s important for individual database clusters and absolutely crucial for multi-tenancy. In order to keep latency low, one can utilize a WebAssembly runtime with async support. One such runtime is Wasmtime, a Rust project perfectly capable of running WebAssembly functions cooperatively and asynchronously. This talk briefly describes WebAssembly and Wasmtime, and shows how to integrate them into a C++ project in a latency-friendly manner, while implementing the core runtime for user-defined functions in async Rust.
Distributed Database Design Decisions to Support High Performance Event Streaming
In an article earlier this year, I talked about comparing distributed databases in general. All of those considerations remain salient for choosing a database right for your use case. Yet for today let’s focus in on a particular aspect of capability: whether a database is a strong fit for an event streaming architecture.
The transition from batch-oriented processing to event streaming architectures is at the heart of what’s fundamentally shifting in this next tech cycle. Kafka was initially released in 2011, around the same era as the early NoSQL databases. Pulsar was slightly later, being invented at Yahoo in 2013. Many database architectures long predate the event streaming revolution, and their fundamental design decisions may be an anti-pattern for use with event streaming.
Databases are [or should be] designed for specific kinds of data, specific kinds of workloads, and specific kinds of queries. How aligned or far away from your specific use case a database may be in its design & implementation from your desired utility of it determines the resistance of the system. So, sure you can use various databases for tasks they were never designed for — but should you?
For a database to be appropriate for event streaming, it needs to support managing changes to data over time in “real time” — measured in single-digit milliseconds or less.
And where changes to data can be produced at a rate of hundreds of thousands or millions of events per second. (And even greater rates in future.)
There are four major axes that organizations consider when looking at whether a database is suitable for their event streaming use cases:
- Cloud native
- Intrinsic qualities
- Event-driven design
- Best fit
Let’s go through each of these in more detail.
Cloud Native
Earlier this year, a study by Foundry (formerly IDG Communications) found that half of organizations still have the bulk of their workloads mostly on-premises. Another third are “mostly cloud” but still with some on-premises services. Only 7% of organizations are “all cloud.”
This shift to cloud is rapidly underway. “the all-cloud number is expected to increase by some 250% over the next 18 months, to 17% who plan to shift everything they have to the cloud.”
Even if users are not planning to go “all-cloud,” they know their systems need to at least integrate with their cloud services. They need to future proof their technology choices. Thus, “seventy-two percent of IT leaders say that their organization is defaulting to cloud-based services when upgrading or purchasing new technical capabilities.”
Critical to this integration of on-premises services and cloud services is the ability to integrate with event streaming. Indeed, event streaming services are seen as the key component to hold multi-cloud and hybrid cloud deployments together between disparate systems.
What does this mean for a distributed database choice?
For many, this means asking the vendor if they have a Database-as-a-Service (DBaaS) offering. If they do, an organization can run lean. They can focus on developing their applications, and not have to staff specialists in database administration. If they don’t, there’s organizational friction to adoption.
Organizations also want to know if a solution is locked in to a single cloud vendor, or if it is cloud neutral — deployable anywhere. Because while a few organizations are still willing to commit themselves exclusively to a single cloud vendor strategy, about 90% of respondents to a 2022 Flexera survey said their team already has a multicloud strategy. Thus a single cloud vendor database presents another barrier to adoption — it becomes a lock-in that directly contradicts their goal of strategic flexibility.
The next question is whether or how a distributed database can be deployed — only in a single region cluster, or across multiple datacenters and regions.
There’s a huge difference between having a true multi-datacenter architecture versus keeping systems in sync across regions through some sort of cobbled-together cross-datacenter update mechanism. The former is designed to be reliable, secure, and bidirectional (multi-directional); the latter may only be unidirectional, with one primary datacenter flowing updates to other replicas. And, of course, there’s some assembly required which can add to operational complexity and security concerns.
“Some assembly required.”
Being cloud-native means a great many things to different people. It may also infer capabilities like elasticity (dynamic provisioning and the ability to scale up or down throughput and/or storage capacity as needed), to run serverless (without having to worry about specific underlying hardware), or cloud orchestration (Kubernetes). There’s tons more considerations based on various roles and perspectives, ranging from DevSecOps and SREs, to performance and platform engineering.
While many vendors will claim to be “cloud-native,” what they may mean is that they have lifted and shifted their open source or enterprise databases to run in the cloud with a thin veneer of “cloud” painted on top. This is not the same thing as a from-the-ground-up implementation, or putting in all the hard elbow grease to make sure that everything from API and microservices integrations, ecosystem connections, SRE and DevOps tools, to account management, billing and governance was designed to make the system easy to implement and consume.
Intrinsic Qualities
The next considerations are intrinsic qualities of the database in question. These are all the critical “-ilities” that used to be summed up as RAS or RASP — Reliability, Availability, Serviceability, and Performance. While this acronym harkens back to the era of pre-cloud hardware, it, and many other qualities, can be redefined for this cloud-native tech cycle.
Reliability is the critical quality that appeals most to a Site Reliability Engineer. That a database will meet its Service Level Agreements (SLAs) and Service Level Objectives (SLOs). What good is a database you cannot rely upon? You can even drill down upon what “reliability” means, in that it can also stand for “durability” — the resilience of a system to outages. Or the anti-entropy mechanisms that it might include, such as the ability to gracefully handle or recover from outages, repair data, or even catch up a node down for a transient outage using hinted handoffs.
If you are shoving data down a big, fast pipe, and your destination is offline, what do you do? This can be why event sourcing is important — so you can go back step-by-step through all stages of an event stream and find out what your results should be. But it’s vital that your production database doesn’t make you question its viability as a source of truth in the first place.
Availability (high availability) is at the heart of databases like ScyllaDB and all classes of “AP”-mode databases as defined by the CAP theorem. While there is definitely still a place for strongly-consistent (“CP”-mode) databases in an event streaming world — SQL RDBMS are not going away any time soon — high availability systems are often a better match for event driven architectures, being highly asynchronous and optimized for real-time responsiveness.
“AP”-mode databases like ScyllaDB are often a better match for event driven architectures
Serviceability, if adopted to the modern cloud-native milieu, encompasses other elemental qualities such as observability, manageability, usability, facility, and more. Was your database designed so you can tell what it’s doing? Conversely, can you tell it what to do if it’s not behaving properly? Usability is a key factor in adoption. Modern distributed databases are not “easy.” They require expertise in data and query modeling, operations and administration. But is your system needlessly opaque and obtuse? If so, then you might find your team balks at having to learn something utterly orthogonal to current knowledge or behaviors, if not outright actively hostile to adoption.
“Facility,” to me, goes beyond just being “usable.” It asks, “is it easy to use and administer?” Does it feel bulletproof to run? Or is it a cantankerous beast? Does running it feel like a joy, or does it saddle organizations like constantly suffering from the miasma of a low-grade fever?
This can be exemplified by the annual Stack Overflow survey, which asks tens of thousands of developers what their most loved and most dreaded databases are. Developers who feel their current choice is a dreaded chore will be looking for something, anything, to ameliorate their pain over time. Those that enjoy their work are less likely to churn to a new technology, or abandon an organization that they believe is woefully behind the times. Users who love the technology they’ve deployed on don’t have to think too heavily about it. The database becomes “invisible,” allowing them to focus instead on the technical problems they are trying to solve.
Performance is also a key part of this critical factor. Let’s look more at this in specific respect to an impedance match for your event-driven architecture.
Event-Driven
Let’s now focus on whether the database is truly designed for an event-driven architecture. Does its features and capabilities actually help you evolve from a batch-oriented world to a real-time streaming reality?
There are two sides to this: sink (consumer) and source (producer).
Acting as a consumer (sink) seems pretty straightforward — it’s just data ingestion, right? — but there are many subtleties involved. How does the system handle time series data? How does it partition such data and distribute it within the system? Do all of the writes go to a single hot shard that melts with the throughput? Do the other shards hold “stale” data that is essentially “write once, read hardly ever?” (This can be especially problematic on primary-replica systems that only allow one node to be the leader that can be written to; all the others are read-only copies.)
As a producer (source), there’s more requisite infrastructure work, such as support for Change Data Capture (CDC) so that an alteration to the database can be published to a topic. CDC implementations can vary widely, with some being well-designed and engineered; efficient and comprehensive. Yet others can be afterthoughts or bolt-ons, with significant performance impacts and kludge-y in implementation. You really want to double-click on the word “CDC” when you see it in a list of supported features, because no two database vendor implementations will be the same.
Also, while it is subtle, it is important to note that to truly be an “event sourcing” solution, a database needs to keep a record of all data states, event-after-event. So you may need to keep pre-values as well as post-values of data records; not just the diffs. Plus your use case will determine how long those records need to be kept around.
Impedance matching in event streaming is important to keep the “current” of your organization’s data flowing without resistance or signal loss
To truly pair with your event streaming system, both the database you are using and the event streaming architecture you are implementing need to have an “impedance match.” If the database is too lightweight or too sluggish to keep up with the volumes of your event streaming, it will quickly become overwhelmed and unusable. Conversely, if your event streaming cannot keep up with the pace of updates to your operational transactional database, you may find that you can’t keep the rest of your enterprise in sync with the latest changes occurring in your source of truth.
Best Fit for Use Case
Just because you are familiar with a particular database doesn’t mean it’s the best fit for the job. Maybe you are familiar with SQL. But does the data structures and high availability and throughput of this use case require you to consider a NoSQL database instead? Are you familiar with NoSQL, but really, for this highly consistent use case you will need tables and joins and third normalized forms? Then that sounds like a job for SQL.
Here, users should think about what sorts of applications they are designing. What sort of throughput and latency requirements they have. Data set scaling (gigabytes? terabytes? petabytes?), both in total, and growth-over-time.
They should then look at the data model and data query languages required to most efficiently and easily obtain the data results they are looking for. Is this a read- or write-heavy workload, or is it mixed read-write? Is it transactional updates, or is it analytical range or full table scans? How fast does the data processing have to happen? Are results needed in the sub-millisecond, millisecond, second or even minute+ range?
Speed can also be measured in terms of throughput. Do you have hundreds of queries per second, or thousands or a million or more? And throughput itself can either be seen as operations or transactions per second, or it can be looked at based on total volume or payload. Because the result set for a full table scan is going to be vastly different than a single object or row result query.
Finally, and of primary concern to all, what is the price/cost that the organization is willing to bear for this solution? What sort of TCO or ROI goals does it have? Because what may sound lovely for performance purposes might make you blanch at your monthly bill when it comes due.
For example, a DBaaS option may seem more expensive than open source (“it’s free, right?”), but the commercial service might have features that can actually save your organization more in the long run, either in terms of operational or administrative overhead.
With all of these considerations in mind, you now have a general rubric against which you can grade your short list of options for your use case.
Event Streaming Journey of a NoSQL Database: ScyllaDB
So let’s see how this rubric “grades” ScyllaDB in our own quest to be the best database for your event streaming architecture. This is not to say that we are a perfect exemplar against which all other databases should be measured. I’ll try to be fair and candid in assessment. If you are using a different database, just consider for yourself how your own technology choice stacks up.
First, let’s look at where ScyllaDB began its journey towards event streaming. Work on ScyllaDB began in earnest in 2015 — well into the event streaming era. Its founders were cognizant of Kafka and Pulsar, and blogs early in the history of ScyllaDB (and Confluent) spoke about the virtues of connecting Kafka and ScyllaDB (see here, here, here and here).
The reason was straightforward: ScyllaDB was built on “good bones.” Kafka was a highly-scalable, performant event streaming architecture, and ScyllaDB was a highly-scalable, performant NoSQL database.
ScyllaDB was designed to get single-digit P99 latencies at millions of operations per second. It could scale out not just across any number of servers, but, given its foundations on the Seastar framework, also scale up to any core count per server.
Its peer-to-peer active-active topology meant there was no single point of failure or throughput bottleneck of a leader-replica design. ScyllaDB also had high availability, topology awareness and multi-datacenter designs built in from the get-go.
(You can read more here if you want to be more familiar with ScyllaDB’s architecture.)
And best of all, it was compatible with Cassandra CQL — and later, Amazon DynamoDB — so it was already comfortable to those who were familiar with its query languages and data models.
Because it was compatible with Apache Cassandra it immediately inherited the ecosystem connectors that were already available for it.
Our customers, such as Numberly, developed their own systems to integrate ScyllaDB and Kafka. Fortunately, ScyllaDB and Kafka already played well together.
We Can Do Better
While that was sufficient to get users up and running connecting ScyllaDB with Apache Kafka, we knew there were features that users would truly benefit from unique to our own designs and implementation. For example, building our own custom Sink and Source connectors.
The sink is the easier of the two, and can seem pretty straightforward. But even then, you have subtleties. For example, ScyllaDB’s Kafka Sink Connector is far from “vanilla.” It takes advantage of ScyllaDB’s shard-per-core design by being shard-aware, directing writes not only to the right node, but to the exact right vCPU associated with a data partition in a cluster. This minimizes cross-CPU traffic for fastest write efficiency, lowering latencies and maximizing throughput. It also has other features, such as the ability to manage the schema within ScyllaDB as you write topics to the database, or even setting the Time-to-Live (TTL) on the data you’re writing.
To be an event source, though, required some pretty radical work on ScyllaDB that took quite a bit of time. First was the implementation of Change Data Capture. We spoke repeatedly about the design and implementation in 2020, later again in 2020, then more on CDC best practices in 2021 when it went GA, and later again in 2021 when we had the Kafka Source Connector built on Debezium. All in all, it took more than two full years of engineering to really have the CDC implementation we knew customers would be happy with.
ScyllaDB’s Journey to Event Streaming — Starting with Kafka
- Shard-Aware Kafka Sink Connector [January 2020]
- Change Data Capture [January 2020 – October
2021]
- January 2020: ScyllaDB Open Source 3.2 — Experimental
- Course of 2020 – 3.3, 3.4, 4.0, 4.1, 4.2 — Experimental iterations
- January 2021: 4.3: Production-ready, new API
- March 2021: 4.4: new API
- October 2021: 4.5: performance & stability
- CDC Kafka Source Connector [April 2021]
By late 2021, with both Sink and Source connectors, and with the CDC interface fully stable and performant, the integration was bidirectional between the systems and complete.
That solved for open source Apache Kafka and enterprise Confluent Kafka, yet those are not the only options in the event streaming world. Fortunately, Redpanda is Kafka-compatible, so it immediately inherits compatibility to our existing connectors.
ScyllaDB’s Journey to Event Streaming — with Pulsar
How about Apache Pulsar? Or a commercialized Pulsar, such as the StreamNative Platform? Fortunately, out of the box Pulsar ships with a Cassandra Sink Connector. Though this isn’t a shard-aware sink connector, like we have for Kafka. Using the Pulsar Producer, you can wrap or adapt it to ingest data from Kafka topics. However, the Pulsar wrapper is admittedly a “some assembly required” solution.
So future potential developments include a native Pulsar shard-aware Consumer, and a native CDC Pulsar Producer.
- Pulsar Consumer: Cassandra Sink Connector
- Comes by default with Pulsar
- ScyllaDB is Cassandra CQL compatible
- Docs: https://pulsar.apache.org/docs/io-cassandra-sink/
- Github: https://github.com/apache/pulsar/blob/master/site2/docs/io-cassandra-sink.md
- Pulsar Producer: Can use ScyllaDB CDC Source Connector
using Kafka Compatibility
- Pulsar makes it easy to bring Kafka topics into Pulsar
- Docs: https://pulsar.apache.org/docs/adaptors-kafka/
- Potential Developments
- Native Pulsar Shard-Aware ScyllaDB Consumer Connector — even faster ingestion
- Native CDC Pulsar Producer — unwrap your topics
Summary
Readers can use the ideas presented herein to compare the suitability of existing or proposed databases for their own event streaming architectures.
Event streaming is a trend that is unmistakable and growing, and presently many users deploy event streaming side-by-side with large enterprise-grade databases. This is especially true for ScyllaDB users, such as Numberly, Grab, Nauto, Amdocs and many, many more.
ScyllaDB was created concurrently to, and has grown and matured in line with the event streaming revolution. ScyllaDB has a robust, performant implementation of Change Data Capture which allows users great flexibility in how they wish to consume data from ScyllaDB. Its integration with Apache Kafka includes performant and advanced sink and source connectors. However, while ScyllaDB is compatible with Apache Cassandra connectors for Apache Pulsar, ScyllaDB lacks native connectors, which remain a future consideration for more performant and easy-to-deploy implementation.
How Are You Using Event Streaming in Your Organization?
If you are a self-starter, feel free to read the docs on how to integrate Kafka, or take a free course on Kafka and Change Data Capture in ScyllaDB University. Or if Pulsar is more your speed, check out the tutorial on connecting Cassandra and Pulsar. It should work the same for ScyllaDB.
We’d love to hear how you are implementing event streaming in your organization alongside ScyllaDB, or to hear your questions on your path to getting started. There’s two ways you can connect with us. You can contact us privately through our online form or chat function, or you can join your fellow database monsters in our Slack community.
Chatting KVM, Cassandra, Discord, DBaaS, DynamoDB on GCP & More
Whenever ScyllaDB CEO Dor Laor and SADA CTO Miles Ward get together, their mutual obsession over tapping the power of modern cloud infrastructure rings loud and clear. Case in point: the recent podcast, Cloud N Clear.
Here are some highlights from their conversation covering ScyllaDB’s origins in KVM, why and how ScyllaDB built an alternative Cassandra and then DynamoDB, how ScyllaDB supports business-critical use cases like Discord’s, the rise of database-as-a-service, and more. The transcript is lightly edited for brevity.
Pivoting from KVM to a “Monstrously Fast and Scalable” Database
Dor: We decided to create an operating system for virtualization from scratch with our own kernel because we [Avi Kivity and I] did the KVM hypervisor before. We saw everybody running just one app per OS and we thought, “Let’s create a new OS.” But Docker came along and forced us to pivot. We ran a lot of databases on our OS and we felt that Apache Cassandra was not as fast as it should be. It’s got a super-nice scale-out architecture, but it’s not efficient at all. It’s written in Java, which is a nice language for management apps but not for high-end speed. So, we figured, “Let’s rewrite this from scratch; take the OS knowledge we’ve got and just do it.”
Miles: [chuckling] For everyone in the audience, you should internalize the scope of what he’s saying: “Oh, we think this is a good app – we should just build a faster version of it.” I’m super impressed by the technical depth on this team. This is the same technical group that designed the KVM hypervisor, which among other places, is in use in production at Google Cloud – powering every GCP instance and a huge number of internal services.
Dor: We didn’t know how complicated it would be. We thought, “We wrote a hypervisor, how much more complicated can a database be?” It’s actually much more complicated than a hypervisor… GCP published a benchmark of Cassandra running 330 small virtual machines to run 1 million Cassandra query language (CQL) operations per second. When we were at Red Hat and working on the KVM hypervisor, we broke the world record of doing filesystem IO using a single VM virtual machine: we did 1.6 million operations per second with a single virtual machine. It’s an unfair comparison because it’s only file system IO, not replicated, versus a million database operations with 3X replication. But really…330? Something is not right. Ahead of time, we were sure that we could 10x the Apache Cassandra performance – and we have.
Open Source and the Rise of Database as a Service
Miles: ScyllaDB is available not only as an open source project, but also as a managed service. Can you talk about the relationship between those two and how ScyllaDB manages the managed service offering?
Dor: From the very beginning, we’ve been open source fans. We believe in open source; we like open source, we think it’s fun. My management style actually took a lot of processes and values from open source. In addition to an Open Source version of ScyllaDB, we have an Enterprise version based on it (similar to the Red Hat offering). Also, a couple of years ago, we saw the need to have a managed database as a service. This is basically where the entire market is going – in databases and beyond. ScyllaDB Cloud is a managed ScyllaDB Enterprise with a lot of code around it to automate the management aspects of it. When a machine fails, when a data center fails, you don’t need to wake up at night. You don’t need to worry about backups. Not everyone does backups, and even fewer make sure that the restore actually works before a disaster.
Miles: Backups are worthless – it’s restores that are so nice, right? That’s why the “as a service model” is so valuable to so many customers. It aligns incentives properly. If you can make an investment in operational efficiency that pays dividends over all of your customers, of course, you’re incentivized to do that work. It benefits every customer. So, you end up in this spot where you’ve got the right reasons to be making a better and better product over time.
The Responsibility of Being “The Beating Heart in the Middle of a Business” like Discord
Miles: Discord is running ScyllaDB on GCP. Mark Smith, Discord’s Director of Infrastructure, said they’ve got over 150 million monthly active users – a really scaled system for communicating while you’re playing games, open source projects, etc. I use it all the time with some of our companies that want to collaborate in that way. His quote is, “We trust ScyllaDB’s speed and reliability to help with some of our most business-critical use cases including all of our core product experience, our anti-abuse systems and more.” Talk to me about that responsibility. You’re building a product that powers other products. SADA is a company that helps companies build their products. But what are you doing at ScyllaDB to make sure that it can keep those kinds of promises?
Dor: That’s definitely not a simple undertaking. It takes a while to gain the trust of customers. Take Discord, for example. They started with us at a small scale. Over time, after a lot of testing, they started to shift their services off Cassandra and onto ScyllaDB. Now, they’ve moved pretty much everything to ScyllaDB, and we’re super proud. We work hard every day to satisfy all customer needs, even minor issues. Sometimes it’s hard to figure out if an issue is coming from the database, or the infrastructure, or the client. We don’t worry about those boundaries; we focus on solving the problems and making the customer happy. One time, an on-prem customer reported that a single node suddenly went to 50% capacity. I had seen this before in my past, so I told them to go check if the second power supply was still on. That turned out to be the issue. We’ve solved driver issues in the OS, bugs in non-ScyllaDB database drivers, and so on. It all needs to work end-to-end.
Miles: You’re exactly right. I think I remember when the term “full stack engineer” made its way into the market of ideas and how we talk about these roles. Every time I met one that said they were full stack I was like, “Oh no, you don’t mean full stack full stack. You don’t know power supplies, you don’t know what’s going on inside of memory registers…There’s too much to know – the scale of context that’s required to really make all of this technology do what it’s supposed to do is enormously broad. SADA knows that; all day long, we’ve got people that ask us for support for every different Google product, all the ISVs that sit around them, all of the surrounding technologies they layer on top of that. It really is an incredible thing to watch in practice as businesses try to make value out of this crazy set of zeros and ones that they throw around.
Continue Watching/Listening: CAP Theorem, DynamoDB, and More
That’s just the beginning of this lively chat. Watch/listen (you can also get it on your favorite podcast platform) to hear what Dor and Miles have to say about:
- What ScyllaDB has been working on (spoiler: consistency and elasticity feature quite prominently) (12:16)
- Miles’ interactions with Eric Brewer of CAP theorem fame (15:51)
- How ScyllaDB’s open source DynamoDB-compatible API enables customers to take DynamoDB to GCP and on-prem (17:00)
- The “fun” budding relationship between SADA and ScyllaDB (23:40)
Also, to learn more about ScyllaDB’s partnership with Google Cloud and SADA (the award-winning Google Cloud Premier Partner), see ScyllaDB’s Google Cloud partner page.
P99 CONF 2022 Agenda Revealed
P99 CONF is shaping up to be an incredible event. Already thousands of people have registered to attend. Best of all, because it’s all online and free, there’s no limits to attendance! You can register right now for two days of keynotes and interactive tech talks on all things P99.
P99 CONF is a free virtual event scheduled for Wednesday and Thursday, October 19th and 20th, 2022, from 8:00 AM to 1:00 PM Pacific Daylight Time (PDT) — 16:00 – 20:00 UTC.
What You Get for Registering
As a registered attendee of P99 CONF you gain full and free access to all of the following:
- Two days of scheduled online sessions, including keynotes and breakouts
- Presentation decks
- Our “Speaker’s Lounge,” a live interactive virtual talk show with our guests of honor
- Extensive, exclusive on-demand only content
- Entry into contests for awesome prizes
REGISTER NOW FOR P99 CONF 2022
Watch the Sessions from P99 CONF 2021
To get a taste of what our 2022 event will be like, you can catch up on any or all of last year’s sessions now available on demand.
Zeotap: Building a Privacy Compliant Customer Data Platform (CDP) with ScyllaDB
Today’s article highlights the reasons why Zeotap moved their workload to ScyllaDB after trying a few other technologies first. It is drawn from a talk given at ScyllaDB Summit 2022 earlier this year. Note that you can watch all of the talks from that virtual event for free.
Zeotap’s Mission
Zeotap is a customer intelligence platform, also known as a Customer Data Platform (CDP) focusing on the needs of European marketers. It helps brands better understand their customers and predict behavior. Founded in Germany in 2014, as a third party data aggregation platform it evolved into a next-generation CDP. Their mission is to give a 360º view to their customers, allowing them to branch data and make better marketing decisions. To enable their rapid growth Zeotap built their brand on Google Cloud.
Zeotap rose quickly to the front ranks of the digital marketing industry, advancing the state-of-the-art through innovation while still meeting the stringent privacy compliance requirements for operating in the European Union. For example, Zeotap announced General Data Protection Regulation (GDPR) compliance in July 2016 — nearly two full years ahead of the mandatory May 2018 compliance date.
Another emerging aspect of privacy compliance is eliminating the currently near-ubiquitous 3rd party cookies that watch users as they traverse within sites and browse from site-to-site. Since HTTP cookies first debuted in 1994 at Netscape, an entire generation of technologists and companies have come to rely upon them for user identification and behavior attribution.
This time around the compliance requirement was not driven by government-mandated regulation but by industry fiat. Since Google’s Chrome browser accounts for the great majority of web clients — somewhere between 62% to 65% of detectable browser usage — the company can effectively dictate policy.
Google first announced in January 2020 it would phase out supporting third-party cookies by the start of 2022. However, this sent a panic through the tech industry. Many business models were based upon cookies and dependent on their implementation. Late in 2021, Google pushed out the deadline for compliance to mid-to-late 2023. Yet this was only a relatively brief reprieve which still has many companies scrambling.
Zeotap Leads the Way
Facing a world with a cookieless future, Zeotap tapped into its innovative spirit, once again pioneering how to implement compliant systems. At ScyllaDB Summit 2022 we had the pleasure of hosting Shubham Patil and Safal Pandita, two engineers from Zeotap who explained how their cookieless CDP system was implemented in their talk Building Zeotap’s Privacy Compliant Customer Data Platform (CDP) with ScyllaDB.
Shubham noted, “Today’s customer journeys are more complex than ever.” For example, a user might stumble across a company’s social media post which directs them to a blog, and then finally lands them on the e-commerce site. “With such unpredictable journeys, how can the business keep track of their customers?” Shubham asked, “and how can they deliver personalized experiences across an ever-expanding set of touch points?”
This is where the Customer Data Platform comes in. It defines unique users, and aggregates information related to those users’ detected online and offline activities. This means ingesting, processing, storing and distributing correlated data from hundreds of consented-to data sources. These can be first, second, or third party sources. For example, it could come from your Customer Relationship Management (CRM) platform, an adtech Data Management Platform (DMP), or your data lake or data warehouse. Or it could be a mobile application today, or an IoT-enabled device in the future.
This aggregated data is made into what are called “golden records.” These golden records which can then be segmented into target audiences, have Machine Learning (ML) algorithms run against them, and then be utilized, for example, in a paid social media advertising campaign.
This data could come from bulk data batch processing, or real time event streaming. These weren’t just for ingestion, but also for data export. Beyond that, the database also needed to support strong privacy compliance measures, including user opt-outs and explicit consent management systems.
Zeotap had a list of other technical requirements as well, including:
- Multi-region, multi-tenant
- Low-latency (sub-second) real-time reads and writes
- Point lookups
- Scalable from megabytes to petabytes
- Mature monitoring stack
- Spark integration support
- Control of cluster sizing
- Encryption-at-rest
- Control of data model
- Simple SQL-like query
- Enterprise support
Fortunately, ScyllaDB met each of these requirements. Whereas their original system, based on Google BigQuery and detailed below, could not even provide sub-second query response, with ScyllaDB their responses were measurable in milliseconds.
First Version
Zeotap implemented the first version of their CDP with very simplified goals using Google BigQuery for the data store for ingestion of all the client data. But there were some problems with this, Shubham observed. Their business requirements included on-the-fly user unification from data drawn from different sources with sub-second latencies, and this system could not manage that performance threshold. Here’s the scorecard that Zeotap applied to their first implementation.
“We needed to do better,” Shubham admitted. “In this version one implementation our sub-second read writes were not possible. Point lookups were not possible. We also did not have a very good monitoring stack and also did not have complete control on sizing of the cluster. So we started exploring wide column OLTP databases.”
Second Version
“Our next POC [Proof of Concept] was with JanusGraph using both HBase and ScyllaDB [as backend storage layers]. With this new tech we were able to solve most of our problems but then we also started facing some other issues. Specifically issues where we did not have complete control over the data model, scans were multi-row and expensive, and there was no enterprise support to any of the issues that we were facing. And there was no transparency to the transaction failure and back-end errors.”
After releasing version 2 with JanusGraph this is what their tech matrix looked like:
Shubham shared, “Some of the problems like sub-second reads and deletes were actually solved. But the other problems like point lookups and ‘works for data at every scale’ and a matured monitoring stack was still not there. Then again there were some other problems that came into the picture, like we did not have a simple SQL-like query interface and, again, we did not have complete control on the data model.”
JanusGraph uses Gremlin/Tinkerpop as its query language, which can appear quite foreign to developers familiar with SQL. ScyllaDB, on the other hand, uses Cassandra Query Language (CQL), which has a passing similarity to SQL, even though its syntax differs significantly. For example, CQL does not support table JOINs. But it did fit Zeotap’s requirements far better.
Third Version
The Zeotap team decided at that point to use ScyllaDB natively as the storage layer using its CQL interface for queries, and removed the JanusGraph layer on top.
At last all their main technical requirements were met. All the problems faced in their first and second iterations were resolved. Latencies which had gotten down to 600 milliseconds using JanusGraph (nominally acceptable according to Zeotap’s “sub-second” requirement) now dropped far further in ScyllaDB — to only 30 milliseconds.
Plus, Zeotap now had a very mature and transparent observability platform in the form of ScyllaDB Monitoring Stack, which is comprised of open source components like Prometheus and Grafana. ScyllaDB also offered Apache Spark integration. Zeotap now had complete control on sizing of their cluster, their underlying data model, all with enterprise support.
After moving to ScyllaDB, Zeotap’s tech matrix finally looked like this:
Explore How Zeotap Implemented ScyllaDB
Shubham then introduced Senior Software Engineer Safal Pandita, who took the audience through the rest of the presentation describing Zeotap’s implementation of ScyllaDB. You won’t want to miss it! Fortunately, you can watch the entire video on demand.
Learn more about their data model, their queries, the performance optimizations they employed, and details of their operational deployment. Just to give you a sense of Zeotap’s scale, they now run four clusters across continental Europe, the UK, India, and the United States, with each cluster comprising six nodes of n2-highmem-64 instances.
If you have your own use case you have questions about and want to see if ScyllaDB is right for you, feel free to drop in to our Slack user community, or reach out to contact us privately.
Benchmarking ScyllaDB 5.0 on AWS i4i.4xlarge
Earlier this year, we compared ScyllaDB 4.4.3’s performance on early-access instances of AWS’s storage-optimized I4i instances for IO-heavy workloads. Running ScyllaDB 4.4.3 on both the i4i.16xlarge and i3.16xlarge, we observed up to 2.7x higher throughput per vCPU for reads. With an even mix of reads and writes, we observed 2.2x higher throughput per vCPU on the new i4i series, with a 40% reduction in average latency than I3 instances. [Read the results from this i4i.16xlarge benchmark].
Since then, the complete family of I4i instances went into general availability – opening up access to the i4i.4xlarge instances that are closely related to the i3.4xlarge instances used in our recent ScyllaDB vs Cassandra benchmarks. Also, a major new release of ScyllaDB went to general availability with performance optimizations such as a new IO scheduler that’s optimized for mixed read/write workloads.
Curious to quantify the combined performance impact of the new I4is plus ScyllaDB’s latest performance optimizations, we recently kicked off a new series of benchmarks. This blog shares the first results from that project: ScyllaDB Open Source 5.0 on i4i.4xlarge vs. ScyllaDB 4.4.3 on the i3.4xlarge we used in previous benchmarks. We are working on expanding these benchmarks – stay tuned to our blog for additional comparisons and findings.
What’s So Interesting About the AWS I4i Instances?
The new I4i instances are powered by the latest generation Intel Xeon Scalable (Ice Lake) Processors with an all-core turbo frequency of 3.5 GHz. Compare this to the I3’s Intel Xeon E5-2686 v4 (Broadwell) @ 2.3 GHz (2.7 GHz). Moreover, they use AWS’s directly-attached, NVMe-based Nitro SSD devices that were designed to minimize latency and maximize transactions per second. AWS found that they offer up to 30% better compute price performance, 60% lower storage I/O latency, and 75% lower storage I/O latency variability compared to I3 instances. Another difference: there’s also a new, larger instance size (i4i.32xlarge) that features 2x the RAM of the i3.metal as well as 2x the SSD — a whopping 128 vCPUs backed by a terabtye of RAM and 30 terabtyes of storage per node.
Here are the I4i technical specs:
Our Setup
The benchmarking was a series of simple invocations of Cassandra-stress with CL=QUORUM. The procedure was:
- Set up the cluster from intact AMIs and spin up powerful loaders.
- Load the cluster with 3TB of data.
- Warm up the cache by ~3 hours of reads.
- Apply the data distributions (described below) in gradual increments of 10k ops/s.
- Stop when p90 latency exceeds 1000 ms.
See the earlier ScyllaDB vs Cassandra benchmark for all the setup details
For this round of benchmarking, we ran a total of 10 scenarios spanning different data distributions and workloads. The data distributions used were:
- Uniform distribution, with a close-to-zero cache, hit ratio
- “Real-life” (Gaussian) distribution, with sensible cache-hit ratios of ~30%
The workloads were read only, write only, and mixed (50% read, 50% write). For each of these 3 variations, we ran disk-intensive workloads as well as memory-intensive workloads (e.g., the entire dataset could fit in RAM).
Benchmark Results: i4i.4xlarge vs i3i.4xlarge
Here’s a summary of the results (maximum throughput in k/s with single-digit ms P99 latencies).
Disk-intensive write-only workload
With a disk-intensive workload of 100% writes, ScyllaDB 5.0 on the i4i achieved 2X (100%) greater throughput with single-digit ms P99 latencies: 350k vs 170k.
Disk-intensive read-only workload
For 100% reads, ScyllaDB 5.0 on the i4i achieved 67% greater throughput with single-digit ms P99 latencies: 100k vs 60k.
Disk-intensive mixed workload – write latencies
With a 50% read / 50% write workload, ScyllaDB 5.0 on the i4i achieved 37% greater throughput with single-digit ms P99 latencies for writes: 110k vs 80k.
Disk-intensive mixed workload – read latencies
With a 50% read / 50% write workload, ScyllaDB 5.0 on the i4i achieved 62% greater throughput with single-digit ms P99 latencies for reads: 110k vs 68k.
Disk-intensive Gaussian mixed workload – write latencies
With the Gaussian distribution mixed workload, ScyllaDB 5.0 on the i4i achieved 12.5% greater throughput with single-digit ms P99 latencies for writes: 90k vs 80k.
Disk-intensive Gaussian mixed workload – read latencies
With the Gaussian distribution mixed workload, ScyllaDB 5.0 on the i4i achieved 28% greater throughput with single-digit ms P99 latencies for reads: 90k vs 70k.
Memory-intensive write-only workload
Now, over to the memory-intensive workloads, where the entire dataset could fit in RAM.
With a disk-intensive workload of 100% writes, ScyllaDB 5.0 on the i4i achieved 2X (100%) greater throughput with single-digit ms P99 latencies: 400k vs 200k. This 2X improvement in throughput matches the gain from the disk-intensive write-only workload.
Memory-intensive read-only workload
For 100% reads, ScyllaDB 5.0 on the i4i achieved 83% greater throughput with single-digit ms P99 latencies: 440k vs 240k.
Memory-intensive mixed workload – write latencies
With a 50% read / 50% write workload, ScyllaDB 5.0 on the i4i achieved 112% greater throughput with single-digit ms P99 latencies for writes: 340k vs 160k.
Memory-intensive mixed workload – read latencies
With a 50% read / 50% write workload, ScyllaDB 5.0 on the i4i achieved 112% greater throughput with single-digit ms P99 latencies for reads: 340k vs 160k.
Surprising Latencies for Administrative Operations
Finally, we measured the latencies during “Add Node” and “Major Compaction” operations. We used 3x i4i.4xlarge for “Add Node,” 1x i4i.4xlarge for “Major Compaction, ScyllaDB 5.0, ~50% ScyllaDB load of background workload, and mixed disk-intensive workload. We aimed at 30% CPU load but overshot.
Here are the results for adding a node:
| Write latency (in ms) | Read latency (in ms) | |
| Mean | 1.17 | 2.41 |
| p90 | 2.28 | 4.41 |
| p99 | 2.72 | 5.52 |
And here are the results for major compaction:
| Write latency (in ms) | Read latency (in ms) | |
| Mean | 1.11 | 3.79 |
| p90 | 1.35 | 5.21 |
| p99 | 1.52 | 6.06/span> |
Yes – compaction unexpectedly improved the P99 write latencies ever so slightly. There are cases where it’s better for ScyllaDB to process data than to go back to the idle loop.
Final Thoughts
Of course, your mileage will vary. We strongly encourage you to perform your own benchmarks with your own workloads. In the meantime, we’ll be performing more benchmarking as well.
We want to delve into some of the more interesting results (for example the discrepancy across the read and write latencies on the Gaussian distribution and the dip in read latencies for the memory-intensive workload). Additionally, we want to compare ScyllaDB’s performance on I4is vs that of other databases. We are also considering more matchups like our “4 x 40” benchmark: ScyllaDB on 4 nodes vs. Cassandra on 40 nodes.
Happn: Falling in Love with ScyllaDB
Happn was born in 2014 with the mission to make finding the people you cross paths with easier. To the Happn founders, fate plays a large part in where we find ourselves in life. Happn helps by making you aware of those nearby who may share the same romantic inclinations as you. Their vision is that the love you find online should be as authentic as if you met face-to-face in real life; technology just gives a helping hand to destiny by allowing singles who have crossed paths to find each other on the app.
Happn has grown from 1 million users in 2015 to over 128 million users in 2022.
Their strategy has resonated with singles around the world. Happn is one of the most downloaded online dating apps on the planet, active in forty countries with users primarily located in Western Europe, South America, Turkey and India. To date, 126 million people have signed up for the Happn app.
Happn has been part of the FT120 ranking of the French tech sector for two years in a row. Their team reflects their corporate ethos. They are young and cosmopolitan; the average employee age is 31 and 40% of their staff are women. Happn’s staff, now grown to around 100 employees, comprises 18 nationalities.
Falling In Love with ScyllaDB
One of the staff at Happn, Senior Platform Engineer Robert Czupiol, described their path to falling in love with our database at ScyllaDB Summit 2022.
Early in their growth Happn adopted Apache Cassandra. Yet by 2021 the relationship had soured. There were many issues with using Cassandra:
- Total Cost of Ownership (TCO)
- Technical debt
- Managing growing volumes of data
- Minimizing latencies
- Real-time monitoring
Technical debt was incurred in great part by running a very old version of Cassandra 2.1, originally released in 2014. They were also running a down-rev version of Debian 8 (released in 2015) which they knew they needed to upgrade. Yet because of the other compounding factors the Happn team looked beyond just upgrading to the current Cassandra and Debian releases to solve their broader issues. After internal analysis the decision was made to move to ScyllaDB.
Happn migrated fourteen of their clusters from Cassandra to ScyllaDB, the main one of which stores over 20 terabytes of data and at peak times hitting over 300,000 operations per second. While setting up dual writes and then making a cut-over is a standard procedure for migrations, one cluster proved a bit more complicated.
The second biggest cluster was known as the “Crossings Cluster.”
This was responsible for tracking when two individuals who might be
potentially romantically interested in each other came within
physical proximity — an important event the server wanted to count.
This Cassandra cluster, running on Google Cloud, comprised 48
e2-highmem-4 nodes, each of which had 4 CPUs, 32 GB of RAM, and 1
TB of networked persistent SSD (PD-SSD). This
Crossings_count table alone comprised 68 billion
records, and collectively they had over 88 billion records total to
migrate.
The problem with migrating counter data from Cassandra to ScyllaDB was to ensure they weren’t double counting across the two systems. For example, if you had records with values of 26, 27, and 28 on Cassandra, and just started up a corresponding counter table on ScyllaDB, it would have initial values of 1, 2, 3. If you tried to later merge the Cassandra table data, you could end up double-counting or totally miscounting activities.
To solve this, Happn invented their own Java-based counter-migrator, which did a compare-and-set operation. This double-checked the latest value in Cassandra, and set ScyllaDB to that latest value. However, with their scale they still had issues with performance and memory consumption of the Java app, so they eventually moved their code to Golang.
When finally in full production operations on ScyllaDB, the results were significantly better. P99 latencies dropped from 80 ms to only 20 ms — four times better. P90 latencies dropped from 50ms down to 15ms — over three times better.
Happn also saved significantly on disk space. With Cassandra they were using upwards of 48 TB total storage. With ScyllaDB that figure dropped to only 18 TB — about a third. These savings were attributable to three main factors:
- The more compact “md” format for SSTables
- More aggressive compaction ScyllaDB could manage with its custom IO scheduler, and
- ScyllaDB’s support for Zstandard (Zstd) compression.
Because ScyllaDB could vertically scale up to bigger nodes and use significantly less storage overall, Happn was able to shrink its cluster footprint from 48 nodes of Cassandra to just 6 nodes with ScyllaDB.
Happn also reduced their Google Cloud Storage (GCS) budget by going to incremental snapshot backups. Finally, knowing they could at last manage the scale of their growth they moved to committed annual contracts. Overall Happn reduced their total cost of ownership (TCO) by 75%.
The Romance Continues to Bloom
Robert’s presentation at ScyllaDB Summit 2022 was held the week just before Valentine’s Day in February. Much has changed since then. I had the opportunity to interview Robert to find out how the romance has continued to bloom between Happn and ScyllaDB.
To serve their global community Happn now performs between 200K to 350k read ops/second, and 45k to 80k write ops/second, depending on the time of day.
They have continued to shift workloads to ScyllaDB. Now, their total node count is 51 n2 series servers using fast local NVMe SSD. These can store 91 TB of total capacity. 60 TBs of that capacity is currently occupied, with the biggest cluster now being 25 TB of data.
Block diagram showing the extents of ScyllaDB in use at Happn. Red rectangles are ScyllaDB clusters. Green rectangles are microservices.
Heat map of where Happn’s current global daily requests come from
Looking to Build a Community of a Hundred Million or More?
Happn isn’t alone in falling in love with ScyllaDB. Along with Asian-based dating app Tantan, Happn shows how ScyllaDB can be used to build relationships within communities of a hundred million people or more using our fast and scalable NoSQL database.
Beyond online dating, Discord has built their platform using ScyllaDB as its core storage engine for over 350 million users, and online shopping giant Rakuten has grown their product catalog to more than a billion products which you can browse through and buy.
If you are interested in scaling your own business we’d love to hear. Please contact us directly, or join our Slack community to discuss your use case with your fellow big data monsters.