18 February 2025, 1:39 pm by
ScyllaDB
Monster SCALE Summit speakers have amassed a rather
impressive list of publications, including quite a few books. This
blog highlights 10+ of them. If you’ve seen the
Monster
SCALE Summit agenda, you know that the stars have aligned
nicely. In just two half days, from anywhere you like, you can
learn from 60+ outstanding speakers – all exploring extreme scale
engineering challenges from a variety of angles. Distributed
databases, event streaming, AI/ML, Kubernetes, Rust…it’s all on the
agenda. If you read the bios of our speakers, you’ll note that many
have written books. This blog highlights eleven of those Monster
SCALE Summit speakers’ books – plus two new books by past
conference speakers. Once you register for the conference (it’s
free + virtual), you’ll gain 30-day full access to the complete
O’Reilly library (thanks to O’Reilly, a conference media sponsor).
And Manning Publications is also a media sponsor. They are offering
the Monster SCALE community a nice 50% discount on all Manning
books . One more bonus: conference attendees who participate in the
speaker chat will be eligible to win book bundles, courtesy of
Manning.
See the agenda and
register – it’s free Designing Data-Intensive Applications, 2nd
Edition
By Martin Kleppmann and Chris Riccomini
O’Reilly ETA: December 2025 Data is at the center of many
challenges in system design today. Difficult issues such as
scalability, consistency, reliability, efficiency, and
maintainability need to be resolved. In addition, there’s an
overwhelming variety of tools and analytical systems, including
relational databases, NoSQL datastores, plus data warehouses and
data lakes. What are the right choices for your application? How do
you make sense of all these buzzwords? In this second edition,
authors Martin Kleppmann and Chris Riccomini build on the
foundation laid in the acclaimed first edition, integrating new
technologies and emerging trends. You’ll be guided through the maze
of decisions and trade-offs involved in building a modern data
system, from choosing the right tools like Spark and Flink to
understanding the intricacies of data laws like the GDPR. Peer
under the hood of the systems you already use, and learn to use
them more effectively Make informed decisions by identifying the
strengths and weaknesses of different tools Navigate the trade-offs
around consistency, scalability, fault tolerance, and complexity
Understand the distributed systems research upon which modern
databases are built Peek behind the scenes of major online
services, and learn from their architectures
Martin and Chris
are presenting “Designing Data-Intensive Applications in 2025”
Think Distributed Systems
Dominik Tornow
ETA: Fall 2025
Manning
(use code SCALE2025 for 50% off) All modern software is
distributed. Let’s say that again—all modern software is
distributed. Whether you’re building mobile utilities,
microservices, or massive cloud native enterprise applications,
creating efficient distributed systems requires you to think
differently about failure, performance, network services, resource
usage, latency, and much more. This clearly-written book guides you
into the mindset you’ll need to design, develop, and deploy
scalable and reliable distributed systems. In Think Distributed
Systems you’ll find a beautifully illustrated collection of mental
models for: Correctness, scalability, and reliability Failure
tolerance, detection, and mitigation Message processing
Partitioning and replication Consensus
Dominik is presenting
“The Mechanics of Scale” Latency: Reduce Delay in
Software Systems
Pekka Enberg ETA: Summer 2025
Manning (use
code SCALE2025 for 50% off) Slow responses can kill good software.
Whether it’s recovering microseconds lost while routing messages on
a server or speeding up page loads that keep users waiting, finding
and fixing latency can be a frustrating part of your work as a
developer. This one-of-a-kind book shows you how to spot,
understand, and respond to latency wherever it appears in your
applications and infrastructure. This book balances theory with
practical implementations, turning academic research into useful
techniques you can apply to your projects. In Latency you’ll learn:
What latency is—and what it is not How to model and measure latency
Organizing your application data for low latency Making your code
run faster Hiding latency when you can’t reduce it
Pekka
presented “Patterns
of Low Latency” at P99 CONF 2024. And his Turso co-founder
Glauber Costa will be presenting “Who Needs One Database Anyway?”
at Monster SCALE Summit Writing for Developers: Blogs
That Get Read
By Piotr Sarna and Cynthia Dunlop
January 2025
Amazon |
Manning (use code SCALE2025 for 50% off) This book is a
practical guide to writing more compelling engineering blog posts.
We discuss strategies for nailing all phases of the technical
blogging process. And we have quite a bit of fun exploring the core
blog post patterns that are most common across engineering blogs
today, like “The Bug Hunt,” “How We Built It,” “Lessons Learned,”
“We Rewrote It in X,” “Thoughts on Trends,” etc. Each “pattern”
chapter includes an analysis of real-world examples as well as
specific dos/don’ts for that particular pattern. There’s a section
on moving from blogging into opportunities such as article writing,
conference speaking, and book writing. Finally, we wrap with a
critical (and often amusing) look at generative AI blogging uses
and abuses. Oh…and there’s also a foreword by Bryan Cantrill and an
afterword by Scott Hanselman! Readers will learn how to: Pinpoint
topics that make intriguing posts Apply popular blog post design
patterns Rapidly plan, draft, and optimize blog posts Make your
content clearer and more convincing to technical readers Tap AI for
revision while avoiding misuses and abuses Increase the impact of
all your technical communications
Piotr is presenting “A Dist
Sys Programmer’s Journey Into AI” ScyllaDB in Action
Bo Ingram October 2024
Amazon |
Manning
(use code SCALE2025 for 50% off) |
ScyllaDB
(free chapters) ScyllaDB in Action is your guide to everything you
need to know about ScyllaDB, from your very first queries to
running it in a production environment. It starts you with the
basics of creating, reading, and deleting data and expands your
knowledge from there. You’ll soon have mastered everything you need
to build, maintain, and run an effective and efficient database.
This book teaches you ScyllaDB the best way—through hands-on
examples. Dive into the node-based architecture of ScyllaDB to
understand how its distributed systems work, how you can
troubleshoot problems, and how you can constantly improve
performance.You’ll learn how to: • Read, write, and delete data in
ScyllaDB • Design database schemas for ScyllaDB • Write performant
queries against ScyllaDB • Connect and query a ScyllaDB cluster
from an application • Configure, monitor, and operate ScyllaDB in
production
Bo’s colleagues Ethan Donowitz and Vicki Niu are
both presenting at Monster SCALE Summit Data
Virtualization in the Cloud Era
Dr. Daniel Abadi and Andrew
Mott July 2024
O’Reilly Data virtualization had been held back by complexity
for decades until recent advances in cloud technology, data lakes,
networking hardware, and machine learning transformed the dream
into reality. It’s becoming increasingly practical to access data
through an interface that hides low-level details about where it’s
stored, how it’s organized, and which systems are needed to
manipulate or process it. You can combine and query data from
anywhere and leave the complex details behind. In this practical
book, authors Dr. Daniel Abadi and Andrew Mott discuss in detail
what data virtualization is and the trends in technology that are
making data virtualization increasingly useful. With this book,
data engineers, data architects, and data scientists will explore
the architecture of modern data virtualization systems and learn
how these systems differ from one another at technical and
practical levels. By the end of the book, you’ll understand: The
architecture of data virtualization systems Technical and practical
ways that data virtualization systems differ from one another Where
data virtualization fits into modern data mesh and data fabric
paradigms Modern best practices and case study use cases
Daniel
is presenting “Two Leading Approaches to Data Virtualization: Which
Scales Better?” Bonus: Read Daniel Abadi’s
article on
the PACELC theorem. Database Performance at Scale
By Felipe Cardeneti Mendes, Piotr Sarna, Pavel Emelyanov,
and Cynthia Dunlop October 2023
Amazon |
ScyllaDB
(free) Discover critical considerations and best practices for
improving database performance based on what has worked, and
failed, across thousands of teams and use cases in the field. This
book provides practical guidance for understanding the
database-related opportunities, trade-offs, and traps you might
encounter while trying to optimize data-intensive applications for
high throughput and low latency. Whether you’re building a new
system from the ground up or trying to optimize an existing use
case for increased demand, this book covers the essentials. The
ultimate goal of the book is to help you discover new ways to
optimize database performance for your team’s specific use cases,
requirements, and expectations. Understand often overlooked factors
that impact database performance at scale Recognize data-related
performance and scalability challenges associated with your project
Select a database architecture that’s suited to your workloads, use
cases, and requirements Avoid common mistakes that could impede
your long-term agility and growth Jumpstart teamwide adoption of
best practices for optimizing database performance at scale
Felipe is presenting “ScyllaDB is No Longer “Just a Faster
Cassandra” Piotr is presenting “A Dist Sys Programmer’s
Journey Into AI” Algorithms and Data Structures for
Massive Datasets
Dzejla Medjedovic, Emin Tahirovic, and
Ines Dedovic May 2022
Amazon |
Manning (use code SCALE2025 for 50% off) Algorithms and Data
Structures for Massive Datasets reveals a toolbox of new methods
that are perfect for handling modern big data applications. You’ll
explore the novel data structures and algorithms that underpin
Google, Facebook, and other enterprise applications that work with
truly massive amounts of data. These effective techniques can be
applied to any discipline, from finance to text analysis. Graphics,
illustrations, and hands-on industry examples make complex ideas
practical to implement in your projects—and there’s no mathematical
proofs to puzzle over. Work through this one-of-a-kind guide, and
you’ll find the sweet spot of saving space without sacrificing your
data’s accuracy. Readers will learn: Probabilistic sketching data
structures for practical problems Choosing the right database
engine for your application Evaluating and designing efficient
on-disk data structures and algorithms Understanding the
algorithmic trade-offs involved in massive-scale systems Deriving
basic statistics from streaming data Correctly sampling streaming
data Computing percentiles with limited space resources
Dzejla
is presenting “Read- and Write-Optimization in Modern Database
Infrastructures” Kafka: The Definitive Guide, 2nd
Edition
By Gwen Shapira, Todd Palino, Rajini Sivaram, Krit
Petty November 2021
Amazon |
O’Reilly Engineers from Confluent and LinkedIn responsible for
developing Kafka explain how to deploy production Kafka clusters,
write reliable event-driven microservices, and build scalable
stream processing applications with this platform. Through detailed
examples, you’ll learn Kafka’s design principles, reliability
guarantees, key APIs, and architecture details, including the
replication protocol, the controller, and the storage layer. You’ll
learn: Best practices for deploying and configuring Kafka Kafka
producers and consumers for writing and reading messages Patterns
and use-case requirements to ensure reliable data delivery Best
practices for building data pipelines and applications with Kafka
How to perform monitoring, tuning, and maintenance tasks with Kafka
in production The most critical metrics among Kafka’s operational
measurements Kafka’s delivery capabilities for stream processing
systems
Gwen is presenting “The Nile Approach: Re-engineering
Postgres for Millions of Tenants” The Missing README: A
Guide for the New Software Engineer
by Chris Riccomini and
Dmitriy Ryaboy
Amazon |
O’Reilly August 2021 For new software engineers, knowing how to
program is only half the battle. You’ll quickly find that many of
the skills and processes key to your success are not taught in any
school or bootcamp. The Missing README fills in that gap—a
distillation of workplace lessons, best practices, and engineering
fundamentals that the authors have taught rookie developers at top
companies for more than a decade. Early chapters explain what to
expect when you begin your career at a company. The book’s middle
section expands your technical education, teaching you how to work
with existing codebases, address and prevent technical debt, write
production-grade software, manage dependencies, test effectively,
do code reviews, safely deploy software, design evolvable
architectures, and handle incidents when you’re on-call. Additional
chapters cover planning and interpersonal skills such as Agile
planning, working effectively with your manager, and growing to
senior levels and beyond. You’ll learn: How to use the legacy code
change algorithm, and leave code cleaner than you found it How to
write operable code with logging, metrics, configuration, and
defensive programming How to write deterministic tests, submit code
reviews, and give feedback on other people’s code The technical
design process, including experiments, problem definition,
documentation, and collaboration What to do when you are on-call,
and how to navigate production incidents Architectural techniques
that make code change easier Agile development practices like
sprint planning, stand-ups, and retrospectives
Chris and Martin
Kleppmann are presenting “Designing Data-Intensive Applications in
2025” The DynamoDB Book
By Alex Debrie
April 2020
Amazon |
Direct
DynamoDB is a highly available, infinitely scalable NoSQL database
offering from AWS. But modeling with a NoSQL database like DynamoDB
is different than modeling with a relational database. You need to
intentionally design for your access patterns rather than creating
a normalized model that allows for flexible querying later. The
DynamoDB Book is the authoritative resource in the space, and it’s
the recommended resource within Amazon for learning DynamoDB. Rick
Houlihan, the former head of the NoSQL Blackbelt team at AWS, said
The DynamoDB Book is “definitely a must read if you want to
understand how to correctly model data for NoSQL apps.” The
DynamoDB takes a comprehensive approach to teaching DynamoDB,
including: Discussion of key concepts, underlying infrastructure
components, and API design; Explanations of core strategies for
data modeling, including one-to-many and many-to-many
relationships, filtering, sorting, aggregations, and more; 5 full
walkthrough examples featuring complex data models and a large
number of access patterns.
Alex is presenting “DynamoDB Cost
Optimization Considerations and Strategies” RESTful Java
Patterns and Best Practices: Learn Best Practices to Efficiently
Build Scalable, Reliable, and Maintainable High Performance Restful
Services
By Bhakti Mehta
Amazon September, 2014 This book provides an overview of the
REST architectural style and then dives deep into best practices
and commonly used patterns for building RESTful services that are
lightweight, scalable, reliable, and highly available. It’s
designed to help application developers get familiar with REST. The
book explores the details, best practices, and commonly used REST
patterns as well as gives insights on how Facebook, Twitter,
PayPal, GitHub, Stripe, and other companies are implementing
solutions with RESTful services.
13 February 2025, 4:55 pm by
ScyllaDB
How strategic database migration + data (re)modeling
improved latencies and cut database costs 5X ZEE is
India’s largest media and entertainment business, covering
broadcast TV, films, streaming media, and music. ZEE5 is
their premier OTT streaming service, available in over 190
countries with ~150M monthly active users. And every user’s
playback experience, security, and recommendations rely upon a
“heartbeat API” that processes a whopping 100B+ heartbeats per day.
The engineers behind the system knew that continued business growth
would stress their infrastructure (as well as the people reviewing
the database bills). So, the team decided to rethink the system
before it inflicted any heart attacks. TL;DR, they designed a
system that’s loved internally and by users. And Jivesh Threja
(Tech Lead) and Srinivas Shanmugam (Principal Architect) joined us
on Valentine’s Day last year to share their experiences. They
outlined the technical requirements for the replacement (cloud
neutrality, multi-tenant readiness, simplicity of onboarding new
use cases, and high throughput and low latency at optimal costs)
and how that led to ScyllaDB. Then, they explained how they
achieved their goals through a new stream processing pipeline, new
API layer, and data (re)modeling. The initial results of their
optimization:
5X cost savings (from $744K to $144K
annually) and single-digit millisecond P99 read latency.
Wrapping up, they shared lessons learned that could benefit anyone
considering or using ScyllaDB. Here are some highlights from that
talk… What’s a Heartbeat? “Heartbeat” refers to a request that’s
fired at regular intervals during video playback on the ZEE5 OTT
platform. These simple requests track what users are watching and
how far they’ve progressed in each video. They’re essential for
ZEE5’s “continue watching” functionality, which lets users pause
content on one device then resume it on any device. They’re also
instrumental for calculating key metrics, like concurrent
viewership for a big event or the top shows this week. Why Change?
ZEE5’s original heartbeat system was a web of different databases,
each handling a specific part of the streaming experience. Although
it was technically functional, this approach was expensive and
locked them into a specific vendor ecosystem. The team recognized
an opportunity to streamline their infrastructure– and they went
for it. They wanted a system that wasn’t locked into any particular
cloud provider, would cost less to operate, and could handle their
massive scale with consistently fast performance – specifically,
single-digit millisecond responses. Plus, they wanted the
flexibility to add new features easily and the ability to offer
their system to other streaming platforms. As Srinivas put it: “It
needed to be multi-tenant ready so it could be reused for any OTT
provider. And it needed to be easily extensible to new use cases
without major architectural changes.” System Architecture, Before
and After Here’s a look at their original system architecture with
multiple databases: DynamoDB to store the basic heartbeat data
Amazon RDS to store next and previous episode information Apache
Solr to store persistent metadata One Redis instance to cache
metadata Another Redis instance to store viewership details
Click for a detailed view The ZEE5 team considered
four main database options for this project: Redis, Cassandra,
Apache Ignite, and ScyllaDB. After evaluation and benchmarking,
they chose ScyllaDB. Some of the reasons Srinivas cited for this
decision: “We don’t need an extra cache layer on top of the
persistent database. ScyllaDB manages both the cache layer and the
persistent database within the same infrastructure, ensuring low
latency across regions, replication, and multi-cloud readiness. It
works with any cloud vendor, including Azure, AWS, and GCP, and now
offers managed support with a turnaround time of less than one
hour.” The new architecture simplifies and flattens the previous
system architecture structure.
Click for a detailed view Now, all heartbeat events
are pushed into their heartbeat topic, processed through stream
processing, and ingested into ScyllaDB Cloud using ScyllaDB
connectors. Whenever content is published, it’s ingested into their
metadata topic and then inserted into ScyllaDB Cloud via metadata
connectors. Srinivas concludes: “With this new architecture,
we successfully migrated workloads from DynamoDB, RDS, Redis, and
Solr to ScyllaDB. This has resulted in a
5x cost reduction,
bringing our monthly expenses down from $62,000 to around
$12,000.” Deeper into the Design Next Jivesh shared more
about their low-level design… Real-time stream processing pipeline
In the real-time stream processing pipeline, heartbeats are sent to
ScyllaDB at regular intervals. The heartbeat interval is set to 60
seconds, meaning that every frontend client sends a heartbeat every
60 seconds while a user is watching a video. These heartbeats pass
through the playback stream processing system, business logic
consumers transform that data into the required format – then the
processed data is stored in ScyllaDB. Scalable API layer The first
component in the scalable API layer is the heartbeat service, which
is responsible for handling large volumes of data ingestion. Topics
process the data, then it passes through a connector service and is
stored in ScyllaDB. Another notable API layer service is the
Concurrent Viewership Count service. This service uses ScyllaDB to
retrieve concurrent viewership data – either per user or per asset
(e.g., per ID). For example, if a movie is released, this service
can tell how many users are watching the movie at any given moment.
Metadata management use case One of the first major challenges ZEE5
faced was managing metadata for their massive OTT platform.
Initially, they relied on a combination of three different
databases – Solr, Redis, and Postgres – to handle their extensive
metadata needs. Looking to optimize and simplify, they redesigned
their data model to work with ScyllaDB instead – using ID as the
partition key, along with materialized views. Here’s a look at
their metadata model:
create keyspace.meta_data ( id text,
title text, show_id text, …, …, PRIMARY KEY((id),show_id) ) with
compaction = {‘class’: ‘LeveledCompactionStrategy’ }; In
this model, the ID serves as the partition key. Since this table
experiences relatively few writes (a write occurs only when a new
asset is released) but significantly more reads, they used Leveled
Compaction Strategy to optimize performance. And, according to
Jivesh, “Choosing the right partition and clustering keys helped us
get a single-digit millisecond latency.” Viewership count use case
Viewership Count is another use case that they moved to ScyllaDB.
Viewership count can be tracked per user or per asset ID. ZEE5
decided to design a table where the user ID served as the partition
key and the asset ID as the sort key – allowing viewership data to
be efficiently queried. They set ScyllaDB’s TTL to match the
60-second heartbeat interval, ensuring that data automatically
expires after the designated time. Additionally, they used
ScyllaDB’s Time-Window Compaction Strategy to efficiently manage
data in memory, clearing expired records based on the configured
TTL. Jivesh explained, “This table is continuously updated with
heartbeats from every front end and every user. As heartbeats
arrive, viewership counts are tracked in real time and
automatically cleared when the TTL expires. That lets us
efficiently retrieve live viewership data using ScyllaDB.” Here’s
their viewership count data model:
CREATE TABLE
keyspace.USER_SESSION_STREAM ( USER_ID text, DEVICE_ID text,
ASSET_ID text, TITLE text, …, PRIMARY KEY((USER_ID), ASSET_ID) )
WITH default_time_to_live = 60 and compaction = { 'class' :
'TimeWindowCompactionStrategy' }; ScyllaDB Results and
Lessons Learned The following load test report shows a throughput
of 41.7K requests per second. This benchmark was conducted during
the database selection process to evaluate performance under high
load. Jivesh remarked, “Even with such a high throughput, we could
achieve a microsecond write latency and average microsecond read
latency. This really gave us a clear view of what ScyllaDB could do
– and that helped us decide.” He then continued to share some facts
that shed light on the scale of ZEE5’s ScyllaDB deployment: “We
have around 9TB on ScyllaDB. Even with such a large volume of data,
it’s able to handle latencies within microseconds and a
single-digit millisecond, which is quite tremendous. We have a
daily peak concurrent viewership count of 1 million. Every second,
we are writing so much data into ScyllaDB and getting so much data
out of it We process more than 100 billion heartbeats in a day.
That’s quite huge.” The talk wrapped with the following lessons
learned: Data modeling is the single most critical factor in
achieving single-digit millisecond latencies. Choose the right
quorum setting and compaction strategy. For example, does a
heartbeat need to be written to every node before it can be read,
or is a local quorum sufficient? Selecting the right quorum ensures
the best balance between latency and SLA requirements. Choose
Partition and Clustering Keys wisely – it’s not easy to modify them
later. Use Materialized Views for faster lookups and avoid filter
queries. Querying across partitions can degrade performance. Use
prepared statements to improve efficiency. Use asynchronous queries
for faster query processing. For instance, in the metadata model,
20 synchronous queries were executed in parallel, and ScyllaDB
handled them within milliseconds. Zone-aware ScyllaDB clients help
reduce cross-AZ (Availability Zone) network costs. Fetching data
within the same AZ minimizes latency and significantly reduces
network expenses.
12 February 2025, 7:18 pm by
ScyllaDB
Given that we’re hosting
Monster SCALE Summit…with
tech talks on
extreme-scale engineering…many of
which feature our
monstrously fast and scalable
database, a big announcement is probably expected? We hope
this meets your super-sized expectations. Monster SCALE Summit 2025
will be featuring 60+ tech talks including: Just-added keynotes by
Kelsey Hightower and Rachel Stephens + Adam Jacob Previously-teased
keynotes by Avi Kivity, Martin Kleppmann + Chris Riccomini, Gwen
Shapira, and Dor Laor Engineering talks by gamechangers like Uber,
Slack, Canva, Atlassian, Wise, and Booking.com 14 talks by ScyllaDB
users such as Medium, Disney+, ShareChat, Yieldmo, Clearview AI,
and more – plus two talks by Discord The latest from ScyllaDB
engineering: including object storage, vector search, and “ScyllaDB
X Cloud” Like other ScyllaDB-hosted conferences (e.g.,
P99 CONF), this conference will be
free and virtual so that everyone can participate.
See the agenda and
register – it’s free Mark your calendar for March 11 and 12
because – in addition to all those great talks – you can… Chat
directly with speakers and connect with ~20K of your peers
Participate in some monster scale global distributed system
challenges – with prizes for winners, of course Learn from
ScyllaDB’s top experts, who are eager to answer your toughest
database performance questions in our lively lounge – and preparing
special interactive training courses for the occasion Win
conference swag, sea monster plushies, book bundles, and other cool
giveaways It’s a lot. But hey, it’s Monster SCALE Summit. 🙂
Details, Details Beyond what’s on the agenda, here’s some
additional detail on a few recently-added sessions (see more in our
“tiny peek” blog post) How Discord Performs Database Upgrades
at Scale
Ethan Donowitz, Senior Software Engineer,
Persistence Infrastructure at Discord Database upgrades
are high-risk but high-reward. Upgrading to a newer version can
make your database faster, cheaper, and more reliable; however,
without thorough planning and testing, upgrades can be risky.
Because databases are stateful, it is often not possible to roll
back if you encounter problems after the upgrade due to backwards
incompatible changes across versions. While new versions typically
mean improved query latencies, changes in query planning or cache
behavior across versions can cause unexpected differences in
performance in places one might not expect. Discord relies on
ScyllaDB to serve millions of reads per second across many
clusters, so we needed a comprehensive strategy to sufficiently
de-risk upgrades to avoid impact to our users. To accomplish this,
we use what we call “shadow clusters.” A shadow cluster contains
roughly the same data as its corresponding cluster in production,
and traffic to the primary cluster is mirrored to the shadow
cluster. Running a real production workload on a shadow cluster can
expose differences in performance and resource usage across
versions. When mirroring reads, we also have the ability to perform
“read validations,” where the results for a query issued to the
primary cluster and the shadow cluster are checked for equality.
This gives us confidence that data has not been corrup How Discord
Indexes Trillions of Messages: Scaling Search Infrastructure
Vicki Niu, Senior Software Engineer at Discord
When Discord first built messages search in 2017, we designed our
infrastructure to handle billions of messages sent by millions of
users. As our platform grew to trillions of messages, our search
system failed to keep up. We thus set out to rebuild our message
search platform to meet these new scaling needs using our learnings
and some new technologies. This talk will share how we scaled
Discord’s message search infrastructure using Rust, Kubernetes, and
a multi-cluster Elasticsearch architecture to achieve better
performance, operability, and reliability, while also enabling new
search features for Discord users. ted due to differences in
behavior across versions. Testing with shadow clusters has been
paramount to de-risking complicated upgrades for one of the most
important pieces of infrastructure at Discord. Route It Like It’s
Hot: Scaling Payments Routing at American Express
Benjamin
Cane, Distinguished Engineer at American Express In 2023,
there were over 723 billion credit card transactions. Whenever
someone taps, swipes, dips, or clicks a credit or debit card, a
payment switch ensures the transaction arrives safely and securely
at the correct financial institution.These payment switches are the
backbone of the worldwide payments ecosystem. Join the American
Express Payment Acquiring and Network team as they share their
experiences from building their Global Transaction Router, which is
responsible for switching and routing payments at the scale of
American Express. They will explore how they’ve designed, built,
and operated this Global Transaction Router to perform during
record-breaking shopping holidays, ticket sales, and unexpected
customer behavior. The audience will leave with a deep
understanding of the unique challenges of a payments switch (E.g.,
routing ISO 8583 transactions as fast as possible), some of our
design choices (E.g., using containers and avoiding logging), and a
deep dive into a few implementation challenges (E.g., Inefficient
use of Goroutines and Channels) we found along the way. How Yieldmo
Cut Database Costs and Cloud Dependencies Fast
Todd
Coleman, Chief Architect and Co-founder at Yieldmo
Yieldmo’s business relies on processing hundreds of billions of
daily ad requests with subsecond latency responses. Our services
initially depended on DynamoDB, and we valued its simplicity and
stability. However, DynamoDB costs were becoming unsustainable,
latencies were not ideal, and we sought greater flexibility in
deploying services to other cloud providers. In this session, we’ll
walk you through the various options we considered to address these
challenges and share why and how we ultimately moved forward with
ScyllaDB’s DynamoDB-compatible API.
See more
session details
4 February 2025, 1:19 pm by
ScyllaDB
Let’s focus on the performance-related complexities that
teams commonly face with write-heavy workloads and discuss your
options for tackling them Write-heavy database workloads
bring a distinctly different set of challenges than read-heavy
ones. For example: Scaling writes can be costly, especially if you
pay per operation and writes are 5X more costly than reads Locking
can add delays and reduce throughput I/O bottlenecks can lead to
write amplification and complicate crash recovery Database
backpressure can throttle the incoming load While cost matters –
quite a lot, in many cases – it’s not a topic we want to cover
here. Rather, let’s focus on the performance-releated complexities
that teams commonly face and discuss your options for tackling
them. What Do We Mean by “a Real-Time Write Heavy Workload”? First,
let’s clarify what we mean by a “real-time write-heavy” workload.
We’re talking about workloads that: Ingest a large amount of data
(e.g., over 50K OPS) Involve more writes than reads Are bound by
strict latency SLAs (e.g., single-digit millisecond P99 latency) In
the wild, they occur across everything from online gaming to
real-time stock exchanges. A few specific examples:
Internet of Things (IoT) workloads tend to involve
small but frequent append-only writes of time series data. Here,
the ingestion rate is primarily determined by the number of
endpoints collecting data. Think of smart home sensors or
industrial monitoring equipment constantly sending data streams to
be processed and stored.
Logging and Monitoring
systems also deal with frequent data ingestion, but they
don’t have a fixed ingestion rate. They may not necessarily append
only, as well as may be prone to hotspots, such as when one
endpoint misbehaves.
Online Gaming platforms need
to process real-time user interactions, including game state
changes, player actions, and messaging. The workload tends to be
spiky, with sudden surges in activity. They’re extremely latency
sensitive since even small delays can impact the gaming experience.
E-commerce and Retail workloads are typically
update-heavy and often involve batch processing. These systems must
maintain accurate inventory levels, process customer reviews, track
order status, and manage shopping cart operations, which usually
require reading existing data before making updates.
Ad
Tech and Real-time Bidding systems require split-second
decisions. These systems handle complex bid processing, including
impression tracking and auction results, while simultaneously
monitoring user interactions such as clicks and conversions. They
must also detect fraud in real time and manage sophisticated
audience segmentation for targeted advertising.
Real-time
Stock Exchange systems must support high-frequency trading
operations, constant stock price updates, and complex order
matching processes – all while maintaining absolute data
consistency and minimal latency. Next, let’s look at key
architectural and configuration considerations that impact write
performance. Storage Engine Architecture The choice of storage
engine architecture fundamentally impacts write performance in
databases. Two primary approaches exist: LSM trees and B-Trees.
Databases known to handle writes efficiently – such as ScyllaDB,
Apache Cassandra, HBase, and Google BigTable – use Log-Structured
Merge Trees (LSM). This architecture is ideal for handling large
volumes of writes. Since writes are immediately appended to memory,
this allows for very fast initial storage. Once the “memtable” in
memory fills up, the recent writes are flushed to disk in sorted
order. That reduces the need for random I/O. For example, here’s
what the ScyllaDB write path looks like: With B-tree structures,
each write operation requires locating and modifying a node in the
tree – and that involves both sequential and random I/O. As the
dataset grows, the tree can require additional nodes and
rebalancing, leading to more disk I/O, which can impact
performance. B-trees are generally better suited for workloads
involving joins and ad-hoc queries. Payload Size Payload size also
impacts performance. With small payloads, throughput is good but
CPU processing is the primary bottleneck. As the payload size
increases, you get lower overall throughput and disk utilization
also increases. Ultimately, a small write usually fits in all the
buffers and everything can be processed quite quickly. That’s why
it’s easy to get high throughput. For larger payloads, you need to
allocate larger buffers or multiple buffers. The larger the
payloads, the more resources (network and disk) are required to
service those payloads. Compression Disk utilization is something
to watch closely with a write-heavy workload. Although storage is
continuously becoming cheaper, it’s still not free. Compression can
help keep things in check – so choose your compression strategy
wisely. Faster compression speeds are important for write-heavy
workloads, but also consider your available CPU and memory
resources. Be sure to look at the
compression chunk size parameter. Compression basically splits
your data into smaller blocks (or chunks) and then compresses each
block separately. When tuning this setting, realize that larger
chunks are better for reads while smaller ones are better for
writes, and take your payload size into consideration. Compaction
For LSM-based databases, the compaction strategy you select also
influences write performance. Compaction involves merging multiple
SSTables into fewer, more organized files, to optimize read
performance, reclaim disk space, reduce data fragmentation, and
maintain overall system efficiency. When selecting compaction
strategies, you could aim for low read amplification, which makes
reads as efficient as possible. Or, you could aim for low write
amplification by avoiding compaction from being too aggressive. Or,
you could prioritize low space amplification and have compaction
purge data as efficiently as possible. For example, ScyllaDB offers
several compaction strategies (and Cassandra offers similar
ones):
Size-tiered compaction strategy (STCS):
Triggered when the system has enough (four by default) similarly
sized SSTables.
Leveled compaction strategy (LCS):
The system uses small, fixed-size (by default 160 MB) SSTables
distributed across different levels.
Incremental Compaction
Strategy (ICS): Shares the same read and write
amplification factors as STCS, but it fixes its 2x temporary space
amplification issue by breaking huge sstables into SSTable runs,
which are comprised of a sorted set of smaller (1 GB by default),
non-overlapping SSTables.
Time-window compaction strategy
(TWCS): Designed for time series data. For write-heavy
workloads, we warn users to avoid leveled compaction at all costs.
That strategy is designed for read-heavy use cases. Using it can
result in a regrettable 40x write amplification. Batching In
databases like ScyllaDB and Cassandra, batching can actually be a
bit of a trap – especially for write-heavy workloads. If you’re
used to relational databases, batching might seem like a good
option for handling a high volume of writes. But it can actually
slow things down if it’s not done carefully. Mainly, that’s because
large or unstructured batches end up creating a lot of coordination
and network overhead between nodes. However, that’s really not what
you want in a distributed database like ScyllaDB. Here’s how to
think about batching when you’re dealing with heavy writes:
Batch by the Partition Key: Group your writes by
the partition key so the batch goes to a coordinator node that also
owns the data. That way, the coordinator doesn’t have to reach out
to other nodes for extra data. Instead, it just handles its own,
which cuts down on unnecessary network traffic.
Keep
Batches Small and Targeted: Breaking up large batches into
smaller ones by partition keeps things efficient. It avoids
overloading the network and lets each node work on only the data it
owns. You still get the benefits of batching, but without the
overhead that can bog things down.
Stick to Unlogged
Batches: Considering you follow the earlier points, it’s
best to use unlogged batches. Logged batches add extra consistency
checks, which can really slow down the write. So, if you’re in a
write-heavy situation, structure your batches carefully to avoid
the delays that big, cross-node batches can introduce. Wrapping Up
We offered quite a few warnings, but don’t worry. It was easy to
compile a list of lessons learned because so many teams are
extremely successful working with real-time write-heavy workloads.
Now you know many of their secrets, without having to experience
their mistakes. 🙂 If you want to learn more, here are some
firsthand perspectives from teams who tackled quite interesting
write-heavy challenges:
Zillow: Consuming records from multiple data
producers, which resulted in out-of-order writes that could result
in incorrect updates
Tractian: Preparing for 10X growth in high-frequency
data writes from IoT devices
Fanatics: Heavy write operations like handling orders,
shopping carts, and product updates for this online sports retailer
Also, take a look at the following video, where we go into even
greater depth on these write-heavy challenges and also walk you
through what these workloads look like on ScyllaDB.
30 January 2025, 1:28 pm by
ScyllaDB
See the engineering behind real-time personalization at
Tripadvisor’s massive (and rapidly growing) scale What
kind of traveler are you? Tripadvisor tries to assess this as soon
as you engage with the site, then offer you increasingly relevant
information on every click—within a matter of milliseconds. This
personalization is powered by advanced ML models acting on data
that’s stored on ScyllaDB running on AWS. In this article, Dean
Poulin (Tripadvisor Data Engineering Lead on the AI Service and
Products team) provides a look at how they power this
personalization. Dean shares a taste of the technical challenges
involved in delivering real-time personalization at Tripadvisor’s
massive (and rapidly growing) scale. It’s based on the following
AWS re:Invent talk: Pre-Trip Orientation
In Dean’s words …
Let’s start with a quick snapshot of who Tripadvisor is, and the
scale at which we operate. Founded in 2000, Tripadvisor has become
a global leader in travel and hospitality, helping hundreds of
millions of travelers plan their perfect trips. Tripadvisor
generates over $1.8 billion in revenue and is a publicly traded
company on the NASDAQ stock exchange. Today, we have a talented
team of over 2,800 employees driving innovation, and our platform
serves a staggering 400 million unique visitors per month – a
number that’s continuously growing. On any given day, our system
handles more than 2 billion requests from 25 to 50 million users.
Every click you make on Tripadvisor is processed in real time.
Behind that, we’re leveraging machine learning models to deliver
personalized recommendations – getting you closer to that perfect
trip. At the heart of this personalization engine is ScyllaDB
running on AWS. This allows us to deliver millisecond-latency at a
scale that few organizations reach. At peak traffic, we hit around
425K operations per second on ScyllaDB with P99 latencies
for reads and writes around 1-3 milliseconds. I’ll be
sharing how Tripadvisor is harnessing the power of ScyllaDB, AWS,
and real-time machine learning to deliver personalized
recommendations for every user. We’ll explore how we help travelers
discover everything they need to plan their perfect trip: whether
it’s uncovering hidden gems, must-see attractions, unforgettable
experiences, or the best places to stay and dine. This [article] is
about the engineering behind that – how we deliver seamless,
relevant content to users in real time, helping them find exactly
what they’re looking for as quickly as possible. Personalized Trip
Planning Imagine you’re planning a trip. As soon as you land on the
Tripadvisor homepage, Tripadvisor already knows whether you’re a
foodie, an adventurer, or a beach lover – and you’re seeing spot-on
recommendations that seem personalized to your own interests. How
does that happen within milliseconds?
As you browse around Tripadvisor, we start to personalize what
you see using Machine Learning models which calculate scores based
on your current and prior browsing activity. We recommend hotels
and experiences that we think you would be interested in. We sort
hotels based on your personal preferences. We recommend popular
points of interest near the hotel you’re viewing. These are all
tuned based on your own personal preferences and prior browsing
activity. Tripadvisor’s Model Serving Architecture Tripadvisor runs
on hundreds of independently scalable microservices in Kubernetes
on-prem and in Amazon EKS. Our ML Model Serving Platform is exposed
through one of these microservices.
This gateway service abstracts over 100 ML Models from the
Client Services – which lets us run A/B tests to find the best
models using our experimentation platform. The ML Models are
primarily developed by our Data Scientists and Machine Learning
Engineers using Jupyter Notebooks on Kubeflow. They’re managed and
trained using ML Flow, and we deploy them on Seldon Core in
Kubernetes. Our Custom Feature Store provides features to our ML
Models, enabling them to make accurate predictions The Custom
Feature Store The Feature Store primarily serves User Features and
Static Features. Static Features are stored in Redis because they
don’t change very often. We run data pipelines daily to load data
from our offline data warehouse into our Feature Store as Static
Features.
User Features are served in real time through a platform
called Visitor Platform. We execute dynamic CQL queries against
ScyllaDB, and
we do not need a caching layer because
ScyllaDB is so fast. Our Feature Store serves up to 5
million Static Features per second and half a million User Features
per second. What’s an ML Feature? Features are input variables to
the ML Models that are used to make a prediction. There are Static
Features and User Features. Some examples of Static Features are
awards that a restaurant has won or amenities offered by a hotel
(like free Wi-Fi, pet friendly or fitness center). User Features
are collected in real time as users browse around the site. We
store them in ScyllaDB so we can get lightning fast queries. Some
examples of user features are the hotels viewed over the last 30
minutes, restaurants viewed over the last 24 hours, or reviews
submitted over the last 30 days. The Technologies Powering Visitor
Platform ScyllaDB is at the core of Visitor Platform. We use
Java-based Spring Boot microservices to expose the platform to our
clients. This is deployed on AWS ECS Fargate. We run Apache Spark
on Kubernetes for our daily data retention jobs, our offline to
online jobs. Then we use those jobs to load data from our offline
data warehouse into ScyllaDB so that they’re available on the live
site. We also use Amazon Kinesis for processing streaming user
tracking events. The Visitor Platform Data Flow The following
graphic shows how data flows through our platform in four stages:
produce, ingest, organize, and activate.
Data is produced by our website and our mobile apps. Some of
that data includes our Cross-Device User Identity Graph, Behavior
Tracking events (like page views and clicks) and streaming events
that go through Kinesis. Also, audience segmentation gets loaded
into our platform. Visitor Platform’s microservices are used to
ingest and organize this data. The data in ScyllaDB is stored in
two keyspaces: The Visitor Core keyspace, which contains the
Visitor Identity Graph The Visitor Metric keyspace, which contains
Facts and Metrics (the things that the people did as they browsed
the site) We use daily ETL processes to maintain and clean up the
data in the platform. We produce Data Products, stamped daily, in
our offline data warehouse – where they are available for other
integrations and other data pipelines to use in their processing.
Here’s a look at Visitor Platform by the numbers:
Why Two Databases? Our online database is focused on
the real-time, live website traffic. ScyllaDB fills this role by
providing very low latencies and high throughput. We use short term
TTLs to prevent the data in the online database from growing
indefinitely, and our data retention jobs ensure that we only keep
user activity data for real visitors. Tripadvisor.com gets a lot of
bot traffic, and we don’t want to store their data and try to
personalize bots – so we delete and clean up all that data.
Our offline data warehouse retains historical data used for
reporting, creating other data products, and training our ML
Models. We don’t want large-scale offline data processes impacting
the performance of our live site, so we have two separate databases
used for two different purposes. Visitor Platform Microservices We
use 5 microservices for Visitor Platform:
Visitor
Core manages the cross-device user identity graph based on
cookies and device IDs.
Visitor Metric is our
query engine, and that provides us with the ability for exposing
facts and metrics for specific visitors. We use a domain specific
language called visitor query language, or VQL. This example VQL
lets you see the latest commerce click facts over the last three
hours.
Visitor Publisher and
Visitor
Saver handle the write path, writing data into the
platform. Besides saving data in ScyllaDB, we also stream data to
the offline data warehouse. That’s done with Amazon Kinesis.
Visitor Composite simplifies publishing data in
batch processing jobs. It abstracts Visitor Saver and Visitor Core
to identify visitors and publish facts and metrics in a single API
call. Roundtrip Microservice Latency This graph illustrates how our
microservice latencies remain stable over time.
The average latency is only 2.5 milliseconds, and our P999 is
under 12.5 milliseconds. This is impressive performance, especially
given that we handle over 1 billion requests per day. Our
microservice clients have strict latency requirements. 95% of the
calls must complete in 12 milliseconds or less. If they go over
that, then we will get paged and have to find out what’s impacting
the latencies. ScyllaDB Latency Here’s a snapshot of ScyllaDB’s
performance over three days.
At peak, ScyllaDB is handling 340,000 operations per second
(including writes and reads and deletes) and the CPU is hovering at
just 21%. This is high scale in action! ScyllaDB delivers
microsecond writes and millisecond reads for us. This level of
blazing fast performance is exactly why we chose ScyllaDB.
Partitioning Data into ScyllaDB This image shows how we partition
data into ScyllaDB.
The Visitor Metric Keyspace has two tables: Fact and Raw
Metrics. The primary key on the Fact table is Visitor GUID, Fact
Type, and Created At Date. The composite partition key is the
Visitor GUID and Fact Type. The clustering key is Created At Date,
which allows us to sort data in partitions by date. The attributes
column contains a JSON object representing the event that occurred
there. Some example Facts are Search Terms, Page Views, and
Bookings. We use ScyllaDB’s Leveled Compaction Strategy because:
It’s optimized for range queries It handles high cardinality very
well It’s better for read-heavy workloads, and we have about 2-3X
more reads than writes Why ScyllaDB? Our solution was originally
built using Cassandra on-prem. But as the scale increased, so did
the operational burden. It required dedicated operations support in
order for us to manage the database upgrades, backups, etc. Also,
our solution requires very low latencies for core components. Our
User Identity Management system must identify the user within 30
milliseconds – and for the best personalization, we require our
Event Tracking platform to respond in 40 milliseconds. It’s
critical that our solution doesn’t block rendering the page so our
SLAs are very low. With Cassandra, we had impacts to performance
from garbage collection. That was primarily impacting the tail
latencies, the P999 and P9999 latencies.
We ran a Proof of Concept with ScyllaDB and found the
throughput to be much better than Cassandra and the operational
burden was eliminated. ScyllaDB gave us a monstrously fast live
serving database with the lowest possible latencies. We wanted a
fully-managed option, so we migrated from Cassandra to ScyllaDB
Cloud, following a dual write strategy. That allowed us to migrate
with zero downtime while handling 40,000 operations or requests per
second. Later, we migrated from ScyllaDB Cloud to ScyllaDB’s “Bring
your own account” model, where you can have the ScyllaDB team
deploy the ScyllaDB database into your own AWS account. This gave
us improved performance as well as better data privacy. This
diagram shows what ScyllaDB’s BYOA deployment looks like.
In the center of the diagram, you can see a 6-node ScyllaDB
cluster that is running on EC2. And then there’s two additional EC2
instances. ScyllaDB Monitor gives us Grafana dashboards as well as
Prometheus metrics. ScyllaDB Manager takes care of infrastructure
automation like triggering backups and repairs. With this
deployment, ScyllaDB could be co-located very close to our
microservices to give us even lower latencies as well as much
higher throughput and performance. Wrapping up, I hope you now have
a better understanding of our architecture, the technologies that
power the platform, and how ScyllaDB plays a critical role in
allowing us to handle Tripadvisor’s extremely high scale.
28 January 2025, 1:55 pm by
ScyllaDB
Build a shopping cart app with ScyllaDB– and learn how to
use ScyllaDB’s Change Data Capture (CDC) feature to query and
export the history of all changes made to the tables.This
blog post showcases one of ScyllaDB’s sample applications: a
shopping cart
app. The project uses
FastAPI as the backend
framework and ScyllaDB as the database. By cloning the
repository and
running the application, you can explore an example of an API
server built on top of ScyllaDB for a CRUD app. Additionally,
you’ll see how to use ScyllaDB’s
Change Data Capture (CDC) feature to query and export the
history of all changes made to the tables.What’s inside the
shopping cart sample app?The application has two components: an API
server and a database.API server: Python + FastAPIThe backend is
built with Python and FastAPI, a modern Python web framework known
for its speed and ease of use. FastAPI ensures that you have a
framework that can deliver relatively high performance if used with
the right database. At the same time, due to its exceptional
developer experience, you can easily understand the code of the
project and how it works even if you’ve never used it before.The
application exposes multiple API endpoints to perform essential
operations like:Adding products to the cartRemoving products from
the cartUploading new productsUpdating product information (e.g.
price)Database: ScyllaDBAt the core of this application is
ScyllaDB, a low-latency NoSQL database that provides predictable
performance. ScyllaDB excels in handling large volumes of data with
single-digit millisecond latency, making it ideal for large-scale
real-time applications.ScyllaDB acts as the foundation for a
high-performance low-latency app. Moreover, it has additional
capabilities that can help you maintain low p99 latency as well as
analyze user behavior. ScyllaDB’s CDC feature tracks changes in the
database and you can query historical operations. For e-commerce
applications, this means you can capture insights into user
behavior:What products are being added or removed from the cart and
when?How do users interact with the cart?What does a typical
journey look like for a user who actually buys something?These and
other insights are invaluable for personalizing the user
experience, optimizing the buying journey, and increasing
conversion rates.Using ScyllaDB for an ecommerce applicationAs
studies have shown, low latency is critical for achieving high
conversion rates and delivering a smooth user experience. For
instance, shopping cart operations – such as adding, updating, and
retrieving products – require high performance to prevent cart
abandonment.Data modeling, being the foundation for
high-performance web applications, must remain a top priority. So
let’s start with the process of creating a performant data
model.Design a Shopping Cart data modelWe emphasize a practical
“query-first” approach to NoSQL data modeling: start with your
application’s queries, then design your schema around them. This
method ensures your data model is optimized for your specific use
cases and the database can provide a reliable and single-digit p99
latency at any scale.Let’s review the specific CRUD operations and
queries a typical shopping cart application performs.ProductsList,
add, edit and remove products.GET /products?limit=?SELECT * FROM
product LIMIT {limit}GET /products/{product_id}SELECT * FROM
product WHERE id = ?POST /productsINSERT INTO product () values
()PUT /products/{product_id}UPDATE product SET ? WHERE id = ?DELETE
/products/{product_id}DELETE FROM product WHERE id = ?Based on
these requirements, you can create a table to store products. You
can notice what value is often used to filter products: product id.
This is a good indicator that product id should be the partition
key or at least part of it.The Product table:Our application is
simple, so a single column will suffice as the partition key.
However, if your use case requires additional queries and filtering
by additional columns, you can consider using a composite partition
key or adding a
clustering key to the table.CartList, add, remove products from
user’s cart.GET /cart/{user_id}SELECT * FROM cart_items WHERE
user_id = ? AND cart_id = ?POST /cart/{user_id}INSERT INTO cart()
VALUES ()Here we don’t need cart id because the user can only have
one active cart at a time. (You could also build another endpoint
to list past purchases by the user – that endpoint would require
the cart id as well)DELETE /cart/{user_id}DELETE FROM cart_items
WHERE user_id = ? AND cart_id = ? AND product_id = ?POST checkout
/cart/{user_id}/checkoutUPDATE cart SET is_active = false WHERE
user_id = ? AND cart_id = ?The cart-related operations contain a
slightly more complicated logic behind the scenes. We have two
values that we use to query by: user id and cart id. Those can be
used together as composite partition keys.Additionally, one user
can have multiple carts – one they’re using right now to shop and
possibly other ones that they had in the past that they already
paid for. For this reason, we need to have a way to efficiently
find the user’s active cart. This query requirement will be handled
by a
secondary index on the is_active column.The Cart
table:Additionally, we also need to create a table which connects
the Product and Cart tables. Without this table, it would be
impossible to retrieve products from a cart.The Cart_items table:We
enable Change Data Capture for this table. This feature logs all
data operations performed on the table into another table,
cart_items_scylla_cdc_log. Later, we can query this log to retrieve
the table’s historical operations. This data can be used to analyze
user behavior, such as the products users add or remove from their
carts.Final database schema:Now that we’ve covered the data
modeling aspect of the project, you can
clone the
repository and get started with building.Getting
startedPrerequisites:Python 3.8+ScyllaDB cluster (with
ScyllaDB Cloud or
use Docker)Connect to your ScyllaDB cluster using
CQLSH and create the
schema:Then,
install the Python requirements in a new environment:Modify
config.py to match your database credentials:Run the
server:Generate sample user data:This script populates your
ScyllaDB tables with sample data. This is necessary for the next
step, where you will run CDC queries to analyze user
behavior.Analyze user behavior with CDCCDC records every data
change, including deletes, offering a comprehensive view of your
data evolution without affecting database performance. For a
shopping cart application, some potential use cases for CDC
include:Analyzing a specific user’s buying behaviorTracking user
actions leading to checkoutEvaluating product popularity and
purchase frequencyAnalyzing active and abandoned cartsBeyond these
business-specific insights, CDC data can also be exported to
external platforms, such as
Kafka, for further processing and analysis.Here are a couple of
useful tips when working with CDC:The CDC log table contains
timeuuid values, which can be converted to readable timestamps
using the toTimestamp() CQL function.The cdc$operation column helps
filter operations by type. For instance, a value of 2 indicates an
INSERT
operation.The most efficient and scalable way to query CDC data
is to use the
ScyllaDB
source connector and set up an
integration with Kafka.Now, let’s explore a couple of quick
questions that CDC can help answer.How many times did users add
more than 2 of the same product to the cart?How many carts contain
a particular product?Set up ScyllaDB CDC with Kafka ConnectTo
provide a scalable way for you to analyze ScyllaDB CDC logs, you
can use Kafka to receive messages sent by ScyllaDB. Then, you can
use an analytics tool, like Elasticsearch, to get insights. To send
CDC logs to Kafka, you need to install the
ScyllaDB CDC source connector, and create a new ScyllaDB
connection in Kafka Connect.Install the ScyllaDB source connector
on the machine/container that’s running Kafka:Then use the
following ScyllaDB related parameters when you create the
connection:Make sure to enable CDC on each table you want to send
messages from. You can do this by executing the following CQL:Try
it out yourselfIf you are interested in trying out this application
yourself, check out the dedicated documentation site:
shopping-cart.scylladb.com
and the
GitHub
repository.If you have any questions about this project or
ScyllaDB, submit
a question
in ScyllaDB the forum.
22 January 2025, 1:09 pm by
ScyllaDB
Big things have been happening behind the scenes for the premier
Monster SCALE Summit. Ever since we introduced it at P99 CONF, the
community response has been overwhelming. We’re now faced with the
“good” problem of determining how to fit all the selected speakers
into the two half-days we set aside for the event. 😅 If you missed
the intro last year, Monster Scale Summit is a highly technical
conference that connects the community of professionals designing,
implementing, and optimizing performance-sensitive data-intensive
applications. It focuses on exploring “monster scale” engineering
challenges with respect to extreme levels of throughput, data, and
global distribution. The two-day event is free, intentionally
virtual, and highly interactive.
Register – it’s
free and virtual We’ll be announcing the agenda next month. But
we’re so excited about the speaker lineup that we can’t wait to
share a taste of what you can expect. Here’s a preview of 12 of the
60+ sessions that you can join on March 11 and 12… Designing
Data-Intensive Applications in 2025
Martin Kleppmann and
Chris Riccomini (Designing Data-Intensive Applications
book) Join us for an informal chat with Martin Kleppmann
and Chris Riccomini, who are currently revising the famous book
Designing Data-Intensive Applications. We’ll cover how
data-intensive applications have evolved since the book was first
published, the top tradeoffs people are negotiating today, and what
they believe is next for data-intensive applications. Martin and
Chris will also provide an inside look at the book writing and
revision process. The Nile Approach: Re-engineering Postgres for
Millions of Tenants
Gwen Shapira (Nile) Scaling
relational databases is a notoriously challenging problem. Doing so
while maintaining consistent low latency, efficient use of
resources and compatibility with Postgres may seem impossible. At
Nile, we decided to tackle the scaling challenge by focusing on
multi-tenant applications. These applications require not only
scalability, but also a way to isolate tenants and avoid the noisy
neighbor problem. By tackling both challenges, we developed an
approach, which we call “virtual tenant databases”, which gives us
an efficient way to scale Postgres to millions of tenants while
still maintaining consistent performance. In this talk, I’ll
explore the limitations of traditional scaling for multi-tenant
applications and share how Nile’s virtual tenant databases address
these challenges. By combining the best of Postgres existing
capabilities, distributed algorithms and a new storage layer, Nile
re-engineered Postgres for multi-tenant applications at scale. The
Mechanics of Scale
Dominik Tornow (Resonate HQ) As
distributed systems scale, the complexity of their development and
operation skyrockets. A dependable understanding of the mechanics
of distributed systems is our most reliable parachute. In this
talk, we’ll use systems thinking to develop an accurate and concise
mental model of concurrent, distributed systems, their core
challenges, and the key principles to address these challenges.
We’ll explore foundational problems such as the tension between
consistency and availability, and essential techniques like
partitioning and replication. Whether you are building a new system
from scratch or scaling an existing system to new heights, this
talk will provide the understanding to confidently navigate the
intricacies of modern, large-scale distributed systems. Feature
Store Evolution Under Cost Constraints: When Cost is Part of the
Architecture
Ivan Burmistrov and David Malinge
(ShareChat) At P99 CONF 23, the ShareChat team presented
the scaling challenges for the ML Feature Store so it could handle
1 billion features per second. Once the system was scaled to handle
the load, the next challenge the team faced was extreme cost
constraints: it was required to make the same quality system much
cheaper to run. Ivan and David will talk about approaches the team
implemented in order to optimize for cost in the Cloud environment
while maintaining the same SLA for the service. The talk will touch
on such topics as advanced optimizations on various levels to bring
down the compute, minimizing the waste when running on Kubernetes,
autoscaling challenges for stateful Apache Flink jobs, and others.
The talk should be useful for those who are either interested in
building or optimizing an ML Feature Store or in general looking
into cost optimizations in the cloud environment. Time Travelling
at Scale
Richard Hart (Antithesis) Antithesis is a
continuous reliability platform that autonomously searches for
problems in your software within a simulated environment. Every
problem we find can be perfectly reproduced, allowing for efficient
debugging of even the most complex problems. But storing and
querying histories of program execution at scale creates monster
large cardinalities. Over a ~10 hour test run, we generate ~1bn
rows. The solution: our own tree-database. 30B Images and Counting:
Scaling Canva’s Content-Understanding Pipelines
Dr. Kerry
Halupka (Canva) As the demand for high-quality, labeled
image data grows, building systems that can scale content
understanding while delivering real-time performance is a
formidable challenge. In this talk, I’ll share how we tackled the
complexities of scaling content understanding pipelines to support
monstrous volumes of data, including backfilling labels for over 30
billion images. At the heart of our system is an extreme label
classification model capable of handling thousands of labels and
scaling seamlessly to thousands more. I’ll dive into the core
components: candidate image search, zero-shot labelling using
highly trained teacher models, and iterative refinement with visual
critic models. You’ll learn how we balanced latency, throughput,
and accuracy while managing evolving datasets and continuously
expanding label sets. I’ll also discuss the tradeoffs we faced—such
as ensuring precision in labelling without compromising speed—and
the techniques we employed to optimise for scale, including
strategies to address data sparsity and performance bottlenecks. By
the end of the session, you’ll gain insights into designing,
implementing, and scaling content understanding systems that meet
extreme demands. Whether you’re working with real-time systems,
distributed architectures, or ML pipelines, this talk will provide
actionable takeaways for pushing large-scale labelling pipelines to
their limits and beyond. How Agoda Scaled 50x Throughput with
ScyllaDB
Worakarn Isaratham (Agoda) In this talk,
we will explore the performance tuning strategies implemented at
Agoda to optimize ScyllaDB. Key topics include enhancing disk
performance, selecting the appropriate compaction strategy, and
adjusting SSTable settings to match our usage profile. Who Needs
One Database Anyway?
Glauber Costa (Turso)
Developers need databases. That’s how you store your data. And
that’s usually how it goes: you have your large fleet of services,
and they connect to one database. But what if it wasn’t like that?
What if instead of one database, one application would create one
million databases, or even more? In this talk, we’ll explore the
market trends that give rise to use cases where this pattern is
beneficial, and the infrastructure changes needed to support it.
How We Boosted ScyllaDB’s Data Streaming by 25x
Asias He
(ScyllaDB) To improve elasticity, we wanted to speed up
streaming, the process of scaling out/in to other nodes used to
analyze every partition. Enter file-based streaming, a new
feature that optimizes tablet movement. This new approach streams
the entire SSTable files without deserializing SSTable files into
mutation fragments and re-serializing them back into SSTables on
receiving nodes. As a result, significantly less data is streamed
over the network, and less CPU is consumed – especially for
data models that contain small cells. This session will share the
engineering behind this optimization and look at the performance
impact you can expect in common situations. Evolving Atlassian
Confluence Cloud for Scale, Reliability, and Performance
Bhakti Mehta (Atlassian) This session covers the
journey of Confluence Cloud – the team workspace for collaboration
and knowledge sharing used by thousands of companies – and how we
aim to take it to the next level, with scale, performance, and
reliability as the key motivators. This session presents a deep
dive to provide insights into how the Confluence architecture has
evolved into its current form. It discusses how Atlassian deploys,
runs, and operates at scale and all challenges encountered along
the way. I will cover performance and reliability at scale starting
with the fundamentals of measuring everything, re-defining metrics
to be insightful of actual customer pain, auditing end-to-end
experiences. Beyond just dev-ops and best practices, this means
empowering teams to own product stability through practices and
tools. Two Leading Approaches to Data Virtualization: Which Scales
Better?
Dr. Daniel Abadi (University of Maryland)
You have a large dataset stored in location X, and some code to
process or analyze it in location Y. What is better: move the code
to the data, or move the data to the code? For decades, it has
always been assumed that the former approach is more scalable.
Recently, with the rise of cloud computing, and the push to
separate resources for storage and compute, we have seen data
increasingly being pushed to code, flying in face of conventional
wisdom. What is behind this trend, and is it a dangerous idea? This
session will look at this question from academic and practical
perspectives, with a particular focus on data virtualization, where
there exists an ongoing debate on the merits of push-based vs.
pull-based data processing. Scaling a Beast: Lessons from 400x
Growth in a High-Stakes Financial System
Dmytro Hnatiuk
(Wise) Scaling a system from 66 million to over 25 billion
records is no easy feat—especially when it’s a core financial
system where every number has to be right, and data needs to be
fresh right now. In this session, I’ll share the ups and downs of
managing this kind of growth without losing my sanity. You’ll learn
how to balance high data accuracy with real-time performance,
optimize your app logic, and avoid the usual traps of database
scaling. This isn’t about turning you into a database expert—it’s
about giving you the practical, no-BS strategies you need to scale
your systems without getting overwhelmed by technical headaches.
Perfect for engineers and architects who want to tackle big
challenges and come out on top.
14 January 2025, 9:55 am by
ScyllaDB
How a team of just two engineers tackled real-time
persisted events for hundreds of millions of players With
just two engineers, Supercell took on the daunting task of growing
their basic account system into a social platform connecting
hundreds of millions of gamers. Account management, friend
requests, cross-game promotions, chat, player presence tracking,
and team formation – all of this had to work across their five
major games. And they wanted it all to be covered by a single
solution that was simple enough for a single engineer to maintain,
yet powerful enough to handle massive demand in real-time.
Supercell’s Server Engineer, Edvard Fagerholm, recently shared how
their mighty team of two tackled this task. Read on to learn how
they transformed a simple account management tool into a
comprehensive cross-game social network infrastructure that
prioritized both operational simplicity and high performance.
Note: If you enjoy hearing about engineering
feats like this, join us at Monster Scale
Summit (free + virtual). Engineers from Disney+/Hulu,, Slack,
Canva, Uber, Salesforce, Atlassian and more will be sharing
strategies and case studies. Background: Who’s Supercell?
Supercell is the Finland-based company behind the hit games Hay
Day, Clash of Clans, Boom Beach, Clash Royale and Brawl Stars. Each
of these games has generated $1B in lifetime revenue.
Somehow they manage to achieve this with a super small staff. Until
quite recently, all the account management functionality for games
servicing hundreds of millions of monthly active users was being
built and managed by just two engineers. And that brings us to
Supercell ID. The Genesis of Supercell ID Supercell ID was born as
a basic account system – something to help users recover accounts
and move them to new devices. It was originally implemented as a
relatively simple HTTP API. Edvard explained, “The client could
perform HTTP queries to the account API, which mainly returned
signed tokens that the client could present to the game server to
prove their identity. Some operations, like making friend requests,
required the account API to send a notification to another player.
For example, ‘Do you approve this friend request?’ For that
purpose, there was an event queue for notifications. We would post
the event there, and the game backend would forward the
notification to the client using the game socket.” Enter Two-Way
Communication After Edvard joined the Supercell ID project in late
2020, he started working on the notification backend – mainly for
cross-promotion across their five games. He soon realized that they
needed to implement two-way communication themselves, and built it
as follows: Clients connected to a fleet of proxy servers, then a
routing mechanism pushed events directly to clients (without going
through the game). This was sufficient for the immediate goal of
handling cross-promotion and friend requests. It was fairly simple
and didn’t need to support high throughput or low latency. But it
got them thinking bigger. They realized they could use two-way
communication to significantly increase the scope of the Supercell
ID system. Edvard explained, “Basically, it allowed us to implement
features that were previously part of the game server. Our goal was
to take features that any new games under development might need
and package them into our system – thereby accelerating their
development.” With that, Supercell ID began transforming into a
cross-game social network that supported features like friend
graphs, teaming up, chat, and friend state tracking. Evolving
Supercell ID into Cross-Game Social Network At this point, the
Social Network side of the backend was still a single-person
project, so they designed it with simplicity in mind. Enter
abstraction. Finding the right abstraction “We wanted to have only
one simple abstraction that would support all of our uses and could
therefore be designed and implemented by a single engineer,”
explained Edvard. “In other words, we wanted to avoid building a
chat system, a presence system, etc. We wanted to build one thing,
not many.” Finding the right abstraction was key. And a
hierarchical key-value store with Change Data Capture fit the bill
perfectly. Here’s how they implemented it: The top-level keys in
the key-value store are topics that can be subscribed to. There’s a
two-layer map under each top-level key –
map(string,
map(string, string)). Any change to the data under a
top-level key is broadcast to all that key’s subscribers. The
values in the innermost map are also timestamped. Each data source
controls its own timestamps and defines the correct order. The
client drops any update with an older timestamp than what it
already has stored. A typical change in the data would be something
like ‘level equals 10’ changes to ‘level equals 11’. As players
play, they trigger all sorts of updates like this, which means a
lot of small writes are involved in persisting all the events.
Finding the Right Database They needed a database that would
support their technical requirements and be manageable, given their
minimalist team. That translated to the following criteria: Handles
many small writes with low latency Supports a hierarchical data
model Manages backups and cluster operations as a service ScyllaDB
Cloud turned out to be a great fit. (ScyllaDB Cloud is the
fully-managed version of ScyllaDB, a database known for delivering
predictable low latency at scale). How it All Plays Out For an idea
of how this plays out in Supercell games, let’s look at two
examples. First, consider chat messages. A simple chat message
might be represented in their data model as follows: <room
ID> -> <timestamp_uuid> -> message -> “hi there”
metadata
-> …
reactions
-> … Edvard explained, “The top-level key that’s subscribed to
is the chat room ID. The next level key is a timestamp-UID, so we
have an ordering of each message and can query chat history. The
inner map contains the actual message together with other data
attached to it.” Next, let’s look at “presence”, which is used
heavily in Supercell’s new (and highly anticipated) game, mo.co.
The goal of presence, according to Edvard: “When teaming up for
battle, you want to see in real-time the avatar and the current
build of your friends – basically the weapons and equipment of your
friends, as well as what they’re doing. If your friend changes
their avatar or build, goes offline, or comes online, it should
instantly be visible in the ‘teaming up’ menu.” Players’ state data
is encoded into Supercell’s hierarchical map as follows: <player
ID> -> “presence” -> weapon -> sword
level
-> 29
status
-> in battle Note that: The top level is the player ID, the
second level is the type, and the inner map contains the data.
Supercell ID doesn’t need to understand the data; it just forwards
it to the game clients. Game clients don’t need to know the friend
graph since the routing is handled by Supercell ID. Deeper into the
System Architecture Let’s close with a tour of the system
architecture, as provided by Edvard. “The backend is split into
APIs, proxies, and event routing/storage servers. Topics live on
the event routing servers and are sharded across them. A client
connects to a proxy, which handles the client’s topic subscription.
The proxy routes these subscriptions to the appropriate event
routing servers. Endpoints (e.g., for chat and presence) send their
data to the event routing servers, and all events are persisted in
ScyllaDB Cloud. Each topic has a primary and backup shard. If the
primary goes down, the primary shard maintains the memory sequence
numbers for each message to detect lost messages. The secondary
will forward messages without sequence numbers. If the primary is
down, the primary coming up will trigger a refresh of state on the
client, as well as resetting the sequence numbers. The API for the
routing layers is a simple post-event RPC containing a batch of
topic, type, key, value tuples. The job of each API is just to
rewrite their data into the above tuple representation. Every event
is written in ScyllaDB before broadcasting to subscribers. Our APIs
are synchronous in the sense that if an API call gives a successful
response, the message was persisted in ScyllaDB. Sending the same
event multiple times does no harm since applying the update on the
client is an idempotent operation, with the exception of possibly
multiple sequence numbers mapping to the same message. When
connecting, the proxy will figure out all your friends and
subscribe to their topics, same for chat groups you belong to. We
also subscribe to topics for the connecting client. These are used
for sending notifications to the client, like friend requests and
cross promotions. A router reboot triggers a resubscription to
topics from the proxy. We use Protocol Buffers to save on bandwidth
cost. All load balancing is at the TCP level to guarantee that
requests over the same HTTP/2 connection are handled by the same
TCP socket on the proxy. This lets us cache certain information in
memory on the initial listen, so we don’t need to refetch on other
requests. We have enough concurrent clients that we don’t need to
separately load balance individual HTTP/2 requests, as traffic is
evenly distributed anyway, and requests are about equally expensive
to handle across different users. We use persistent sockets between
proxies and routers. This way, we can easily send tens of thousands
of subscriptions per second to a single router without an issue.”
But It’s Not Game Over If you want to watch the complete tech talk,
just press play below: And if you want to read more about
ScyllaDB’s role in the gaming world, you might also want to read:
Epic Games: How Epic Games uses ScyllaDB as a
binary cache in front of NVMe and S3 to accelerate global
distribution of large game assets used by Unreal Cloud DDC.
Tencent Games: How Tencent Games built service
architecture based on CQRS and event sourcing patterns with Pulsar
and ScyllaDB.
Discord: How Discord uses ScyllaDB to power
their massive growth, moving from a niche gaming platform to one of
the world’s largest communication platforms.
