Books by Monster SCALE Summit 25 Speakers: Distributed Data Systems & Beyond

Monster SCALE Summit speakers have amassed a rather impressive list of publications, including quite a few books. This blog highlights 10+ of them. If you’ve seen the Monster SCALE Summit agenda, you know that the stars have aligned nicely. In just two half days, from anywhere you like, you can learn from 60+ outstanding speakers – all exploring extreme scale engineering challenges from a variety of angles. Distributed databases, event streaming, AI/ML, Kubernetes, Rust…it’s all on the agenda. If you read the bios of our speakers, you’ll note that many have written books. This blog highlights eleven of those Monster SCALE Summit speakers’ books – plus two new books by past conference speakers. Once you register for the conference (it’s free + virtual), you’ll gain 30-day full access to the complete O’Reilly library (thanks to O’Reilly, a conference media sponsor). And Manning Publications is also a media sponsor. They are offering the Monster SCALE community a nice 50% discount on all Manning books . One more bonus: conference attendees who participate in the speaker chat will be eligible to win book bundles, courtesy of Manning. See the agenda and register – it’s free Designing Data-Intensive Applications, 2nd Edition By Martin Kleppmann and Chris Riccomini O’Reilly ETA: December 2025 Data is at the center of many challenges in system design today. Difficult issues such as scalability, consistency, reliability, efficiency, and maintainability need to be resolved. In addition, there’s an overwhelming variety of tools and analytical systems, including relational databases, NoSQL datastores, plus data warehouses and data lakes. What are the right choices for your application? How do you make sense of all these buzzwords? In this second edition, authors Martin Kleppmann and Chris Riccomini build on the foundation laid in the acclaimed first edition, integrating new technologies and emerging trends. You’ll be guided through the maze of decisions and trade-offs involved in building a modern data system, from choosing the right tools like Spark and Flink to understanding the intricacies of data laws like the GDPR. Peer under the hood of the systems you already use, and learn to use them more effectively Make informed decisions by identifying the strengths and weaknesses of different tools Navigate the trade-offs around consistency, scalability, fault tolerance, and complexity Understand the distributed systems research upon which modern databases are built Peek behind the scenes of major online services, and learn from their architectures Martin and Chris are presenting “Designing Data-Intensive Applications in 2025”   Think Distributed Systems Dominik Tornow ETA: Fall 2025 Manning (use code SCALE2025 for 50% off) All modern software is distributed. Let’s say that again—all modern software is distributed. Whether you’re building mobile utilities, microservices, or massive cloud native enterprise applications, creating efficient distributed systems requires you to think differently about failure, performance, network services, resource usage, latency, and much more. This clearly-written book guides you into the mindset you’ll need to design, develop, and deploy scalable and reliable distributed systems. In Think Distributed Systems you’ll find a beautifully illustrated collection of mental models for: Correctness, scalability, and reliability Failure tolerance, detection, and mitigation Message processing Partitioning and replication Consensus Dominik is presenting “The Mechanics of Scale”   Latency: Reduce Delay in Software Systems Pekka Enberg ETA: Summer 2025 Manning (use code SCALE2025 for 50% off) Slow responses can kill good software. Whether it’s recovering microseconds lost while routing messages on a server or speeding up page loads that keep users waiting, finding and fixing latency can be a frustrating part of your work as a developer. This one-of-a-kind book shows you how to spot, understand, and respond to latency wherever it appears in your applications and infrastructure. This book balances theory with practical implementations, turning academic research into useful techniques you can apply to your projects. In Latency you’ll learn: What latency is—and what it is not How to model and measure latency Organizing your application data for low latency Making your code run faster Hiding latency when you can’t reduce it Pekka presented “Patterns of Low Latency” at P99 CONF 2024. And his Turso co-founder Glauber Costa will be presenting “Who Needs One Database Anyway?” at Monster SCALE Summit   Writing for Developers: Blogs That Get Read By Piotr Sarna and Cynthia Dunlop January 2025 Amazon | Manning (use code SCALE2025 for 50% off) This book is a practical guide to writing more compelling engineering blog posts. We discuss strategies for nailing all phases of the technical blogging process. And we have quite a bit of fun exploring the core blog post patterns that are most common across engineering blogs today, like “The Bug Hunt,” “How We Built It,” “Lessons Learned,” “We Rewrote It in X,” “Thoughts on Trends,” etc. Each “pattern” chapter includes an analysis of real-world examples as well as specific dos/don’ts for that particular pattern. There’s a section on moving from blogging into opportunities such as article writing, conference speaking, and book writing. Finally, we wrap with a critical (and often amusing) look at generative AI blogging uses and abuses. Oh…and there’s also a foreword by Bryan Cantrill and an afterword by Scott Hanselman! Readers will learn how to: Pinpoint topics that make intriguing posts Apply popular blog post design patterns Rapidly plan, draft, and optimize blog posts Make your content clearer and more convincing to technical readers Tap AI for revision while avoiding misuses and abuses Increase the impact of all your technical communications Piotr is presenting “A Dist Sys Programmer’s Journey Into AI”   ScyllaDB in Action Bo Ingram October 2024 Amazon | Manning  (use code SCALE2025 for 50% off) | ScyllaDB (free chapters) ScyllaDB in Action is your guide to everything you need to know about ScyllaDB, from your very first queries to running it in a production environment. It starts you with the basics of creating, reading, and deleting data and expands your knowledge from there. You’ll soon have mastered everything you need to build, maintain, and run an effective and efficient database. This book teaches you ScyllaDB the best way—through hands-on examples. Dive into the node-based architecture of ScyllaDB to understand how its distributed systems work, how you can troubleshoot problems, and how you can constantly improve performance.You’ll learn how to: • Read, write, and delete data in ScyllaDB • Design database schemas for ScyllaDB • Write performant queries against ScyllaDB • Connect and query a ScyllaDB cluster from an application • Configure, monitor, and operate ScyllaDB in production Bo’s colleagues Ethan Donowitz and Vicki Niu are both presenting at Monster SCALE Summit   Data Virtualization in the Cloud Era Dr. Daniel Abadi and Andrew Mott July 2024 O’Reilly Data virtualization had been held back by complexity for decades until recent advances in cloud technology, data lakes, networking hardware, and machine learning transformed the dream into reality. It’s becoming increasingly practical to access data through an interface that hides low-level details about where it’s stored, how it’s organized, and which systems are needed to manipulate or process it. You can combine and query data from anywhere and leave the complex details behind. In this practical book, authors Dr. Daniel Abadi and Andrew Mott discuss in detail what data virtualization is and the trends in technology that are making data virtualization increasingly useful. With this book, data engineers, data architects, and data scientists will explore the architecture of modern data virtualization systems and learn how these systems differ from one another at technical and practical levels. By the end of the book, you’ll understand: The architecture of data virtualization systems Technical and practical ways that data virtualization systems differ from one another Where data virtualization fits into modern data mesh and data fabric paradigms Modern best practices and case study use cases Daniel is presenting “Two Leading Approaches to Data Virtualization: Which Scales Better?” Bonus: Read Daniel Abadi’s article on the PACELC theorem.   Database Performance at Scale By Felipe Cardeneti Mendes, Piotr Sarna, Pavel Emelyanov, and Cynthia Dunlop October 2023 Amazon | ScyllaDB (free) Discover critical considerations and best practices for improving database performance based on what has worked, and failed, across thousands of teams and use cases in the field. This book provides practical guidance for understanding the database-related opportunities, trade-offs, and traps you might encounter while trying to optimize data-intensive applications for high throughput and low latency. Whether you’re building a new system from the ground up or trying to optimize an existing use case for increased demand, this book covers the essentials. The ultimate goal of the book is to help you discover new ways to optimize database performance for your team’s specific use cases, requirements, and expectations. Understand often overlooked factors that impact database performance at scale Recognize data-related performance and scalability challenges associated with your project Select a database architecture that’s suited to your workloads, use cases, and requirements Avoid common mistakes that could impede your long-term agility and growth Jumpstart teamwide adoption of best practices for optimizing database performance at scale Felipe is presenting “ScyllaDB is No Longer “Just a Faster Cassandra” Piotr is presenting “A Dist Sys Programmer’s Journey Into AI”   Algorithms and Data Structures for Massive Datasets Dzejla Medjedovic, Emin Tahirovic, and Ines Dedovic May 2022 Amazon | Manning (use code SCALE2025 for 50% off) Algorithms and Data Structures for Massive Datasets reveals a toolbox of new methods that are perfect for handling modern big data applications. You’ll explore the novel data structures and algorithms that underpin Google, Facebook, and other enterprise applications that work with truly massive amounts of data. These effective techniques can be applied to any discipline, from finance to text analysis. Graphics, illustrations, and hands-on industry examples make complex ideas practical to implement in your projects—and there’s no mathematical proofs to puzzle over. Work through this one-of-a-kind guide, and you’ll find the sweet spot of saving space without sacrificing your data’s accuracy. Readers will learn: Probabilistic sketching data structures for practical problems Choosing the right database engine for your application Evaluating and designing efficient on-disk data structures and algorithms Understanding the algorithmic trade-offs involved in massive-scale systems Deriving basic statistics from streaming data Correctly sampling streaming data Computing percentiles with limited space resources Dzejla is presenting “Read- and Write-Optimization in Modern Database Infrastructures”   Kafka: The Definitive Guide, 2nd Edition By Gwen Shapira, Todd Palino, Rajini Sivaram, Krit Petty November 2021 Amazon | O’Reilly Engineers from Confluent and LinkedIn responsible for developing Kafka explain how to deploy production Kafka clusters, write reliable event-driven microservices, and build scalable stream processing applications with this platform. Through detailed examples, you’ll learn Kafka’s design principles, reliability guarantees, key APIs, and architecture details, including the replication protocol, the controller, and the storage layer. You’ll learn: Best practices for deploying and configuring Kafka Kafka producers and consumers for writing and reading messages Patterns and use-case requirements to ensure reliable data delivery Best practices for building data pipelines and applications with Kafka How to perform monitoring, tuning, and maintenance tasks with Kafka in production The most critical metrics among Kafka’s operational measurements Kafka’s delivery capabilities for stream processing systems Gwen is presenting “The Nile Approach: Re-engineering Postgres for Millions of Tenants”   The Missing README: A Guide for the New Software Engineer by Chris Riccomini and Dmitriy Ryaboy Amazon | O’Reilly August 2021 For new software engineers, knowing how to program is only half the battle. You’ll quickly find that many of the skills and processes key to your success are not taught in any school or bootcamp. The Missing README fills in that gap—a distillation of workplace lessons, best practices, and engineering fundamentals that the authors have taught rookie developers at top companies for more than a decade. Early chapters explain what to expect when you begin your career at a company. The book’s middle section expands your technical education, teaching you how to work with existing codebases, address and prevent technical debt, write production-grade software, manage dependencies, test effectively, do code reviews, safely deploy software, design evolvable architectures, and handle incidents when you’re on-call. Additional chapters cover planning and interpersonal skills such as Agile planning, working effectively with your manager, and growing to senior levels and beyond. You’ll learn: How to use the legacy code change algorithm, and leave code cleaner than you found it How to write operable code with logging, metrics, configuration, and defensive programming How to write deterministic tests, submit code reviews, and give feedback on other people’s code The technical design process, including experiments, problem definition, documentation, and collaboration What to do when you are on-call, and how to navigate production incidents Architectural techniques that make code change easier Agile development practices like sprint planning, stand-ups, and retrospectives Chris and Martin Kleppmann are presenting “Designing Data-Intensive Applications in 2025”   The DynamoDB Book By Alex Debrie April 2020 Amazon | Direct DynamoDB is a highly available, infinitely scalable NoSQL database offering from AWS. But modeling with a NoSQL database like DynamoDB is different than modeling with a relational database. You need to intentionally design for your access patterns rather than creating a normalized model that allows for flexible querying later. The DynamoDB Book is the authoritative resource in the space, and it’s the recommended resource within Amazon for learning DynamoDB. Rick Houlihan, the former head of the NoSQL Blackbelt team at AWS, said The DynamoDB Book is “definitely a must read if you want to understand how to correctly model data for NoSQL apps.” The DynamoDB takes a comprehensive approach to teaching DynamoDB, including: Discussion of key concepts, underlying infrastructure components, and API design; Explanations of core strategies for data modeling, including one-to-many and many-to-many relationships, filtering, sorting, aggregations, and more; 5 full walkthrough examples featuring complex data models and a large number of access patterns. Alex is presenting “DynamoDB Cost Optimization Considerations and Strategies” RESTful Java Patterns and Best Practices: Learn Best Practices to Efficiently Build Scalable, Reliable, and Maintainable High Performance Restful Services By Bhakti Mehta Amazon September, 2014 This book provides an overview of the REST architectural style and then dives deep into best practices and commonly used patterns for building RESTful services that are lightweight, scalable, reliable, and highly available. It’s designed to help application developers get familiar with REST. The book explores the details, best practices, and commonly used REST patterns as well as gives insights on how Facebook, Twitter, PayPal, GitHub, Stripe, and other companies are implementing solutions with RESTful services.

Tracking Millions of Heartbeats on ZEE’s Streaming Platform

How strategic database migration + data (re)modeling improved latencies and cut database costs 5X ZEE is India’s largest media and entertainment business, covering broadcast TV, films,  streaming media, and music. ZEE5 is their premier OTT streaming service, available in over 190 countries with ~150M monthly active users. And every user’s playback experience, security, and recommendations rely upon a “heartbeat API” that processes a whopping 100B+ heartbeats per day. The engineers behind the system knew that continued business growth would stress their infrastructure (as well as the people reviewing the database bills). So, the team decided to rethink the system before it inflicted any heart attacks. TL;DR, they designed a system that’s loved internally and by users. And Jivesh Threja (Tech Lead) and Srinivas Shanmugam (Principal Architect) joined us on Valentine’s Day last year to share their experiences. They outlined the technical requirements for the replacement (cloud neutrality, multi-tenant readiness, simplicity of onboarding new use cases, and high throughput and low latency at optimal costs) and how that led to ScyllaDB. Then, they explained how they achieved their goals through a new stream processing pipeline, new API layer, and data (re)modeling. The initial results of their optimization: 5X cost savings (from $744K to $144K annually) and single-digit millisecond P99 read latency. Wrapping up, they shared lessons learned that could benefit anyone considering or using ScyllaDB. Here are some highlights from that talk… What’s a Heartbeat? “Heartbeat” refers to a request that’s fired at regular intervals during video playback on the ZEE5 OTT platform. These simple requests track what users are watching and how far they’ve progressed in each video. They’re essential for ZEE5’s “continue watching” functionality, which lets users pause content on one device then resume it on any device. They’re also instrumental for calculating key metrics, like concurrent viewership for a big event or the top shows this week. Why Change? ZEE5’s original heartbeat system was a web of different databases, each handling a specific part of the streaming experience. Although it was technically functional, this approach was expensive and locked them into a specific vendor ecosystem. The team recognized an opportunity to streamline their infrastructure– and they went for it. They wanted a system that wasn’t locked into any particular cloud provider, would cost less to operate, and could handle their massive scale with consistently fast performance – specifically, single-digit millisecond responses. Plus, they wanted the flexibility to add new features easily and the ability to offer their system to other streaming platforms. As Srinivas put it: “It needed to be multi-tenant ready so it could be reused for any OTT provider. And it needed to be easily extensible to new use cases without major architectural changes.” System Architecture, Before and After Here’s a look at their original system architecture with multiple databases: DynamoDB to store the basic heartbeat data Amazon RDS to store next and previous episode information Apache Solr to store persistent metadata One Redis instance to cache metadata Another Redis instance to store viewership details Click for a detailed view The ZEE5 team considered four main database options for this project: Redis, Cassandra, Apache Ignite, and ScyllaDB. After evaluation and benchmarking, they chose ScyllaDB. Some of the reasons Srinivas cited for this decision: “We don’t need an extra cache layer on top of the persistent database. ScyllaDB manages both the cache layer and the persistent database within the same infrastructure, ensuring low latency across regions, replication, and multi-cloud readiness. It works with any cloud vendor, including Azure, AWS, and GCP, and now offers managed support with a turnaround time of less than one hour.” The new architecture simplifies and flattens the previous system architecture structure. Click for a detailed view Now, all heartbeat events are pushed into their heartbeat topic, processed through stream processing, and ingested into ScyllaDB Cloud using ScyllaDB connectors. Whenever content is published, it’s ingested into their metadata topic and then inserted into ScyllaDB Cloud via metadata connectors. Srinivas concludes:  “With this new architecture, we successfully migrated workloads from DynamoDB, RDS, Redis, and Solr to ScyllaDB. This has resulted in a 5x cost reduction, bringing our monthly expenses down from $62,000 to around $12,000.” Deeper into the Design Next Jivesh shared more about their low-level design… Real-time stream processing pipeline In the real-time stream processing pipeline, heartbeats are sent to ScyllaDB at regular intervals. The heartbeat interval is set to 60 seconds, meaning that every frontend client sends a heartbeat every 60 seconds while a user is watching a video. These heartbeats pass through the playback stream processing system, business logic consumers transform that data into the required format – then the processed data is stored in ScyllaDB. Scalable API layer The first component in the scalable API layer is the heartbeat service, which is responsible for handling large volumes of data ingestion. Topics process the data, then it passes through a connector service and is stored in ScyllaDB. Another notable API layer service is the Concurrent Viewership Count service. This service uses ScyllaDB to retrieve concurrent viewership data – either per user or per asset (e.g., per ID). For example, if a movie is released, this service can tell how many users are watching the movie at any given moment. Metadata management use case One of the first major challenges ZEE5 faced was managing metadata for their massive OTT platform. Initially, they relied on a combination of three different databases – Solr, Redis, and Postgres – to handle their extensive metadata needs. Looking to optimize and simplify, they redesigned their data model to work with ScyllaDB instead – using ID as the partition key, along with materialized views. Here’s a look at their metadata model: create keyspace.meta_data ( id text, title text, show_id text, …, …, PRIMARY KEY((id),show_id) ) with compaction = {‘class’: ‘LeveledCompactionStrategy’ }; In this model, the ID serves as the partition key. Since this table experiences relatively few writes (a write occurs only when a new asset is released) but significantly more reads, they used Leveled Compaction Strategy to optimize performance. And, according to Jivesh, “Choosing the right partition and clustering keys helped us get a single-digit millisecond latency.” Viewership count use case Viewership Count is another use case that they moved to ScyllaDB. Viewership count can be tracked per user or per asset ID. ZEE5 decided to design a table where the user ID served as the partition key and the asset ID as the sort key – allowing viewership data to be efficiently queried. They set ScyllaDB’s TTL to match the 60-second heartbeat interval, ensuring that data automatically expires after the designated time. Additionally, they used ScyllaDB’s Time-Window Compaction Strategy to efficiently manage data in memory, clearing expired records based on the configured TTL. Jivesh explained, “This table is continuously updated with heartbeats from every front end and every user. As heartbeats arrive, viewership counts are tracked in real time and automatically cleared when the TTL expires. That lets us efficiently retrieve live viewership data using ScyllaDB.” Here’s their viewership count data model: CREATE TABLE keyspace.USER_SESSION_STREAM ( USER_ID text, DEVICE_ID text, ASSET_ID text, TITLE text, …, PRIMARY KEY((USER_ID), ASSET_ID) ) WITH default_time_to_live = 60 and compaction = { 'class' : 'TimeWindowCompactionStrategy' }; ScyllaDB Results and Lessons Learned The following load test report shows a throughput of 41.7K requests per second. This benchmark was conducted during the database selection process to evaluate performance under high load. Jivesh remarked, “Even with such a high throughput, we could achieve a microsecond write latency and average microsecond read latency. This really gave us a clear view of what ScyllaDB could do – and that helped us decide.” He then continued to share some facts that shed light on the scale of ZEE5’s ScyllaDB deployment: “We have around 9TB on ScyllaDB. Even with such a large volume of data, it’s able to handle latencies within microseconds and a single-digit millisecond, which is quite tremendous. We have a daily peak concurrent viewership count of 1 million. Every second, we are writing so much data into ScyllaDB and getting so much data out of it We process more than 100 billion heartbeats in a day. That’s quite huge.” The talk wrapped with the following lessons learned: Data modeling is the single most critical factor in achieving single-digit millisecond latencies. Choose the right quorum setting and compaction strategy. For example, does a heartbeat need to be written to every node before it can be read, or is a local quorum sufficient? Selecting the right quorum ensures the best balance between latency and SLA requirements. Choose Partition and Clustering Keys wisely – it’s not easy to modify them later. Use Materialized Views for faster lookups and avoid filter queries. Querying across partitions can degrade performance. Use prepared statements to improve efficiency. Use asynchronous queries for faster query processing. For instance, in the metadata model, 20 synchronous queries were executed in parallel, and ScyllaDB handled them within milliseconds. Zone-aware ScyllaDB clients help reduce cross-AZ (Availability Zone) network costs. Fetching data within the same AZ minimizes latency and significantly reduces network expenses.

First Look at the Monster SCALE Summit Agenda

Given that we’re hosting Monster SCALE Summit…with tech talks on extreme-scale engineering…many of which feature our monstrously fast and scalable database, a big announcement is probably expected? We hope this meets your super-sized expectations. Monster SCALE Summit 2025 will be featuring 60+ tech talks including: Just-added keynotes by Kelsey Hightower and Rachel Stephens + Adam Jacob Previously-teased keynotes by Avi Kivity, Martin Kleppmann + Chris Riccomini, Gwen Shapira, and Dor Laor Engineering talks by gamechangers like Uber, Slack, Canva, Atlassian, Wise, and Booking.com 14 talks by ScyllaDB users such as Medium, Disney+, ShareChat, Yieldmo, Clearview AI, and more – plus two talks by Discord The latest from ScyllaDB engineering: including object storage, vector search, and “ScyllaDB X Cloud” Like other ScyllaDB-hosted conferences (e.g., P99 CONF), this conference will be free and virtual so that everyone can participate. See the agenda and register – it’s free Mark your calendar for March 11 and 12 because – in addition to all those great talks – you can… Chat directly with speakers and connect with ~20K of your peers Participate in some monster scale global distributed system challenges – with prizes for winners, of course Learn from ScyllaDB’s top experts, who are eager to answer your toughest database performance questions in our lively lounge – and preparing special interactive training courses for the occasion Win conference swag, sea monster plushies, book bundles, and other cool giveaways It’s a lot. But hey, it’s Monster SCALE Summit. 🙂 Details, Details Beyond what’s on the agenda, here’s some additional detail on a few recently-added sessions (see more in our “tiny peek” blog post) How Discord Performs Database Upgrades at Scale Ethan Donowitz, Senior Software Engineer, Persistence Infrastructure at Discord Database upgrades are high-risk but high-reward. Upgrading to a newer version can make your database faster, cheaper, and more reliable; however, without thorough planning and testing, upgrades can be risky. Because databases are stateful, it is often not possible to roll back if you encounter problems after the upgrade due to backwards incompatible changes across versions. While new versions typically mean improved query latencies, changes in query planning or cache behavior across versions can cause unexpected differences in performance in places one might not expect. Discord relies on ScyllaDB to serve millions of reads per second across many clusters, so we needed a comprehensive strategy to sufficiently de-risk upgrades to avoid impact to our users. To accomplish this, we use what we call “shadow clusters.” A shadow cluster contains roughly the same data as its corresponding cluster in production, and traffic to the primary cluster is mirrored to the shadow cluster. Running a real production workload on a shadow cluster can expose differences in performance and resource usage across versions. When mirroring reads, we also have the ability to perform “read validations,” where the results for a query issued to the primary cluster and the shadow cluster are checked for equality. This gives us confidence that data has not been corrup How Discord Indexes Trillions of Messages: Scaling Search Infrastructure Vicki Niu, Senior Software Engineer at Discord When Discord first built messages search in 2017, we designed our infrastructure to handle billions of messages sent by millions of users. As our platform grew to trillions of messages, our search system failed to keep up. We thus set out to rebuild our message search platform to meet these new scaling needs using our learnings and some new technologies. This talk will share how we scaled Discord’s message search infrastructure using Rust, Kubernetes, and a multi-cluster Elasticsearch architecture to achieve better performance, operability, and reliability, while also enabling new search features for Discord users. ted due to differences in behavior across versions. Testing with shadow clusters has been paramount to de-risking complicated upgrades for one of the most important pieces of infrastructure at Discord. Route It Like It’s Hot: Scaling Payments Routing at American Express Benjamin Cane, Distinguished Engineer at American Express In 2023, there were over 723 billion credit card transactions. Whenever someone taps, swipes, dips, or clicks a credit or debit card, a payment switch ensures the transaction arrives safely and securely at the correct financial institution.These payment switches are the backbone of the worldwide payments ecosystem. Join the American Express Payment Acquiring and Network team as they share their experiences from building their Global Transaction Router, which is responsible for switching and routing payments at the scale of American Express. They will explore how they’ve designed, built, and operated this Global Transaction Router to perform during record-breaking shopping holidays, ticket sales, and unexpected customer behavior. The audience will leave with a deep understanding of the unique challenges of a payments switch (E.g., routing ISO 8583 transactions as fast as possible), some of our design choices (E.g., using containers and avoiding logging), and a deep dive into a few implementation challenges (E.g., Inefficient use of Goroutines and Channels) we found along the way. How Yieldmo Cut Database Costs and Cloud Dependencies Fast Todd Coleman, Chief Architect and Co-founder at Yieldmo Yieldmo’s business relies on processing hundreds of billions of daily ad requests with subsecond latency responses. Our services initially depended on DynamoDB, and we valued its simplicity and stability. However, DynamoDB costs were becoming unsustainable, latencies were not ideal, and we sought greater flexibility in deploying services to other cloud providers. In this session, we’ll walk you through the various options we considered to address these challenges and share why and how we ultimately moved forward with ScyllaDB’s DynamoDB-compatible API. See more session details  

Real-Time Write Heavy Workloads: Considerations & Tips

Let’s focus on the performance-related complexities that teams commonly face with write-heavy workloads and discuss your options for tackling them Write-heavy database workloads bring a distinctly different set of challenges than read-heavy ones. For example: Scaling writes can be costly, especially if you pay per operation and writes are 5X more costly than reads Locking can add delays and reduce throughput I/O bottlenecks can lead to write amplification and complicate crash recovery Database backpressure can throttle the incoming load While cost matters – quite a lot, in many cases – it’s not a topic we want to cover here. Rather, let’s focus on the performance-releated complexities that teams commonly face and discuss your options for tackling them. What Do We Mean by “a Real-Time Write Heavy Workload”? First, let’s clarify what we mean by a “real-time write-heavy” workload. We’re talking about workloads that: Ingest a large amount of data (e.g., over 50K OPS) Involve more writes than reads Are bound by strict latency SLAs (e.g., single-digit millisecond P99 latency) In the wild, they occur across everything from online gaming to real-time stock exchanges. A few specific examples: Internet of Things (IoT) workloads tend to involve small but frequent append-only writes of time series data. Here, the ingestion rate is primarily determined by the number of endpoints collecting data. Think of smart home sensors or industrial monitoring equipment constantly sending data streams to be processed and stored. Logging and Monitoring systems also deal with frequent data ingestion, but they don’t have a fixed ingestion rate. They may not necessarily append only, as well as may be prone to hotspots, such as when one endpoint misbehaves. Online Gaming platforms need to process real-time user interactions, including game state changes, player actions, and messaging. The workload tends to be spiky, with sudden surges in activity. They’re extremely latency sensitive since even small delays can impact the gaming experience. E-commerce and Retail workloads are typically update-heavy and often involve batch processing. These systems must maintain accurate inventory levels, process customer reviews, track order status, and manage shopping cart operations, which usually require reading existing data before making updates. Ad Tech and Real-time Bidding systems require split-second decisions. These systems handle complex bid processing, including impression tracking and auction results, while simultaneously monitoring user interactions such as clicks and conversions. They must also detect fraud in real time and manage sophisticated audience segmentation for targeted advertising. Real-time Stock Exchange systems must support high-frequency trading operations, constant stock price updates, and complex order matching processes – all while maintaining absolute data consistency and minimal latency. Next, let’s look at key architectural and configuration considerations that impact write performance. Storage Engine Architecture The choice of storage engine architecture fundamentally impacts write performance in databases. Two primary approaches exist: LSM trees and B-Trees. Databases known to handle writes efficiently – such as ScyllaDB, Apache Cassandra, HBase, and Google BigTable – use Log-Structured Merge Trees (LSM). This architecture is ideal for handling large volumes of writes. Since writes are immediately appended to memory, this allows for very fast initial storage. Once the “memtable” in memory fills up, the recent writes are flushed to disk in sorted order. That reduces the need for random I/O. For example, here’s what the ScyllaDB write path looks like: With B-tree structures, each write operation requires locating and modifying a node in the tree – and that involves both sequential and random I/O. As the dataset grows, the tree can require additional nodes and rebalancing, leading to more disk I/O, which can impact performance. B-trees are generally better suited for workloads involving joins and ad-hoc queries. Payload Size Payload size also impacts performance. With small payloads, throughput is good but CPU processing is the primary bottleneck. As the payload size increases, you get lower overall throughput and disk utilization also increases. Ultimately, a small write usually fits in all the buffers and everything can be processed quite quickly. That’s why it’s easy to get high throughput. For larger payloads, you need to allocate larger buffers or multiple buffers. The larger the payloads, the more resources (network and disk) are required to service those payloads. Compression Disk utilization is something to watch closely with a write-heavy workload. Although storage is continuously becoming cheaper, it’s still not free. Compression can help keep things in check – so choose your compression strategy wisely. Faster compression speeds are important for write-heavy workloads, but also consider your available CPU and memory resources. Be sure to look at the compression chunk size parameter. Compression basically splits your data into smaller blocks (or chunks) and then compresses each block separately. When tuning this setting, realize that larger chunks are better for reads while smaller ones are better for writes, and take your payload size into consideration. Compaction For LSM-based databases, the compaction strategy you select also influences write performance. Compaction involves merging multiple SSTables into fewer, more organized files, to optimize read performance, reclaim disk space, reduce data fragmentation, and maintain overall system efficiency. When selecting compaction strategies, you could aim for low read amplification, which makes reads as efficient as possible. Or, you could aim for low write amplification by avoiding compaction from being too aggressive. Or, you could prioritize low space amplification and have compaction purge data as efficiently as possible. For example, ScyllaDB offers several compaction strategies (and Cassandra offers similar ones): Size-tiered compaction strategy (STCS): Triggered when the system has enough (four by default) similarly sized SSTables. Leveled compaction strategy (LCS): The system uses small, fixed-size (by default 160 MB) SSTables distributed across different levels. Incremental Compaction Strategy (ICS): Shares the same read and write amplification factors as STCS, but it fixes its 2x temporary space amplification issue by breaking huge sstables into SSTable runs, which are comprised of a sorted set of smaller (1 GB by default), non-overlapping SSTables. Time-window compaction strategy (TWCS): Designed for time series data. For write-heavy workloads, we warn users to avoid leveled compaction at all costs. That strategy is designed for read-heavy use cases. Using it can result in a regrettable 40x write amplification. Batching In databases like ScyllaDB and Cassandra, batching can actually be a bit of a trap – especially for write-heavy workloads. If you’re used to relational databases, batching might seem like a good option for handling a high volume of writes. But it can actually slow things down if it’s not done carefully. Mainly, that’s because large or unstructured batches end up creating a lot of coordination and network overhead between nodes. However, that’s really not what you want in a distributed database like ScyllaDB. Here’s how to think about batching when you’re dealing with heavy writes: Batch by the Partition Key: Group your writes by the partition key so the batch goes to a coordinator node that also owns the data. That way, the coordinator doesn’t have to reach out to other nodes for extra data. Instead, it just handles its own, which cuts down on unnecessary network traffic. Keep Batches Small and Targeted: Breaking up large batches into smaller ones by partition keeps things efficient. It avoids overloading the network and lets each node work on only the data it owns. You still get the benefits of batching, but without the overhead that can bog things down. Stick to Unlogged Batches: Considering you follow the earlier points, it’s best to use unlogged batches. Logged batches add extra consistency checks, which can really slow down the write. So, if you’re in a write-heavy situation, structure your batches carefully to avoid the delays that big, cross-node batches can introduce. Wrapping Up We offered quite a few warnings, but don’t worry. It was easy to compile a list of lessons learned because so many teams are extremely successful working with real-time write-heavy workloads. Now you know many of their secrets, without having to experience their mistakes. 🙂 If you want to learn more, here are some firsthand perspectives from teams who tackled quite interesting write-heavy challenges: Zillow: Consuming records from multiple data producers, which resulted in out-of-order writes that could result in incorrect updates Tractian: Preparing for 10X growth in high-frequency data writes from IoT devices Fanatics: Heavy write operations like handling orders, shopping carts, and product updates for this online sports retailer Also, take a look at the following video, where we go into even greater depth on these write-heavy challenges and also walk you through what these workloads look like on ScyllaDB.

Inside Tripadvisor’s Real-Time Personalization with ScyllaDB + AWS

See the engineering behind real-time personalization at Tripadvisor’s massive (and rapidly growing) scale What kind of traveler are you? Tripadvisor tries to assess this as soon as you engage with the site, then offer you increasingly relevant information on every click—within a matter of milliseconds. This personalization is powered by advanced ML models acting on data that’s stored on ScyllaDB running on AWS. In this article, Dean Poulin (Tripadvisor Data Engineering Lead on the AI Service and Products team) provides a look at how they power this personalization. Dean shares a taste of the technical challenges involved in delivering real-time personalization at Tripadvisor’s massive (and rapidly growing) scale. It’s based on the following AWS re:Invent talk: Pre-Trip Orientation In Dean’s words … Let’s start with a quick snapshot of who Tripadvisor is, and the scale at which we operate. Founded in 2000, Tripadvisor has become a global leader in travel and hospitality, helping hundreds of millions of travelers plan their perfect trips. Tripadvisor generates over $1.8 billion in revenue and is a publicly traded company on the NASDAQ stock exchange. Today, we have a talented team of over 2,800 employees driving innovation, and our platform serves a staggering 400 million unique visitors per month – a number that’s continuously growing. On any given day, our system handles more than 2 billion requests from 25 to 50 million users. Every click you make on Tripadvisor is processed in real time. Behind that, we’re leveraging machine learning models to deliver personalized recommendations – getting you closer to that perfect trip. At the heart of this personalization engine is ScyllaDB running on AWS. This allows us to deliver millisecond-latency at a scale that few organizations reach. At peak traffic, we hit around 425K operations per second on ScyllaDB with P99 latencies for reads and writes around 1-3 milliseconds. I’ll be sharing how Tripadvisor is harnessing the power of ScyllaDB, AWS, and real-time machine learning to deliver personalized recommendations for every user. We’ll explore how we help travelers discover everything they need to plan their perfect trip: whether it’s uncovering hidden gems, must-see attractions, unforgettable experiences, or the best places to stay and dine. This [article] is about the engineering behind that – how we deliver seamless, relevant content to users in real time, helping them find exactly what they’re looking for as quickly as possible. Personalized Trip Planning Imagine you’re planning a trip. As soon as you land on the Tripadvisor homepage, Tripadvisor already knows whether you’re a foodie, an adventurer, or a beach lover – and you’re seeing spot-on recommendations that seem personalized to your own interests. How does that happen within milliseconds? As you browse around Tripadvisor, we start to personalize what you see using Machine Learning models which calculate scores based on your current and prior browsing activity. We recommend hotels and experiences that we think you would be interested in. We sort hotels based on your personal preferences. We recommend popular points of interest near the hotel you’re viewing. These are all tuned based on your own personal preferences and prior browsing activity. Tripadvisor’s Model Serving Architecture Tripadvisor runs on hundreds of independently scalable microservices in Kubernetes on-prem and in Amazon EKS. Our ML Model Serving Platform is exposed through one of these microservices. This gateway service abstracts over 100 ML Models from the Client Services – which lets us run A/B tests to find the best models using our experimentation platform. The ML Models are primarily developed by our Data Scientists and Machine Learning Engineers using Jupyter Notebooks on Kubeflow. They’re managed and trained using ML Flow, and we deploy them on Seldon Core in Kubernetes. Our Custom Feature Store provides features to our ML Models, enabling them to make accurate predictions The Custom Feature Store The Feature Store primarily serves User Features and Static Features. Static Features are stored in Redis because they don’t change very often. We run data pipelines daily to load data from our offline data warehouse into our Feature Store as Static Features. User Features are served in real time through a platform called Visitor Platform. We execute dynamic CQL queries against ScyllaDB, and we do not need a caching layer because ScyllaDB is so fast. Our Feature Store serves up to 5 million Static Features per second and half a million User Features per second. What’s an ML Feature? Features are input variables to the ML Models that are used to make a prediction. There are Static Features and User Features. Some examples of Static Features are awards that a restaurant has won or amenities offered by a hotel (like free Wi-Fi, pet friendly or fitness center). User Features are collected in real time as users browse around the site. We store them in ScyllaDB so we can get lightning fast queries. Some examples of user features are the hotels viewed over the last 30 minutes, restaurants viewed over the last 24 hours, or reviews submitted over the last 30 days. The Technologies Powering Visitor Platform ScyllaDB is at the core of Visitor Platform. We use Java-based Spring Boot microservices to expose the platform to our clients. This is deployed on AWS ECS Fargate. We run Apache Spark on Kubernetes for our daily data retention jobs, our offline to online jobs. Then we use those jobs to load data from our offline data warehouse into ScyllaDB so that they’re available on the live site. We also use Amazon Kinesis for processing streaming user tracking events. The Visitor Platform Data Flow The following graphic shows how data flows through our platform in four stages: produce, ingest, organize, and activate. Data is produced by our website and our mobile apps. Some of that data includes our Cross-Device User Identity Graph, Behavior Tracking events (like page views and clicks) and streaming events that go through Kinesis. Also, audience segmentation gets loaded into our platform. Visitor Platform’s microservices are used to ingest and organize this data. The data in ScyllaDB is stored in two keyspaces: The Visitor Core keyspace, which contains the Visitor Identity Graph The Visitor Metric keyspace, which contains Facts and Metrics (the things that the people did as they browsed the site) We use daily ETL processes to maintain and clean up the data in the platform. We produce Data Products, stamped daily, in our offline data warehouse – where they are available for other integrations and other data pipelines to use in their processing. Here’s a look at Visitor Platform by the numbers:   Why Two Databases? Our online database is focused on the real-time, live website traffic. ScyllaDB fills this role by providing very low latencies and high throughput. We use short term TTLs to prevent the data in the online database from growing indefinitely, and our data retention jobs ensure that we only keep user activity data for real visitors. Tripadvisor.com gets a lot of bot traffic, and we don’t want to store their data and try to personalize bots – so we delete and clean up all that data. Our offline data warehouse retains historical data used for reporting, creating other data products, and training our ML Models. We don’t want large-scale offline data processes impacting the performance of our live site, so we have two separate databases used for two different purposes. Visitor Platform Microservices We use 5 microservices for Visitor Platform: Visitor Core manages the cross-device user identity graph based on cookies and device IDs. Visitor Metric is our query engine, and that provides us with the ability for exposing facts and metrics for specific visitors. We use a domain specific language called visitor query language, or VQL. This example VQL lets you see the latest commerce click facts over the last three hours. Visitor Publisher and Visitor Saver handle the write path, writing data into the platform. Besides saving data in ScyllaDB, we also stream data to the offline data warehouse. That’s done with Amazon Kinesis. Visitor Composite simplifies publishing data in batch processing jobs. It abstracts Visitor Saver and Visitor Core to identify visitors and publish facts and metrics in a single API call. Roundtrip Microservice Latency This graph illustrates how our microservice latencies remain stable over time. The average latency is only 2.5 milliseconds, and our P999 is under 12.5 milliseconds. This is impressive performance, especially given that we handle over 1 billion requests per day. Our microservice clients have strict latency requirements. 95% of the calls must complete in 12 milliseconds or less. If they go over that, then we will get paged and have to find out what’s impacting the latencies. ScyllaDB Latency Here’s a snapshot of ScyllaDB’s performance over three days. At peak, ScyllaDB is handling 340,000 operations per second (including writes and reads and deletes) and the CPU is hovering at just 21%. This is high scale in action! ScyllaDB delivers microsecond writes and millisecond reads for us. This level of blazing fast performance is exactly why we chose ScyllaDB. Partitioning Data into ScyllaDB This image shows how we partition data into ScyllaDB. The Visitor Metric Keyspace has two tables: Fact and Raw Metrics. The primary key on the Fact table is Visitor GUID, Fact Type, and Created At Date. The composite partition key is the Visitor GUID and Fact Type. The clustering key is Created At Date, which allows us to sort data in partitions by date. The attributes column contains a JSON object representing the event that occurred there. Some example Facts are Search Terms, Page Views, and Bookings. We use ScyllaDB’s Leveled Compaction Strategy because: It’s optimized for range queries It handles high cardinality very well It’s better for read-heavy workloads, and we have about 2-3X more reads than writes Why ScyllaDB? Our solution was originally built using Cassandra on-prem. But as the scale increased, so did the operational burden. It required dedicated operations support in order for us to manage the database upgrades, backups, etc. Also, our solution requires very low latencies for core components. Our User Identity Management system must identify the user within 30 milliseconds – and for the best personalization, we require our Event Tracking platform to respond in 40 milliseconds. It’s critical that our solution doesn’t block rendering the page so our SLAs are very low. With Cassandra, we had impacts to performance from garbage collection. That was primarily impacting the tail latencies, the P999 and P9999 latencies. We ran a Proof of Concept with ScyllaDB and found the throughput to be much better than Cassandra and the operational burden was eliminated. ScyllaDB gave us a monstrously fast live serving database with the lowest possible latencies. We wanted a fully-managed option, so we migrated from Cassandra to ScyllaDB Cloud, following a dual write strategy. That allowed us to migrate with zero downtime while handling 40,000 operations or requests per second. Later, we migrated from ScyllaDB Cloud to ScyllaDB’s “Bring your own account” model, where you can have the ScyllaDB team deploy the ScyllaDB database into your own AWS account. This gave us improved performance as well as better data privacy. This diagram shows what ScyllaDB’s BYOA deployment looks like. In the center of the diagram, you can see a 6-node ScyllaDB cluster that is running on EC2. And then there’s two additional EC2 instances. ScyllaDB Monitor gives us Grafana dashboards as well as Prometheus metrics. ScyllaDB Manager takes care of infrastructure automation like triggering backups and repairs. With this deployment, ScyllaDB could be co-located very close to our microservices to give us even lower latencies as well as much higher throughput and performance. Wrapping up, I hope you now have a better understanding of our architecture, the technologies that power the platform, and how ScyllaDB plays a critical role in allowing us to handle Tripadvisor’s extremely high scale.

How to Build a High-Performance Shopping Cart App with ScyllaDB

Build a shopping cart app with ScyllaDB– and learn how to use ScyllaDB’s Change Data Capture (CDC) feature to query and export the history of all changes made to the tables.This blog post showcases one of ScyllaDB’s sample applications: a shopping cart app. The project uses FastAPI as the backend framework and ScyllaDB as the database. By cloning the repository and running the application, you can explore an example of an API server built on top of ScyllaDB for a CRUD app. Additionally, you’ll see how to use ScyllaDB’s Change Data Capture (CDC) feature to query and export the history of all changes made to the tables.What’s inside the shopping cart sample app?The application has two components: an API server and a database.API server: Python + FastAPIThe backend is built with Python and FastAPI, a modern Python web framework known for its speed and ease of use. FastAPI ensures that you have a framework that can deliver relatively high performance if used with the right database. At the same time, due to its exceptional developer experience, you can easily understand the code of the project and how it works even if you’ve never used it before.The application exposes multiple API endpoints to perform essential operations like:Adding products to the cartRemoving products from the cartUploading new productsUpdating product information (e.g. price)Database: ScyllaDBAt the core of this application is ScyllaDB, a low-latency NoSQL database that provides predictable performance. ScyllaDB excels in handling large volumes of data with single-digit millisecond latency, making it ideal for large-scale real-time applications.ScyllaDB acts as the foundation for a high-performance low-latency app. Moreover, it has additional capabilities that can help you maintain low p99 latency as well as analyze user behavior. ScyllaDB’s CDC feature tracks changes in the database and you can query historical operations. For e-commerce applications, this means you can capture insights into user behavior:What products are being added or removed from the cart and when?How do users interact with the cart?What does a typical journey look like for a user who actually buys something?These and other insights are invaluable for personalizing the user experience, optimizing the buying journey, and increasing conversion rates.Using ScyllaDB for an ecommerce applicationAs studies have shown, low latency is critical for achieving high conversion rates and delivering a smooth user experience. For instance, shopping cart operations – such as adding, updating, and retrieving products – require high performance to prevent cart abandonment.Data modeling, being the foundation for high-performance web applications, must remain a top priority. So let’s start with the process of creating a performant data model.Design a Shopping Cart data modelWe emphasize a practical “query-first” approach to NoSQL data modeling: start with your application’s queries, then design your schema around them. This method ensures your data model is optimized for your specific use cases and the database can provide a reliable and single-digit p99 latency at any scale.Let’s review the specific CRUD operations and queries a typical shopping cart application performs.ProductsList, add, edit and remove products.GET /products?limit=?SELECT * FROM product LIMIT {limit}GET /products/{product_id}SELECT * FROM product WHERE id = ?POST /productsINSERT INTO product () values ()PUT /products/{product_id}UPDATE product SET ? WHERE id = ?DELETE /products/{product_id}DELETE FROM product WHERE id = ?Based on these requirements, you can create a table to store products. You can notice what value is often used to filter products: product id. This is a good indicator that product id should be the partition key or at least part of it.The Product table:Our application is simple, so a single column will suffice as the partition key. However, if your use case requires additional queries and filtering by additional columns, you can consider using a composite partition key or adding a clustering key to the table.CartList, add, remove products from user’s cart.GET /cart/{user_id}SELECT * FROM cart_items WHERE user_id = ? AND cart_id = ?POST /cart/{user_id}INSERT INTO cart() VALUES ()Here we don’t need cart id because the user can only have one active cart at a time. (You could also build another endpoint to list past purchases by the user – that endpoint would require the cart id as well)DELETE /cart/{user_id}DELETE FROM cart_items WHERE user_id = ? AND cart_id = ? AND product_id = ?POST checkout /cart/{user_id}/checkoutUPDATE cart SET is_active = false WHERE user_id = ? AND cart_id = ?The cart-related operations contain a slightly more complicated logic behind the scenes. We have two values that we use to query by: user id and cart id. Those can be used together as composite partition keys.Additionally, one user can have multiple carts – one they’re using right now to shop and possibly other ones that they had in the past that they already paid for. For this reason, we need to have a way to efficiently find the user’s active cart. This query requirement will be handled by a secondary index on the is_active column.The Cart table:Additionally, we also need to create a table which connects the Product and Cart tables. Without this table, it would be impossible to retrieve products from a cart.The Cart_items table:We enable Change Data Capture for this table. This feature logs all data operations performed on the table into another table, cart_items_scylla_cdc_log. Later, we can query this log to retrieve the table’s historical operations. This data can be used to analyze user behavior, such as the products users add or remove from their carts.Final database schema:Now that we’ve covered the data modeling aspect of the project, you can clone the repository and get started with building.Getting startedPrerequisites:Python 3.8+ScyllaDB cluster (with ScyllaDB Cloud or use Docker)Connect to your ScyllaDB cluster using CQLSH and create the schema:Then, install the Python requirements in a new environment:Modify config.py to match your database credentials:Run the server:Generate sample user data:This script populates your ScyllaDB tables with sample data. This is necessary for the next step, where you will run CDC queries to analyze user behavior.Analyze user behavior with CDCCDC records every data change, including deletes, offering a comprehensive view of your data evolution without affecting database performance. For a shopping cart application, some potential use cases for CDC include:Analyzing a specific user’s buying behaviorTracking user actions leading to checkoutEvaluating product popularity and purchase frequencyAnalyzing active and abandoned cartsBeyond these business-specific insights, CDC data can also be exported to external platforms, such as Kafka, for further processing and analysis.Here are a couple of useful tips when working with CDC:The CDC log table contains timeuuid values, which can be converted to readable timestamps using the toTimestamp() CQL function.The cdc$operation column helps filter operations by type. For instance, a value of 2 indicates an INSERT operation.The most efficient and scalable way to query CDC data is to use the ScyllaDB source connector and set up an integration with Kafka.Now, let’s explore a couple of quick questions that CDC can help answer.How many times did users add more than 2 of the same product to the cart?How many carts contain a particular product?Set up ScyllaDB CDC with Kafka ConnectTo provide a scalable way for you to analyze ScyllaDB CDC logs, you can use Kafka to receive messages sent by ScyllaDB. Then, you can use an analytics tool, like Elasticsearch, to get insights. To send CDC logs to Kafka, you need to install the ScyllaDB CDC source connector, and create a new ScyllaDB connection in Kafka Connect.Install the ScyllaDB source connector on the machine/container that’s running Kafka:Then use the following ScyllaDB related parameters when you create the connection:Make sure to enable CDC on each table you want to send messages from. You can do this by executing the following CQL:Try it out yourselfIf you are interested in trying out this application yourself, check out the dedicated documentation site: shopping-cart.scylladb.com and the GitHub repository.If you have any questions about this project or ScyllaDB, submit a question in ScyllaDB the forum.

A Tiny Peek at Monster SCALE Summit 2025

Big things have been happening behind the scenes for the premier Monster SCALE Summit. Ever since we introduced it at P99 CONF, the community response has been overwhelming. We’re now faced with the “good” problem of determining how to fit all the selected speakers into the two half-days we set aside for the event. 😅 If you missed the intro last year, Monster Scale Summit is a highly technical conference that connects the community of professionals designing, implementing, and optimizing performance-sensitive data-intensive applications. It focuses on exploring “monster scale” engineering challenges with respect to extreme levels of throughput, data, and global distribution. The two-day event is free, intentionally virtual, and highly interactive. Register – it’s free and virtual We’ll be announcing the agenda next month. But we’re so excited about the speaker lineup that we can’t wait to share a taste of what you can expect. Here’s a preview of 12 of the 60+ sessions that you can join on March 11 and 12… Designing Data-Intensive Applications in 2025 Martin Kleppmann and Chris Riccomini (Designing Data-Intensive Applications book) Join us for an informal chat with Martin Kleppmann and Chris Riccomini, who are currently revising the famous book Designing Data-Intensive Applications. We’ll cover how data-intensive applications have evolved since the book was first published, the top tradeoffs people are negotiating today, and what they believe is next for data-intensive applications. Martin and Chris will also provide an inside look at the book writing and revision process. The Nile Approach: Re-engineering Postgres for Millions of Tenants Gwen Shapira (Nile) Scaling relational databases is a notoriously challenging problem. Doing so while maintaining consistent low latency, efficient use of resources and compatibility with Postgres may seem impossible. At Nile, we decided to tackle the scaling challenge by focusing on multi-tenant applications. These applications require not only scalability, but also a way to isolate tenants and avoid the noisy neighbor problem. By tackling both challenges, we developed an approach, which we call “virtual tenant databases”, which gives us an efficient way to scale Postgres to millions of tenants while still maintaining consistent performance. In this talk, I’ll explore the limitations of traditional scaling for multi-tenant applications and share how Nile’s virtual tenant databases address these challenges. By combining the best of Postgres existing capabilities, distributed algorithms and a new storage layer, Nile re-engineered Postgres for multi-tenant applications at scale. The Mechanics of Scale Dominik Tornow (Resonate HQ) As distributed systems scale, the complexity of their development and operation skyrockets. A dependable understanding of the mechanics of distributed systems is our most reliable parachute. In this talk, we’ll use systems thinking to develop an accurate and concise mental model of concurrent, distributed systems, their core challenges, and the key principles to address these challenges. We’ll explore foundational problems such as the tension between consistency and availability, and essential techniques like partitioning and replication. Whether you are building a new system from scratch or scaling an existing system to new heights, this talk will provide the understanding to confidently navigate the intricacies of modern, large-scale distributed systems. Feature Store Evolution Under Cost Constraints: When Cost is Part of the Architecture Ivan Burmistrov and David Malinge (ShareChat) At P99 CONF 23, the ShareChat team presented the scaling challenges for the ML Feature Store so it could handle 1 billion features per second. Once the system was scaled to handle the load, the next challenge the team faced was extreme cost constraints: it was required to make the same quality system much cheaper to run. Ivan and David will talk about approaches the team implemented in order to optimize for cost in the Cloud environment while maintaining the same SLA for the service. The talk will touch on such topics as advanced optimizations on various levels to bring down the compute, minimizing the waste when running on Kubernetes, autoscaling challenges for stateful Apache Flink jobs, and others. The talk should be useful for those who are either interested in building or optimizing an ML Feature Store or in general looking into cost optimizations in the cloud environment. Time Travelling at Scale Richard Hart (Antithesis) Antithesis is a continuous reliability platform that autonomously searches for problems in your software within a simulated environment. Every problem we find can be perfectly reproduced, allowing for efficient debugging of even the most complex problems. But storing and querying histories of program execution at scale creates monster large cardinalities. Over a ~10 hour test run, we generate ~1bn rows. The solution: our own tree-database. 30B Images and Counting: Scaling Canva’s Content-Understanding Pipelines Dr. Kerry Halupka (Canva) As the demand for high-quality, labeled image data grows, building systems that can scale content understanding while delivering real-time performance is a formidable challenge. In this talk, I’ll share how we tackled the complexities of scaling content understanding pipelines to support monstrous volumes of data, including backfilling labels for over 30 billion images. At the heart of our system is an extreme label classification model capable of handling thousands of labels and scaling seamlessly to thousands more. I’ll dive into the core components: candidate image search, zero-shot labelling using highly trained teacher models, and iterative refinement with visual critic models. You’ll learn how we balanced latency, throughput, and accuracy while managing evolving datasets and continuously expanding label sets. I’ll also discuss the tradeoffs we faced—such as ensuring precision in labelling without compromising speed—and the techniques we employed to optimise for scale, including strategies to address data sparsity and performance bottlenecks. By the end of the session, you’ll gain insights into designing, implementing, and scaling content understanding systems that meet extreme demands. Whether you’re working with real-time systems, distributed architectures, or ML pipelines, this talk will provide actionable takeaways for pushing large-scale labelling pipelines to their limits and beyond. How Agoda Scaled 50x Throughput with ScyllaDB Worakarn Isaratham (Agoda) In this talk, we will explore the performance tuning strategies implemented at Agoda to optimize ScyllaDB. Key topics include enhancing disk performance, selecting the appropriate compaction strategy, and adjusting SSTable settings to match our usage profile. Who Needs One Database Anyway? Glauber Costa (Turso) Developers need databases. That’s how you store your data. And that’s usually how it goes: you have your large fleet of services, and they connect to one database. But what if it wasn’t like that? What if instead of one database, one application would create one million databases, or even more? In this talk, we’ll explore the market trends that give rise to use cases where this pattern is beneficial, and the infrastructure changes needed to support it. How We Boosted ScyllaDB’s Data Streaming by 25x Asias He (ScyllaDB) To improve elasticity, we wanted to speed up streaming, the process of scaling out/in to other nodes used to analyze every partition. Enter file-based streaming,  a new feature that optimizes tablet movement. This new approach streams the entire SSTable files without deserializing SSTable files into mutation fragments and re-serializing them back into SSTables on receiving nodes. As a result, significantly less data is streamed over the network, and less CPU is consumed –  especially for data models that contain small cells. This session will share the engineering behind this optimization and look at the performance impact you can expect in common situations. Evolving Atlassian Confluence Cloud for Scale, Reliability, and Performance Bhakti Mehta (Atlassian) This session covers the journey of Confluence Cloud – the team workspace for collaboration and knowledge sharing used by thousands of companies – and how we aim to take it to the next level, with scale, performance, and reliability as the key motivators. This session presents a deep dive to provide insights into how the Confluence architecture has evolved into its current form. It discusses how Atlassian deploys, runs, and operates at scale and all challenges encountered along the way. I will cover performance and reliability at scale starting with the fundamentals of measuring everything, re-defining metrics to be insightful of actual customer pain, auditing end-to-end experiences. Beyond just dev-ops and best practices, this means empowering teams to own product stability through practices and tools. Two Leading Approaches to Data Virtualization: Which Scales Better? Dr. Daniel Abadi (University of Maryland) You have a large dataset stored in location X, and some code to process or analyze it in location Y. What is better: move the code to the data, or move the data to the code? For decades, it has always been assumed that the former approach is more scalable. Recently, with the rise of cloud computing, and the push to separate resources for storage and compute, we have seen data increasingly being pushed to code, flying in face of conventional wisdom. What is behind this trend, and is it a dangerous idea? This session will look at this question from academic and practical perspectives, with a particular focus on data virtualization, where there exists an ongoing debate on the merits of push-based vs. pull-based data processing. Scaling a Beast: Lessons from 400x Growth in a High-Stakes Financial System Dmytro Hnatiuk (Wise) Scaling a system from 66 million to over 25 billion records is no easy feat—especially when it’s a core financial system where every number has to be right, and data needs to be fresh right now. In this session, I’ll share the ups and downs of managing this kind of growth without losing my sanity. You’ll learn how to balance high data accuracy with real-time performance, optimize your app logic, and avoid the usual traps of database scaling. This isn’t about turning you into a database expert—it’s about giving you the practical, no-BS strategies you need to scale your systems without getting overwhelmed by technical headaches. Perfect for engineers and architects who want to tackle big challenges and come out on top.

How Supercell Handles Real-Time Persisted Events with ScyllaDB

How a team of just two engineers tackled real-time persisted events for hundreds of millions of players With just two engineers, Supercell took on the daunting task of growing their basic account system into a social platform connecting hundreds of millions of gamers. Account management, friend requests, cross-game promotions, chat, player presence tracking, and team formation – all of this had to work across their five major games. And they wanted it all to be covered by a single solution that was simple enough for a single engineer to maintain, yet powerful enough to handle massive demand in real-time. Supercell’s Server Engineer, Edvard Fagerholm, recently shared how their mighty team of two tackled this task. Read on to learn how they transformed a simple account management tool into a comprehensive cross-game social network infrastructure that prioritized both operational simplicity and high performance. Note: If you enjoy hearing about engineering feats like this, join us at Monster Scale Summit (free + virtual). Engineers from Disney+/Hulu,, Slack, Canva, Uber, Salesforce, Atlassian and more will be sharing strategies and case studies. Background: Who’s Supercell? Supercell is the Finland-based company behind the hit games Hay Day, Clash of Clans, Boom Beach, Clash Royale and Brawl Stars. Each of these games has generated $1B in lifetime revenue.   Somehow they manage to achieve this with a super small staff. Until quite recently, all the account management functionality for games servicing hundreds of millions of monthly active users was being built and managed by just two engineers. And that brings us to Supercell ID. The Genesis of Supercell ID Supercell ID was born as a basic account system – something to help users recover accounts and move them to new devices. It was originally implemented as a relatively simple HTTP API. Edvard explained, “The client could perform HTTP queries to the account API, which mainly returned signed tokens that the client could present to the game server to prove their identity. Some operations, like making friend requests, required the account API to send a notification to another player. For example, ‘Do you approve this friend request?’ For that purpose, there was an event queue for notifications. We would post the event there, and the game backend would forward the notification to the client using the game socket.” Enter Two-Way Communication After Edvard joined the Supercell ID project in late 2020, he started working on the notification backend – mainly for cross-promotion across their five games. He soon realized that they needed to implement two-way communication themselves, and built it as follows: Clients connected to a fleet of proxy servers, then a routing mechanism pushed events directly to clients (without going through the game). This was sufficient for the immediate goal of handling cross-promotion and friend requests. It was fairly simple and didn’t need to support high throughput or low latency. But it got them thinking bigger. They realized they could use two-way communication to significantly increase the scope of the Supercell ID system. Edvard explained, “Basically, it allowed us to implement features that were previously part of the game server. Our goal was to take features that any new games under development might need and package them into our system – thereby accelerating their development.” With that, Supercell ID began transforming into a cross-game social network that supported features like friend graphs, teaming up, chat, and friend state tracking. Evolving Supercell ID into Cross-Game Social Network At this point, the Social Network side of the backend was still a single-person project, so they designed it with simplicity in mind. Enter abstraction. Finding the right abstraction “We wanted to have only one simple abstraction that would support all of our uses and could therefore be designed and implemented by a single engineer,” explained Edvard. “In other words, we wanted to avoid building a chat system, a presence system, etc. We wanted to build one thing, not many.” Finding the right abstraction was key. And a hierarchical key-value store with Change Data Capture fit the bill perfectly. Here’s how they implemented it: The top-level keys in the key-value store are topics that can be subscribed to. There’s a two-layer map under each top-level key – map(string, map(string, string)). Any change to the data under a top-level key is broadcast to all that key’s subscribers. The values in the innermost map are also timestamped. Each data source controls its own timestamps and defines the correct order. The client drops any update with an older timestamp than what it already has stored. A typical change in the data would be something like ‘level equals 10’ changes to ‘level equals 11’. As players play, they trigger all sorts of updates like this, which means a lot of small writes are involved in persisting all the events. Finding the Right Database They needed a database that would support their technical requirements and be manageable, given their minimalist team. That translated to the following criteria: Handles many small writes with low latency Supports a hierarchical data model Manages backups and cluster operations as a service ScyllaDB Cloud turned out to be a great fit. (ScyllaDB Cloud is the fully-managed version of ScyllaDB, a database known for delivering predictable low latency at scale). How it All Plays Out For an idea of how this plays out in Supercell games, let’s look at two examples. First, consider chat messages. A simple chat message might be represented in their data model as follows: <room ID> -> <timestamp_uuid> -> message -> “hi there”                                  metadata -> …                                  reactions -> … Edvard explained, “The top-level key that’s subscribed to is the chat room ID. The next level key is a timestamp-UID, so we have an ordering of each message and can query chat history. The inner map contains the actual message together with other data attached to it.” Next, let’s look at “presence”, which is used heavily in Supercell’s new (and highly anticipated) game, mo.co. The goal of presence, according to Edvard: “When teaming up for battle, you want to see in real-time the avatar and the current build of your friends – basically the weapons and equipment of your friends, as well as what they’re doing. If your friend changes their avatar or build, goes offline, or comes online, it should instantly be visible in the ‘teaming up’ menu.” Players’ state data is encoded into Supercell’s hierarchical map as follows: <player ID> -> “presence” -> weapon -> sword                              level -> 29                              status -> in battle Note that: The top level is the player ID, the second level is the type, and the inner map contains the data. Supercell ID doesn’t need to understand the data; it just forwards it to the game clients. Game clients don’t need to know the friend graph since the routing is handled by Supercell ID. Deeper into the System Architecture Let’s close with a tour of the system architecture, as provided by Edvard. “The backend is split into APIs, proxies, and event routing/storage servers. Topics live on the event routing servers and are sharded across them. A client connects to a proxy, which handles the client’s topic subscription. The proxy routes these subscriptions to the appropriate event routing servers. Endpoints (e.g., for chat and presence) send their data to the event routing servers, and all events are persisted in ScyllaDB Cloud. Each topic has a primary and backup shard. If the primary goes down, the primary shard maintains the memory sequence numbers for each message to detect lost messages. The secondary will forward messages without sequence numbers. If the primary is down, the primary coming up will trigger a refresh of state on the client, as well as resetting the sequence numbers. The API for the routing layers is a simple post-event RPC containing a batch of topic, type, key, value tuples. The job of each API is just to rewrite their data into the above tuple representation. Every event is written in ScyllaDB before broadcasting to subscribers. Our APIs are synchronous in the sense that if an API call gives a successful response, the message was persisted in ScyllaDB. Sending the same event multiple times does no harm since applying the update on the client is an idempotent operation, with the exception of possibly multiple sequence numbers mapping to the same message. When connecting, the proxy will figure out all your friends and subscribe to their topics, same for chat groups you belong to. We also subscribe to topics for the connecting client. These are used for sending notifications to the client, like friend requests and cross promotions. A router reboot triggers a resubscription to topics from the proxy. We use Protocol Buffers to save on bandwidth cost. All load balancing is at the TCP level to guarantee that requests over the same HTTP/2 connection are handled by the same TCP socket on the proxy. This lets us cache certain information in memory on the initial listen, so we don’t need to refetch on other requests. We have enough concurrent clients that we don’t need to separately load balance individual HTTP/2 requests, as traffic is evenly distributed anyway, and requests are about equally expensive to handle across different users. We use persistent sockets between proxies and routers. This way, we can easily send tens of thousands of subscriptions per second to a single router without an issue.” But It’s Not Game Over If you want to watch the complete tech talk, just press play below: And if you want to read more about ScyllaDB’s role in the gaming world, you might also want to read: Epic Games: How Epic Games uses ScyllaDB as a binary cache in front of NVMe and S3 to accelerate global distribution of large game assets used by Unreal Cloud DDC. Tencent Games: How Tencent Games built service architecture based on CQRS and event sourcing patterns with Pulsar and ScyllaDB. Discord: How Discord uses ScyllaDB to power their massive growth, moving from a niche gaming platform to one of the world’s largest communication platforms.