Agent Memory at Monster Scale with Mem0 and ScyllaDB Cloud
Combine Mem0’s memory management with ScyllaDB’s persistence features to deploy large-scale AI agents Scaling AI agents to handle persistent memory for a large number of users (e.g. 500k+ DAU) introduces engineering bottlenecks. These complications generally compound across two areas: how context is filtered for the model and how that data is stored globally. Mem0 addresses context efficiency by processing raw interactions into concise, entity-based facts. Instead of storing entire chat transcripts, it extracts and updates specific user preferences and historical context across sessions. This extraction method reduces the volume of data sent to the LLM, keeping prompt size manageable while retaining long-term user context. To operate at a global scale, this memory layer requires a database that handles high-throughput, low-latency vector search across multiple regions. ScyllaDB Cloud provides this distributed database infrastructure with high availability and strict P99 guarantees. Combining Mem0’s memory management with ScyllaDB’s persistence features allows teams to deploy large-scale AI agents. This post shows you how Mem0 and ScyllaDB Cloud can be used together. Memory: A Core Building Block of Agentic Applications An agent might run for minutes, make dozens of tool calls, coordinate with other agents, and serve the same user across days or weeks. At each step, it needs context: what did this user say before? What facts are relevant right now? What happened earlier in this run? Agent memory (semantic memory) covers facts, preferences, and learned context that persist across sessions and agents. “Tom is vegetarian.” “Nicole prefers concise answers.” This layer needs vector search so the agent can retrieve relevant memories by meaning, not just by key lookup. An agent memory framework solves the application side of this problem. It extracts facts from conversations, stores them as vector embeddings, and retrieves the most relevant ones at query time. You also need a database that the framework will use to store the embeddings and memories. In the case of Mem0, you have plenty to choose from. Global, large-scale agentic use cases require the memory layer to be highly available, geographically distributed, and scale with the application. This is why choosing the right database backing your memory infrastructure matters as much, or even more, as the memory framework itself. Another important context layer element is agent state. Agent state covers task progress, intermediate results, checkpoints, and tool outputs. A great stack for agent state is ScyllaDB + LangGraph. You can learn more about that in this post. What Is Mem0? Mem0 is an open-source memory layer for AI agents. It sits between your agent code and your storage backend. Mem0 handles the hard parts of agent memory: Extraction: given a conversation or a piece of text, Mem0 calls an LLM to extract discrete facts. Deduplication: before storing a new fact, Mem0 checks whether a semantically similar memory already exists and updates it instead of creating a duplicate. Retrieval:
search() embeds a query
and runs an ANN lookup to return the most relevant memories for a
given user or agent. Lifecycle management:
add(), get(), get_all(),
search(), update(),
delete(). A clean API that abstracts the underlying
vector store. Mem0 supports multiple LLM providers (OpenAI, Groq,
Anthropic, etc.) and multiple vector store backends. One of those
backends is ScyllaDB. What Is ScyllaDB Cloud? ScyllaDB is a
distributed high-performance database with vector support built for
high-throughput, multi-region workloads. ScyllaDB
Cloud is a fully managed database-as-a-service. Once connected,
the ScyllaDB team handles infrastructure, monitoring, autoscaling,
and backups automatically. The properties that make ScyllaDB Cloud
a good fit for an agent memory backend: Vector
Search: vector<float, N> is a
first-class column type. HNSW indexes are created with a single
CREATE CUSTOM INDEX statement and support multiple
similarity functions. Millions of ops/sec:
ScyllaDB is designed to handle millions of operations per second
with low and predictable latency. Horizontal
scalability: add new nodes or even new regions to your
cluster without complexity. Autoscaling
capabilities: automatically scales up and down based on
real-time load, ensuring consistent performance for unpredictable
agentic workloads. High availability: data is
replicated across nodes automatically. The cluster continues to
serve reads and writes even if a node goes down. There is no single
point of failure. Battle-tested at scale: major
enterprises such as
Discord,
Zillow, and Freshworks
depend on ScyllaDB for their data-intensive workloads.
Trusted for AI: industry leaders such as
Tripadvisor, Tubi, and
ShareChat rely on ScyllaDB to power their AI-driven workloads.
Serving Global AI Apps with Mem0 and ScyllaDB Cloud Let’s say you
are expecting thousands, hundreds of thousands, and millions of
agent loops, messages, and threads daily with hundreds of thousands
of concurrent sessions across multiple continents. Your database
that powers the memory layer is very likely to become the
bottleneck. It’s a good idea to decouple your components —
application, LLM, and database — so you can scale each part
individually to meet demand. Mem0 and ScyllaDB Cloud work well
together to allow this kind of decoupling. ScyllaDB Cloud offers
multi-region support out of the box so you can serve memories from
clusters that are closest to your application instance and users.
ScyllaDB Cloud also helps you optimize and scale two distinct kinds
of workloads independently: Standard key-based lookups (get thread
by id) Vector search queries (memory semantic search) In ScyllaDB
Cloud, non-vector search queries are served by data nodes. Vector
search queries are served by vector search nodes. This design
allows you to optimize resources and reduce infrastructure costs.
Decoupling your database from your memory layer lets you scale it
on your own terms: add nodes (horizontal scaling) or CPUs (vertical
scaling) to absorb load without touching application code or
accepting latency regressions. How ScyllaDB and Mem0 Work Together
Mem0 handles fact extraction, deduplication, embedding, and
retrieval. ScyllaDB provides the storage: high-availability,
multi-region, low-latency storage with vector search. On
add(), Mem0 sends the input to an LLM to pull out
structured facts, embeds each fact, and writes the resulting
vectors to ScyllaDB. On search(), Mem0 embeds the
query and runs an ANN query against ScyllaDB’s vector index,
returning the most semantically relevant memories. The agent does
not manage embeddings or indexes directly; instead, it calls
add() and search(). Schema The schema
below creates a memories table with a
vector<float, 384> column (matching
all-MiniLM-L6-v2) and a vector index for ANN search. Secondary
indexes on user_id support get_all()
queries. CREATE KEYSPACE mem0; CREATE TABLE mem0.memories (
id UUID, user_id TEXT, agent_id TEXT, run_id TEXT, hash TEXT, data
TEXT, metadata TEXT, vector vector<float, 384>, created_at
TIMESTAMP, updated_at TIMESTAMP, PRIMARY KEY (id) ); CREATE CUSTOM
INDEX IF NOT EXISTS memories_vector_idx ON mem0.memories (vector)
USING 'vector_index' WITH OPTIONS = { 'similarity_function':
'COSINE' }; CREATE INDEX IF NOT EXISTS memories_user_id_idx ON
mem0.memories (user_id); ScyllaDB and Mem0 Configuration
ScyllaDB is CQL (Cassandra Query Language) compatible. You can
integrate ScyllaDB with Mem0 using the following configuration:
import os from dotenv import load_dotenv from
cassandra.policies import DCAwareRoundRobinPolicy from mem0 import
Memory load_dotenv() config = { "vector_store": { "provider":
"cassandra", "config": { "contact_points": [
os.environ["SCYLLADB_ADDRESS"] ], "port": 9042, "username":
os.environ["SCYLLADB_USERNAME"], "password":
os.environ["SCYLLADB_PASSWORD"], "keyspace": "mem0",
"collection_name": "memories", # DC-aware routing
"load_balancing_policy": DCAwareRoundRobinPolicy(
local_dc=os.environ.get( "SCYLLADB_DATACENTER", "AWS_US_EAST_1" )
), # matches all-MiniLM-L6-v2 "embedding_model_dims": 384, }, },
"llm": { "provider": "groq", "config": { "model":
"mllama-3.1-70b-versatile", "api_key": os.environ["GROQ_API_KEY"],
}, }, "embedder": { "provider": "huggingface", "config": { "model":
"sentence-transformers/all-MiniLM-L6-v2" }, }, } m =
Memory.from_config(config) After configuration, you can
start using the Mem0 functions: add(),
search(), update(),
delete(). Adding Memories USER_ID = "alice" #
Add a single fact m.add( "I love hiking in the mountains,
especially in autumn.", user_id=USER_ID, ) # Add a conversation —
Mem0 extracts facts automatically m.add( [ { "role": "user",
"content": "What's a good pasta recipe?", }, { "role": "assistant",
"content": ( "Try cacio e pepe — " "it only needs three
ingredients." ), }, { "role": "user", "content": "I'm vegetarian,
by the way.", }, ], user_id=USER_ID, ) Mem0 calls the LLM to
extract facts from the conversation (“user is vegetarian”, “user
was recommended cacio e pepe”), embeds each fact using
sentence-transformers as defined in the embedder configuration, and
writes them to ScyllaDB. If a semantically equivalent memory
already exists for this user, Mem0 updates it instead. Searching
and Retrieving Memories # Semantic search results = m.search(
"outdoor activities", filters={"user_id": USER_ID}, limit=3, ) for
entry in results["results"]: print(f"[{entry['score']:.3f}]
{entry['memory']}") # [0.91] User loves hiking in the mountains,
especially in autumn. # List all memories for a user all_memories =
m.get_all(filters={"user_id": USER_ID}) # Update or delete a
specific memory m.update( all_memories["results"][0]["id"], "I love
hiking and trail running in the mountains.", )
m.delete(all_memories["results"][-1]["id"]) In an agentic
loop, call m.search(user_query, filters={"user_id":
user_id}) at the start of each turn to inject relevant
context into the prompt, and m.add(conversation_turn,
user_id=user_id) at the end to persist what was learned. Try
ScyllaDB + Mem0 The full example code is available in the
repository. To get started, provision a ScyllaDB Cloud cluster
at cloud.scylladb.com.
Resources: ScyllaDB Cloud free trial
Mem0 documentation ScyllaDB
Vector Search docs scylla-driver on PyPI What’s Coming in Cassandra 6? Key Apache Cassandra CEPs to Watch
IntroductionApache Cassandra’s contributors continue to push the database forward, and the Cassandra Enhancement Proposal (CEP) process is where that work takes shape. A CEP is a proposal to design, discuss, and build a meaningful change, with the author signaling real intent to implement it and to gather community consensus along the way.
We have previously covered CEPs here, some of which are anticipated to be present in Apache Cassandra 6 (currently in alpha). In this article we look at five CEPs that together touch many layers of Cassandra: replica consistency (CEP-45: Mutation Tracking), data placement and balancing across a cluster (CEP-60: Flexible Placements), cluster administration (CEP-38: CQL Management API and CEP-62: Cassandra Configuration Management via Sidecar), and efficient use of the underlying hardware (CEP-49: Hardware-accelerated compression).
All of these CEPs have been accepted, but as with all open source development, inclusion in a future release depends on successful implementation, community consensus, testing, and approval by project committers.
CEPs Discussed- CEP-38: CQL Management API
- CEP-45: Mutation Tracking
- CEP-49: Hardware Accelerated Compression
- CEP-60: Flexible Placements
- CEP-62: Configuration Management via Sidecar
What it does: Adds a native Cassandra CQL interface so operators can run cluster administration tasks directly through CQL instead of depending on JMX-backed tooling.
Most Cassandra admin tasks, from taking a snapshot to running a compaction, run through JMX MBeans, and the tools operators depend on, like nodetool and the Cassandra Sidecar, all speak to them over JMX. The ecosystem has worked around this over the years by wrapping JMX in REST APIs or bypassing it with Java agents, but these layers sit on top of an internal API that was never designed as a stable contract. There are further drawbacks to that coupling, from JMX’s security exposure and operational complexity to the cost of maintaining nodetool and the lack of structured command metadata from the server.
CEP-38 intends to make CQL the management interface to run commands directly, removing the dependence on JMX and external tooling in addition to aligning administration with the same interface developers already use. Defining each command once in a single registry with structured metadata gives the agents and plugins that expose a REST API something solid to build on.
Here’s an example using the CQL syntax, per the CEP’s documentation:
EXECUTE COMMAND forcecompact WITH keyspace=distributed_test_keyspace AND table=tbl AND keys=["k4", "k2", "k7"];
Just as important, it moves command execution toward an asynchronous, observable model: instead of executing and blocking, a command can be submitted, return an identifier to track it, and have its result observed afterward.
This flow is intended to lay the groundwork for automation and higher-level workflows in the future. Underpinning both the CQL interface and this execution model is a single command registry that defines each command once and exposes it consistently across interfaces. This would prevent drift between JMX, the CLI, and any REST layer.
With behavior centralized, nodetool and cqlsh stop being separate implementations and become thin entry points over the same operations. A dedicated management surface that can be reached on its own admin port gives the control plane a clear boundary that higher-level orchestration can build on.
This CEP benefits operators administering clusters and developers building and working with management tooling. Crucially, the CEP doesn’t aim to remove or deprecate the existing MBeans or CLI tools. JMX will keep working, but it nudges the project toward a state where deprecating JMX could eventually become feasible.
CEP-45: Cassandra mutation tracking for replica consistencyWhat it does: Tracks individual writes by ID rather than comparing whole partitions across replicas.
Cassandra has two ways of catching writes that didn’t reach every replica, repair and read repair, but both work by pulling stored data from multiple nodes and comparing it, which is expensive.
Repair ships whole partitions between nodes when it finds a discrepancy, driving up streaming and compaction work and making very large partitions impractical. On the other hand, read repair only fixes the slice of a partition a query touched. This can leave a write half-applied and undermine Cassandra’s partition-level write atomicity. It also can’t provide read monotonicity—an important property of quorum reads and writes—for witness replicas without read-repairing nearly every read, which has limited their usefulness.
CEP-45 takes a different approach. Rather than comparing data on disk, each write is tracked individually: the coordinator stamps every write with a unique ID that travels to the replicas, and each replica records which IDs it has applied. At read time, one replica returns the data plus a summary of its applied IDs while the others return only that summary; matching summaries mean the data is accurate, and any gaps are filled by sending the specific missing writes. A background process continually reconciles these IDs across replicas and establishes a lower bound (kind of like a watermark) that signals older log entries can be cleaned up.
The feature is enabled per keyspace or table through a new replication-type setting and reuses Accord’s addressable commit log, which adds an index over Cassandra’s commit log so individual entries can be retrieved by ID.
Repair and read repair have long been operational burdens for Cassandra operators. Reconciling at the level of individual writes instead of whole partitions should cut streaming and compaction cost and ease partition-size limits.
CEP-49: Hardware-accelerated compressionWhat it does: Offloads compression work to hardware accelerators where available, freeing CPU for other tasks.
Cassandra ships with four compressors (LZ4, Zstd, Deflate, and Snappy) and compressing and decompressing data eats a meaningful share of CPU during flush and compaction. Compression can also apply to the commitlog and to data moving across the network.
Some newer processors carry built-in accelerators for this work, such as Intel’s QuickAssist Technology (QAT) on Intel Xeon chips, which can accelerate LZ4, Zstd, and Deflate. The proposal intends to hand compression off to that hardware where it exists, freeing CPU for other tasks and speeding up compression itself.
CEP-49 adds a framework that uses the accelerator when present and reverts to the software compressor otherwise, with room to plug in other accelerators later. Backends ship as separate plugins that Cassandra discovers at startup and it falls back to the standard compressor if a plugin fails.
The main beneficiaries are operators running compression-heavy workloads on capable hardware, though the framework is also designed to support other hardware-based compressors in the future.
Note: The hardware must already be configured correctly, and anything not functioning falls back to default software-based compression.
CEP-60: Flexible Placements for cluster scaling and balancingWhat it does: Decouples data placement from token ring position, enabling steadier cluster utilization and more granular scaling.
In the current model, a node’s token positions on the ring determine which data it owns, which causes several problems. Growing a cluster cheaply tends to require doubling it; while a node joins, its range is temporarily served by an extra replica, adding load. On vnodes, tokens can’t be moved, so fixing an unbalanced ring falls on the operator and there’s no way to plan a large change as one operation or break a long one into smaller retriable steps.
CEP-60 decouples placement from token position, making the “tablet”—a range for a specific keyspace/table pair—the unit of ownership, with replicas assigned directly. Built on CEP-21’s transactional cluster metadata, it lets bootstrap and streaming run in smaller resumable steps. It also decides where data lives using per-range load and capacity metrics, moving away from ring percentage.
A central benefit is better node density. Because the cluster stays close to balanced at any size, operators can run nodes at higher, steadier utilization. By contrast, token-based growth produces a sawtooth pattern, forcing operators to provision for the peak and pay for idle capacity. Keeping nodes near a target utilization, and growing a few nodes at a time, translates into fewer wasted machines and potentially lower cost.
CEP-62: Cassandra configuration management via SidecarWhat it does: Adds a Sidecar REST API for programmatically reading and modifying cassandra.yaml and JVM options files.
Many Cassandra settings, like memtable configuration, SSTable
options, and storage_compatibility_mode, live in
cassandra.yaml and can’t be changed while a node is
running. Additionally, startup tuning such as heap size and garbage
collection lives in JVM options files. Runtime settings can be
adjusted through JMX, but for these on-disk files Cassandra offers
no programmatic interface, leaving operators to edit them by hand
or with custom scripts. An earlier addition let the Cassandra
Sidecar start and stop instances, but it still couldn’t touch the
configuration those instances read at boot.
CEP-62 fills that gap with a Sidecar REST API for reading and
changing cassandra.yaml and JVM options. It layers a
sparse “overlay” of explicit changes on top of a base template and
merges the two into the configuration a node actually uses; a
pluggable provider can keep overlays locally or in a central system
like etcd or Consul. A version-aware check rejects settings a given
Cassandra version wouldn’t recognize, reducing the risk that a typo
or unsupported setting leaves a node unable to start. Changes apply
on the next restart, and everything lives in Sidecar with Cassandra
left untouched.
This helps operators managing configuration across many nodes, especially those wiring Cassandra into centralized configuration tooling. It’s disabled by default and purely additive, so existing deployments and anyone not running Sidecar are unaffected, and it lays groundwork for later work on driving Cassandra upgrades through Sidecar.
ConclusionAltogether, these proposals show a project investing in the things that matter most to the people who run it: reliability, operability, and efficiency. Mutation tracking and flexible placements aim to make data consistency and cluster scaling less costly and less manual. The CQL management API and Sidecar-based configuration management give operators stabler, more programmable ways to administer their clusters. Hardware-accelerated compression squeezes more out of modern hardware.
Each feature aims to lower the operational challenges of running Cassandra at scale both for self-hosted clusters and managed Cassandra providers such as NetApp Instaclustr, and several CEPs lay groundwork that future enhancements will build on.
Following the CEP process is one of the best ways to see where Cassandra is headed. We’ll keep tracking these proposals as they move through implementation, and we look forward to seeing them land in users’ hands in future releases.
Ready to run Cassandra without the operational complexity? Try NetApp Instaclustr for Apache Cassandra free for 30 days today! Our managed platform handles the infrastructure, configuration, and operational heavy lifting so your team can focus on building applications.
The post What’s Coming in Cassandra 6? Key Apache Cassandra CEPs to Watch appeared first on Instaclustr.
ScyllaDB vs Aerospike, Wide-Column vs. Key/Value
Wide-column flexibility doesn’t have to come at the expense of performance — see where the two models differ, where each one wins, and why you no longer have to choose Aerospike published a paid benchmark to show it’s faster. Color me surprised…it defined the winner as itself. Aerospike benchmarked the one workload its architecture is built for. This article shares the fuller picture: what a wide-column database does that a key-value store can’t, why that’s a genuinely harder engineering problem, and how — even with broader capabilities — ScyllaDB still beats Aerospike at its own (rather limited) game. Let’s start by giving Aerospike its due. It’s a very good key/value database. Caching keys in RAM while values sit on fast NVMe is a clever design for a simple K/V model: you get excellent throughput and superb low latency. But when a vendor enters your backyard with a benchmark, the right response is to educate the market on what the numbers leave out. The challenge also inspired us to build a pure K/V option of our own. (More on that later.) ScyllaDB is a more sophisticated database. It belongs to the wide-column NoSQL family, a strict superset of key/value. The model gives every row a full set of clustering keys. A single row can hold millions of independent entries, and any one cell can be read or written without touching the rest. High availability and disaster recovery are baked into the model, along with configurable replication, local and global indexes, best-in-class elasticity, and a managed service with bring-your-own-cloud, encryption, bring-your-own-keys, and security built in. This post explains why wide-column flexibility doesn’t have to come at the expense of performance. We’ll show where the two models differ, where each one wins, and why you no longer have to choose. Wide-column (KKV) vs key/value (KV) A key/value store is a collection of rows, each with a key and a value. That value might be a single element or a set of bins, but they all share one unit of allocation. A wide-column database, by contrast, can host millions of independent cells in each row. Think of it as a key-key-value: a partition key locates the row, and ordered clustering keys locate multiple cells within it. Clustering keys unlock real access patterns Because clustering keys allow you to sort, ScyllaDB is a natural time-series database. Partition bysensor_id, cluster by timestamp, and
“fetch every reading for sensor A between Tuesday and Friday”
becomes an efficient range scan. The same shape powers messaging,
event logs, IoT, and ML feature stores. A few
of the patterns this enables: Time-series: latest-N is a
cheap head read 
PRIMARY KEY ((sensor_id),
reading_ts) WITH CLUSTERING ORDER BY (reading_ts DESC) -- Range
slice: WHERE sensor_id=? AND reading_ts >= ? AND reading_ts <
? Use partition keys as partition IDs and time buckets as
clustering keys. It’s a classic time-series use case with features
like TTL, time window compaction strategies, and easy tiering of
time-based storage. Messaging (the Discord shape): order
comes for free PRIMARY KEY ((channel_id, bucket),
message_id) WITH CLUSTERING ORDER BY (message_id DESC) And
more patterns, not just time-based… Leaderboard: the top of
the partition IS the ranking PRIMARY KEY ((game_id,
season), score, player_id) WITH CLUSTERING ORDER BY (score
DESC) Key-key-value / generic attribute
store Emulate a map of independently updatable fields per
entity by clustering on the attribute name. This is the
cell-level-update advantage over plain K/V — no
read-modify-write, explained below. PRIMARY KEY ((entity_id),
attr_name) -- (attr_name, value) rows Adjacency list
/ graph edges Partition per vertex, cluster by edge type
then neighbor, so you can slice “all FOLLOWS edges for user X.”
PRIMARY KEY ((vertex_id), edge_type, target_id)
Append-only event log / event sourcing Immutable
events ordered within an aggregate. Replay is a single partition
scan. PRIMARY KEY ((aggregate_id), event_seq) WITH CLUSTERING
ORDER BY (event_seq ASC) Partition/topic sharding
for smaller building blocks PRIMARY KEY
((celebrity_id, shard), post_ts) A key/value store can only
approximate some of these patterns, typically by relying on
secondary indexes and their overhead, or by stuffing ever-growing
lists into a single bin (which bloats records and creates
bottlenecks). Look Ma, no read-modify-write! When a K/V store needs
to update a single element inside a large value, it can’t touch
just that element (since the element shares storage units with
other fields written in the past or even in parallel). It has to
read the whole object, merge the change in memory, and write the
whole thing back. That’s the read-modify-write (RMW) tax, and it
shows up as write amplification. Aerospike provides optimized
concurrency control for users through a generation number exchanged
between client and server:
When concurrent accesses touch the same cell, this makes a ton
of sense. But within a large partition, it produces an enormous
amount of false sharing. ScyllaDB
updates a single cell directly. A log-structured merge (LSM) tree
reconciles cells later, in the background, using I/O schedulers and
compaction controllers that run on low-priority idle time. No user
involvement is required, and there’s no impact on latency. KKV
handles more Values in the K/V model are limited in size and
treated as an atomic blob, which is exactly what exposes them to
the RMW cost above, but also what makes them easy to cache and
locate on disk. Aerospike takes a better approach than Redis: it
caches keys in RAM and fetches values from disk. Since value size
is bounded, a single disk access is enough. It’s a simple,
effective design, and it performs well. Latency is excellent: at
most one I/O is required to bring data into RAM, and modern NVMe
can do that in tens of microseconds. Wide-column adds complexity, a
challenge for the engineers who build the database but a benefit
for the people who use it. A single key can point to a giant
partition; ScyllaDB has customers running terabyte partitions. We
don’t recommend it, but the flexibility is there. The ScyllaDB
engine does much more than fetch a blob. Since partition size is
unknown ahead of time, multiple memory allocations happen during
parsing, and multiple I/O accesses may be needed to fetch a
partition range. For an example of the data structure and I/O
accesses required for such partition ranges, see
How ScyllaDB’s Trie-Based Index Delivers Up to 3X More
Throughput. To guarantee consistency for parallel readers and
writers, ScyllaDB uses MVCC (multi-version concurrency control), so
readers get consistent results. Users can range-scan subsets of
partitions, reading terabytes in a single query. The cache has to
operate at row level, not partition level. Tombstones, the markers
for deleted data, can’t be a single marker; ScyllaDB supports range
tombstones. Altogether, this is a fundamentally harder problem.
Operational costs of keeping the index in RAM Aerospike stores its
primary index in RAM at a fixed 64 bytes per key. That design
decision has three consequences: slow boot, poor efficiency on
small values, and wasted storage density. Slow
boot. Because the model assumes every key lives in RAM, a
node must rebuild its entire index on startup. We’ve seen this take
30 minutes. A new node joining the cluster has to build its index
before it’s useful, which slows elasticity, and rolling restarts
become operationally heavier as a result. Poor efficiency
on small values. Aerospike’s index costs 64 bytes of RAM
per key regardless of value size. At 1 KB records, that’s a 1:16
RAM-to-disk ratio — reasonable compared to Redis, but a costly
limiting factor as datasets grow. At 256 B records, the ratio drops
to 1:4, and you end up provisioning enormous amounts of RAM for a
mediocre result. ScyllaDB commonly runs at a 1:100 RAM-to-disk
ratio, and we’re pushing that toward 1:1000 through ongoing
engineering work: moving metadata objects from RAM to disk and
expanding tiered storage support. For large datasets, that ratio
determines how many nodes you provision. There’s an interactive
tool on this
comparison page that lets you plug in your own record size and
key count.
The same 6 TB dataset: eight emptier Aerospike nodes vs
two dense ScyllaDB nodes Dense instances go
wasted. A RAM-bound index caps how much disk a node can
usefully hold, so Aerospike can’t make effective use of
storage-dense instance families like AWS i8ge, i7ie, or i3en. The
sponsored Aerospike benchmark reveals this directly: 6 TB of raw
data spread across 8 i4g.4xlarge nodes, with roughly 15 TB of
storage per zone. That’s about 40% utilization; you’re paying for
empty disk on a deliberately less dense instance family. ScyllaDB
can host the same 6 TB on 2 i8ge.3xlarge nodes, one per zone, at 12
vCPUs and 7.5 TB each, running at 80% utilization and capable of
going up to 90%. Most deployments become data-bound over time as
datasets grow; a RAM-bound index forces you to run more, emptier
machines to compensate. How the more complex model performs Despite
all that complexity, ScyllaDB matches, and in a fair fight beats,
Aerospike’s best published benchmark numbers. Aerospike’s software
license doesn’t permit third-party benchmarking of their database,
so our methodology is to reproduce their published ScyllaDB
benchmarks on equivalent hardware and rerun them ourselves. In
August 2021, Aerospike published a petabyte-scale whitepaper
claiming 5 million TPS for a read-only workload and 3.7 million TPS
for an 80/20 read-write mix, on 20 i3en.24xlarge nodes. We
reproduced the test on 20 i3en.metal nodes with the same CPU count,
RAM, and storage. ScyllaDB sustained 7 million operations per
second for reads or writes at single-digit millisecond latencies,
and 7.5 million inserts per second at 4ms P99 latency. Aerospike
had a slight edge on P99 latency; ScyllaDB had 40% higher
throughput.
Benchmarking Petabyte-Scale NoSQL Workloads with ScyllaDB (July
2022)
Operating at Monstrous New Scales: Benchmarking Petabyte Workloads
on ScyllaDB (February 2022)
Running Operational Workloads with Aerospike at Petabyte Scale in
the Cloud on 20 Nodes (August 2021) In early 2026, Aerospike
published another benchmark, this one run by McKnight Group on
Aerospike’s behalf. Unsurprisingly, its database came out ahead,
the result of a long list of misconfigurations. A few examples:
Replication factor 3 for ScyllaDB against replication factor 2 for
Aerospike, even though ScyllaDB supports RF=2. An old, unmaintained
YCSB Cassandra client instead of a current ScyllaDB driver with
shard and tablet awareness. A modern ScyllaDB driver performs 3–4x
better. Several other configuration choices that stacked the deck
the same way. The new benchmark also references our 2022
measurements, and in doing so explains why ScyllaDB showed superior
throughput back then: storage density. That benchmark used i3en
nodes, with a RAM-to-storage ratio of 1:78. At that ratio,
Aerospike’s standard mode can’t work, so the vendor’s own benchmark
runners switched it to flash mode, storing keys on disk as well.
The result was ScyllaDB with 40% better throughput. In similar
storage-bound cases, the TCO gap widens further, since Aerospike
charges high per-terabyte rates. We reran the ScyllaDB side with a
corrected configuration and matched Aerospike’s throughput using
our standard KKV log-structured merge tree. ScyllaDB’s P99 latency
came in 1–2ms higher than Aerospike’s, only at P99, not at P50. The
KKV model does more work per operation and may require multiple
disk I/O accesses. We’re not going to pretend otherwise. One note
on what that benchmark actually tested: key/value only, the one
schema Aerospike supports, with large 2 KB values that mask the
all-index-in-RAM problem. It skipped single-cell RMW updates, node
failures, and scaling events, precisely the conditions where the
model difference shows up. We’re proud of the shard-per-core
architecture that lets us extract maximum throughput from the
hardware, and the tablet architecture that drives elastic scale.
But the 1–2ms P99 difference on that workload was real. It’s the
cost of running a more complex KKV model that does more work per
operation, and there are plenty of pure K/V use cases like it.
Teams running those workloads on Redis or Aerospike often end up
overprovisioning RAM to compensate. That’s what led us to build a
storage format that keeps ScyllaDB’s shard-per-core architecture,
schedulers, and elastic scale, while cutting I/O accesses to the
minimum for simple K/V workloads: Logstor. Meet Logstor,
ScyllaDB’s new native K/V format Logstor inherits everything that
makes ScyllaDB ScyllaDB: tablets, the managed service, advanced
consistency options, and adds a simple value type to push
performance further. It’s named after the LSA (log-structured
allocator) that already manages ScyllaDB’s in-RAM memory; Logstor
applies that same algorithm to managing space on disk. For some
workloads, the wide-column KKV format is the right tool. For
others, genuinely simple key/value lookups, the new K/V format is
the better choice, and in those cases Logstor outperforms
Aerospike. We’re completing results on the current i8g generation
and will publish them soon. Either way, teams can now run both
workload types on a single, elastic, fully managed database:
ScyllaDB. What the ScyllaDB managed service brings Teams also
choose ScyllaDB for reasons beyond performance. ScyllaDB Cloud has
run data-intensive production workloads since 2019 for teams
including
Tripadvisor,
SAS, Freshworks, and
Supercell. It offers: Elasticity via tablets and ScyllaDB X
Cloud: scale from 500K to 2M OPS in about 10 minutes. Schema for
type enforcement and safety, plus a more relaxed
DynamoDB-compatible API. Configurable consistency: global quorum,
each-quorum (per data center), and per-query consistency levels,
along with advanced networking, compression, and workload
prioritization. Native multi-DC with eventually consistent
active-active replication and local reads and writes, not CDC
bolted on between data centers. Control over
availability-versus-cost trade-offs, from replication factor 2 to
quorum-based majority of 3 or 5. Global, local, and materialized
view indexes. Simultaneous scale up and out: fully utilize large
nodes while adding nodes in parallel. 24×7 support with a 15-minute
SLA and the leading rating among NoSQL providers on G2. API
compatibility with both Cassandra and DynamoDB. You don’t need to
compromise Your team can have a best-in-class managed service,
elasticity, and high availability, along with access to both a
feature-rich wide-column database and a pure key/value store that
holds its own in the industry. Pick the model that fits each
workload and run both on the same elastic, fully managed ScyllaDB
platform. Like every vendor, we can show you a synthetic benchmark
where we win. We’d rather prove it on your own workload. Our
engineers can help you get
started and share strategies for an accurate comparison. 
Cutting P99 Latency 1000X During Connection Storms by Hardening ScyllaDB Admission Control
ScyllaDB successfully mitigated performance-degrading connection storms by optimizing caching, throttling, and password hashing to achieve a 1000x reduction in tail latency The story begins with a customer-visible problem: node restarts spiked P99 latency to 5 seconds for a full minute, while typical latency for the cluster was only 4 milliseconds. The cause was a flood of new connections. For ScyllaDB to achieve ultra-low latency, new client connections go directly to each CPU core. For instance, on clusters with 9 nodes and 64 cores (vCPUs), that’s 500+ connections per client. The number of client processes can also range from a few processes to tens of thousands of processes (in large k8s systems). Then, the effective number of connections can range from tens of thousands to millions. This isn’t a problem in a steady state, but – during topology operations or a customer’s application restart – it could trigger massive surges of new connections. While ScyllaDB can handle millions of operations per second without any problem, we couldn’t do the same with connections. Establishing a new connection requires additional work and triggers code paths that can become problematic at scale, as we’ll discuss later. Clients could create new connections gradually; this way, no excessive strain would be put on the database. But ScyllaDB needs to handle even cases where clients are not behaving well. Worse, the problem can quickly snowball: a surge bloats the task queues, connections miss their timeouts, and clients retry… which means even more connections and additional pressure on the system. We don’t want to contribute to a cascading failure when graceful startup sequences aren’t implemented or fail for any reason. Moreover, we try to be as fast as possible; deployment changes should not be delayed by the database warming up when it’s not absolutely necessary. For all the above reasons, we decided to improve efficiency around connection creation. This led us to investigate and optimize the code on multiple fronts. From TCP SYN to Query-Ready The flow of establishing a new connection looks like this: We introduced improvements at nearly every stage of this process. Each tackles connection load differently: Throttling limits how many connections work at once Caching removes redundant work Hashing makes unavoidable CPU work yield A dedicated service level isolates background traffic In the next sections, we’ll walk through this pipeline step by step. Connection throttling To fully utilize the CPU, ScyllaDB divides work into tasks, and the CPU scheduler switches between them. Tasks live in queues, from which they are picked for execution. A much more in-depth dive into our CPU scheduler can be found in this P99 CONF talk by our co-founder/CTO Avi Kivity. When a new connection request arrives, we create a task to handle it. The scheduler gives it some CPU time, then — after it depletes or we need to wait for some I/O – we switch to another task. When many new connections are attempted, units of CPU are sent to the scheduler’s task queue. The scheduler tries to divide the work across time slices, but its task queue is bloated. We strive to fairly progress all connections at once. However, in the end, most of them are not fully established within the timeout. That leads to client retries and even more overload. As a first line of defense, we added a connection throttling mechanism. We already had request throttling and shedding, but nothing that controlled connection admission. The main design objective was that it should work automatically; we didn’t want the user to benchmark their system and put some rigid number in the configuration. We split new connection work into two states: “CPU” state and “networking” state. A connection is marked as being in a networking state while sending or receiving packets from the driver. And since our bottleneck is CPU, we look at how many connections are in a CPU state when we’re accepting new connections, and we allow only a small fixed number (the default is 8). Although this limit can be configured via the
uninitialized_connections_semaphore_cpu_concurrency
option in scylla.yaml, we don’t expect that users will need to
adjust it. In practice, there may be more connections in “CPU”
state than this fixed number (because once a connection is
admitted, it can switch the states without blocking). Once we spend
some CPU on handling the connection, it’s more important to
continue with it than to switch focus to some newer connection.
This ensures that the system can make progress and we don’t hit
timeouts. When new connections are waiting over 1 minute to be
admitted, we start shedding them. In the end, there is a certain
number of new connections we can handle. The result is that the
system keeps making progress under a storm instead of thrashing
across thousands of half-finished handshakes and timing them all
out. A unified permissions cache We noticed that, in some cases,
the cluster generated ~3k ops/s of internal reads just to refresh
the permissions cache. Moreover, role information – which is only
needed during connection setup (unlike permissions) – was not
cached at all. We started with low-hanging fruits. The roles table
was queried redundantly because permissions form a tree of
inherited grants; to populate a single cache entry, we issued
several selects on the roles table. By improving the way inherited
roles were gathered, we decreased the load for the roles table by
50%. But there shouldn’t be any load at all; roles
and permissions tables are usually very small compared to users’
tables. There is no need to involve the full database query
machinery. We already had a cache for permissions, but it had
several drawbacks. The primary one was that it was based on a
periodic refresh of entries. This was causing the internal load
mentioned above, together with some unpredictable latency. And our
cache didn’t cover per-connection authentication, since it didn’t
store the roles table. We decided to create a unified cache. A
typical problem to tackle with any cache is cache
coherence. Luckily, we already solved this when we changed the
replication of auth tables to Raft. Writes to those tables are
linearized by the Raft consensus protocol, and they are applied in
the same order on all nodes. Tapping into this mechanism allowed us
to have a coherent, write-updated cache. It also eliminated the
need for periodic refresh. After the change, the internal load
spikes briefly during loading and then disappears completely.
Making hashing yield For one of our customers, we noticed that node
restarts were causing elevated latency for a short period of time.
The P95 latency spiked to 5 seconds for around a minute, while the
typical latency for this cluster was only 4 milliseconds. We take
latency seriously at ScyllaDB, so we had to fix this. After
investigation, we realized that our next improvement target has to
be password hashing. The CQL protocol has this odd property that,
during connection authentication, passwords are sent and hashed on
the server side. We can’t cache hashing because we don’t want to
store raw passwords in memory for security reasons. The problem
wasn’t CPU usage by itself, but rather how it was allocated.
ScyllaDB uses futures-based execution with cooperative scheduling.
This allows us to write asynchronous code easily. Typically, yield
points are triggered by an IO operation. But when CPU-intensive
code (such as hashing) runs, we need to manually insert yield
points into the code. Otherwise, the scheduler can’t fairly switch
CPU execution to other tasks, hence elevated request latency.
Unfortunately, we can’t do that easily: we use libcrypt, an
external library. Maintaining our own hashing function in such a
security-sensitive context is not something we were keen on doing.
Initially, we tried to use something we call an alien thread. It’s
a standard OS thread where execution is subject to preemptive
scheduling. We decided to have a single thread per instance.
Otherwise, we’d proportionally decrease OS time for our own
internal scheduler, defeating its purpose. But the cure was worse
than the disease. In our tests with just 10 CPUs, we reduced
connection-per-shard rates 6X. And on larger nodes, the decline
would be even bigger. Our second solution was to fork hashing code
from the musl library and then add yield points. This is a middle
ground between writing our own hashing code and an ineffective
attempt to use library code without modifications. Connections per
second per shard were measured as follows: Configuration Rate
Initially 244 conns/s Alien thread 41 conns/s Modified musl 198
conns/s We sacrificed around 20% of the hashing
speed, since musl is less efficient than libcrypt. It impacted only
our connection rate, not our request throughput. That’s an easy
trade to kill a multi-second latency cliff. Isolating startup
traffic ScyllaDB has a workload prioritization feature where
multiple workloads can be isolated so they don’t affect one
another’s performance. More information about it can be found
here in the documentation. But, as described in the From
TCP SYN to Query-Ready section, service level assignment
happens quite late during connection establishment. It needs an
authenticated role to find an appropriate service level. Any
work done before that runs with the main scheduling group and can
compete with user workloads. Moreover, each driver instance keeps a
control connection for tracking schema changes, topology,
heartbeats, etc. Historically, such work was done in the main
scheduling group. With the default setup, it could affect user
workloads. That’s why we introduced a new driver service level that
has reduced the number of shares. This service level is
automatically selected at connection startup and then switched to
the appropriate level once it is known. Our synthetic tests showed
that, during heavy connection storms, this change alone reduced the
maximum latency of CQL requests from 225 ms to
41 ms. Benchmarking We measured the performance
gains at two levels: targeted micro-benchmarks that stress
individual parts of the implementation, and cluster-wide load tests
for the final result. Here is the impact of just the cache
optimizations: Metric BEFORE new cache AFTER new cache Delta
Instructions / Conn 4,226,658 3,861,944 -8.63% Allocs / Conn ~407.0
~182.5 -55.16% Tasks / Conn ~139.3 ~82.0
-41.13% This benchmark runs a single ScyllaDB node
and then floods it with connections. For each connection, it
performs a CQL startup sequence (including authentication), and
then closes the connection. We see that stress caused by connection
storms to the memory subsystem is reduced by half, and stress to
task queues was reduced by almost half. The following charts show
the final result of all the improvements to mitigate
connection storm impact on the cluster: Before
After
Here, we evaluated ScyllaDB’s ability to maintain availability and
performance during a node restart under sustained, production-like
load. The benchmark was conducted on a nine-node cluster of
i4i.2xlarge instances deployed in a single availability zone and
distributed across three racks, with a replication factor of three.
To simulate a demanding client workload, 20 c7i.2xlarge loader
instances generated approximately 380,000 concurrent
connections—around 6,000 per shard. The load was 74,000 read
operations per second and 12,000 write operations per second.
Client request timeout was set to 10 seconds, connection timeout to
11 seconds, and control connection timeout to 6 seconds. Prior to
the benchmark, the cluster was populated with approximately 65 GB
of data. The workload was then executed continuously for 10
minutes, after which one randomly selected node was restarted while
the load remained active, allowing us to observe the cluster’s
behavior and recovery characteristics during a node restart. The
red lines denote the time the node was restarting; since all the
driver instances need to reconnect, this is a common case which
causes connection storms. Latency is now 1000x
lower than it was before, making the impact of new connections
barely visible to the cluster. Add it to your ScyllaDB deployment
Our goal is simple: you should be able to focus on your application
logic, not on babysitting database connection parameters. With
these changes, you should be able to deploy and restart with
confidence, knowing that your cluster is designed to handle traffic
spikes gracefully. All these improvements are in the new 2026.2
release; we strongly encourage you to upgrade and share how it’s
working in your environment. 
How ScyllaDB’s Trie-Based Index Delivers Up to 3X More Throughput
By transitioning from separate summary and index files to a prefix tree, we optimized cache efficiency, reduced disk I/O, and reduced memory overhead Trie-based SSTable index format was added in ScyllaDB 2025.4. Since then, it has evolved and matured to become the default index format in ScyllaDB 2026.2. In this post, we deep dive into the format change, present its pros and cons, and show our latest benchmark results of the legacy vs. Trie index formats. For benchmarking, we chose four different read workloads that would benefit from the Trie index format in different degrees. For all four, Trie indexes demonstrated better performance. They achieved 30% to 230% higher throughput and 31% to 63% lower latency compared to legacy indexes. The impact of Trie index on the write path is negligible. Trie Index Usage in ScyllaDB Before explaining the new format, we will cover the legacy index format and its challenges. Legacy Three-Layer Lookup (me/md format) Until ScyllaDB 2026.2 the default storage format was the SSTable version md and me. Every SSTable lookup in the me/md format traverses three or four structures:Summary.db
(entirely in RAM) Binary search in Index.db Sequential
read in Data.db Both the partition index and the
clustering-row index are stored in Index.db. The
partition index is partially represented in memory by the sampled
Summary.db, while the clustering-row index exists as a
promoted index for large partitions. ┌──────── MEMORY
──────────────────────────────┐ │ │ │ Summary.db (entirely in RAM)
│ │ ───────────────────────────── │ │ Sampled at ~1 byte per ~2000
bytes of │ │ Data.db │ │ │ │ "aardvark" → Index.db byte 0 │ │
"kangaroo" → Index.db byte 1,048,576 │ │ "platypus" → Index.db byte
2,097,152 │ │ "zebra" → Index.db byte 3,145,728 │ │ │
└──────────────────┬───────────────────────────┘ │ binary search →
window in Index.db │ ▼ ┌──────── DISK
────────────────────────────────┐ │ │ │ Index.db │ │ ───────── │ │
"kangaroo" → Data.db: 4,096,000 │ │ "koala" → Data.db: 4,097,280 │
│ "kookaburra" → Data.db: 4,098,560 │ │ "lemur" → Data.db:
4,099,840 ← found │ │ ... (up to ~800 entries per 1 MB scan) │ │ │
└──────────────────┬───────────────────────────┘ │ 1 seek +
sequential read │ ▼ ┌──────── DISK
────────────────────────────────┐ │ Data.db │ │ <partition
data> │ └──────────────────────────────────────────────┘
For partitions containing enough clustering rows, a fourth
structure is involved: Index.db entry for a large partition
┌──────────────────────────────────────────┐ │ partition key →
Data.db offset │ │ promoted_index (flat list of CK blocks) │ │
block 0: ck_start="aaa", ck_end="azz" │ │ block 1: ck_start="baa",
ck_end="bzz" │ │ ... │ │ block N: ck_start=..., ck_end=... │ │
offsets[0..N] ← binary search here │
└──────────────────────────────────────────┘ The New Trie
Index Format The Trie index (#25626)
replaces Summary.db + Index.db with a
single on-disk prefix
tree. The storage format is compatible with Apache Cassandra’s
BTI (Big Trie Index) format, implemented using ScyllaDB’s Seastar
architecture. Trie indexes are used for both partition indexes and
clustering key indexes. What Is a Trie? A trie (prefix tree) stores
keys character-by-character. Shared prefixes occupy a single path,
eliminating redundancy: Keys: "kangaroo", "koala",
"kookaburra", "lemur", "lion" [root] / \ 'k' 'l' | | [k] [l] / \ /
\ 'a' 'o''e' 'i' | | | | 'n' [o]'m' 'i' | / \ | | 'g''a' 'o''u' 'o'
| | | | | 'a''l' 'k''r' 'n' | | | * 'r''a' 'a' * = leaf node | * |
(payload: Data.db offset) 'o' 'b' | | 'o' 'u' * | 'r' | 'r' | 'a' *
"k", "ko", "koo" — shared prefixes stored ONCE New SSTable
Files The ms/mt format replaces Summary.db and
Index.db with two purpose-built files: SSTable
(ms format) ├── Data.db │ unchanged — partition and row data, │
same binary layout as me/md │ ├── Partitions.db ← NEW │ Trie index:
│ partition key → Data.db offset │ (small partitions) │ partition
key → Rows.db offset │ (large partitions) │ │ ┌── Page 0 (4,096
bytes) ───────┐ │ │ trie root node [1] │ │ │ + children (fan-out ≤
256) │ │ │ + their children (packed) │ │
└───────────────────────────────┘ │ ┌── Page 1 (4,096 bytes)
───────┐ │ │ subtree for keys 'a'–'g' │ │
└───────────────────────────────┘ │ ... │ ┌── Footer
─────────────────────┐ │ │ first_key (raw bytes) │ │ │ last_key
(raw bytes) │ │ │ partition_count (uint64) │ │ │ trie_root_pos
(uint64) │ │ └───────────────────────────────┘ │ ├── Rows.db ← NEW
│ Per-partition clustering-key │ tries, concatenated. │ Each
sub-trie: │ clustering key → byte-offset │ within partition │
(replaces flat "promoted index" │ in Index.db) │ ├── Filter.db
bloom filter — unchanged ├── Statistics.db statistics — unchanged
└── Scylla.db ScyllaDB metadata — unchanged [1] Parent nodes
are always written after their child nodes. Parents point to
children, so child positions must be known before parents are
written. How a Partition Lookup Works Query: SELECT * FROM
orders WHERE order_id = 'ORD-20240611-98765'
┌─────────────────────────────────────────┐ │ Step 1 — Key
Translation │ │ │ │ 'ORD-20240611-98765' │ │ ↓
bti_key_translation.cc │ │ comparable byte sequence │ │
(lexicographic order matches │ │ CQL semantic order) │
└──────────────────┬──────────────────────┘ │
┌──────────────────▼──────────────────────┐ │ Step 2 — Trie
Traversal │ │ in Partitions.db │ │ │ │ Read root page (4 KB) │ │ ←
usually in OS page cache │ │ │ │ [root] ──'O'──> [node] (page 0)
│ │ [node] ──'R'──> [node] (page 0) │ │ [node] ──'D'──>
[node] (fetch page 1) │ │ [node] ──'-'──> [node] (page 1) │ │
... │ │ [leaf] payload = Data.db or │ │ Rows.db pos 2,097,152 │ │ │
│ Typical: 2–6 page fetches. │ │ Top pages cached → often 0–1 disk
I/Os │ └──────────────────┬──────────────────────┘ │
┌──────────────────▼──────────────────────┐ │ Step 3 — Read Data.db
│ │ Seek to offset 2,097,152 │ │ → read partition header │ │ For
large partitions: read Rows.db │
└──────────────────┬──────────────────────┘ │ (range/clustering
queries only) ┌──────────────────▼──────────────────────┐ │ Step 4
— Read Data.db at position │ │ returned from index │
└─────────────────────────────────────────┘ Page Layout:
Packing Parent and Children Together The most critical write-time
optimization is ensuring that a node and its children land on the
same 4 KB page. This means an entire trie
neighborhood is readable in a single I/O, even on the first (cold)
access. ┌──────── 4,096-byte page ────────────────┐ │ │ │ [A]
──'p'──> [B] ──'p'──> [C] │ │ │ │ │ │ └──'r'──> [D]
├──'l'──>[E]* │ │ │ │ │ └──'y'──>[F]* │ │ │ │ (* = leaf node
with payload) │ │ (padding bytes to align next subtree │ │ to page
boundary) │ └─────────────────────────────────────────┘ trie_writer
algorithm (trie_writer.hh): 1. Maintain rightmost path root →
current node (_stack) 2. On each new key: branch off rightmost
path, add new nodes 3. Accumulate nodes until a finished subtree
exceeds a page 4. Flush child subtrees with padding so each subtree
fits within one page ScyllaDB vs. Cassandra reference impl:
Cassandra: one character per node ScyllaDB: characters grouped into
"chains" (up to 300 bytes) → dramatically faster writes for long
keys, same read-side page structure Old vs. New:
Side-by-Side Summary Legacy me/md Trie ms/mt On-disk
files Summary.db + Index.db
Partitions.db + Rows.db In-memory
component Summary (always loaded, never evicted) None.
Trie top-nodes live in OS page cache (evictable) Index
structure Flat sorted list Prefix tree Partition
lookup (cold cache) Binary search in summary (RAM) + scan
of Index.db window Trie traversal byte-by-byte: O(key_length) page
reads Key storage Full key per entry Shared
prefixes stored once Clustering key lookup Flat
promoted-index list (binary search) Sub-trie in
Rows.db (trie traversal) Benchmark Results The
following table shows the key results, measured at client-side P99
≤ 10 ms on a 3-node AWS production-class cluster. All tests were
run on 3 × i8g.2xlarge instances, each on a different zone, with
Replication Factor (RF) of 3. For a full description of the test
cases and setup, see the Appendix below. Test case Legacy (me)
Throughput Legacy (me) P99 Trie (ms) Throughput Trie (ms) P99
Throughput gain Test 1: Typical (~20% row cache) 130k ops/s 5.1 ms
170k ops/s 1.9 ms +31% Test 2: Key / Value 90k
ops/s 5.2 ms 300k ops/s 3.6 ms +233% Test 3: Large
Partitions 23k ops/s 7.7 ms 37k ops/s 4.6 ms +61%
Test 4: Long shared clustering key prefixes 22k ops/s 5.4 ms 38k
ops/s 3.3 ms +73% Discussion: Why Trie Indexes
Improve Performance, and When to Use Them ScyllaDB CTO Avi Kivity
has mentioned three reasons why Trie indexes improve performance:
Improved cacheability: the index is denser, so it
is more likely to fit in cache, requiring no I/O for the index
itself. Fewer I/O operations after cache miss: if
the index is not in cache, fewer I/O operations are required to
fetch it since the index is more compact and shallower. This is
especially true for large partitions, often seen in materialized
view workloads. CPU efficiency: less CPU is needed
to process the index during reads. A possible downside is that more
CPU is needed to create the index during memtable flush and
compaction. However, this is more than offset by the read-side
advantages. For proof of Avi’s first point, see the Disk Read
panels from ScyllaDB Monitoring’s OS metric dashboard for legacy
vs. Trie index format. Each step represents increasing workload
throughput. For the first load, using the same throughput, Disk
Reads for legacy indexes is ~240 MB/s; for Trie indexes, it is ~33
MB/s. The Trie index consumes only ~1/7th of the storage bandwidth.
Note
that the Trie index has a negligible effect for 100% cache hit rate
or 0% cache hit rate workloads. For both, the in-memory index
representation is irrelevant. Write workload is also only
marginally affected by the index format change. Summary ScyllaDB
2026.2 has adopted a Trie-based index format as its new default,
replacing the legacy index structure to significantly enhance read
path performance. By transitioning from separate summary and index
files to a prefix tree, this design optimizes cache efficiency,
reduces disk I/O, and reduces memory overhead. Benchmarks indicate
that this architectural change delivers throughput improvements
ranging up to 3x across various workloads, offering a more scalable
and efficient solution than the legacy index format. As
always, actual results depend on your workload. To evaluate the
gain of Trie index, we highly recommend testing it yourself with
the latest ScyllaDB releases. Appendix: The Full Test
Setup All tests compare me (legacy) vs. ms (trie) format on
identical code, hardware, and dataset. Metric: maximum
throughput (ops/s) at which client-side P99 latency stays below 10
ms. Hardware and Infrastructure 
Component
Specification ────────────────────────────────────────── Cloud
provider AWS us-east-1 DB nodes i8g.2xlarge × 3, RF=3, 3 racks
Loaders (normal) c7i.4xlarge × 4 (Tests 1, 3) Loaders (large)
c7i.4xlarge × 3 (Tests 2, 4) Row cache ~20% of dataset Kernel
7.0.0-1006-aws SSTable formats me (legacy) vs ms (trie)
────────────────────────────────────────── Software Versions
Component Version ScyllaDB 2026.3.0~dev cassandra-stress 3.20.6
latte 0.48.0-scylladb Java driver 3.11.5.14 Rust driver (latte)
1.6.0 Python driver 3.29.10 Test 1: Typical (~20% Row Cache)
Description: Standard cassandra-stress read
workload with ~20% row cache. Closest analogue to a general
production workload. Schema and dataset:
CREATE TABLE keyspace1.standard1 ( key blob PRIMARY KEY, C0
blob, C1 blob, C2 blob, C3 blob, C4 blob ); -- 5 columns × 256
bytes ≈ 1,280 bytes/row -- 650,000,004 rows -- (written with CL=ALL
across 4 loaders) Workload parameters:
Stress tool: cassandra-stress Consistency: QUORUM Access:
Sequential on 2 loaders, Gaussian on 2 loaders Threads: 620 total
Throttle: 70k–250k ops/s in 10k steps, 30 min per step SCT config:
test-cases/trie/ perf-steps-neutral.yaml
Builds: Format ScyllaDB revision Build date Build
ID me 1fdd379bf99c 2026-06-03 8c9256ba ms
1fdd379bf99c 2026-06-03 8c9256ba
Results: Legacy me Trie ms Max Throughput 130k
ops/s 170k ops/s P99 Latency 6.28 ms 3.19 ms Throughput
gain — +31% Latency at
saturation 6.28 ms 3.19 ms (−49%) Note:
Even at the same 130k ops/s, Trie P99 is 3.19 ms vs 6.28 ms — half
the latency, with substantial headroom before the 10ms ceiling.
Test 2: Key / Value Description: Deliberately
favorable for the trie. Schema has only a partition key with no
clustering columns. Tiny rows (~8 bytes payload) mean extremely
high partition density. The trie’s prefix compression yields much
higher effective cache density than the legacy flat list.
Schema and dataset: CREATE TABLE
trie_test.fav ( key blob PRIMARY KEY, col blob -- FIXED(1024) in
stress profile ); -- ~8 bytes/row effective (key-only access) --
650,000,004 rows Workload parameters:
Stress tool: cassandra-stress (custom profile) Throttle: me:
70k–100k ops/s ms: 230k–350k ops/s (non-overlapping by design — me
saturates far below ms) Step duration: 30 minutes SCT config:
test-cases/trie/ perf-steps-fav.yaml
Builds: Format ScyllaDB revision Build date Build
ID me 1fdd379bf99c 2026-06-03 8c9256ba ms
d8de7268e7f4 2026-06-05 b6e8613d
Results: Legacy me Trie ms Max Throughput 90k
ops/s >280k ops/s P99 Latency 6.12 ms 4.50 ms Throughput
gain — >+211% The trie can serve 3×
the request rate at lower latency because the same OS page cache
budget covers 3× more trie top-nodes than the equivalent
Index.db window. Test 3: Large Partitions
Description: Each partition holds 460,000 rows
with a large clustering key. With only ~100 total partitions,
per-shard throughput (not index access) is the bottleneck. Tests
the trie row index (Rows.db) rather than the partition
index. Schema and dataset: CREATE TABLE
trie_test.large ( pk blob, ck blob, value blob, PRIMARY KEY (pk,
ck) ); -- 46M total rows -- 460,000 rows per partition, ~100
partitions -- Accessed via: latte large-partition.rn
Workload parameters: Stress tool: latte
0.48.0-scylladb (large-partition.rn) Threads: 1 Throttle: 10k–70k
ops/s in 10k steps, 30 min per step SCT config: test-cases/trie/
perf-steps-large.yaml Loaders: 3 × c7i.4xlarge
Builds: Format ScyllaDB revision Build date Build
ID me e5b4f43ec1c8 2026-06-02 c32c9fc9 ms
e5b4f43ec1c8 2026-06-02 c32c9fc9
Results: Legacy me Trie ms Max Throughput ~19,998
ops/s ~29,998 ops/s P99 Latency 2.87 ms 3.80 ms Saturation point
~24k (at 30k target) ~37k (at 40k target) Throughput
gain — +50% Test 4: Long Shared
Clustering Key Prefixes Description: The worst
case for the trie. Clustering keys are 2048 bytes long with a long
common prefix — only the final bytes differ. This maximizes trie
depth, erodes prefix sharing, and inflates node fan-out near the
top. Same schema as Large Partitions with 10× fewer rows per
partition (46,000 vs 460,000). Schema and dataset:
CREATE TABLE trie_test.unfav ( pk blob, -- 4 bytes ck blob,
-- 2048 bytes, long shared prefix value blob, -- 1024 bytes PRIMARY
KEY (pk, ck) ); -- 46M total rows -- 46,000 rows per partition,
~1,000 partitions -- Accessed via: latte unfavorable.rn
Workload parameters: Stress tool: latte
0.48.0-scylladb (unfavorable.rn) Threads: 1 Throttle: 10k–80k ops/s
in 10k steps, 30 min per step SCT config: test-cases/trie/
perf-steps-unfav.yaml Loaders: 3 × c7i.4xlarge
Builds: Format ScyllaDB revision Build date Build
ID me a0e160db8a7d 2026-06-04 f1e189b2 ms
a0e160db8a7d 2026-06-04 f1e189b2 ScyllaDB 2026.2: DynamoDB Streams and Vector Search, Trie Indexes, and Strongly Consistent Tables
ScyllaDB 2026.2 brings a combination of GA new features, exciting experimental features, and multiple stability and external use case improvements. The updates include: DynamoDB compatible API (Alternator) enhancements: DynamoDB compatible Streams and Vector Search extension. Performance and Stability: Trie index performance improvement, and streamlining connection storm handling. ScyllaDB Cloud: New Private Link connectivity feature and Vector Search integrations (LangChain, LlamaIndex, Spring AI, Agno, and more). Experimental features: Strongly Consistent Tables and vNode-to-Tablet online migration. 2026.2 is more stable, works faster, and better than any past release. You are encouraged to upgrade to it, and use it for any new deployment. For the full release notes, see this forum post. DynamoDB compatible API (Alternator) enhancements Alternator is a native DynamoDB-compatible API on ScyllaDB. ScyllaDB 2026.2 includes two major additions to that API: DynamoDB Streams compatibility and a Vector Search extension. Streams are Generally Available Alternator Streams, ScyllaDB’s DynamoDB Streams-compatible change data capture interface, is now Generally Available (GA). Applications can capture ordered item-level changes from Alternator tables for event-driven architectures, CDC pipelines, and replication workflows. Vector Search Extension A new extension to the Amazon DynamoDB API allows you to use Semantic Vector Search directly on your data. For users in the DynamoDB environment, implementing vector search has been overly complicated. Amazon’s “Zero ETL” forces a dual-service approach (managing both DynamoDB and OpenSearch) and requires using two separate APIs just for Vector Semantic Search queries. ScyllaDB eliminated the heavy lifting by integrating vector search capabilities into Alternator, our DynamoDB-compatible API. This gives DynamoDB users high-performance similarity search within their familiar API, without the need for extra clusters or constant API context-switching. Vector Search is available in ScyllaDB Cloud. Performance: Trie index by default The Trie-based SSTable index format was added in ScyllaDB 2025.4. Since then it has evolved and matured to become the default index format in ScyllaDB 2026.2. The Trie index format increases throughput by up to x3 with lower latency compared to legacy index formats. To read more, see our blog post, How ScyllaDB’s Trie-Based Index Delivers Up to 3X More Throughput. Stability: Connection Storm Mitigation A connection storm is the result of thousands of applications reconnecting to the cluster after a node restart or a network issue. This issue is rare. But, when it happens, it’s important that it doesn’t cause latency to spike. In 2026.2, we implemented multiple mechanisms to mitigate this effect. Some of them are: Coordinator side connection throttling Unified authentication and authorization cache Improved password hashing Using a lower default service level Replica-Side Load Shedding As shown below, the effect is dramatic. From a P99 of 5 seconds during a connection storm (after a node restart) to 3.5 milliseconds. For more details, see our blog post, Cutting P99 Latency 1000X During Connection Storms by Hardening ScyllaDB Admission Controls. ScyllaDB Cloud ScyllaDB continues to expand its Vector Search capabilities with integrations to popular tools like LangChain, LlamaIndex, Spring AI, Agno and more. For more details, see our upcoming blog post, Build Durable Chat Memory for RAG Using ScyllaDB and LangChain. New experimental features Strong Consistency: Raft Group per Tablet ScyllaDB has been using the Raft consensus algorithm for metadata like cluster topology and data distribution (Tablets) for years now. In this release, ScyllaDB started using Raft for data consistency as well. You can now create a globally consistent Keyspace, and enjoy strong consistency – with performance that’s very close to eventual consistency, and superior to the existing LWT approach. You can read more about this in Riding the Raft to Strong Consistency in ScyllaDB. vNodes to Tablets Migration Tablets, ScyllaDB’s new dynamic algorithm for data distribution, powers ScyllaDB’s extreme elasticity. With 2026.1, tablets achieved full feature parity and became recommended for all new clusters. But what about exiting production clusters? In 2026.2, ScyllaDB provides a zero downtime migration from legacy vNode to Tablets, with minimal resource usage. See Migrate a Keyspace from Vnodes to Tablets in the ScyllaDB documentation for details. Want to use one of the experimental features? See the documentation for details on how to enable them, or contact the ScyllaDB Cloud support team.
Riding the Raft to Strong Consistency in ScyllaDB
How ScyllaDB is using per-tablet Raft groups to bring strong consistency to data, without sacrificing the parallelism that makes it fast Distributed systems do not give us simple guarantees for free Distributed databases live in a world where failure is normal. Nodes fail. Networks could have partitions. Clocks might be different in each area that you’re working in. Messages can be delayed or never arrive because of the network itself. A request that looks simple from the application side may cross replicas, shards, data centers, coordinators, and recovery paths before the database can safely answer it. Consistency is one of the most important contracts between the database and the application…and it’s hard. When the database says a write succeeded, what exactly does that mean? When a second client reads the same key, what state should it see? When two updates race, who wins, and can the application reason about the result? For many workloads, eventual consistency is the right approach. It gives a distributed database room to stay highly available and fast, even when part of the system is under stress. For now, these workloads are where ScyllaDB shines. But for other workloads, “eventually correct” creates too much ambiguity. Those workloads need a stronger contract. Eventual consistency was the right foundation for ScyllaDB ScyllaDB started as a high-performance, Cassandra-compatible, eventually consistent database. That model is powerful: the system is leaderless, work is distributed across shards, and applications choose consistency levels such as ONE, QUORUM, or ALL (depending on their needs). In this model, a client sends a write to a coordinator, the coordinator forwards it to replicas, and the operation is acknowledged according to the selected consistency level. Reads follow a similar pattern: replicas respond, and the coordinator waits for enough responses to satisfy the requested consistency level. Figure 1: Write and read patterns This design gives applications a flexible tradeoff between latency, availability, and consistency. It remains valuable, especially for high-scale workloads that prioritize availability and throughput. But some parts of a database should not be “eventual” Eventual consistency becomes harder when the data being changed is not naturally commutative, when different observers must agree on one order of events, or when a wrong answer is expensive. Metadata is the clearest example. Schema, topology, tablet placement, and cluster state describe the database’s internal operations. If nodes disagree about this information, the system becomes harder to operate safely. The same pattern appears in application data. Counters, account balances, inventory reservations, entitlement checks, idempotency records, and conditional updates all become simpler when the database can provide a clear, strongly ordered answer. Without that guarantee, application developers often compensate by adding retries, custom conflict resolution, reconciliation jobs, or application-side locks. Figure 2: Gossip-based topology spreads membership information without a single global source of truth, which is powerful for availability but eventually consistent by design. For example, “last-write-wins” sounds simple: keep the newest update. But if two clients update the same value at the same time, one update can disappear.
Initial value: {} Client
1 writes: {A} Client 2 writes: {B} Final value: {B} Both
writes may have been accepted, but only one survives. The problem
is that last-write-wins avoids coordination, but it does not merge
intent. For mutable data, this can mean lost updates. To learn
more: https://aphyr.com/posts/294-jepsen-cassandra
Strong consistency gives the system one accepted order An
alternative approach is strong consistency. Instead of allowing
independent updates to happen concurrently and converge later, the
system establishes one accepted order for operations. In practical
terms, this means that once an operation is committed, later
operations observe that committed state according to the
consistency model. This provides correctness as well as simplicity.
Developers can reason about the database as if there is one clear
sequence of operations. Operators can reason about topology and
schema changes as deterministic state transitions. And the database
can remove entire classes of ambiguity because it no longer needs
to guess which version of the world is the right one. Strong
consistency is an all-around simpler programming model for the
parts of the application where correctness, ordering, and
predictability matter more than the lowest possible write latency.
Why Raft is the right building block Raft provides the foundation
for this stronger consistency model. At a high level, a Raft group
chooses one node to be the leader. The leader receives the request,
records it, and makes sure most of the group has the same record.
Only then does the group treat the request as accepted.
Figure 3: Raft request access by majority process Because
all nodes follow the same ordered list of accepted requests, they
all reach the same result in a clear and predictable way. This
model is especially useful in a distributed database because it
gives the system a clear answer to a hard question: Who is
allowed to decide what happens next? In Raft, the leader
proposes the order and the majority confirms it. ScyllaDB adopted
Raft because some database operations are too important to be left
to eventually converging metadata. Schema changes, topology
changes, and strongly consistent data operations all need a clear,
agreed-upon order. Without that, two nodes may observe changes in
different orders, or the system may need complex recovery logic to
repair disagreement after the fact.
Figure 4: Raft turns a distributed decision into an ordered
log: leader election, append, replicate to a majority, commit, and
apply. This is also important for elasticity. ScyllaDB’s
tablet architecture is designed for flexible data distribution
across the cluster, and Raft-managed topology allows operations
such as adding nodes and moving data to be coordinated safely.
ScyllaDB uses tablets, together with consistent topology updates,
as a foundation for faster and more flexible scaling. In other
words, ScyllaDB adopted Raft not just because Raft is a well-known
consensus algorithm, but because it gives the database a reliable
coordination layer. It replaces “every node eventually figures it
out” with “the group agrees on the order first, then applies the
result.” That is the foundation needed for strong consistency;
there’s just one: Agreed order. Committed history. Deterministic
path from request to replicated state. At ScyllaDB, the first step
in adopting Raft was to use it for topology and metadata changes.
This gave those critical operations strong consistency guarantees
while also reducing complexity across the system.
Figure 5: In ScyllaDB, shard 0 runs Raft for metadata tables
while the other shards continue doing user-data and storage-engine
work. The next milestone for ScyllaDB: strongly consistent
tables Our next step is bringing the same idea to user data through
strongly consistent tables. A strongly consistent table is built on
top of the Raft log. A write is routed to the Raft leader for the
relevant tablet group, appended to that group’s log, replicated to
a majority, committed, and applied to the storage engine.
Figure 6: High-level strongly consistent write path: route to
the tablet group leader, append to the Raft log, replicate to a
majority, commit, and apply to storage. This is a major shift
in how users can model correctness in ScyllaDB. Instead of treating
strong guarantees as a special case workaround, users can choose a
consistency model at the data-modeling level. Tables that need the
classic high throughput eventual consistency behavior can keep it.
Tables that need stronger ordering can use the strong path. It’s
important to note: ScyllaDB is not replacing eventual consistency.
We are offering multi-consistency. Our users will be able to choose
the right consistency model for each workload. And we will deliver
strong performance in both cases. Why a Raft Group for each tablet?
In ScyllaDB, we already use Raft for important system-level
coordination. For example, Raft is used to safely sequence topology
and schema metadata changes. That makes cluster operations
consistent and reliable. So a natural question is: If we
already have Group 0, why not use it as the single synchronization
point for all strongly consistent data operations? At
first glance, that sounds simpler. We already have one Raft group
in the system. We could use it as the “master sync point” for every
read, write, and data-related operation. But, unfortunately, it’s
not that simple. ScyllaDB clusters can contain many tablets.
Tablets are the basic units of data distribution: each tablet owns
a portion of a table’s data and can be independently managed and
moved across the cluster. To understand the issue, imagine a busy
system with many tablets, heavy reads, heavy writes, topology
activity, background work, maintenance operations, and user traffic
all happening at the same time. If all of these operations had to
pass through a single Raft group, that group would become a global
synchronization bottleneck. Because Raft is a consensus protocol,
operations must be ordered and committed consistently. (That’s what
gives us correctness). But if one global group is responsible for
ordering everything, then unrelated operations are forced into the
same queue. A write to one tablet may have to wait behind work for
another tablet. A read or write on one part of the dataset may be
delayed by background activity somewhere else in the cluster. All
this hurts both latency and throughput. A better option is to
“divide and conquer.” Since the tablet is already the natural unit
of data ownership, we can also make it the natural unit of
synchronization. Instead of forcing all strongly consistent
operations through one global Raft group, each tablet gets its own
Raft group. This means each tablet handles its own coordination,
replication, and bookkeeping. Operations on one tablet do not need
to block unrelated operations on another tablet. The system can
make progress in parallel, across many independent Raft groups,
instead of serializing everything through a single global queue.
The result is a much more scalable architecture with: Lower
contention: Each Raft group handles only the operations
for its own tablet. Better parallelism: Many
tablets can process strongly consistent operations at the same
time. Improved throughput: The system is no longer
limited by one global synchronization point. Lower
latency: Unrelated operations do not wait behind each
other as often. Better user experience: Strong
consistency becomes practical without turning the entire database
into a single serialized pipeline. This is the same basic reason
ScyllaDB’s tablet architecture exists in the first place: splitting
data into smaller independent units allows the system to scale,
rebalance, and operate in parallel. Tablets were designed to
support faster, more flexible scaling by separating data ownership
from fixed server ownership. A single global Raft group is nice
because it is easier to reason about. However, ScyllaDB is built
for high-throughput, low-latency workloads. A single global queue
for all of them would immediately become the bottleneck. By
assigning a Raft group to each tablet, we keep the correctness
properties of Raft while preserving the parallelism that makes
ScyllaDB fast. Each tablet becomes an independent unit of
consistency. The cluster as a whole can continue to behave like a
distributed, parallel database rather than a single synchronized
queue. In short: Group 0 is great for global
metadata. Per-tablet Raft groups are what make
strong consistency scalable for user data. Raft
leaders and followers: one voice for the group Raft is a consensus
protocol designed around a simple idea: instead of allowing every
replica to independently decide what should happen next, the group
elects one replica to act as the leader. The other
replicas become followers. This leader-based model
makes the system easier to reason about because all changes flow
through a single authority for that Raft group. The original Raft paper describes
this as one of Raft’s main design choices: decomposing consensus
into understandable pieces, especially leader election and log
replication. In normal operation, the leader is
responsible for accepting new operations, appending them to its
log, and replicating those log entries to the followers. A follower
does not independently decide the order of operations. Instead, it
follows the leader’s log and acknowledges replicated entries. Once
an entry is safely replicated to a majority of the group, it can be
considered committed and applied to the replicated state machine.
This is the core mechanism that allows multiple machines to behave
as if they agreed on one ordered history of changes. The important
point is that the leader is not “more correct” than the followers.
It is simply the replica currently elected to coordinate the group.
Followers still store the replicated state, validate the leader’s
messages according to the Raft rules, and participate in making
progress by acknowledging replicated entries. The leader can only
commit entries when it has agreement from a quorum. This is why
Raft depends on a majority of replicas being available; without a
quorum, the group cannot safely make new decisions. ScyllaDB’s
Raft documentation also highlights this quorum requirement for
Raft-managed operations. A useful way to think about it is this:
The leader proposes the order. The followers confirm and
persist that order. The majority makes it durable. Raft
leaders also send periodic heartbeat messages to followers. These
heartbeats tell the followers that the leader is still alive and
still responsible for the current term. As long as followers keep
receiving valid communication from the leader, they remain
followers. If a follower stops hearing from the leader for long
enough, it assumes the leader may have failed and starts an
election by becoming a candidate. If that candidate receives votes
from a majority, it becomes the new leader. This election mechanism
allows the group to recover automatically when the current leader
crashes or becomes unreachable. This distinction between leader and
follower is especially important in distributed databases. A
database cluster is not running on one machine. It is running
across many machines, and those machines can fail, restart,
disconnect, or see events in different orders. Without a clear
coordination model, two replicas could make conflicting decisions
at the same time. Raft avoids that by ensuring that, for a given
term and Raft group, there is one leader responsible for sequencing
new changes. In ScyllaDB, Raft has already been used to make
important metadata operations safer and more consistent, including
schema and topology changes. ScyllaDB’s work on Raft-managed
topology means topology operations are internally sequenced
consistently, rather than relying on each node to independently
converge on the same result. ScyllaDB also has one place that
coordinates topology changes together with the Raft leader. If that
leader goes down, another leader can take over and continue from
the same shared information, instead of guessing or starting from a
different view. For strongly consistent data, the same basic idea
applies: the leader of the relevant Raft group is the place where
the ordered history of that group is created. A write is not just
“sent somewhere and eventually copied.” It is placed into a
replicated log, agreed on by a majority, and then applied in the
same order by the replicas. Followers are not passive backups in
the weak sense; they are active participants in preserving the
agreed history. This model gives us a clean mental picture:
Without Raft: replicas may need to reconcile
different views after the fact. With Raft:
replicas agree on the order first, then apply the result. That is
the key difference. Raft does not remove the complexity of
distributed systems; failures, latency, partitions, and recovery
still exist. But it gives the system a disciplined way to handle
that complexity. The leader gives each Raft group a single
coordination point, the followers provide durable replicated state,
and the quorum rule ensures that progress is made only when enough
replicas agree. In other words, Raft leader/follower replication is
not about creating one “special” node forever. It is about creating
a temporary, elected coordinator that gives the group one
consistent voice. If that voice disappears, the group elects
another one. The result is a system that can keep a strongly
ordered history of changes even while individual machines come and
go. Leader awareness: sending requests to the right place Now that
we’ve seen how every Raft group has a leader, let’s look at the
next important question: How does the client know where to
send the request? In a Raft-based system, the leader is
the replica that coordinates the work for the group. It decides the
order of operations, appends new entries to the replicated log, and
drives replication to the followers. For ScyllaDB Strong
Consistency, where every tablet has its own Raft group, this means
that every tablet also has a current Raft leader. That creates an
important difference from eventual consistency. With eventual
consistency, a client request can usually be sent to one of the
replicas, and that replica can act as the coordinator for the
operation. The driver does not necessarily need to know which
replica is “special,” because there is no Raft leader that must
order the operation first. With strong consistency, the situation
is different. If a request reaches a follower, that follower cannot
independently decide the order of the operation. The leader must
coordinate the operation. The follower may have the data, and it
may be part of the Raft group, but it is not the replica currently
responsible for sequencing new writes or strongly consistent
operations. So the request has to reach the leader. Request
forwarding To make this work, ScyllaDB implements request
forwarding. It works like this: The client sends a request
to one of the replicas. If that replica is the Raft leader for the
tablet, it handles the request directly. If that replica is a
follower, it acts as a proxy for the request. The leader processes
the request through Raft. The result is returned back through the
forwarding replica to the client. This gives us an important
correctness property: Even if the client reaches the wrong
replica, the operation is still coordinated by the Raft
leader. That is exactly what we want. The leader remains
the single place where the ordered history of the tablet is
created. Followers can help route the request, but they do not
bypass the leader or make independent decisions. This forwarding
mechanism is especially useful because leadership can change. A
leader may fail, restart, become unreachable, or step down. When
that happens, the Raft group elects a new leader. Request
forwarding gives the system a way to continue operating even when
the client does not yet know about the new leadership state. The
cost of forwarding However, request forwarding has a cost. If the
client sends the request to a follower, the request has to make an
extra network hop: client → follower → leader → follower →
client Instead of the simpler path: client → leader →
client That extra communication adds latency. It also
increases the number of messages inside the cluster. For occasional
requests, this may be acceptable. But for a high-performance
database, especially under heavy read and write workloads,
unnecessary network hops matter. This is where leader awareness
becomes important.
Leader-aware drivers A leader-aware driver allows
the client driver itself to learn which replica is the leader for a
given tablet. Instead of blindly sending requests to any replica
and relying on forwarding, the driver can send future requests
directly to the leader. The first request may still go to any
replica. If it reaches a follower, the follower can forward it to
the leader. But when the response comes back, the driver can also
learn: “For this tablet, this replica is currently the
leader.” From that point on, the driver can route requests
directly to the leader. So the flow becomes: The driver sends an
initial request. If needed, ScyllaDB forwards the request to the
current leader. The response includes updated leader information.
The driver remembers the leader for that tablet. Future requests go
directly to the leader. This keeps the correctness benefit of
forwarding, while reducing its performance cost. Forwarding is
still needed Leader-aware drivers do not remove the need for
request forwarding completely. Leadership is not permanent. A Raft
leader can change at any time due to failures, restarts, topology
changes, or elections. When that happens, the driver may
temporarily have stale information. In that case, forwarding is
still the safety net: If the driver sends a request to the
old leader or to a follower, ScyllaDB can forward the request to
the new leader and update the driver again. So forwarding
remains important, but it becomes the exception rather than the
normal path. Instead of paying the forwarding cost on every
request, we mainly pay it at the beginning of
communication, after the leader has changed, or when the driver’s
leader information is stale. That is a much better model
for performance. Balancing correctness and performance Leader
awareness gives us the best of both worlds. Request forwarding
gives us correctness and simplicity: requests always reach the Raft
leader, even if the client does not know who the leader is.
Leader-aware drivers give us performance: once the driver learns
the leader, it can avoid unnecessary hops and send requests
directly to the right replica. For strongly consistent workloads,
this is a major optimization. Strong consistency already requires
coordination, replication, and quorum agreement. We do not want to
add avoidable network hops on top of that. By making the driver
aware of tablet leadership, ScyllaDB can preserve the Raft
correctness model while reducing latency and improving throughput.
In short: Request forwarding makes strong consistency
work. Leader-aware drivers make it fast.
The leader-aware driver is planned for the 2026.3
release. In 2026.2 we release forwarding. The
strongly consistent read path: reading from the leader and using a
barrier After understanding that every Raft group has a leader, and
that writes must be coordinated by that leader, the next natural
question is: What about reads? At first glance,
reads may look simpler than writes. A read does not change the
data, so it may seem safe to read from any replica. But with strong
consistency, this is not always enough. The problem is that
replicas may not all be at exactly the same point in the Raft log
at the same time. A follower can be slightly behind the leader. It
may still be catching up on committed entries. If we read from that
follower without any additional coordination, we may accidentally
read an older version of the data. For eventual consistency, this
may be acceptable depending on the chosen consistency level and
workload. For strong consistency, we need a stronger guarantee:
A read must observe the latest state that was safely
committed before the read. That means the read path also
needs to respect the Raft ordering model. Reading from the leader
The simplest way to make a strongly consistent read is to send the
read to the Raft leader. The leader is the replica that coordinates
the Raft group. It receives operations, orders them, replicates
them, and knows the current progress of the group. Because the
leader is the place where the group’s ordered history is created,
reading through the leader gives us a natural consistency point.
The basic read path looks like this: The client sends a read
request. The request is routed to the Raft leader for the tablet.
The leader makes sure it is still allowed to act as leader. The
leader reads from the state that reflects the committed Raft log.
The result is returned to the client. This keeps the read aligned
with the same ordering model used by writes. In other words:
Writes go through the leader to enter the log.
Reads go through the leader to observe the committed result
of that log. This gives the system a clean and
understandable consistency model. The leader is not only the place
where changes are ordered; it is also the safest place to observe
the latest committed state. Why reading from a follower is not
always enough A follower is still a valid replica. It participates
in the Raft group, stores the replicated log, and applies committed
entries. But a follower can temporarily lag behind the leader. For
example: A write is committed by the leader and a majority of
replicas. One follower has not applied for that committed entry
yet. A client reads from that follower. The client may see the old
value. From the point of view of that follower, nothing “wrong”
happened. It simply has not caught up yet. But from the point of
view of strong consistency, the reading may be stale. This is why
strongly consistent reads cannot blindly read from any replica
without additional coordination. If we want reads to be strongly
consistent, the system must prove that the replica serving the read
is up to date enough for that read. What is implemented today
Today, before serving a strongly consistent read, ScyllaDB runs
read_barrier(). This currently requires network
communication with a quorum and also waits for the Raft state
machine on the leader to apply any previously written commitlog
entries to memtables. In other words, the read must first make sure
the leader has caught up with all committed changes before
returning a result. In the near future, we plan to implement
Raft leases, which will allow the leader to serve
reads locally without additional network hops. Because of this, we
expect strong consistency performance to eventually be on par with
eventual consistency, and in some read-heavy cases, it may even be
better. How the leader keeps followers updated The Raft leader is
also responsible for keeping the followers up to date. When a new
operation is accepted by the leader, the leader appends it to its
local Raft log and then sends it to the followers. The followers
append the same entry to their own logs and acknowledge it back to
the leader. Once the leader receives acknowledgments from a
majority of replicas, the entry is considered committed. After
that, the committed entry can be applied to the actual database
state. The flow looks like this: Client request ↓
Raft leader appends entry to its log ↓ Leader
sends the entry to followers ↓ Followers append
the entry and acknowledge ↓ Majority
confirms ↓ Entry is committed ↓
Replicas apply the committed change The leader has two
jobs: It orders new operations. It
continuously brings followers to the same ordered state.
This is important for reads as well. A follower may temporarily be
behind the leader, even if it is healthy. That is why strongly
consistent reads usually go through the leader, or require an
additional synchronization step such as a barrier before the system
can safely answer from a known up-to-date state. How this impacts
application developers Strongly consistent tables simplify
application logic. Most developers building a payment flow, quota
system, inventory reservation, account state machine, or
idempotency layer don’t want to reason about replica divergence.
They just want the database to protect the invariant. Strong
consistency also improves predictability. Predictability is not
only about latency. It is about knowing what the system will do
when two clients race, when a node restarts, or when an operation
is retried. A deterministic ordering layer makes these cases easier
to explain, test, and debug. Users should also find that strong
consistency makes LWT and transaction workflows much better,
cleaner, and faster.
Figure 7: Application-level benefits: fewer eventual
consistency anomalies, better counters, better predictability,
simpler logic, and a path to better LWT behavior. How to use
strongly consistent tables This feature is still experimental so
please use it with caution.
Strong consistency is selected at the keyspace level. With the
strongly-consistent-tables experimental feature
enabled, create a tablets-based keyspace with consistency =
'global'. Tables created in that keyspace use the strongly
consistent path automatically, so applications continue to use
ordinary CQL reads and writes. There is no separate per-statement
switch for “strong consistency” in the query itself. CREATE
KEYSPACE sc_demo WITH replication = {'class':
'NetworkTopologyStrategy', 'replication_factor': 3} AND tablets =
{'enabled': true} AND consistency = 'global'; CREATE TABLE
sc_demo.orders ( id int PRIMARY KEY, status text, amount int );
INSERT INTO sc_demo.orders (id, status, amount) VALUES (1, 'paid',
100); SELECT * FROM sc_demo.orders WHERE id = 1; This keeps
the usage model simple: choose strong consistency when creating the
keyspace, then use ordinary CQL on the tables inside it. A
fundamental difference from eventual consistency is that strongly
consistent writes do not support user-provided timestamps, so
applications must let ScyllaDB assign them automatically.
More
about LWT Lightweight transactions are one of the places where
users already ask the database for stronger semantics. In
eventually consistent architectures, LWT is commonly implemented
through Paxos, which requires multiple phases such as prepare,
accept, and commit. That can add latency and complexity. A
Raft-backed architecture gives ScyllaDB a path toward a more
unified model: LWT-style behavior on top of a shared replicated
log. This should give you fewer duplicated mechanisms, fewer voting
rounds, and a simpler execution path. The long-term direction is
not “add another special protocol,” but “converge on one ordering
and replication layer where it makes sense.”
Figure 8: LWT can move from a separate Paxos-based path toward
a Raft-backed path with fewer voting rounds and a shared
replication layer. Performance Strong consistency is not free.
A Raft-based write requires a majority, and write latency can
increase compared with the fastest eventually consistent path. That
is the nature of asking the system to commit to one order before
acknowledging the operation. But the right comparison is not only
per-write latency. Strong consistency can also remove repair
overhead, conflict-resolution ambiguity, reconciliation logic, and
application-side complexity. For correctness-sensitive workloads,
paying a predictable coordination cost inside the database can be
far better than paying an unpredictable correctness cost across the
entire application stack.
Figure 9: Architecture convergence: Raft becomes the unified
ordering layer for topology, strongly consistent tables, and
LWT. We don’t expect our users to make every table strongly
consistent; this won’t be the default. Our goal is to give you a
choice so you don’t have to choose between a high-performance
database and a stronger correctness model. What’s available now and
what’s next The backbone implementation for strongly consistent
tables is in ScyllaDB 2026.2. Users can create a keyspace
configured for strong consistency, create strongly consistent
tables, and read and write to them. This is the foundation: the
core path proving that ScyllaDB can execute user-data operations
through a Raft-backed strong-consistency model. But there’s still
more to do. We’re still working on details like tablet migration,
tablet split and merge support, recovery behavior, and leader-aware
driver support. We’re expecting to release these features in
2026.3. That makes 2026.2 an important milestone: not the end of
the journey, but the point where the architecture becomes visible
and usable. ScyllaDB is evolving from a high-performance eventually
consistent database into a high-performance multi-consistency
distributed database.
Conclusion: strong consistency is about removing ambiguity
Distributed systems are already hard enough. The database should
remove ambiguity where ambiguity is dangerous. With Raft-backed
strong consistency, ScyllaDB gives users a clearer model for
workloads that need correctness, ordering, and predictability.
Eventual consistency remains the right choice for many high-scale
workloads. Strong consistency becomes the right choice when the
application needs one answer, one order, and one source of truth.
We’re excited to share this with you and look forward to your
feedback! 
The Evolution of Cassandra Data Movement at Netflix
By Guil Pires, Jennifer Prince, Jose Camacho, Ken Kurzweil, Phanindra Chunduru
Background
In a previous post, we introduced Data Bridge, a unified management plane for batch Data Movement at Netflix. Historically, several bespoke Data Movement connectors were developed across different engineering organizations to fulfill their specific requirements. Over the last few years, the Data Movement team has started centralizing these offerings through an abstraction that provides a catalog of connectors, along with simple UI and APIs to initiate Data Movement jobs.
One such case is the Cassandra to Iceberg connector. Apache Cassandra powers mission critical applications at Netflix, including Member, Billing, Recommendations, Subscriptions and many more. These use cases heavily leverage Data Movement to Apache Iceberg for many analytics and operational tasks, and central to this movement was a connector for Cassandra to Iceberg built in-house named Casspactor. As many Cassandra based Data Abstractions emerged, such as Key Value, Time Series and Graph — the need for larger and more complex Data Movement with transformations became more critical to the business.
Data movements are fundamentally fulfilled by leveraging the existing Cassandra backup infrastructure. Regularly scheduled backups are performed directly on the Apache Cassandra nodes, via a sidecar process managing the upload of all necessary SSTables and associated Metadata files directly into Amazon S3. When a Data Movement job is initiated, the job constructs the specific backup structure it needs by referencing the S3 based metadata, allowing it to precisely locate the SSTable files. The engine then downloads these files, performs the required mutation compaction and processing, and finally writes the fully transformed, compacted data directly into the target Apache Iceberg tables.
Casspactor: The Engine We Outgrew
Casspactor processed roughly 1,200 data movements per day, transferring approximately 3 PB of data from Apache Cassandra into Apache Iceberg tables. It served some of the most critical workloads at Netflix. For years, it worked. Then, two compounding challenges made it clear we needed a fundamentally different architecture.
Fragile Metadata Dependencies
Before Casspactor could move a single record, it needed to answer a deceptively simple question: which backup exists, is it complete, and what does it contain?
Casspactor assembled this answer from multiple independent systems:
Each system had its own failure modes, update cadences, and accuracy guarantees. Casspactor’s view of the world was a composite, and composites diverge from reality.
Metadata fell out of sync with actual backups, causing Casspactor to read stale or incorrect data silently. Routine maintenance on the Cassandra Clusters triggered uncoordinated snapshots, and because Casspactor required all nodes in a region to snapshot at the same clock second, a single node replacement could break data movement for an entire region.
The fix was hiding in plain sight. The answer to “which backup exists and is it complete?” already lived in the backup storage layer (Amazon S3) itself. By reading metadata directly from the backup files, we could replace the entire dependency chain with a single source of truth.
Every Connector Inherited Casspactor’s Limitations
Cassandra at Netflix does not just store raw tables. It backs higher level data abstractions, such as Key Value, Time Series, and others, each with its own data model, access patterns, and semantics. When any of these abstractions needed to move data to Iceberg, they all funneled through Casspactor.
Every abstraction inherited Casspactor’s constraints:
- Skewed partition failures: Casspactor could not handle tables with large partitions, a common pattern in Key Value and Time Series workloads. Jobs crashed with out-of-memory errors on some of Netflix’s largest datasets.
- No data model awareness: Casspactor moved raw Cassandra tables as is. Connectors for Key Value and other abstractions had to bolt on post processing to reconstruct their data models from the raw output — extra cost, extra complexity, and an extra surface for failures.
- Intermediate table bloat: Casspactor wrote to an intermediate Iceberg table before producing the final output. The Key Value connector added another intermediate table and a snapshots table. Connectors for abstractions on top of Key Value added even more. This compounded into significant storage cost overhead.
- Inability to Time Travel: by relying on multiple services to compose a backup unit, Casspactor was unable to restore prior backups in the event of cluster Topology or Keyspace schema changes.
- Monolithic design: Casspactor was built as a single connector, not as an engine. There was no way to build a family of purpose built connectors on a shared foundation.
We needed something fundamentally different: an engine that reads directly from backups in S3, produces standard Spark DataFrames, and lets each data abstraction build its own connector with full awareness of its data model. One foundation, many connectors.
The New Stack: A Layered Architecture
The new architecture, built upon the foundation of Apache Cassandra Analytics and the in-house Move Data framework, represents a fundamental shift toward a layered, purpose-built stack designed for reuse and maintainability. This new engine was conceived with clear separation of concerns, moving away from Casspactor’s monolithic design. The architecture is intentionally layered with the foundation being a core S3 reading capability: the Cassandra Analytics Wrapper, which is built on top of the Open Source Cassandra Analytics with Netflix’s internal backup representation and an S3 Client.
This layer handles the raw data retrieval from backups, translating it into standard Spark DataFrames. Sitting atop this foundation is a “Connector Factory” model, via both Java UDFs and transforms which allows individual data abstractions (Key Value, Time Series, others) to build highly optimized, data model aware connectors that process the generic Spark DataFrames, avoiding the need for complex, expensive, and failure-prone post-processing steps. This layered approach ensures that improvements to the core reading engine benefit all connectors, while the connectors themselves are focused solely on data transformation.
- Handles Skewed Partitions: By moving the mutation compaction and processing to the Executor level within Spark, the new engine can efficiently handle tables with highly skewed or wide partitions, a major pain point for Casspactor. Crucially, this processing occurs without excessive data shuffling, preventing out-of-memory errors and enabling reliable movement of Netflix’s largest datasets.
- Operates at Spark DataFrames (No Intermediary Tables): The new architecture directly generates standard Spark DataFrames from the Cassandra backups. This eliminates the need for Casspactor’s costly, multi-stage intermediate Iceberg tables, which led to storage bloat and operational complexity. This native DataFrame operation enables the “Connector Factory” by providing a universal, easily consumable interface for building diverse, model specific connectors.
- Jobs Auto Size: The engine integrates intelligent auto-sizing capabilities, allowing jobs to dynamically adjust resource consumption based on the source table’s characteristics. This removes the burden of manual tuning from engineering teams, ensuring optimal performance and cost efficiency without sacrificing reliability.
- Reduced Dependencies: By reading metadata directly from the backup files stored in S3, the new stack removes the fragile, multi-service dependency chain that plagued Casspactor. S3 becomes the single, authoritative source of truth for backup existence and completeness, vastly improving data movement reliability and consistency.
- Time Travel: A critical feature of the new stack is the ability to process the schema, cluster topology, and data as a cohesive unit at a specific point in time. This capability provides robust time travel functionality, essential for auditing, debugging, disaster recovery and reproducing past data states.
- Performance: Collectively, these architectural improvements, including native DataFrame processing, optimized partition handling, and streamlined metadata retrieval have resulted in notable performance gains, reducing overall data movement execution runtime and cost compared to the legacy Casspactor system.
- Cost: by eliminating intermediary Iceberg tables and efficient SSTable compaction on Executors, the new stack needs a significantly smaller storage and compute footprint leading to significant cost savings in the order of USD millions.
The Journey Towards a Safe Migration
The successful validation of the new stack was the critical first step, but it only marked the beginning of the most challenging phase: the migration. Large scale data migrations are inherently complex, high-risk undertakings that can be time consuming and often result in customer frustration and service disruption. To navigate the high stakes of decommissioning a mission-critical system like Casspactor and seamlessly replacing it, we needed a strategy that prioritized reliability and transparency above all else.
The migration was fundamentally enabled by a Like-for-Like strategy, which served as the cornerstone of our Platform Engineering philosophy, abstracting complexity. The core tenet was to maintain absolute consistency across the user-facing interface, the output contract, and the final data artifact. This meant ensuring that the data movement parameters defined via the Data Bridge abstraction remained unchanged, and, critically, the schema, metadata, and data within the destination Iceberg tables were identical to the legacy output. By preserving these external contracts, we eliminated the need for complex, time-consuming coordination with dozens of internal teams who relied on these data pipelines. This approach transformed the migration from a distributed, high-risk, multi-team effort into an internal platform implementation detail, allowing us to achieve a transparent, zero-impact transition and accelerate the retirement of the legacy system without requiring any code changes or validation from downstream users.
To navigate this migration, we developed a strategy anchored by three core pillars that serve as a blueprint for successful, large-scale data migrations:
- Validation: Establishing and maintaining absolute confidence in data consistency through rigorous, ongoing validation.
- Visibility: Instrumenting every part of the system to provide a clear, real-time understanding of migration progress and system health.
- Safety: Ensuring user impact is minimized or eliminated, despite the inevitable system failures, by leveraging abstractions and robust fallbacks.
The next section will provide a detailed exploration of these key pillars.
Pillar 1: Validation
Trust is earned, and in data migration, it is earned one row at a time. The first pillar is the most critical: providing a measurable guarantee to users and partners that the data produced by the new system is an exact, row-by-row replica of the data produced by the old one.
Our foundational tactic was deploying the new Move Data connector in a “shadow” testing that ran in parallel with the production Casspactor jobs. This allowed us to validate the new system with real-world, production workloads without any customer impact.
- Let C be the set of rows in the legacy Casspactor output (Iceberg table).
- Let M be the set of rows in the new Move Data output (Iceberg table).
The test for trust: prove that C = M. This required continuously checking for two conditions:
- Rows in C but not in M (C-M): The new system missed data.
- Rows in M but not in C (M-C): The new system introduced phantom or erroneous data.
Any result where the cardinality of these difference sets (the number of differing rows) was greater than zero triggered an immediate, high-priority investigation. The target was 100% similarity.
Uncovering and Resolving Disparities
The shadow mode quickly became a powerful forensic tool, exposing “unknown unknowns”, subtle discrepancies that were not bugs in the new system but rather differences in behavior between the new and old systems. Resolving these was the core work of building trust. For each problem we initiated an investigation log where we captured the details, logs, queries that allowed us to diagnose. Based on the assessment the issues were categorized so that similar differences on other datasets were later resolved affecting many of the shadow pipelines.
Maintaining an investigation log was critical to organize the outstanding issues and effectively communicate to stakeholders the progress and confidence of the new connector so that we effectively measure the appropriate level of “confidence” to initiate the migration.
We observed differences in how connectors leverage reference timestamps for Time-to-Live, Consistency Levels, backup selection, and various internal business logic. This continuous, data-driven cycle of discovery and resolution was the mechanism by which we built confidence in the new architecture.
Pillar 2: Visibility
Trust is built in the background, but an active migration requires real-time insight: Visibility. The second pillar involves instrumenting the system to provide an unambiguous, clear understanding of operational health and migration progress.
We extended our instrumentation to the overall migration workflow and its dependencies:
- Dashboards: We created centralized dashboards to track migration status, visualizing the total number of data movements migrated versus those remaining. The dashboards tracked execution status, average runtime, and cost comparisons between the two connectors.
- Dependency Tracking: Since the new system relied on a new set of APIs to fetch backup metadata, we implemented detailed metrics for failures to keep track of the APIs or dependencies failed.
- Alerting: Proactive alerts were set up for job failures (Move Data or Casspactor), failures on Move Data that triggered a fallback to Casspactor or any data discrepancy being detected.
This comprehensive instrumentation allowed the team to be proactive, fix issues as they emerged during the migration, and gain the necessary confidence to accelerate the migration timeline.
Pillar 3: Safety
Even with perfect data correctness and enhanced visibility, the third pillar, Safety is required for a zero-impact migration. The challenge is ensuring that when a system inevitably fails, the user experience is uninterrupted. Our strategy centered on decoupling the user’s workflow from the underlying connector implementation.
Leveraging Abstraction: The Decider Pattern
To achieve a transparent swap, we leveraged the Maestro workflow orchestration platform to implement the Decider pattern:
- Data Movement Abstraction: From a user’s perspective, their Data Movement job definition remained the same.
- The Decider Step: Internally the workflow responsible to execute the job was modified to include a Decider step. This step took the data movement parameters (source cluster, table name, destination) and invoked a control plane: Connector Controller.
- Connector Controller as the Registry: The control plane served as the dynamic registry. Based on the migration cohort and the data movement attributes, it determined and reported the appropriate connector to use either Casspactor (legacy) or Move Data (new).
This abstraction gave our team complete control. We could upgrade or rollback any connector for any data movement instantly by simply updating a configuration in the controller, with zero modification required to the thousands of downstream customer workflows. Crucially, this abstraction guaranteed the critical safety net: a conditional step in the Maestro workflow logic ensured that if the Move Data step fails, it would immediately execute the Casspactor step.
This pattern would increase the chances that the user’s data movement completes successfully, even if the new connector encountered a bug or transient failure during the initial rollout phases. User impact was completely eliminated; they might see a slightly longer runtime in the event of a failure and fallback, but they would never see a migration failure or suffer from stale data.
Beyond the workflow, the new system architecture itself was inherently more resilient. By building the new data movement connector on Cassandra Analytics and reading backups directly from S3, we removed fragile dependencies on deprecated internal services.
Conclusion
The migration from Casspactor to the new, layered architecture built on Cassandra Analytics and the Move Data connector was more than a typical “tech debt” project; it was a fundamental shift in our approach to data movement reliability and scalability at Netflix.
The legacy system, while serving us well for years, was ultimately constrained by monolithic design, fragile metadata dependencies, and an inability to handle the complexity of modern data abstractions. The new stack resolves these issues by delivering a robust, cost-efficient, and inherently more resilient solution that reads directly from S3, handles wide partitions gracefully, and eliminates costly intermediate tables.
Our blueprint for the migration, anchored by the three pillars of Validation, Visibility, and Safety, ensured a transparent and high-confidence transition. Through rigorous shadow testing and a data-driven audit framework, we achieved the desired data consistency. Enhanced dashboards and alerting provided the real-time operational insight necessary to manage risk. Most critically, the implementation of the Decider pattern within our workflow abstraction minimized the impact for all downstream users.
This successful migration validates a core philosophy: by abstracting complexity at the platform level, we can perform large system migrations without burdening our product engineering partners. The new foundation is now ready to support the next generation of Netflix’s data abstractions.
Looking ahead
This foundational work on the Cassandra Data Movement stack has done more than just replace a legacy system: it has become an accelerator for innovation across the entire Data Movement organization. By providing a reliable, performant engine that standardizes data retrieval into Spark DataFrames, we’ve enabled the rapid development of new, highly optimized connectors. This new “Connector Factory” approach has already delivered a dedicated Key-Value to Iceberg and Time Series connectors, both of which are fully aware of their respective data models, eliminating costly post-processing. This architecture is also paving the way for ambitious new initiatives, including the development of a solution for bulk loading data into Cassandra itself, effectively completing the data movement cycle, and enabling safer fleetwide connector rollout with canaries inspired by the Decider Pattern.
We are incredibly grateful for the extensive collaboration among the Data Movement, Data Bridge, Online Data Stores, Membership, Billing, Subscriber and Ads platform teams at Netflix; this work simply couldn’t have been accomplished without their partnership!
The Evolution of Cassandra Data Movement at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.