Hybrid RAG retrieval in Apache Cassandra: SAI filters plus vector search
IntroductionIn Part 1 of this two-part series, we built the retrieval layer for a Retrieval Augmented Generation (RAG) support bot on Apache Cassandra 5 vector search. The bot returns results that would be sent to an LLM to answer questions about fictional Summit pickup trucks. The application has access to a knowledge base of 21 documents: 12 recalls plus tow ratings, payload figures, maintenance intervals, and warranty terms for the Summit 1500 and 2500.
Part 1 showed how little it takes to enable vector search in Cassandra 5, and where it works well: questions phrased in everyday language. It also showed where vector search alone can fall short. When a question hinges on an exact identifier, or on a detail the user never states (such as the model year), ANN can return the wrong documents with high similarity scores and no warning.
In Part 2, we borrow an idea from hybrid RAG research: ground retrieval in hard facts first, then let vector search handle meaning. We apply that idea inside the same Cassandra table, with no new systems.
What is hybrid RAG?Hybrid RAG combines two retrieval methods. The term comes from the 2024 paper HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction, by researchers at BlackRock and NVIDIA.
Method How it retrieves Strength GraphRAG Queries a knowledge graph (e.g., Neo4j) Precise on structured facts VectorRAG Retrieves passages by semantic similarity (e.g., Pinecone, OpenSearch, Cassandra 5) Strong on meaning and context HybridRAG Combines both, sends merged context to the LLM Best overall balanceHybridRAG sends the context from both methods to the LLM together. The authors tested it on question answering over financial earnings call transcripts. HybridRAG scored 0.96 on both faithfulness (how well the answer is supported by the retrieved context) and answer relevance. VectorRAG scored 0.94 and 0.91, and GraphRAG scored 0.96 and 0.89. No single method won every metric. HybridRAG had the lowest context precision (0.79), because merging two sets of context also brings in some noise. Overall, though, it gave the best balance.
A note on terminology: in search engines, “hybrid search” usually means combining keyword ranking (BM25) with vector ranking. The approach in this post borrows from both ideas. Exact facts constrain the results, and vectors rank them.
Cassandra and hybrid RAGCassandra is not a graph database and has no native GraphRAG. The core idea, grounding the answer in facts and then adding meaning, doesn’t require a graph. Store facts such as model, model year, category, and recall ID as columns on the same row as the embedding. Cassandra can then filter on those facts and rank by vector similarity in a single query. This method keeps the same architecture from Part 1: one database and one table.
Storage-Attached Indexing (SAI) is what makes this work. In Part 1, we used SAI only to index the embedding column, where it uses JVector for approximate nearest neighbor (ANN) search. SAI can also index ordinary columns. SAI indexes are attached to the table’s own storage and are built as memtables and SSTables are written. Compared with Cassandra’s legacy secondary indexes, they are designed to reduce write-time overhead and disk usage. A single query can use several SAI indexes.
In this design, SAI is the filter and ANN is the
ranking. One CQL statement can restrict candidates to rows
that match exact facts (WHERE) and then order the
remaining rows by vector similarity (ORDER BY … ANN
OF). The facts decide which documents are eligible, and the
vectors decide their order. SAI and vector search are complementary
parts of the same query, not competing features.
We build on the schema from Part 1. The
companion app is cassandra-hybrid-retrieval-demo.
It uses the same 21 documents and still uses one table, with two
changes: a new keywords column, and SAI indexes on the metadata
columns.
CREATE TABLE docs ( id text PRIMARY KEY, title text, body text, category text, keywords set<text>, model text, model_year int, embedding vector<float, 384> ); CREATE INDEX docs_keywords_sai ON docs (keywords) USING 'sai'; CREATE INDEX docs_category_sai ON docs (category) USING 'sai' WITH OPTIONS = {'case_sensitive': 'false', 'normalize': 'true'}; CREATE INDEX docs_model_sai ON docs (model) USING 'sai'; CREATE INDEX docs_model_year_sai ON docs (model_year) USING 'sai'; CREATE INDEX docs_embedding_sai ON docs (embedding) USING 'sai' WITH OPTIONS = {'similarity_function': 'COSINE'};
Here’s what each addition is for:
keywordsis a set<text> of exact terms curated for each document, such as the recall ID (24v-330), model numbers, years, and component names. The app pulls exact tokens out of the question and matches them with keywords CONTAINS ?. Those tokens are identifiers like 24V-330, three- or four-digit numbers like 1500 or 2024, and a short list of domain terms like tow and recall.category,model, andmodel_yearare the fact filters, our stand-in for the graph side of hybrid RAG. The app infers category from words in the question. Model and model year represent customer context, such as the truck on file in the customer’s account, rather than something parsed from the question. In the demo, you pass them with--modeland--model-year.USING 'sai'is required on every index. In Cassandra 5.0, CREATE INDEX without it creates a legacy secondary index, unless the cluster’sdefault_secondary_indexsetting has been changed to sai.
Before you design filters, know which query operators SAI supports in Cassandra 5.0:
- Numerics:
=,<,>,<=,>= - Scalar strings:
= - Predicates can be combined with
AND - Collections:
CONTAINS,CONTAINS KEY - Not supported:
LIKE, scalar-string substring search
Cassandra 5.0 has no negation operators in WHERE
clauses. NOT IN, NOT CONTAINS and
NOT CONTAINS KEY arrive in Cassandra 6.0 (CEP-29),
usually with ALLOW FILTERING; NOT LIKE was proposed
but not included. On 5.0, aside from reversing range comparisons,
excluding specific values has to happen in application code.
The demo answers each question three ways against the same table, so you can compare the context each approach would hand to an LLM:
- Keyword: runs
keywords CONTAINS ?for each exact token in the question. The app ranks results by how many tokens each document matched. SAI returns matches, not relevance scores, so documents with the same match count are in no meaningful order. - Vector ANN: runs
ORDER BY embedding ANN OF ?, exactly as in Part 1. - Hybrid: runs SAI filters and ANN ordering in
one statement. It filters on the most selective token in the
question (an identifier such as
24v-330, if there is one), the inferred category, and the customer’s model and year when provided.
For example, this is the hybrid statement the app runs for “What is recall 24V-330?”:
SELECT id, title, category, body, model, model_year, similarity_cosine(embedding, ?) AS score FROM docs WHERE keywords CONTAINS [BB3] '24v-330' AND category = 'recalls' ORDER BY embedding ANN OF ? LIMIT 5;
Note: the index keywords is case sensitive while category is case insensitive when we created our indexes in the Setup section.
Revisiting the Part 1 questionsLet’s ask the same three questions from Part 1.
Question 1: “Is there a recall on the tailgate?”
All three approaches rank Recall 24V-113 (tailgate latch) first.
Vector search already handled this in Part 1, scoring it 0.7767
against 0.7032 for the runner-up, because embeddings are good at
matching meaning. The question contains no identifier, so the
hybrid query filters only on category = 'recalls', and
ANN still puts the tailgate recall on top.
This baseline is worth keeping. Before adding filters to fix the hard cases, confirm they don’t break the cases vector search already gets right.
Now for the questions where vector search struggled.
In Part 1, vector search ranked the matching document ninth, with a score of 0.7332. That put it below eight other recall notices that never mention 24V-330, even though the question contained the exact recall ID. The 12 recall documents follow one template, so the IDs differ by only a few characters and the embeddings sit very close together. Cosine similarity has no concept of an exact token match.
SAI fills that gap. The keyword lane ranks
recall-24v-330 first, because it’s the only document
that matches both 24v-330 and recall.
Ranks 2 through 5 match only recall, so they aren’t
really ranked, just matched. The hybrid query (keywords
CONTAINS '24v-330' AND category = 'recalls') returns a
single row, and it’s the right one. The identifier decides which
documents qualify instead of competing with semantic
similarity.
In Part 1, all four of the top vector results looked relevant, scoring within 0.08 of each other.
The keyword lane does no better. It ranks the 2023 truck first,
on a two-way tie that the app breaks by the order results came back
rather than by relevance. Without customer context, the hybrid
query can filter only on category = 'towing'. That
removes the payload document, but it still mixes model years and
models.
The question never says which model year, and no ranking method
can infer it. When the facts are available, use them to filter
first. Here, the customer’s truck is on file. Passing --model
1500 --model-year 2024 narrows the hybrid query to a single
row, summit-1500-2024-tow (11,300 lb), with a score of
0.8990. Here’s what the data says:
Retrieval still runs either way. If you don’t check what comes back, it’s easy to miss that the top results describe different trucks. When the application doesn’t know a fact the answer depends on, it should look it up from the customer record or ask the user, rather than letting the LLM choose among conflicting chunks.
Limitations to know before you build- SAI is not BM25. It returns matches, not relevance scores. A dedicated search engine still wins on full-text relevance, facets, and highlighting.
- Cassandra is not a graph database. Following relationships, such as related recalls or citations, happens in application code.
- Filters only help if the facts are right. The metadata on each row must be accurate, and the application must be able to extract or supply the filter values.
Version note: Queries that combine
WHERE filters with ANN had correctness and timeout
issues before Cassandra 5.0.7 (CASSANDRA-20086). The demo runs on
5.0.9, but it will also work on Instaclustr’s latest supported
version of Cassandra.
If your data already lives in Cassandra 5, combining SAI filters with vector search on the same table lets you do two things in one query: ground retrieval in exact facts and rank results by meaning. You don’t need a separate vector database, or a search cluster just to match identifiers. Embeddings sit on the same row as the data they describe, and SAI indexes each write synchronously, so new rows are searchable as soon as the write is acknowledged. However, be aware that an ANN (top-K) query does not keep the consistency level you asked for: Cassandra downgrades it to LOCAL_ONE when the request is datacenter-local, and to ONE otherwise.
Try it yourselfRun the examples yourself from the companion app on
GitHub. Cassandra 5.0.9 runs in Docker, and the Python app
(3.10–3.13) runs locally with the all-MiniLM-L6-v2
embedding model, so you don’t need an API key. The first run
downloads the model.
You can also run the same CQL on the NetApp Instaclustr Managed
Platform with a 30-day free trial,
which covers one small cluster at a time. Point the app at your
cluster by setting CASSANDRA_HOST to the cluster’s
contact point. You will also need to modify your setup to accept
authentication to the Instaclustr Cassandra cluster, shown
here.
The post Hybrid RAG retrieval in Apache Cassandra: SAI filters plus vector search appeared first on Instaclustr.
Why Real-Time AI at Scale is So Hard
The failure patterns that surface as your app ramps up, and how to avoid them Real-time AI at scale is harder than it looks. Pipelines that hum along in development routinely hit problems in production. It’s always easy to blame the model for all your problems. But issues like rising latency and degrading accuracy can usually be traced back to the data pipeline. My colleague Tim Koopmans and I… SourceP99 CONF 2026 Recap: A Tail of Two Days
Our Aussie host’s take on P99 CONF around the clock and (now) anytime on demand P99 CONF… you’ve got to love this conference, which just completed its 6th edition. I’ve been lucky to help host the last 4 of them and I reckon the format is bang on. As a speaker, you get to pre-record and then listen to yourself “live” while the audience asks you questions in chat. You can even “heckle yourself,”… SourceFeast Feature and Vector Serving at ScyllaDB Speed
A new and tighter integration between Feast and ScyllaDB online feature store, including vector search support and better performance Starting with Feast v0.65, you can build with a new and tighter integration between Feast and ScyllaDB online feature stores, including vector search support and better performance. This post covers what changed, what tends to push teams to outgrow their online… Source5 questions to ask before your next DataStax Enterprise renewal
Renewing a DataStax Enterprise (DSE) contract used to be a formality. Check the box. Sign the term. Get back to shipping.
That’s changed. IBM’s acquisition of DataStax didn’t touch your cluster. It touched the company on the other side of the contract. Your application runs the way it did last quarter, but the agreement you’re about to sign describes a different relationship than the one you signed a few years ago.
The decision is also bigger than renew or leave. You can renew DSE, move to IBM’s Hyper-Converged Database (HCD), or move to managed Apache Cassandra®.
A DataStax Enterprise renewal should be evaluated as a three-way decision: renew DSE, adopt IBM’s HCD, or migrate from DataStax to managed Apache Cassandra. The right choice depends on your support costs, DSE-specific dependencies, operational requirements, and tolerance for vendor lock-in. Five questions will help you choose.
1. What exactly are you renewing?IBM announced its acquisition of DataStax on February 25, 2025 and closed it on May 28, 2025. Since November 1, 2025, new DataStax transactions run on IBM contracts. IBM sells DSE and HCD as components of its watsonx.data Premium offering, not as a standalone Cassandra product line. DSE 6.x was withdrawn from market on October 31, 2025.
None of that means you’re cut off. IBM’s published lifecycle lists standard support through December 31, 2027, and extended support through December 31, 2031. Read the terms closely. Per IBM’s support policy, extended support is a separate purchased add-on, so include it when calculating your total DataStax Enterprise support cost.
Extended support covers critical bug and security fixes in the first year only. After that, you get usage support and existing fixes. Supported and current aren’t the same thing, so read your renewal for what it now describes: a different product relationship.
2. Where is IBM taking the Cassandra portfolio, and does it match your roadmap?IBM tells existing DSE customers they can keep running DSE for several more years and move to HCD “when ready.” HCD is moving, too. HCD 2.0 had its first public release in June 2026, rebased on Cassandra 5.0. IBM’s public DataStax materials now center on watsonx, vector search, and unstructured data for generative AI.
If you want an AI data platform with IBM as the vendor, that’s a coherent direction, and staying put can be the right call. If you adopted DataStax because your application needed Cassandra to run reliably at scale, with predictable latency and uptime your users never notice, ask whether the roadmaps match. They can overlap. They aren’t guaranteed to.
Already planning to move to HCD? Ask what problem it solves. HCD stays inside IBM’s platform, so the vendor dependency comes with you. It’s self-managed, so upgrades, incident response, and tuning stay with your team. It isn’t a like-for-like move, either: DSE’s integrated Search and Graph don’t come along. If you have to rework the architecture anyway, compare HCD with a fully managed, vendor-independent option before you commit.
3. Which DSE-specific features do you actually use?Your answer reshapes the decision more than any other question here. For some teams, Search, Graph, Spark integration, OpsCenter, and DSE Advanced Replication keep applications compliant, observable, and fast. Others pay for the bundle and use a fraction of it.
Leaving DSE doesn’t mean leaving an active project. Apache Cassandra is under active, independent development: versions 5.0.9, 4.1.12, and 4.0.21 all shipped on August 7, 2026. 5.0 is the Apache Software Foundation’s current generally available release line, and 6.0 is in development, with an alpha already published. If your application relies on core Cassandra plus enterprise-grade support rather than the DSE-specific layer on top, you want to know that before you renew.
Building on Astra’s AI tooling? Check what’s actually tied to Astra. Langflow is open source and vector-store agnostic, and its Cassandra integration works with open source Cassandra. The narrower boundary is RAGStack and Knowledge Graph RAG, which don’t port over. If you rely on DSE Graph, treat it as its own decision and scope it separately.
4. What does another contract term cost you in flexibility?Renewing isn’t automatically wrong. But every multiyear DataStax Enterprise renewal is also a multiyear commitment not to reevaluate, and that has a cost even when the sticker price doesn’t move. Vendor lock-in is a business issue as much as a technical one. An application that depends on predictable performance also depends on your team having room to negotiate, migrate, or scale on its own timeline. Before you sign, price the next 3 years from every angle: what it costs to stay, what it costs to move, and what you give up in leverage either way.
5. Do you know what leaving would cost, or are you estimating?Most teams renew DSE because nobody priced the alternative, not because they compared the options and chose to stay. Has your team calculated what it would take to migrate from DataStax to Cassandra, or is it renewing without a documented cost and dependency assessment?
NetApp® Instaclustr has migrated more than 10,000 DataStax nodes across dozens of customers to open source Cassandra. Tesouro reports a 75% reduction in its storage footprint after migrating. Pega reports roughly 60,000 Cassandra management hours saved a year, hours that now go to the application instead of the database layer underneath it. Clear Capital completed its migration with zero downtime. These outcomes are customer-reported, and results vary by workload.
Migration isn’t simple. DSE-specific features add real complexity, and that complexity has to be scoped. Experience is what makes it manageable: the path exists, teams have completed it at scale, and you can price it with real numbers. Learn how to migrate from DataStax to Cassandra.
Before you renew, price the exitYou don’t need a research project. You need someone who has done this before to look at your environment and tell you what staying costs, what leaving costs, and what you’d give up either way.
The DSE Exit and Renewal Assessment from NetApp Instaclustr does that. Fill out a short intake, then spend 30 to 45 minutes on a call with a migration engineer. You get a written summary of your DSE dependency, directional migration complexity, and a ballpark cost comparison. It’s free, with no obligation. If your answer is to renew or move to HCD, the assessment shows you that too. If the call turns up real complexity, the next step is a deeper scoping conversation for your account.
Book my free assessment today.
The post 5 questions to ask before your next DataStax Enterprise renewal appeared first on Instaclustr.
TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads
By Oleksii Tkachuk, Rajiv Shringi, Chris Lohfink, Abe Ratnofsky, Kartik Sathyanarayanan
Introduction
In an earlier post, we introduced the TimeSeries Abstraction, our high-throughput, low-latency platform designed to store and query temporal event data at scale. It has since become the operational backbone for Netflix’s largest datasets, which now span multiple petabytes.
However, managing such large datasets in hot storage, tuned for low-millisecond latency access, is not cheap. At Netflix, cost-efficiency is a continuous discipline rather than a temporary initiative. This post shares two solutions that significantly reduce storage costs by continuously moving older, less frequently accessed data into cheaper object storage:
- Solution 1: Uses Apache Kafka and Flink to stream data directly to S3
- Solution 2: Leverages Cassandra backups in S3 and pluggable Memtables to read data directly from S3
We’ll start with the Kafka/Flink-based design, then describe how we evolved to a native Cassandra-based cold tier.
Challenges
As Netflix’s TimeSeries datasets have expanded, we’ve encountered several compounding challenges:
- Growing retention demands: Users often push for increasing data retention to serve a broader range of use cases, which in turn increases the storage costs.
- Escalating costs: Maintaining massive, dense clusters for infrequently accessed historical data is prohibitively expensive.
- Sparse historical access: Most queries target recent events, but older data must remain readily available for occasional audits and investigations.
- Operational strain: High storage density on hot nodes complicates scaling, deployment, and cluster stability.
- Latency requirements: Even historical data sometimes requires sub-second query performance for internal tooling and analysis.
This creates a clear opportunity: keep the most recent data on fast storage and migrate older, rarely read data to cheaper storage while preserving a consistent API. Done well, this can meet user and operational needs while greatly reducing our infrastructure spend.
TimeSeries Abstraction
Before we dive into the cold storage solutions, here is a quick recap of how TimeSeries partitions its dataset into multiple tables, each covering a window of time we call Time Slices.
This enables several key optimizations, most notably allowing a table to become immutable once it crosses the write boundary, which lets us safely apply further optimizations. For more details, refer to our previous post.
This slice information is critical and is maintained in a dedicated metadata table:
Creating new Time Slices and maintaining their metadata is a primary responsibility of the TimeSeries control plane. The TimeSeries data plane uses this information to correctly route the read and write traffic to the provisioned tables.
With that background, let’s look at the different solutions we built for data tiering.
Solution 1: Streaming with Apache Kafka and Flink
Our first solution followed a familiar streaming pattern using Apache Kafka and Flink, where we streamed data to S3 in realtime while simultaneously persisting it to Cassandra.
Getting Data Into S3 Efficiently
To optimize data streaming to S3, the system hash-partitions Time Series IDs into buckets. This partitions the stream and lets dedicated Flink tasks handle each bucket, ultimately batching and reducing the write throughput to S3. Flink workers batch events into Parquet files grouped by event time within the hour. These files are great for high-throughput streaming writes, but not for low-latency reads.
Enter the Compaction job.
Compaction Job to Optimize Reads
The compaction job runs daily and merges small Parquet files into a single, optimized daily Parquet file per namespace.
The job further optimizes the Parquet files in the following ways:
- Small Row Groups: Creates small (4–8 MB) row groups to minimize lookup cost. We can then use the Parquet Java library to perform range reads of row groups from S3. This is intentionally well below Parquet’s recommended row group floor — smaller groups were better because our reads are dominated by point/range lookups rather than full scans.
- Event Sorting: Sorts events within a row group by time, eliminating brute-force scans when querying for a time range.
- Bloom Filters: Caches Bloom filters and file metadata in EVCache, allowing the TimeSeries query planner to efficiently skip unnecessary reads from S3.
Once the compaction job completes, all compacted files per partition are recorded in the TimeSeries control plane, and the hot storage slices can now be truncated!
On some of our largest datasets, this job typically takes about 12 hours, prioritizing lower compute costs over execution speed.
Streaming Cold Tier Read
Once compacted files are optimized in cold storage, they are ready to be read. The TimeSeries data plane fans out a given query across both the hot and cold tiers and merges their result. The users still use the same query interface:
The Cold Tier Data Plane uses compaction metadata for aggressive planning and pruning, which dramatically reduces the S3 query fan-out. As a result, retrieving a full day of data requires at most three sequential S3 reads, even in the uncommon case where an ID spans two row groups. This yields a p50 latency of ~150 ms and a p99 latency of ~500 ms per read.
At the time of writing, the streaming cold tier implementation was processing up to 1,250 reads per second from cold storage globally:
Why not stop here?
The streaming cold tier has delivered real savings, shrinking both Cassandra’s EBS footprint and the Cassandra backup footprint in S3. But the streaming infrastructure itself is expensive and adds moving pieces that widen the team’s on-call scope:
We wanted to improve this further and asked ourselves:
- Can we avoid streaming data into the object store and the associated data transfer costs?
- Can we reduce the compute and operational overhead of new components?
That led to a more radical question: Can we read directly from existing Cassandra backups in S3?
Solution 2: Cassandra Cold Storage for Immutable Data
At Netflix, we back up Cassandra data into S3 at regular intervals for disaster recovery. The idea for tiering is simple:
Read directly from these S3 backups instead of paying a separate data transfer cost to S3.
Once TimeSeries marks a hot Time Slice as ready for cold storage, we can customize Cassandra to read directly from its backups, allowing us to delete the on-disk data copy. This would also incur minimal changes to the TimeSeries read path!
To make this viable in production, we had to answer a series of “How…?” questions.
How Can We Read the Backups Efficiently?
We introduced a new compaction strategy in Cassandra, which we call ReadOnlyCompactionStrategy (ROCS). Here is how it works:
Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable. This immutability unlocks significant optimization opportunities:
- SSTable overlap: Standard strategies (LCS, TWCS) often produce SSTables that overlap in the token space, and so a single partition key may exist across multiple files.
- Read fan‑out: Because a read must touch every SSTable covering the key’s token range, this overlap turns a single point-read into multiple lookups, increasing latency across all tiers.
ROCS re-compacts all SSTables once, aligning them precisely with token ranges:
ROCS applies more broadly to all TimeSeries use cases, driving tail latency down on all storage tiers by reducing the SSTable fan-out during read operations:
Once the immutable Time Slices have been optimized, it is time to trigger backups.
How Would We Leverage Backups?
Cassandra backs up every node in the cluster, organizing each backup on S3 as a declarative manifest, along with the SSTables backing the keyspace’s tables:
{bucket}/
└── {prefix}/ ← cluster identity (env_backup/hash_app)
└── {token}/ ← node identity (ring token)
├── META_V2/ ← manifests, one per snapshot
│ ├── {snapshotInstantMs}.json
│ ├── {snapshotInstantMs}.json
│ └── ...
└── SST_V2/ ← actual SSTable files
└── {backupTimestamp}/ ← snapshot timestamp
└── {keyspace}/
└── {table-with-uuid}/
└── {compression}/
└── {encryption}/
├── {component-file}
└── ...
The backup manifest orchestrates the backup, listing the paths to all its SSTables:
manifest.json
├── info
│ ├── appName "cass-cluster"
│ ├── region "us-east-1"
│ ├── snapshotInstantMs 1748908800000
│ ├── backupPathPrefix "prod_backup/1234_cass/token"
└── data[]
└── [0]
├── keyspaceName "ks"
├── columnfamily "tbl-a1b2..."
└── sstables[]
└── [0]
└── components[]
├── na-1-big-Data.db
├── na-1-big-Index.db
├── na-1-big-Filter.db
├── na-1-big-Statistics.db
├── na-1-big-Summary.db
├── na-1-big-CompressionInfo.db
Alongside the large Data.db file, backups include a compact set of index, metadata, and integrity files. This structure allows us to quickly locate, validate, and restore specific partitions or data ranges without scanning the entire dataset. Furthermore, since the manifest catalogs every S3 path, we can download most of these components directly to the Cassandra node (excluding the Data.db file) where they occupy negligible space. These local components enable efficient query planning before any data is read from S3.
How Would Cassandra Read From Backups?
Cassandra reads typically pull data from two sources: the Memtable, which holds recent unflushed writes, and flushed SSTables on durable storage. As of Cassandra 4.1, we can leverage pluggable Memtables to redirect reads from “cold” tables directly to the backup storage in S3.
-- create table for a given Time Slice
CREATE TABLE data_20260616_00_04 (
...
...,
PRIMARY KEY ((time_series_id, time_bucket, event_bucket), event_time, event_id, key)
) WITH CLUSTERING ORDER BY (event_time DESC, event_id DESC, key DESC)
...
AND compaction = ...
AND compression = ...
AND memtable = 'backupmemtable:timestamp=1781953200000' ← Custom memtable
...
The timestamp marks the point at which the backup is guaranteed to cover the full table, and the table is guaranteed to be immutable.
Opening a backup Memtable kicks off a state-machine initialization that pulls the index data from S3 onto the node once:
This process ensures all SSTable components are downloaded reliably through several resilience mechanisms:
- Retries and backoff: All S3 operations and file-system cache writes use retries with exponential backoff.
- Resource isolation: Initialization runs on a dedicated thread pool, isolated from normal Cassandra operations.
- Partial download protection: If a downloaded component’s size doesn’t match the manifest, it is deleted and re-downloaded.
- DDL change protection: A .timestamp file marks completed initialization and ties it to the backup timestamp. If the table’s backup timestamp changes, this file is removed and initialization restarts for the new timestamp.
- Read protection: Memtable initialization is asynchronous; reads to an initializing memtable block for at most half the read RPC timeout.
Once initialization has pulled every SSTable component (except Data.db) from S3 onto the local disk, the Memtable is ready to serve reads. Here is what the read process looks like:
To minimize latency, the system parallelizes S3 reads across requests, SSTables, and individual chunks. A shared cache deduplicates fetches and prefetches data, so chunks are ready before they are needed.
We further optimize cluster and table settings to reduce the number of S3 calls:
- Larger compression chunks: Reduce S3 calls per partition by reading fewer, larger chunks (accepting some over-fetch on small reads).
- Read repair disabled: Avoid expensive, unnecessary repair traffic against immutable, pre‑repaired backup data.
- Speculative retries disabled: Prevent S3’s higher latency from triggering redundant, overlapping reads.
How Would We Know When It Is Safe To Read From Backups?
However, before the Memtable can point to a backup, we must confirm two things across the whole ring, and we use Cassandra Virtual Tables to do that.
First, that ROCS re-compaction succeeded:
cqlsh> SELECT * FROM netflix_views.cluster_view
WHERE keyspace_name = 'system_views' AND table_name = 'sstable_tasks';
keyspace_name | table_name | host | value
--------------+---------------+----------+------------------------------------------------
system_views | sstable_tasks | 10.0.0.1 | [{"table_name":"<.>","kind":"compaction", ...}]
system_views | sstable_tasks | 10.0.0.2 | [] <---- no active compactions
system_views | sstable_tasks | 10.0.0.3 | []
And second, that backups were fully captured (denoted by uploaded=True):
cqlsh> SELECT * FROM netflix_views.complete_backups
WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';
keyspace_name | table_name | timestamp | num_tokens | num_uploaded | uploaded
--------------+------------+---------------+------------+--------------+----------
my_ks | my_table | 1765211400000 | 3 | 3 | True
my_ks | my_table | 1765213200000 | 3 | 2 | False
How Would We Automate This Entire Flow?
The TimeSeries control plane automates the entire transition as slices cross the immutable boundary:
Each slice can be in any lifecycle stage at any time without impacting system availability, and the system remains resilient to delayed backups or lagging compactions.
How Would Cassandra Handle The Additional Load?
Serving data from S3 is much more CPU-intensive than reading from local storage. A sudden spike in read requests can overwhelm the Cassandra cluster and affect the hot path, turning a cost-saving feature into a reliability risk.
To avoid this, cold reads can optionally run on a separate Cassandra cluster that points its backup Memtables at the hot cluster’s backups:
The cold Cassandra cluster must match the original cluster in topology but need not match it in hardware! We can run the cold cluster on much smaller, cheaper, storage-light instances.
Real World Performance
Cassandra Cold Tier was rolled out to the most storage‑intensive TimeSeries Abstraction clusters. It currently stores over 15 PB of compressed data in Cassandra cold storage and has allowed us to cut attached storage significantly on all affected clusters:
At the same time, it serves thousands of reads per second from cold tier storage in any given region:
Cassandra Cold Tier improves p90 latency by ~30% over the previous cold storage implementation, thanks to more efficient S3 access patterns and smaller read sizes, while keeping p99 latency under ~500 ms. Cassandra already handles bursts of cold reads gracefully via its chunk cache, so this design preserves that same resilience for the S3-backed cold tier:
Acknowledgements: Special thanks to our stunning colleagues who contributed to the TimeSeries Abstraction’s success and collaborated with us on the cold storage solutions: Kaidan Fullerton, Matt Lehman, Sumanth Pasupuleti, Carlos Matias Herrero, Nick Pavlakis, Bryan Keller, Francesco Di Chiara
TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.