Why Real-Time AI at Scale is So Hard
The failure patterns that surface as your app ramps up, and how to avoid them Real-time AI at scale is harder than it looks. Pipelines that hum along in development routinely hit problems in production. It’s always easy to blame the model for all your problems. But issues like rising latency and degrading accuracy can usually be traced back to the data pipeline. My colleague Tim Koopmans and I… SourceP99 CONF 2026 Recap: A Tail of Two Days
Our Aussie host’s take on P99 CONF around the clock and (now) anytime on demand P99 CONF… you’ve got to love this conference, which just completed its 6th edition. I’ve been lucky to help host the last 4 of them and I reckon the format is bang on. As a speaker, you get to pre-record and then listen to yourself “live” while the audience asks you questions in chat. You can even “heckle yourself,”… SourceFeast Feature and Vector Serving at ScyllaDB Speed
A new and tighter integration between Feast and ScyllaDB online feature store, including vector search support and better performance Starting with Feast v0.65, you can build with a new and tighter integration between Feast and ScyllaDB online feature stores, including vector search support and better performance. This post covers what changed, what tends to push teams to outgrow their online… Source5 questions to ask before your next DataStax Enterprise renewal
Renewing a DataStax Enterprise (DSE) contract used to be a formality. Check the box. Sign the term. Get back to shipping.
That’s changed. IBM’s acquisition of DataStax didn’t touch your cluster. It touched the company on the other side of the contract. Your application runs the way it did last quarter, but the agreement you’re about to sign describes a different relationship than the one you signed a few years ago.
The decision is also bigger than renew or leave. You can renew DSE, move to IBM’s Hyper-Converged Database (HCD), or move to managed Apache Cassandra®.
A DataStax Enterprise renewal should be evaluated as a three-way decision: renew DSE, adopt IBM’s HCD, or migrate from DataStax to managed Apache Cassandra. The right choice depends on your support costs, DSE-specific dependencies, operational requirements, and tolerance for vendor lock-in. Five questions will help you choose.
1. What exactly are you renewing?IBM announced its acquisition of DataStax on February 25, 2025 and closed it on May 28, 2025. Since November 1, 2025, new DataStax transactions run on IBM contracts. IBM sells DSE and HCD as components of its watsonx.data Premium offering, not as a standalone Cassandra product line. DSE 6.x was withdrawn from market on October 31, 2025.
None of that means you’re cut off. IBM’s published lifecycle lists standard support through December 31, 2027, and extended support through December 31, 2031. Read the terms closely. Per IBM’s support policy, extended support is a separate purchased add-on, so include it when calculating your total DataStax Enterprise support cost.
Extended support covers critical bug and security fixes in the first year only. After that, you get usage support and existing fixes. Supported and current aren’t the same thing, so read your renewal for what it now describes: a different product relationship.
2. Where is IBM taking the Cassandra portfolio, and does it match your roadmap?IBM tells existing DSE customers they can keep running DSE for several more years and move to HCD “when ready.” HCD is moving, too. HCD 2.0 had its first public release in June 2026, rebased on Cassandra 5.0. IBM’s public DataStax materials now center on watsonx, vector search, and unstructured data for generative AI.
If you want an AI data platform with IBM as the vendor, that’s a coherent direction, and staying put can be the right call. If you adopted DataStax because your application needed Cassandra to run reliably at scale, with predictable latency and uptime your users never notice, ask whether the roadmaps match. They can overlap. They aren’t guaranteed to.
Already planning to move to HCD? Ask what problem it solves. HCD stays inside IBM’s platform, so the vendor dependency comes with you. It’s self-managed, so upgrades, incident response, and tuning stay with your team. It isn’t a like-for-like move, either: DSE’s integrated Search and Graph don’t come along. If you have to rework the architecture anyway, compare HCD with a fully managed, vendor-independent option before you commit.
3. Which DSE-specific features do you actually use?Your answer reshapes the decision more than any other question here. For some teams, Search, Graph, Spark integration, OpsCenter, and DSE Advanced Replication keep applications compliant, observable, and fast. Others pay for the bundle and use a fraction of it.
Leaving DSE doesn’t mean leaving an active project. Apache Cassandra is under active, independent development: versions 5.0.9, 4.1.12, and 4.0.21 all shipped on August 7, 2026. 5.0 is the Apache Software Foundation’s current generally available release line, and 6.0 is in development, with an alpha already published. If your application relies on core Cassandra plus enterprise-grade support rather than the DSE-specific layer on top, you want to know that before you renew.
Building on Astra’s AI tooling? Check what’s actually tied to Astra. Langflow is open source and vector-store agnostic, and its Cassandra integration works with open source Cassandra. The narrower boundary is RAGStack and Knowledge Graph RAG, which don’t port over. If you rely on DSE Graph, treat it as its own decision and scope it separately.
4. What does another contract term cost you in flexibility?Renewing isn’t automatically wrong. But every multiyear DataStax Enterprise renewal is also a multiyear commitment not to reevaluate, and that has a cost even when the sticker price doesn’t move. Vendor lock-in is a business issue as much as a technical one. An application that depends on predictable performance also depends on your team having room to negotiate, migrate, or scale on its own timeline. Before you sign, price the next 3 years from every angle: what it costs to stay, what it costs to move, and what you give up in leverage either way.
5. Do you know what leaving would cost, or are you estimating?Most teams renew DSE because nobody priced the alternative, not because they compared the options and chose to stay. Has your team calculated what it would take to migrate from DataStax to Cassandra, or is it renewing without a documented cost and dependency assessment?
NetApp® Instaclustr has migrated more than 10,000 DataStax nodes across dozens of customers to open source Cassandra. Tesouro reports a 75% reduction in its storage footprint after migrating. Pega reports roughly 60,000 Cassandra management hours saved a year, hours that now go to the application instead of the database layer underneath it. Clear Capital completed its migration with zero downtime. These outcomes are customer-reported, and results vary by workload.
Migration isn’t simple. DSE-specific features add real complexity, and that complexity has to be scoped. Experience is what makes it manageable: the path exists, teams have completed it at scale, and you can price it with real numbers. Learn how to migrate from DataStax to Cassandra.
Before you renew, price the exitYou don’t need a research project. You need someone who has done this before to look at your environment and tell you what staying costs, what leaving costs, and what you’d give up either way.
The DSE Exit and Renewal Assessment from NetApp Instaclustr does that. Fill out a short intake, then spend 30 to 45 minutes on a call with a migration engineer. You get a written summary of your DSE dependency, directional migration complexity, and a ballpark cost comparison. It’s free, with no obligation. If your answer is to renew or move to HCD, the assessment shows you that too. If the call turns up real complexity, the next step is a deeper scoping conversation for your account.
Book my free assessment today.
The post 5 questions to ask before your next DataStax Enterprise renewal appeared first on Instaclustr.
TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads
By Oleksii Tkachuk, Rajiv Shringi, Chris Lohfink, Abe Ratnofsky, Kartik Sathyanarayanan
Introduction
In an earlier post, we introduced the TimeSeries Abstraction, our high-throughput, low-latency platform designed to store and query temporal event data at scale. It has since become the operational backbone for Netflix’s largest datasets, which now span multiple petabytes.
However, managing such large datasets in hot storage, tuned for low-millisecond latency access, is not cheap. At Netflix, cost-efficiency is a continuous discipline rather than a temporary initiative. This post shares two solutions that significantly reduce storage costs by continuously moving older, less frequently accessed data into cheaper object storage:
- Solution 1: Uses Apache Kafka and Flink to stream data directly to S3
- Solution 2: Leverages Cassandra backups in S3 and pluggable Memtables to read data directly from S3
We’ll start with the Kafka/Flink-based design, then describe how we evolved to a native Cassandra-based cold tier.
Challenges
As Netflix’s TimeSeries datasets have expanded, we’ve encountered several compounding challenges:
- Growing retention demands: Users often push for increasing data retention to serve a broader range of use cases, which in turn increases the storage costs.
- Escalating costs: Maintaining massive, dense clusters for infrequently accessed historical data is prohibitively expensive.
- Sparse historical access: Most queries target recent events, but older data must remain readily available for occasional audits and investigations.
- Operational strain: High storage density on hot nodes complicates scaling, deployment, and cluster stability.
- Latency requirements: Even historical data sometimes requires sub-second query performance for internal tooling and analysis.
This creates a clear opportunity: keep the most recent data on fast storage and migrate older, rarely read data to cheaper storage while preserving a consistent API. Done well, this can meet user and operational needs while greatly reducing our infrastructure spend.
TimeSeries Abstraction
Before we dive into the cold storage solutions, here is a quick recap of how TimeSeries partitions its dataset into multiple tables, each covering a window of time we call Time Slices.
This enables several key optimizations, most notably allowing a table to become immutable once it crosses the write boundary, which lets us safely apply further optimizations. For more details, refer to our previous post.
This slice information is critical and is maintained in a dedicated metadata table:
Creating new Time Slices and maintaining their metadata is a primary responsibility of the TimeSeries control plane. The TimeSeries data plane uses this information to correctly route the read and write traffic to the provisioned tables.
With that background, let’s look at the different solutions we built for data tiering.
Solution 1: Streaming with Apache Kafka and Flink
Our first solution followed a familiar streaming pattern using Apache Kafka and Flink, where we streamed data to S3 in realtime while simultaneously persisting it to Cassandra.
Getting Data Into S3 Efficiently
To optimize data streaming to S3, the system hash-partitions Time Series IDs into buckets. This partitions the stream and lets dedicated Flink tasks handle each bucket, ultimately batching and reducing the write throughput to S3. Flink workers batch events into Parquet files grouped by event time within the hour. These files are great for high-throughput streaming writes, but not for low-latency reads.
Enter the Compaction job.
Compaction Job to Optimize Reads
The compaction job runs daily and merges small Parquet files into a single, optimized daily Parquet file per namespace.
The job further optimizes the Parquet files in the following ways:
- Small Row Groups: Creates small (4–8 MB) row groups to minimize lookup cost. We can then use the Parquet Java library to perform range reads of row groups from S3. This is intentionally well below Parquet’s recommended row group floor — smaller groups were better because our reads are dominated by point/range lookups rather than full scans.
- Event Sorting: Sorts events within a row group by time, eliminating brute-force scans when querying for a time range.
- Bloom Filters: Caches Bloom filters and file metadata in EVCache, allowing the TimeSeries query planner to efficiently skip unnecessary reads from S3.
Once the compaction job completes, all compacted files per partition are recorded in the TimeSeries control plane, and the hot storage slices can now be truncated!
On some of our largest datasets, this job typically takes about 12 hours, prioritizing lower compute costs over execution speed.
Streaming Cold Tier Read
Once compacted files are optimized in cold storage, they are ready to be read. The TimeSeries data plane fans out a given query across both the hot and cold tiers and merges their result. The users still use the same query interface:
The Cold Tier Data Plane uses compaction metadata for aggressive planning and pruning, which dramatically reduces the S3 query fan-out. As a result, retrieving a full day of data requires at most three sequential S3 reads, even in the uncommon case where an ID spans two row groups. This yields a p50 latency of ~150 ms and a p99 latency of ~500 ms per read.
At the time of writing, the streaming cold tier implementation was processing up to 1,250 reads per second from cold storage globally:
Why not stop here?
The streaming cold tier has delivered real savings, shrinking both Cassandra’s EBS footprint and the Cassandra backup footprint in S3. But the streaming infrastructure itself is expensive and adds moving pieces that widen the team’s on-call scope:
We wanted to improve this further and asked ourselves:
- Can we avoid streaming data into the object store and the associated data transfer costs?
- Can we reduce the compute and operational overhead of new components?
That led to a more radical question: Can we read directly from existing Cassandra backups in S3?
Solution 2: Cassandra Cold Storage for Immutable Data
At Netflix, we back up Cassandra data into S3 at regular intervals for disaster recovery. The idea for tiering is simple:
Read directly from these S3 backups instead of paying a separate data transfer cost to S3.
Once TimeSeries marks a hot Time Slice as ready for cold storage, we can customize Cassandra to read directly from its backups, allowing us to delete the on-disk data copy. This would also incur minimal changes to the TimeSeries read path!
To make this viable in production, we had to answer a series of “How…?” questions.
How Can We Read the Backups Efficiently?
We introduced a new compaction strategy in Cassandra, which we call ReadOnlyCompactionStrategy (ROCS). Here is how it works:
Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable. This immutability unlocks significant optimization opportunities:
- SSTable overlap: Standard strategies (LCS, TWCS) often produce SSTables that overlap in the token space, and so a single partition key may exist across multiple files.
- Read fan‑out: Because a read must touch every SSTable covering the key’s token range, this overlap turns a single point-read into multiple lookups, increasing latency across all tiers.
ROCS re-compacts all SSTables once, aligning them precisely with token ranges:
ROCS applies more broadly to all TimeSeries use cases, driving tail latency down on all storage tiers by reducing the SSTable fan-out during read operations:
Once the immutable Time Slices have been optimized, it is time to trigger backups.
How Would We Leverage Backups?
Cassandra backs up every node in the cluster, organizing each backup on S3 as a declarative manifest, along with the SSTables backing the keyspace’s tables:
{bucket}/
└── {prefix}/ ← cluster identity (env_backup/hash_app)
└── {token}/ ← node identity (ring token)
├── META_V2/ ← manifests, one per snapshot
│ ├── {snapshotInstantMs}.json
│ ├── {snapshotInstantMs}.json
│ └── ...
└── SST_V2/ ← actual SSTable files
└── {backupTimestamp}/ ← snapshot timestamp
└── {keyspace}/
└── {table-with-uuid}/
└── {compression}/
└── {encryption}/
├── {component-file}
└── ...
The backup manifest orchestrates the backup, listing the paths to all its SSTables:
manifest.json
├── info
│ ├── appName "cass-cluster"
│ ├── region "us-east-1"
│ ├── snapshotInstantMs 1748908800000
│ ├── backupPathPrefix "prod_backup/1234_cass/token"
└── data[]
└── [0]
├── keyspaceName "ks"
├── columnfamily "tbl-a1b2..."
└── sstables[]
└── [0]
└── components[]
├── na-1-big-Data.db
├── na-1-big-Index.db
├── na-1-big-Filter.db
├── na-1-big-Statistics.db
├── na-1-big-Summary.db
├── na-1-big-CompressionInfo.db
Alongside the large Data.db file, backups include a compact set of index, metadata, and integrity files. This structure allows us to quickly locate, validate, and restore specific partitions or data ranges without scanning the entire dataset. Furthermore, since the manifest catalogs every S3 path, we can download most of these components directly to the Cassandra node (excluding the Data.db file) where they occupy negligible space. These local components enable efficient query planning before any data is read from S3.
How Would Cassandra Read From Backups?
Cassandra reads typically pull data from two sources: the Memtable, which holds recent unflushed writes, and flushed SSTables on durable storage. As of Cassandra 4.1, we can leverage pluggable Memtables to redirect reads from “cold” tables directly to the backup storage in S3.
-- create table for a given Time Slice
CREATE TABLE data_20260616_00_04 (
...
...,
PRIMARY KEY ((time_series_id, time_bucket, event_bucket), event_time, event_id, key)
) WITH CLUSTERING ORDER BY (event_time DESC, event_id DESC, key DESC)
...
AND compaction = ...
AND compression = ...
AND memtable = 'backupmemtable:timestamp=1781953200000' ← Custom memtable
...
The timestamp marks the point at which the backup is guaranteed to cover the full table, and the table is guaranteed to be immutable.
Opening a backup Memtable kicks off a state-machine initialization that pulls the index data from S3 onto the node once:
This process ensures all SSTable components are downloaded reliably through several resilience mechanisms:
- Retries and backoff: All S3 operations and file-system cache writes use retries with exponential backoff.
- Resource isolation: Initialization runs on a dedicated thread pool, isolated from normal Cassandra operations.
- Partial download protection: If a downloaded component’s size doesn’t match the manifest, it is deleted and re-downloaded.
- DDL change protection: A .timestamp file marks completed initialization and ties it to the backup timestamp. If the table’s backup timestamp changes, this file is removed and initialization restarts for the new timestamp.
- Read protection: Memtable initialization is asynchronous; reads to an initializing memtable block for at most half the read RPC timeout.
Once initialization has pulled every SSTable component (except Data.db) from S3 onto the local disk, the Memtable is ready to serve reads. Here is what the read process looks like:
To minimize latency, the system parallelizes S3 reads across requests, SSTables, and individual chunks. A shared cache deduplicates fetches and prefetches data, so chunks are ready before they are needed.
We further optimize cluster and table settings to reduce the number of S3 calls:
- Larger compression chunks: Reduce S3 calls per partition by reading fewer, larger chunks (accepting some over-fetch on small reads).
- Read repair disabled: Avoid expensive, unnecessary repair traffic against immutable, pre‑repaired backup data.
- Speculative retries disabled: Prevent S3’s higher latency from triggering redundant, overlapping reads.
How Would We Know When It Is Safe To Read From Backups?
However, before the Memtable can point to a backup, we must confirm two things across the whole ring, and we use Cassandra Virtual Tables to do that.
First, that ROCS re-compaction succeeded:
cqlsh> SELECT * FROM netflix_views.cluster_view
WHERE keyspace_name = 'system_views' AND table_name = 'sstable_tasks';
keyspace_name | table_name | host | value
--------------+---------------+----------+------------------------------------------------
system_views | sstable_tasks | 10.0.0.1 | [{"table_name":"<.>","kind":"compaction", ...}]
system_views | sstable_tasks | 10.0.0.2 | [] <---- no active compactions
system_views | sstable_tasks | 10.0.0.3 | []
And second, that backups were fully captured (denoted by uploaded=True):
cqlsh> SELECT * FROM netflix_views.complete_backups
WHERE keyspace_name = 'my_ks' AND table_name = 'my_table';
keyspace_name | table_name | timestamp | num_tokens | num_uploaded | uploaded
--------------+------------+---------------+------------+--------------+----------
my_ks | my_table | 1765211400000 | 3 | 3 | True
my_ks | my_table | 1765213200000 | 3 | 2 | False
How Would We Automate This Entire Flow?
The TimeSeries control plane automates the entire transition as slices cross the immutable boundary:
Each slice can be in any lifecycle stage at any time without impacting system availability, and the system remains resilient to delayed backups or lagging compactions.
How Would Cassandra Handle The Additional Load?
Serving data from S3 is much more CPU-intensive than reading from local storage. A sudden spike in read requests can overwhelm the Cassandra cluster and affect the hot path, turning a cost-saving feature into a reliability risk.
To avoid this, cold reads can optionally run on a separate Cassandra cluster that points its backup Memtables at the hot cluster’s backups:
The cold Cassandra cluster must match the original cluster in topology but need not match it in hardware! We can run the cold cluster on much smaller, cheaper, storage-light instances.
Real World Performance
Cassandra Cold Tier was rolled out to the most storage‑intensive TimeSeries Abstraction clusters. It currently stores over 15 PB of compressed data in Cassandra cold storage and has allowed us to cut attached storage significantly on all affected clusters:
At the same time, it serves thousands of reads per second from cold tier storage in any given region:
Cassandra Cold Tier improves p90 latency by ~30% over the previous cold storage implementation, thanks to more efficient S3 access patterns and smaller read sizes, while keeping p99 latency under ~500 ms. Cassandra already handles bursts of cold reads gracefully via its chunk cache, so this design preserves that same resilience for the S3-backed cold tier:
Acknowledgements: Special thanks to our stunning colleagues who contributed to the TimeSeries Abstraction’s success and collaborated with us on the cold storage solutions: Kaidan Fullerton, Matt Lehman, Sumanth Pasupuleti, Carlos Matias Herrero, Nick Pavlakis, Bryan Keller, Francesco Di Chiara
TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Inside the Database Internals Talks at P99 CONF 2026
Since database performance is so near and dear to ScyllaDB, we reached out to our friends across the community to ensure that a nice range of distributed data systems, approaches, and challenges would be represented at P99 CONF 2026. As you can see from our agenda, the response was overwhelming. Join us at P99 CONF (free + virtual) A quick PSA for the uninitiated: P99 CONF is a free 2-day… SourceBuilding a Faster (Rust-Based) Python Driver for ScyllaDB
Binding Rust to Python with PyO3, zero-copy deserialization with yoke Since Python is one of the most commonly used languages for interacting with a database, having a robust ScyllaDB Python driver is essential. The first ScyllaDB Python driver was created ages ago, back when ScyllaDB was in the 2.x stage, and it’s been used ever since. As part of the ZPP Project at the University of Warsaw… SourceMCP Gateway RBAC tutorial: Add Role-Based Access Control to your AI agent with Auth0 and Apache Cassandra
This is a follow-up to “MCP Gateway tutorial: Connect an AI Agent to Apache Kafka and HTTP Server Backends,” extends the original support chatbot with Auth0-based login, role-based access control (RBAC), and an Apache Cassandra backend.
In this MCP Gateway RBAC tutorial, you will configure MCP Gateway to enforce role-based access control for an AI agent. The walkthrough adds Auth0 authentication, maps user roles to MCP Gateway Access Control Lists, connects an Apache Cassandra backend, and shows how each persona receives only the tools their role is allowed to use.
The goal is to support least-privilege access for AI agent tools at the gateway layer: Auth0 authenticates the user, MCP Gateway maps the user’s role to an Access Control List, and the agent receives only the tools that role is permitted to call.
Get the code: MCP-Gateway/shoe-store-support-chatbot-rbac
Watch the tutorial: Add RBAC to Your AI Agent: MCP Gateway + Auth0 + Apache Cassandra
Recap: where we left offThe previous article built a support chatbot for a shoe
e-commerce company, backed by one MCP (Model Context Protocol)
Gateway Virtual Server (supportchatbot) with two
backends: ordersapi (HTTP) providing get_orders, and
supporttickets (Apache Kafka) providing
submit_request. That version had a single, shared set
of tools for every user.
This article adds Auth0 login and three distinct roles
(support-agent, merchandiser, and
readonly-auditor), plus a third backend type,
Cassandra, so each persona gets exactly the access their job calls
for, configured entirely at the MCP Gateway.
From the previous tutorial, you should already have:
- An MCP Gateway Virtual Server
(
supportchatbot). - The ordersapi (HTTP) backend with a
get_orderstool. - The supporttickets (Kafka) backend with a
submit_requesttool.
Everything involving Auth0, OAuth, roles, and Access Control Lists (ACLs) is new in this article.
Tooling: Python 3.11+, the uv package manager, and an AWS account with Amazon Bedrock model access enabled for the model you’re using, in the region you’re calling (e.g. us-east-1).
The use case: a product catalog with tiered cost visibility-
support-agent(customer-facing staff) gets product info (name, price, stock) to help shoppers, plus order lookups and ticket filing. -
merchandiser(buying/purchasing staff) gets supplier cost and margin data to negotiate pricing, commercially sensitive information relevant to their role, distinct from customer-service tooling. readonly-auditorgets full read visibility across both domains for review purposes, with action (filing a ticket) reserved for support-agent.
Role-Based Access Control means a user’s identity maps to a fixed role, and every access decision is checked against an explicit allow-list. In MCP Gateway, that role determines which tools an AI agent can discover and call, so each persona gets exactly the access their job requires: no more and no less.
This matters especially for an LLM-backed agent, for three reasons:
- It keeps every session’s capabilities predictable, no matter
how the conversation goes. The chatbot handles free-text customer
input, and however that conversation unfolds, a support-agent
session simply can’t reach
catalogdata_get_supplier_cost: it’s not part of that role’s tool set, so there’s nothing to accidentally or deliberately reach for. The boundary is a property of the session, not a judgment call the model has to get right in the moment. - It lets you express two independent things: what a role can
see, and what it can do. merchandiser can see cost data but can’t
file tickets;
readonly-auditorcan see everything but can’t act on any of it. Splitting visibility from action this way maps naturally onto how organizations already assign responsibility. - It moves enforcement to a layer that’s easy to reason about. Rather than relying on prompt instructions like “don’t answer cost questions for this role,” which work well in the common case but depend on the model’s judgment every time, the MCP Gateway checks the ACL before a tool call is honored, giving you one clear, auditable place where access is decided.
Figure 1. Routing role-based AI agent requests to HTTP, Kafka, and Cassandra backends through the MCP Gateway
Step 1: Provision the Cassandra clusterIn the Instaclustr console, Create Cluster → Apache Cassandra, choosing a data center/region and node size appropriate for a demo.
Once the cluster reaches Running, note its Data Center ID from the Details page for Step 3.
Use the console’s Connection Info tab and default superuser credentials to connect via cqlsh for the next step.
Step 2: Set up the Cassandra schema, seed data, and a least-privilege roleCREATE KEYSPACE IF NOT EXISTS catalog WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 3}; CREATE TABLE catalog.products ( productid text PRIMARY KEY, name text, description text, category text, listprice decimal, stockstatus text ); CREATE TABLE catalog.supplier_costs ( productid text PRIMARY KEY, supplierid text, unitcost decimal, marginpercent decimal, suppliercontact text ); CREATE ROLE mcpgateway_catalog WITH PASSWORD = '<password>' AND LOGIN = true; GRANT SELECT ON catalog.products TO mcpgateway_catalog; GRANT SELECT ON catalog.supplier_costs TO mcpgateway_catalog; INSERT INTO catalog.products (productid, name, description, category, listprice, stockstatus) VALUES ('SHOE-001', 'Midnight Runner', 'Black low-top running shoe', 'running', 129.99, 'in_stock'); INSERT INTO catalog.products (productid, name, description, category, listprice, stockstatus) VALUES ('SHOE-002', 'Trailblazer Boot', 'Brown waterproof hiking boot', 'hiking', 189.99, 'backordered'); INSERT INTO catalog.supplier_costs (productid, supplierid, unitcost, marginpercent, suppliercontact) VALUES ('SHOE-001', 'SUP-88', 42.00, 67.7, 'orders@northstarfootwear.com'); INSERT INTO catalog.supplier_costs (productid, supplierid, unitcost, marginpercent, suppliercontact) VALUES ('SHOE-002', 'SUP-14', 71.50, 62.4, 'sales@trailgearco.com');
Connect with the credentials and endpoint from your cluster’s
Connection Info tab before running this (see Step 1).
mcpgateway_catalog is read-only and scoped to just
these two tables: a second, defense-in-depth layer of access
control underneath the MCP Gateway’s own ACLs.
Notes:
- Identifiers are written lowercase throughout, matching how Cassandra represents unquoted identifiers internally: table/column names and named bind markers alike. Writing them lowercase from the start keeps the CQL here consistent with the tool configuration in Step 3.
- Remember to replace <password> in the
CREATE ROLE mcpgateway_catalogstatement with a real, strong password before running this; it’s a placeholder, not a literal value. Keep it somewhere secure, since you’ll need to re-enter it in Step 3 when you create thecatalogdatabackend on the MCP Gateway (that’s where these credentials actually get used).
On the supportchatbot MCP Virtual Server → Create
Backend → name catalogdata, type Cassandra, using the
Data Center ID from Step 1 and the mcpgateway_catalog
credentials from Step 2.
Figure 2. The catalogdata Cassandra backend,
showing its Cassandra Data Center ID, Cassandra Username
(mcpgateway_catalog), and its two configured
tools.
Open the new catalogdata backend and, under Tools,
click Create Tool to create the below two tools.
get_product_details:
SELECT name, description, category, listprice, stockstatus FROM catalog.products WHERE productid = :productid
- Parameter:
productid(string) - Result fields:
name(string),description(string),category(string),listprice(string),stockstatus(string) - Description: “Look up a product’s customer-facing details by product ID, including its name, description, category, list price, and current stock status.”
get_supplier_cost:
SELECT supplierid, unitcost, marginpercent, suppliercontact FROM catalog.supplier_costs WHERE productid = :productid
- Parameter:
productid(string) - Result fields:
supplierid(string),unitcost(string),marginpercent(string),suppliercontact(string) - Description: “Look up internal supplier cost and margin data for a product by product ID. This is sensitive commercial data; only use it when explicitly asked, and only if you have access to this tool.”
Figure 3. The get_product_details tool definition,
showing its CQL query, productid parameter, and result
fields.
Notes:
- Declare Result Field Type as string for any Cassandra decimal column.
- Give every tool a real, descriptive description. Beyond helping the model choose the right tool, AWS Bedrock’s Converse API expects every tool in the list to have one, so it’s worth treating as a required field alongside the CQL and parameters.
- Apache Cassandra forces unquoted bind variable names to
lowercase:
:productIdis interpreted internally as:productid. That’s why every identifier in this tutorial (table/column names and bind markers alike) is written lowercase from the start, keeping the CQL here consistent with the tool configuration above. If you ever need a case-sensitive parameter name, quote it in the query instead, e.g.:"productId".
4a. Application and Post-Login Action: Follow the Auth0 example configuration guide to:
- Create a Regular Web Application and configure its Allowed Callback URIs / Web Origins.
- Create the Post-Login Action that adds a roles claim when the
mcp_rolesscope is requested, add theMCP_GATEWAY_CLIENT_IDsecret, deploy it, and apply it on the post-login trigger. - Note the audience (API Identifier), issuer, and JWKS URI as described there; needed in Step 5.
4b. Create the three Auth0 Roles: User Management → Roles → Create Role, once each for:
support-agentmerchandiserreadonly-auditor
4c. Create three test users, one per role: User Management → Users → Create User (or reuse existing test users), then on each user’s Roles tab, Assign Roles with the matching role from 4b.
Note: Keep role names consistent,
case-for-case, across three places: the Auth0 Role, the
user_roles claim value it produces, and the MCP
Gateway ACL’s Role name field in Step 6. Matching these exactly is
what makes each persona’s tool list show up correctly.
On the supportchatbot MCP Virtual Server’s OAuth
configuration section (refer to the documentation for
more details), enter the values noted in Step 4a:
- Issuer:
https://<domain>/ - JWKS
URI:
https://<domain>/.well-known/jwks.json - Audience: your API Identifier
- Scopes Supported:
mcp_roles - Roles Claim
Name:
https://mcp_gateway/user_roles
Save/apply.
Figure 4. The supportchatbot MCP Virtual Server’s
OAuth configuration with all three backends.
On the MCP Gateway’s ACL screen, you pick each tool by its bare name; the Backend column disambiguates which tool you mean when the same name could theoretically exist on more than one backend:
support-agent
Tool Backendget_orders ordersapi
submit_request supporttickets
get_product_details catalogdata
readonly-auditor
Tool Backendget_orders ordersapi
get_product_details catalogdata
get_supplier_cost catalogdata
merchandiser
Tool Backendget_product_details
catalogdata get_supplier_cost
catalogdata
Figure 5. The support-agent Access Control List, showing its explicitly allowed tools.
Each role has a distinct tool set tailored to its purpose: merchandiser focuses entirely on catalog data, while readonly-auditor sees everything the other two combined can see, with action reserved for support-agent.
Once exposed through the supportchatbot MCP Virtual
Server, the agent sees each tool under a backend-prefixed name
(e.g. catalogdata_get_product_details) to disambiguate
tools coming from different backends; that’s the name that shows up
in the app’s “Tools discovered” sidebar later, and in Bedrock’s
tool-calling requests, even though you select it by its bare name
here.
Figure 6. The completed supportchatbot MCP Virtual
Server, showing its OAuth configuration, all three backends, and
the three role-based Access Control Lists.
None of this required touching the chatbot’s code, because it
was built to discover tools dynamically rather than hardcode
them: mcp_manager.py‘s
connect() attaches the user’s access token as a Bearer
header once, at connection time, and every session it opens rides
on that same authenticated connection for the rest of the login;
get_tool_specs() then simply asks the MCP Gateway
which tools this identity can see and turns whatever comes back
into Bedrock’s toolConfig.tools format, with the MCP
Gateway itself deciding (via the caller’s role and its Access
Control List) which tools are included; and app.py‘s
sidebar just renders that same list, grouping tool names by server
with nothing hardcoded. So, when catalogdata‘s two
tools were added at the MCP Gateway, they simply started showing up
for the roles allowed to see them, with zero code changes anywhere
in this repo.
The full code for this tutorial is available at MCP-Gateway/shoe-store-support-chatbot-rbac in
the instaclustr/code-samples repo.
cp config.example.json config.json # fill in your Auth0 tenant + MCP Gateway URL uv sync export AWS_ACCESS_KEY_ID="..." export AWS_SECRET_ACCESS_KEY="..." export AWS_DEFAULT_REGION="us-east-1" uv run streamlit run src/rbac_shoe_bot/app.py
This opens the app at http://localhost:8501.
Compared to the shared-access version from the previous tutorial,
config.json now includes a populated auth0 section,
and the app opens on a Log in with
Auth0 screen.
Click Log in with Auth0 and sign in as one of your test users. To try a different persona, click Log out and log in again; each login prompts for credentials fresh, so you can move between test users freely.
Figure 7. A readonly-auditor session in the
chatbot, showing its discovered tools in the sidebar and a
successful get_supplier_cost lookup for
SHOE-002.
Notes:
- Enable model access for
bedrock.model_idin the Bedrock console for your region ahead of time; it’s a quick, one-time opt-in per model/region, and worth doing before your first run. - The sidebar’s Tools discovered list is the fastest way to
confirm your ACL setup is working as intended; a
merchandisersession should show only the twocatalogdata_*tools.
Demo script
Using the two seeded products:
support-agent: “What’s the price and stock status of product SHOE-001?” → succeeds. “What do we pay our supplier for it?” → no tool available for this role.merchandiser: “What’s our cost and margin on SHOE-001?” → succeeds. “Show me my recent orders.” → no tool available for this role.readonly-auditor: sees both cost and orders, with ticket filing reserved forsupport-agent.
Note: These prompts reference product IDs
directly rather than descriptions like “black shoes,” since
get_product_details looks up by exact
productid. Referencing known IDs keeps the demo
focused on RBAC rather than search.
Authentication, three distinct roles, and a whole new backend technology were all added here through configuration: Auth0 setup, MCP Gateway OAuth config, ACLs, and the Cassandra backend/tools, without changing the chatbot’s own code. That’s the strength of centralizing access control at the MCP Gateway layer: the same access rules apply consistently, however a conversation unfolds, because the boundary lives outside the agent’s own reasoning.
Try it yourselfIf you want to add role-based access control to your own AI agent, you can spin up MCP Gateway and an Apache Cassandra cluster today: start a free Instaclustr trial, no credit card required. From there:
- Follow the MCP Gateway documentation to provision a Virtual Server and connect your first backend.
- Use the Auth0 example configuration guide to wire up your own identity provider.
- Clone MCP-Gateway/shoe-store-support-chatbot-rbac and swap in your own tools, roles, and backends: the same three steps (Auth0 roles claim, Gateway OAuth config, per-role ACLs) apply no matter what your agent actually does.
If you’d rather start from the simpler, single-role version, the previous tutorial walks through connecting an agent to Kafka and HTTP backends without the auth layer.
The post MCP Gateway RBAC tutorial: Add Role-Based Access Control to your AI agent with Auth0 and Apache Cassandra appeared first on Instaclustr.