Overheard at Scylla Summit 2021, Day Two

14 January 2021, 3:00 pm by ScyllaDB

Based on our record attendance and your feedback, our first online Scylla Summit has been a rousing success! First, if you missed it, make sure you read the blog about Day One of Scylla Summit 2021.

Yesterday, Tuesday, January 12th was the second day of presentations, and included talks from ScyllaDB CTO and Co-Founder Avi Kivity, who spoke about the challenges of Big Data with Scylla, Confluent’s Tim Berglund, who spoke on combining the power of Apache Kafka and Scylla’s new CDC capabilities, and a range of sessions from Scylla and Seastar users and ScyllaDB engineers.

Also, before we get further into the details, we want to give you a timely reminder:

Don’t Miss Our Training Day!

Our free, live and online is happening today, January 14th 2021, from 8:30 – 11:30 AM PST. You can check out the full schedule here.

Our Training Day includes separate tracks for developers and administrators. So whether you are more interested in designing a well-distributed data model, or learning how to manage your clusters using Kubernetes, we’ll have sessions right up your alley. If you’re new to NoSQL or Scylla in particular, we even have a getting started course so you can embark on a whole new career path.

REGISTER FOR SCYLLA SUMMIT 2021 TRAINING DAY

The entire Training Day curriculum, like the rest of Scylla Summit, is free and open to all attendees, so register now! But if you can’t make it at this time, don’t worry! You can always register for Scylla University, which also provides free online self-paced lessons for all ranges of skills and interests. It also serves to reinforce any skills you’ll learn at our Training Day.

GET STARTED IN SCYLLA UNIVERSITY

Avi Kivity: Meeting the Challenges of Big Data

Avi in his keynote began by reiterating the characteristics and initiatives brought together under the umbrella of Project Circe, which we announced the day before.

The first characteristic Avi focused on was performance because it has a direct correlation with the cost of running your database and sustaining your workload. “It directly relates to your expenses. If the database is twice as fast, then your costs are halved. This is the main reason to pick Scylla for your big data.”

Though there’s more to it than just that. “Performance is also the ability to maintain your workload while it is undergoing maintenance operations. Scaling up and down.” Avi noted the importance of maintenance operations not interfering with your main workloads. “So the feature here is performance isolation.”

Isolation also applies to different data types. Simple inserts are very different, in terms of performance costs, than full scans, reading via various indexes, filtering, or reading large partitions. “We are working to improve our support of more and more query types so there are fewer performance-related surprises while using the database.”

Avi also pointed out how we are working on supporting dense nodes. “The more storage your node has, the fewer nodes you will need. And that reduces operational costs of maintaining the database. There are fewer things that can fail. Fewer things to maintain.” To Avi, continuing to make the most out of your hardware is Scylla’s “bread and butter” — utilizing all cores, all memory for caching, and making good use of fast solid state drives (SSDs).

He also pointed out some recent improvements: use of B+tree in cache (instead of a red-black tree), reactor stall eliminations, and the implementation of C++ coroutines.

The B+tree improves the memory footprint of objects in cache, “so we can pack more partitions into cache and increase the hit rate.” It also reduces the cycle cost of reading and inserting into cache, which allows loading more objects per second.

To solve reactor stalls requires breaking down tasks into smaller and smaller pieces that can be preempted. C++ coroutines, a feature of C++20, “allow us to reduce the number of allocations needed to service a query.” And, besides, it’s fun to play with nifty new code. “We enjoy that.”

These are just highlights from the beginning of Avi’s presentation. He ranged onwards to other topics including elasticity, the use of Raft in Scylla, Change Data Capture (CDC), Kubernetes, our partnership with AWS Outposts and more. We will bring you the video and slides of his whole half-hour keynote, as well as all the sessions that delved into each of these individual topics shortly after the Summit concludes.

CDC and Streaming Data from Scylla to Kafka

Speaking about CDC, right after Avi we had two back-to-back sessions that showed users a new paradigm of how to leverage their big data in Scylla using Change Data Capture, which is now a GA feature with the release this week of Scylla Open Source 4.3.

First up was ScyllaDB’s Calle Wilund, who explained best practices for Change Data Capture in Scylla. CDC is enabled in Scylla per table. When enabled it creates a CDC log, which is just another CQL table that records the changes to the base table. It can record all the deltas, as well as the pre- and post-images.

The CDC log is stored distributed on nodes across the cluster, with rows ordered by timestamp and batch sequence. The topology will match the source table — the CDC log is collocated with, and its partition scheme will match the base table. Also, CDC data is transient. It has a Time to Live (TTL) which defaults to 24 hours, but is user-configurable, to ensure that it does not expand unbounded over time.

After Calle next up was Tim Berglund of Confluent, who gave a practical example of how you can harness the power of Scylla’s CDC feature with Apache Kafka. Scylla’s under-development Kafka source connector is built on top of open source Debezium.

With it, you will be able to share data from Scylla CDC tables straight out to Kafka topics. And, along with the Kafka Scylla Connector that serves as a sink connector, you are now able to use Scylla as both a source and a destination in a complete data ecosystem with Apache Kafka.

Tim’s presentation went step-by-step through how the connector works, from setting it up, to showing how data flows from CQL tables to JSON in Kafka topics. For now, the connector only supports delta operations. Preimage and postimage will be added in the future, which Tim pointed out will match nicely with the “before” and “after” fields of Debezium.

A Full Agenda

With each other session being about 10 to 12 minutes long, the rest of the day was filled with the equivalent of “lightning talks” comprised of a mix of ScyllaDB engineers and users/customers.

• Numberly: Getting the Scylla Shard-Aware Drivers Faster — Numberly’s CTO Alexys Jacob is no stranger to the Scylla Summit stage. One of our all-time favorite speakers, this year he described the work that went into improving the performance of Scylla’s shard-aware Python driver. In case you missed it, make sure you read our earlier articles, both part one, which describes the basic goals and methods of making a shard-aware driver, and part two, which covers the implementation. His talk went beyond those earlier posts, covering performance optimizations.
• Scylla Performance Enhancements: Year in Review — ScyllaDB’s Tomasz Grabiec took attendees deeper into the performance improvements Avi touched upon in his keynote. These internal improvements make Scylla faster than ever before: handling large partitions, SSTable index caching, eliminating stalls, B+trees in cache for even faster lookups, and an improved I/O scheduler.
• Mail.Ru: High-Load Storage of Users’ Actions with Scylla and HDDs — Mail.Ru’s Kirill Alekseev had a unique solution for storing petabytes of data, optimizing for best price over performance by employing spinning media — HDDs — while still maintaining single-digit latencies.
• The All-New Scylla Monitoring Stack — ScyllaDB’s Amnon Heiman showed off the new capabilities of Scylla Monitoring Stack, including its integration of Grafana Loki, the new advisor section, overview and CQL dashboards, and log collection analysis.
• Expedia: Our Migration to Scylla — Expedia Group explained why they moved from Apache Cassandra to Scylla, including an analysis of Scylla’s faster software release cycles and clear roadmap, to TCO, to benchmarks showing lower latencies and greater throughput. For Expedia, over and over again, Scylla won hand’s down.
• How We Made Scylla Maintenance Easier, Safer and Faster — ScyllaDB’s Asias He described the efforts behind making Scylla more elastic and even more maintainable. With Scylla Open Source 4.3, we removed the concept of seed nodes (read more in this article). He also spoke about how our improvements in row-level repair led to our repair-based node operations (RBNO), providing a single mechanism for all node operations, from repair, to bootstrapping, replacing, rebuilding, to decommissioning and removing nodes. He also revealed how we will implement a new more resilient “safe mode” feature for removenode, and a redesigned replace operation.
• IOTA: How IOTA Uses Scylla to Store Unbounded Flow — IOTA is a distributed ledger technology, known as The Tangle. It is radically different from blockchains by being minerless and feeless. IOTA Foundation’s Louay Kamel described how they use Scylla and the Rust CQL driver in their architecture to persistently store unbounded flows of data.
• Developing Scylla Applications: Practical Tips — This great session by Kamil Braun focused on application developers. “The way you write your clients can make or break your whole operation. If you want to know how to make clients that keep the cluster healthy, the system performant, and highly available, you’re in the right place.” True story: as Kamil spoke, Numberly’s Alexys Jacob and ScyllaDB’s Israel Fruchter, who collaborated on making the Python shard-aware driver, had such a revelation about adding backoffs they immediately opened a Github issue for a TODO to improve on their existing design!
• Broadpeak: Building a Video Streaming Testing Framework with Seastar — Nicolas Le Scouarnec showed how Broadpeak is using Seastar, the underlying highly-asynchronous engine at the heart of Scylla, to solve a radically different problem: testing a streaming video framework at scale. This was not a server, but a client application — a benchmarking tool focused on network I/O and using the Chromium browser to emulate HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH) players.
• Redpanda: A New Streaming Storage Engine for Modern Hardware — Vectorized CEO Alex Gallego also showcased a unique application of the Seastar framework. Redpanda is a complete open source, from-the-ground-up redesigned streaming engine compatible with Apache Kafka. Analogous to how Scylla’s  reimplementation of Apache Cassandra resulted in 10x performance gains, Redpanda offers 10x performance over Kafka, and 44x better p99 latencies. It uses a thread-per-core architecture, async scheduling, and Raft. Sound familiar?
• Ola’s Journey with ScyllaDB — The last session of the day was from Ola Cabs. Ola, India’s ride hailing giant, has been a customer of Scylla’s from the 1.x days. Anil Yadav took users on a tour of Ola’s use case and data pipeline, and explained the performance they’ve been able to achieve.

Thank You

Scylla Summit 2021 was created with the labor of dozens of speakers, and with the hard work of many behind the scenes staff to put on our first major live virtual event. Without them, the event would not even have been possible. We’d like to thank everyone who helped put on such a successful event.

Yet ultimately, it was the enthusiasm and interest of you — our loyal Scylla database users and customers, and the broader NoSQL community of interest — that made this event so much of a success. Thank you for attending, and giving us your frank and honest feedback, your questions — whether from the newest beginners to the zingers from the experts in the crowd — and, most of all in this busy day and age, your generous time and attention. We’re very happy you enjoyed the event, and grateful for your attendance.

Here’s to a great 2021!

And a reminder! Don’t miss today’s free training!

ATTEND OUR SCYLLA SUMMIT 2021 TRAINING DAY

The post Overheard at Scylla Summit 2021, Day Two appeared first on ScyllaDB.

Overheard at Scylla Summit 2021, Day One

13 January 2021, 3:00 pm by ScyllaDB

Scylla Summit 2021 was held online this year for the first time. Tuesday, January 12th was the first day of presentations, and included keynotes from ScyllaDB CEO and Co-Founder Dor Laor and Jepsen test author and computer safety researcher Kyle Kingsbury, and a range of talks from Scylla users and ScyllaDB engineers. Before we get further into the details, we want to give you this reminder:

Don’t Miss Day Two!

Day Two is happening today, January 13th 2021, from 8:30 – 11:30 AM PST. You can check out the full schedule here.

Scylla Summit 2021 is entirely online, free and open. You can register now to watch all the event proceedings live. We’ll have more sessions, including a keynote from ScyllaDB CTO and Co-Founder Avi Kivity, a presentation on the integration of Scylla’s Change Data Capture (CDC) feature with Apache Kafka by Confluent’s Tim Berglund, and many more sessions from Scylla users and ScyllaDB engineers. You won’t want to miss them!

REGISTER NOW FOR SCYLLA SUMMIT 2021

Prepare to Love Scylla Even More

ScyllaDB CEO and Co-Founder Dor Laor led off the summit yesterday with a keynote, Prepare to Love Scylla Even More. In that session, he outlined the achievements we had over 2020: our feature parity with and advancements beyond Apache Cassandra, adding support for our Project Alternator DynamoDB-compatible API, and our certification to run on AWS Outposts.

Scylla provides NoSQL with no limits. “We are unlocking limits all the time.” This spans the gamut from scalability up to multi-petabyte workload volumes, to unlimited deployments, whether multi-cloud, on-premises, Docker or Kubernetes.

Dor gave great credit to Kyle Kingsbury for working with ScyllaDB engineers conducting Jepsen testing against Scylla. “Now it’s safe to run LWT transactions with Scylla.”

Next, Dor announced the availability of the Scylla Kubernetes Operator 1.0, which has now reached General Availability (GA) status. With it, Dor said, “you can do a bunch of other things:  version upgrades, scaling out, scaling down. It provides a lot of value for those of you who manage your database yourself.”

Then Dor turned to deployability options, announcing that Scylla Cloud can now be deployed to Google Cloud Platform (GCP). This service is already in Beta, and GA will not be far off. You can now run DynamoDB-compatible workloads on GCP to support cross-cloud deployment. Dor also noted that Azure will also be added in due time.

Project Circe

Dor then revealed the roadmap for 2021. The first goals include moving towards better elasticity and maintainability. For instance, to be able to scale up or down by any number of nodes. And to make repairs a “fully solved problem.”

Dor believes Scylla needs to be better. “We want to make it as good as possible in terms of simplicity, elasticity, data consistency, and consistency of performance.”

“In order to do that, we’re launching Project Circe. Just this morning we published a special page on our website for this project.

Explaining the origins of the project’s name Dor noted, “If we go back to the Greek mythology, we’ll discover that Scylla wasn’t a monster to begin with. She was a beautiful nymph. She got cursed by Circe the witch and turned into a monster with twelve tentacles.”

Similar to that legend, Dor explained, “We’d like to transform Scylla from a database into a monstrous database — but in a positive way.”

Besides elasticity and maintainability, other features of Project Circe include the implementation of the Raft consensus protocol, improved performance, stability and deployability.

You can learn more about these initiatives and planned capabilities in our blog post, and by following our project throughout the year on the Project Circe page.

Kyle Kingsbury: Scylla + Jepsen

Aside from Dor’s keynote Kyle Kingsbury’s session on Jepsen testing was, for many, the highlight of the day. It prompted the following Twitter comment from one attendee, Steven Li:

“Super interesting to hear how @ScyllaDB benefited from Jepsen testing, resulting in improvements to areas like LWTs and membership changes. This was a much more technical keynote than I expected. Looking forward to more at #ScyllaSummit!”

Indeed, we learned a lot from Kyle’s testing. Enough to fill an analysis, a blog, and an upcoming webinar on January 28th, on top of his Scylla Summit talk. Make sure you register for it today:

TESTING SCYLLA WITH JEPSEN: RESERVE YOUR SEAT

A Full Agenda

With each other session being about 10 to 12 minutes long, the rest of the day was filled with the equivalent of “lightning talks” comprised of a mix of ScyllaDB engineers and users/customers.

• Raft in Scylla — Immediately following Kyle Kingsbury, ScyllaDB’s own Konstantin “Kostja” Osipov took users on a tour of how the implementation of the Raft protocol is going to expand the application of consensus and consistency far beyond Lightweight Transactions. While we’ll do a full blog about this in the future, for now, refer to our recent Project Circe post and web page.
• Zillow: Optimistic Concurrency Using Write-time Timestamps — Dan Podhola of Zillow discussed their unique method to combine data from multiple data sources and arrive at the latest view of data even without using Lightweight Transactions.
• Empowering the DynamoDB Application Developer with Alternator — ScyllaDB’s Nadav Har’El spoke about Scylla’s implementation of the DynamoDB compatible API in Scylla, with a focus on observability. By the way, did you know we now have a new experimental Alternator Streams feature in Scylla Open Source 4.3? You can read about it in the release notes.
• GE Healthcare: Enabling Precision Health with Edison AI — Sandeep Lakshmipathy took users through the GE Healthcare use case, which employs Scylla’s DynamoDB-compatible Alternator interface. Their hybrid deployment allows critical patient data to remain on-premises to comply with privacy requirements, rather than get stored in a public cloud.
• Scylla Operator 1.0 — Maciej Zimnoch set the Summit abuzz with the news Scylla now has a production-ready Kubernetes (K8s) operator. Want to know more? Read the documentation here and the release notes here. You can also download it now.
• Grab at Scale with Scylla — The team at Grab showed how the “superapp” of Southeast Asia continues to scale with Scylla, including how they conduct real-time aggregation with their counter service, plus a technical tour of their broad array of internal use cases and optimizations.
• Scylla’s Journey Towards Being an Elastic Cloud Native Database — After a brief intermission ScyllaDB’s VP of R&D Shlomi Livne described various requirements and considerations to make Scylla a more elastic distributed database including optimizing data transfers, handling different states as you scale up or down, and organizing data into tablets.
• Scylla @ Disney+ Hotstar — India is one of the fastest growing markets for Disney+, and their Hotstar subsidiary has delivered at scale. They detailed how their service allows users to pause a movie on their television and pick up exactly where they left off on their mobile phone. Such a “continue watching” feature seems pretty simple until you need to scale that to every show watched by over 300 million monthly users across all their devices.
• Scylla Cloud on Display — The Scylla Cloud team showed users the advantages of our managed Database as a Service, including Bring Your Own Account (BYOA), our new Google Cloud Platform beta support, VPC peering, the DynamoDB-compatible Alternator interface, and more. They also gave a demonstration of just how easy it is to provision a cluster under Scylla Cloud.
• Scylla @GumGum: Contextual Ads — Speaking of Scylla Cloud, GumGum showed the audience how they use our managed Database-as-a-Service to deploy and run their global adserver infrastructure.
• Expero/QOMPLX: Using Scylla with JanusGraph for Cyber Security — QOMPLX, and Expero took the audience on a tour of their graph database built with Scylla and JanusGraph to detect real-time problems in their government and customer networks.
• Ticketmaster Performance Test: Scylla, DataStax and Apache Cassandra — Ticketmaster’s Linda Xu finished the day by showing how, side-by-side, Scylla outperformed Apache Cassandra and DataStax Enterprise, especially when it came to handling their spiky immediate needs to meet high load when tickets go on sale.

In the Speaker’s Lounge

I had a chance to speak to Serge Leontiev, who hosted our Speaker’s Lounge. This was an ongoing combined Q&A session that followed right after each speaker’s presentation — a benefit to attending Scylla Summit live. Over time, more and more speakers joined, and they were able to talk shop with each other plus answer the questions of our Summit attendees.

Dor was the first speaker in the lounge after his Summit keynote. “He was very excited to share about Project Circe, and how it would take Scylla to the next level.”

Kostja and Kyle, who worked together closely on Jepsen testing Scylla with Paxos, answered questions about Raft and Scylla. Serge noted, “Kostja was on fire while trying to answer all the questions from attendees about the Raft protocol and how it will be implemented in Scylla.”

Maciej added additional details about Scylla Operator and went to great lengths answering questions about Scylla’s elasticity and performance.

“Shlomi and Tzach both fielded lots of questions about our product roadmap and future releases.” As you might imagine, we had a great deal of community interest around that. And later Noam from our Scylla Cloud team joined the lounge and answered questions about the great demo that was showcased during his session.

Angad from QOMPLX and Brian of Expero fielded questions about JanusGraph on Scylla. Their session on cybersecurity sparked a lot of interest. Angad noted they had gone with JanusGraph and Scylla — versus, say, Neo4J or TigerGraph — specifically because of their commitment to using a true open source solution.

Keith from GumGum shared his experience with Scylla Cloud, highlighting its value for a small DevOps team where ScyllaDB took all the responsibility of cluster management from their shoulders.

The cross-talk between the speakers was enthusiastic and engaging. As Serge told me, “There was a great conversation between Keith Sader at GumGum and Balakrishnan and Vamsi from Disney+ Hotstar about their use of Scylla Cloud. Arun Sandu from Grab, who runs Scylla Enterprise self-managed, was excited to learn about their experiences. That kind of good cross-conversation was unique to the live event.”

“Plus, the presenters from Disney+ Hotstar and Grab both work and live in the same region of India and use each other’s services. What united them was Scylla as the common database running underneath both.”

Meme Machine

During the event we had various contests and prizes. One of our favorites was a meme generator that used illustrations with everyone’s favorite Scylla monster. Users were encouraged to share their own captioned images out to Twitter or LinkedIn with the hashtag of #ScyllaSummit. Here are a few that tickled our fancy:

Stay Tuned For More

Dor noted that there will be much more to come, which would be discussed in other sessions and future updates. For example, he teased that there are already plans afoot to bring CQL into Grafana, the open source metrics and analytics platform used within Scylla Monitoring Stack. “You’ll be able to query virtual tables in it.”

For now we look forward to hosting you for Day Two of Scylla Summit 2021, and also look forward to seeing you back for our Training Day this Thursday, January 14th, with separate developer and administration tracks. If you didn’t register yet, don’t worry! As we said, the event is online and totally free. Sign up now!

REGISTER NOW FOR SCYLLA SUMMIT 2021

The post Overheard at Scylla Summit 2021, Day One appeared first on ScyllaDB.

Making Scylla a Monstrous Database: Introducing Project Circe

12 January 2021, 3:00 pm by ScyllaDB

With more than 30,000 commits in Scylla and Seastar, 78 and 113 committers, respectively, and customers running Scylla in the wild supporting hundreds of million of users, empowering many mission-critical services we all rely on, it’s fair to say that Scylla is now a mature database.

However, as engineers, as much as we’re proud of our results so far, we feel that there is still much more to be done in nearly every dimension of Scylla.

“Cloud native” and “elastic” as concepts are attractive terms, and many bandy them around freely. “How fast can your cloud-native solution scale?” “What’s the cost of your cloud-native solutions?” “How easy is it to maintain such an elastic deployment without committing to a certain vendor’s cloud solution?” But when you get to implementation, you have to make architecture and code to deliver on those buzzwords. How do you restore a database backup to a different number of nodes? Can you run DDL while scaling your cluster? Can you add 1M ops to your cluster in a breeze? Currently Cassandra, DynamoDB and even Scylla do not have perfect answers to all these questions.

What is Project Circe?

In Greek mythology, Scylla was originally a beautiful nymph who was turned into a monster by a witch named Circe. Circe’s enchantment transformed Scylla into a monster. She grew 12 tentacles (as well as a cat’s tail and six dog’s heads around her waist). We now wish to similarly cast our spell on the modern Scylla NoSQL database, and make it into an even bigger and better monster.

Project Circe is a one year effort to improve many aspects of Scylla to bring stronger consistency, elasticity and ease of use to our community. So each month in 2021, we’ll reveal at least one “tentacle,” each representing a significant feature. Some of the new concepts are quite transformative, thus we will provide any available details ahead of time.

The Metamorphosis of Scylla

Scylla began by following the eventual consistency model of Cassandra and DynamoDB. Eventual consistency allows the user to prefer availability and partition tolerance over consistency, and also to tune the consistency for performance; and it is a perfectly valid model for many real-world use cases.

However, lack of consistency comes at a price of complexity. Many times it bothered us as database developers that our lives became more complicated with advanced features such as materialized views, indexes, change data capture and more. For example, materialized views receive asynchronous updates from the base table, in case of some server failures, the view table can go permanently out of sync and requires a full rebuild.

Other issues were observed during our recent Jepsen test of Lightweight Transactions (LWT) – traditionally schema and topology changes weren’t transactional, forcing users to carefully serialize them, making maintenance complicated. It is time for a change!

The Raft Consensus Protocol

Agreement between nodes, or consensus, in a distributed system is complicated but desirable. Scylla gained lightweight transactions (LWT) through Paxos but this protocol has a cost of 3X round trips. Raft allows us to execute consistent transactions without a performance penalty. Unlike LWT, we plan to integrate Raft with most aspects of Scylla, making a leap forward in manageability and consistency.

Raft will become our default option for data manipulation. We will preserve regular operations as well as the Paxos protocol. As of December 2020, the core Raft protocol is implemented in Scylla. The first user-visible value will be in the form of transactional schema changes. Until today Scylla tracked its schema changes using gossip and automatically consolidated schema differences. However, there was no way to heal a conflicting Data Definition Language (DDL) change. Transaction schema change will eliminate schema conflicts and allow full automation of DDL changes under any condition.

The next user-visible improvement is to make topology changes transactional using Raft. Currently Scylla and Cassandra can scale only one node at a time. Scylla can utilize all of the available machine resources and stream data at 10GB/s, thus new nodes can be added quite quickly, however, it can take a long time to double or triple the whole cluster capacity. That’s obviously not the elasticity you’d expect.

Once node range ownership becomes transactional, it will allow many levels of freedom and we plan on improving more aspects of range movements towards tablets and dynamic range splitting for load balancing.

Beyond crucial operational advantages, end users will be able to use the new Raft protocol, gaining strong transaction consistency at the price of a regular operation!

We expect to improve many more aspects of database maintenance. For example, materialized views can be made consistent with their base tables under all scenarios, repairs can be performed faster and cheaper and so forth. We welcome you to follow our progress with Raft on the ScyllaDB users mailing list and to participate in the discussions as well.

Maintainability

Many Scylla maintenance operations require significant data movement between the database nodes in a cluster. It is not an easy task to make the management operations efficient while minimizing impact on the workload all the time. Moreover, the longer the maintenance window is open while moving data between nodes, the less resilient the database might be during those operations.

With its existing IO and CPU scheduler and automatic tuning, Scylla was designed to support smooth maintenance operations. This year we would like to simplify your deployments by removing concepts like seed nodes, which break the homogenous node concept, allowing you to restart nodes or change the cluster size without risk or complexity.

Compaction is a solved problem at Scylla. The compaction controller automatically sets the dynamic priority for the compaction queue and the user doesn’t need to do anything but choose the compaction algorithm. However, we identified cases where this promise isn’t met, especially around repair, node changes and migration. A new internal state called off-strategy compaction will automatically reshape your data to comply with the existing compaction strategy. Such a reshape will also allow you to load SSTables on any node, regardless of the key range ownership; the node will chauffeur the data to the proper replica set. These and many more changes will allow super smooth administration without any glass jaws.

Oh, last but not least, the repair process finally becomes a solved problem for Scylla and Scylla Manager. More changes are coming to repair, starting from an IO bandwidth limiter for poor IO environments, to a new IO scheduler and maintenance window. A key improvement of maintainability is achieved with Repair Based Node Operations (RBNO), which is disabled by default at the moment. Previously, a node operation like adding a new node or a decommission took up to several hours, so if there was a failure, the whole process needed to be restarted from scratch. With RBNO the restart is immediate and the operation will start from the last position — a big improvement. In addition, since the data is streamed by the repair protocol, the stream is consistent with all of the replicas, unlike before where a repair was required at the end of the streaming and there was a window of time with reduced consistency. Since we already use repair for streaming, the process is simpler, too, and saves the additional repair execution time.

See the Performance section (below) as well for how this will improve latency under maintenance operations as well.

Elasticity

Scylla was modeled after Cassandra, which was created in a pre-cloud-native environment. With more workloads running as-a-service, spiky workloads can flip overnight from low usage to peak and back again in moments, requiring a modern distributed database to be as elastic as possible.

One of the basic elasticity improvements we’re now finalizing is the ability to add, remove and replace multiple nodes at a time with full consistency and greater simplicity. This is important not only for database-as-a-service deployments as Scylla Cloud but for all deployments. When cluster manipulation is complex, database admins will refrain from changing the cluster size often. Soon this will no longer be the case. As an example, the remove node operation was hard and complex to execute in a safe way. Beginning with release 4.1 it was safe by default. We automatically hold off on compaction during streaming and wait for the conclusion of the streaming operation, then compact the new data just once at the end.

Related to performance, we’re removing many small, tiny stalls from cluster node operations in order to be able to constantly change the cluster size with no effect on latency. Clusters with 25% of idle time should suffer no P99 latency increase during these operations.

Lastly, by adopting Raft we’re making key range changes transactional. This major shift will allow us to detach the range movement from the actual data movement, which will be asynchronous. In this way, we will be able to add or remove multiple nodes at once. Furthermore, key ranges will be represented as tablets, which will automatically get split or united as a function of the load of the shard.

Performance

Performance and price/performance are fundamental motivations in developing Scylla. Our emphasis on performance this year is around several key improvements:

Uninterrupted performance

Using our stall detector, our scheduler can locate stalls as small as 500usec. We’re eliminating them to ensure a smooth experience under any administration operation, from elasticity to availability.

Large partition performance improvements

Our goal is to make large partitions perform close to small partitions, with almost no overhead for the clustering key scanning. Better performance frees developers from working around their data model and provides consistent performance without any hiccups. We recently implemented a binary search through the SSTable index file, which boosts the speed of reading rows within a large partition. The final step is to cache the index file itself, so we wouldn’t go to read it from the disk.

New IO scheduler

Controlled IO is one of the key principles of Scylla’s Seastar engine. Until now, all of the shards in the system would divide the global capacity of the disks and schedule shard IO within their budget. This design worked well for very fast disks, but slower disks or workloads with extreme latency requirements mixed with large blobs of IO suffered since the shard IO was maxed out. The new scheduler, which recently landed in the Scylla master branch, has a new algorithm with a global view of the system. Shards now receive a budget and keep asking for more once it is close to being drained.

We expect the new IO scheduler to improve latency across the board, improve LWT operations and improve the environment with slow disks such as cloud network drives.

Coroutines

Coroutines are a C++ programming abstraction, recently introduced to Scylla with C++20 and the CLANG compiler. Without diving deeply, coroutines allow Scylla to replace some of the future-promise asynchronous code that extensively allocates memory with larger bulks when it’s appropriate. We expect to broaden the use of coroutines in 2021.

Stability

Many of the above improvements can also be considered stability improvements. For example, making schema changes transactional eliminates a whole family of schema disagreement issues. Same for the elimination of seed nodes — many Scylla users are not fans of read-the-doc and there are times where seeds aren’t configured correctly. So when the time comes, not having enough seeds causes availability issues.

There are however, many improvements directly related to stability. We’re improving the tracking of memory used by queries in order to block/slow-down new queries when memory is limited. Drafts for cancellable IO were published, thus when a node becomes suddenly overloaded, we can cancel pending IO and provide immediate cure. There is ongoing work around tombstones and range tombstones. Scylla will gain a safe mode — a new mode of operation that will reinforce better defaults and reduce the chances for errors. Workload shedding is a family of algorithms that we’re developing in order to prevent overloaded situations, especially when parallelism is unbounded, thus existing mechanisms to slow down clients via backpressure are not effective.

Deployment options

We’re finally ready to move forward and announce our Kubernetes Operator as a fully supported environment. Expect it to work well on all K8s environments, starting with GKE.

Scylla Cloud itself is expanding from AWS-only to GCP and from there to Azure. We will deploy to more instances and develop many more features. We will also add standard machine images for GCP and Azure, and add marketplace options for DBaaS, Enterprise and OSS deployment options. An open source Ansible project exists and we’re adding provisioning options to it.

Even more

There are many more features (such as CDC going GA together with supportive libraries) and improvements we’re excited about. We will boost observability, implement more protocols, improve traceability and much more. check out the talks at our Scylla Summit and the Project Circe page.

The post Making Scylla a Monstrous Database: Introducing Project Circe appeared first on ScyllaDB.

Scylla Summit 2021 Schedule Now Online

5 January 2021, 5:33 pm by ScyllaDB

Scylla Summit is just around the corner! Next week (January 12th – 14th), we’ll hold a two-day virtual conference followed by a day of instructor-led training. All online. All free.

We’ve published the complete schedule so that you can add reminders to your calendar for the sessions you’re most interested in. You can also share sessions via social media or mail them to your colleagues.

Tuesday, January 12

• Prepare to Love Scylla Even More — ScyllaDB CEO Dor Laor will look back on everything that we’ve accomplished since last Scylla Summit, and look ahead to what’s on the horizon for 2021.
• Testing ScyllaDB with Jepsen — Kyle Kingsbury, also known as Aphyr, took Scylla through his in-depth Jepsen testing process. Discover what he found and what we fixed!
• Raft in Scylla — Konstantin “Kostja” Osipov will provide an overview of how Scylla is moving from Paxos to Raft, and what that means for database consistency beyond Lightweight Transactions.
• Zillow: Optimistic Locking Using Write-time Timestamps — Dan Podhola of Zillow will reveal the tricks they used to guarantee in-order delivery of messages for correct and consistent data.
• Empowering DynamoDB Application Developers — ScyllaDB Distinguished Engineer Nadav Har’El will share you can use the advantages of Scylla’s DynamoDB API for greater observability, improved performance and to run your applications at lower cost.
• GE Healthcare: Enabling Precision Health in Edison AI — Learn how GE Healthcare is deploying DynamoDB-compatible instances of Scylla on premises to improve precision healthcare with better patient data confidentiality.
• Scylla Operator Overview — Kubernetes (K8s) has taken cloud native computing by storm. Scylla’s Kubernetes Operator is now generally available. Discover its features and explore its roadmap with ScyllaDB engineer Maciej Zimnoch.
• Grab at Scale with Scylla — Grab is southeast Asia’s favorite “superapp,” providing shopping, ride sharing, payments, and food delivery. Discover all the ways they use Scylla to deliver on their services.
• Becoming an Elastic Cloud Native Database — ScyllaDB VP of R&D Shlomi Livne will share what steps ScyllaDB has already taken to provide elasticity, the capability to scale up or down with minimal disruption, and our plans to deliver further on that vision.
• Scylla Cloud on Display — Find out how easy it is to integrate and connect Scylla Cloud with your applications, and turn over all aspects of management and maintenance to our team of cloud database experts.
• Scylla @ GumGum: Contextual Ads — GumGum’s Keith Sader will describe how and why they moved to Scylla Cloud to meet their growing customer demands with a future proofed solution.
• Expero/QOMPLX: Using Scylla with JanusGraph for Cybersecurity — Recent news shows the importance of managing the cybersecurity of enterprise and government client systems. Learn how QOMPLX, with the help of Expero’s team, designed their intrusion detection system around Scylla and JanusGraph.

Wednesday, January 13

• Meeting the Challenge of OLTP Big Data with Scylla — ScyllaDB CTO Avi Kivity will highlight the advances in Scylla that make it such a great match for big data applications, plus plans for performance, reliability, manageability and ease of use in a modern cloud environment.
• Best Practices for Change Data Capture — Change Data Capture (CDC) allows users to track changes made to a record in Scylla, and even store the pre- and post-image of the data. Learn how to get the most out of CDC with ScyllaDB’s Calle Wilund.
• Confluent: Streaming Data from Scylla to Kafka — Learn how to use the Kafka Scylla Connector to share CDC table data to Kafka topics in this session hosted by Confluent’s Tim Berglund.
• Numberly: Getting the Scylla Shard-Aware Drivers Faster — Find out how you can take best advantage of Scylla’s shard-per-core server architecture with client-side shard-aware drivers. Numberly CTO Alexys Jacob will describe the efforts to make the Scylla Python driver even faster.
• Latest Performance Changes by Scylla — ScyllaDB Distinguished Engineer Tomasz Grabiec will lead a tour of the performance improvements accomplished over the past year in Scylla, including work on the IO Scheduler, and on streaming and repair.
• Mail.Ru: High-Load Storage of User Actions with Scylla and HDDs — Scylla is renowned for its performance on fast NVMe SSDs, but did you know you can also achieve relatively low latencies on low-cost Hard Disk Drives (HDDs)? Find out how Mail.Ru is using this as a low-cost strategy for petabytes of stored activity data.
• The All-New Scylla Monitoring Stack — Scylla Monitoring Stack was designed to be a one-stop-shop for monitoring your cluster. Learn how we’ve extended it by building a CQL interface for monitoring, plus new log collection analysis capabilities.
• Expedia: Our Migration from Cassandra to Scylla — Migrating from one database to another can be quite a journey, even for a drop-in Cassandra replacement like Scylla. But you can rely on the travel experts at Expedia to set forward an itinerary that’s easy and straightforward to follow.
• How We Made Scylla Maintenance Easier, Safer and Faster — Learn how we’ve made Scylla even better, by implementing seedless clusters, repair-based node operations, smarter off-strategy compaction, bandwidth limits for repairs and compactions, and parallel repairs.
• IOTA: Implementing NoSQL in a Persistent Distributed Ledger — IOTA uses a feeless, minerless distributed ledger technology known as “The Tangle.” Discover how and why they implemented Scylla as the persistence layer for the Tangle.
• Developing Scylla Applications: Practical Tips — Even with a high performance scalable system like Scylla there are techniques you can employ to prevent overloading and provide better availability. Learn these pro tips from ScyllaDB engineer Kamil Braun.
• Broadpeak: Building a Video Streaming Testing Framework — Discover how Seastar, the application framework at the heart of Scylla, was employed by Broadpeak to create a benchmarking tool to emulate thousands of streaming video client sessions.
• Vectorized: Building a Distributed Data Streaming Architecture — Vectorized CEO Alex Gallego will introduce the audience to Redpanda, a streaming data engine that utilizes Seastar to provide a Kafka-compatible API with 10x performance improvements.
• Ola Cabs: Real-Time Intelligent Decision Making Using Scylla — India’s leading on-demand ride hailing company has used Scylla since our very beginning. Learn how they scaled their business to support millions of rides every day with Scylla.

Thursday, January 14 — Training Day

This year our training sessions are absolutely free, so make sure you get your coworkers and industry peers to sign up today. We will have full side-by-side tracks for developers and administrators:

Register for Scylla Summit Today!

It’s free. It’s online. It’s all the content you want to take your organization’s NoSQL game to the next level. Database monsters of the world connect!

REGISTER FOR SCYLLA SUMMIT

The post Scylla Summit 2021 Schedule Now Online appeared first on ScyllaDB.

ScyllaDB Developer Hackathon: Managing Scylla Using CQL

29 December 2020, 7:52 pm by ScyllaDB

We had a lot of fun and learned a lot at Scylla’s internal developer conference and hackathon this year. For our hackathon project, we decided to create an API to manage a Scylla cluster using native Cassandra Query Language (CQL) commands.

Background and Motivation

Scylla (and Cassandra for that matter) uses nodetool as its prime command line management tool. Nodetool is a Java-based application that connects over Java’s JMX API and, in Scylla’s case, using a JMX proxy and a RESTful API.

What we were after was to have similar capabilities using Scylla’s native CQL interface. This approach has its advantages:

• CQL is a native API in Scylla, that by definition is already supported by Scylla users
• Makes the management API a first-class citizen
• CQL is easily understood by humans and computers alike
• CQL can be secured, subject to access control
• You can use extra facilities in CQL, such as filtering with SELECT
• Does not require external tools and drops the last Scylla dependency on JVM

The Team

Our team combined Scylla server-side development expertise with the operations expertise of our management and monitoring leaders.

The CQL Management API team, clockwise from upper left included  Kostja Osipov (best known for his work on LWT),  Amnon Heiman from the Scylla Monitoring Stack team, Michal Matczuk from the Scylla Manager team, and Tomasz Grabiec, one of the few black belts of Scylla core.

CQL Syntax

We started with looking at common nodetool commands to see how they can be mapped to CQL. What we found out is that it depends on the command, some commands like nodetool status are tabular in nature and would be best served as selecting from a Virtual table.

Other commands like nodetool repair can be best described as a SQL procedure call and there are some commands like nodetool version that return a single value and would be best implemented as functions.

So we did it all.

Virtual Tables

We use virtual tables for nodetool commands that are tabular in nature and that can benefit from filtering capabilities.

A virtual table is a facade that acts like a regular CQL table that you can select from but the data behind is generated programmatically. For example, consider getting your system status as a CQL table:

As you can see, it acts just like a regular table, cool, right?

Other supported virtual tables are:

• system.describe_ring;
• system.listsnapshots;

Both act like the equivalent nodetool commands (compare here and here).

CALL Syntax

Now to management commands, like repair, take snapshot, etc’.

We used a SQL-like CALL syntax with parameters:

cqlsh> CALL system.take_snapshot(ks=>'keyspace1', tag=>'mysnp');

Procedure can return a result set if needed.

Function Syntax

Finally, there are nodetool commands that return a single value; nodetool version is a good example of this.

We implement those as functions and to make it  easy to use, added the DUAL table that acts as a dummy table that you can query anything from.

SELECT scylla_version() FROM dual;
 scylla_version
---------------------------------------
 666.development-0.20201026.e3d1d458c2

What We Achieved

We set out to create a basic working nodetool-like functionality that is based on CQL. We were able to accomplish this in the course of the Hackathon. However, integrating our work into the Scylla code base, optimizing it, and fleshing it out for production-readiness will take more time and testing. Look forward to this being introduced in Scylla Open Source in the coming future.

Also don’t worry! We are committed to supporting standard nodetool because of its ubiquity and familiarity to users. However, for those who want to do in-band management through CQL, in the days ahead you will have a unique option with Scylla.

Open Issues and Future Steps

During the Hackathon we accomplished a great deal, but we have more ideas for how to make this capability work even better and smoother. Here’s a list of our thoughts on how to improve it more in the days to follow:

• Compile-time schema definition:
  • reduce mutation model boilerplate
  • make the code compile-time safe
• Make “distinct” CQL keyword work (queue_reader::next_partition())
• Proper memory accounting for reader admission control
• Use temporary tables to avoid OOM
• Allow running from a single shard (fix sharders)
• Unit tests
• Ordinal routine parameters
• Parameter markers, UDTs, compounds
• SYSTEM.FUNCTIONS should be SYSTEM.ROUTINES
• ANSI User-Defined Procedures
• ANSI INFORMATION_SCHEMA (must-have for JDBC)
• Oracle-style PERFORMANCE_SCHEMA

See You at Scylla Summit!

Scylla Summit is right around the corner, January 12-14, 2021. You’ll learn more about our future roadmap, and hear how your industry peers are managing Scylla in their production networks. We also have a day of online classes for both application developers and administrators. Best of all, it’s free, virtual and online this year. Sign up today!

REGISTER FOR SCYLLA SUMMIT 2021

The post ScyllaDB Developer Hackathon: Managing Scylla Using CQL appeared first on ScyllaDB.

Jepsen and Scylla: Putting Consistency to the Test

23 December 2020, 7:30 pm by ScyllaDB

At ScyllaDB, we take data consistency seriously. To ensure the correctness of our lightweight transaction implementation we developed a series of tests using Scylla’s existing and new tools, such as Scylla Cluster Test, Gemini and lightest. These tools evaluate data correctness in the face of acute failure scenarios: I/O errors, running out of memory, node crashes and network partitions. We published some of this work in our blog (which has a great explanation of the difference between our normal eventually consistent operations and lightweight transactions, if you need a refresher).

We have also run Jepsen tests in the past. We focused on regular operations since LWT wasn’t supported at the time. Jepsen is an open source project and a service run by Kyle “Aphyr” Kingsbury. Established in 2013, it has by now become the industry’s standard for distributed systems testing, with many SQL and NoSQL vendors using it as a checkmark that their consistency claims are production grade.

Having run our own tests, we had confidence we were going to pass the Jepsen test with flying colors. The result turned out to be surprising.

Finding consistency violations in databases is an NP-complete problem – the cost grows exponentially with the number of transactions. Sometimes it is outright impossible to do for an external observer. Multiple violations can interact and produce the right results, while masking the underlying problems. ScyllaDB tests create scenarios with data constraints and check these constraints after running a workload with fault injections. Jepsen takes a more holistic approach. It observes not only whether or not there was a consistency fault, but also, with a high degree of certainty, the steps that brought it about. One tool that proved invaluable in particular was Elle, which finds anomalies in linear time.

But Jepsen testing, as a process, goes beyond just its powerful toolset — Kyle’s personal contribution into the testing is invaluable. Where else can you find such an opinionated, independent expert and also the author of one of the most cited Cassandra evaluations? With Kyle, we had an objective set of eyes.

Kyle’s testing was so thorough it ended up going beyond testing lightweight transactions. Scylla documentation for the eventual consistency model had grey areas in semantics of INSERTs (#7082), conflict resolution in case of duplicate timestamp (#7170), safe practices for topology changes, and the correct use of nodetool. All of these topics now have better documentation.

The vast majority of Scylla (and Cassandra) use cases are absolutely satisfied with eventual consistency. Regular operations enjoy better availability and better performance, thus conflict-free replicated data types, last-write-wins and even lack of row isolation are the right compromises for these workloads. QUORUM writes ensure durability, while QUORUM reads guarantee the application never reads dirty data. Background repair processes are responsible for eventually weeding out conflicts. Lightweight transactions are still invaluable when it’s necessary to not just guarantee durability, but linearizability — a specific (strict) order of changes, conditioning a change on the state of the database.

Some nasty LWT consistency bugs were found, too. In one instance (#7116), a typo in the data checksum loop introduced in Scylla 2.2 led to slight differences in data staying invisible to repair. LWT, as well as standard repair and streaming, depend greatly on the correctness of this code. Changing even one letter in the column name would make the bug disappear, meaning Kyle’s special talent for breaking systems had to play a role. Indeed, our own tests used null values in just the same way, but did not observe any errors just because the column holding nulls had a different name! This issue is fixed in Scylla 4.2 and the fix is now backported to all Scylla Enterprise versions.

Another ancient bug (#7611) was revealed in list append and prepend code, which historically generated its own timestamps for append/prepend values, ignoring Paxos timestamps. Using the Paxos timestamps was essential to preserve linearizability of list operations. And while the code was and still is working this way in Cassandra, that’s not good enough for us. The Scylla fix will appear in an upcoming release.

Truthful to the eventual consistency philosophy, Scylla does not run a strong coordination during topology changes. If you know what you’re doing, you can repair a cluster split across multiple availability zones, adding and removing nodes at will. This also allows you to shoot oneself in the foot.

The problem that manifested itself in the test was that Jepsen could start a new topology change after the previous one had failed – and by failure I also mean an operation timeout. Network partitioning and/or node failures, accompanied by a still ongoing node removal and a fresh node addition confused Paxos’s idea of what replicas constitute a quorum.

Despite our objections that this is a wrong way to operate Scylla, Kyle was nonplussed. There are circumstances in which even a human operator could not know if the previous operation has ended. Besides, the rules were never explicit in the docs.

Scylla guarantees that LWT operations will be strictly serializable during a single membership change. We refreshed the standard operational procedures with a new documentation page. If multiple changes are concurrent, the cluster is vulnerable to operator errors – a problem that is, fortunately, less important if one uses Scylla Cloud.

Jepsen testing spurred us to redouble our efforts to switch topology changes from eventual to strong consistency (#1247, #1207).  With topology changes based on Raft, Scylla will be able to add and remove multiple nodes at a time, as well as detect and reject unsafe topology transitions requested by a DBA. I’ll talk more about it at our upcoming Scylla Summit.

Our confidence in LWT quality had some basis – the testing found no issues in the trickiest domain of lightweight transactions implementation, the Paxos algorithm. With Jepsen we learned yet again that database quality has to be all-round, and maturity is achieved through testing across many product features and their interactions, rather than focusing on a single piece.

To conclude, let us restate ScyllaDB’s commitment to ultimate product quality – specifically, to fixing all of the consistency issues found by Jepsen testing, and further improving the product and documentation to avoid usability traps. As of today, all the found issues have been resolved, and Scylla documentation clarified. We run Jepsen on a regular basis to ensure no regressions in upcoming releases, and plan to revisit and extend the coverage once strongly consistent topology changes are in.

Thanks Kyle!

READ THE TEST RESULTS AT JEPSEN.IO

Learn More at Scylla Summit

Both Kyle Kingsbury and Konstantin Osipov will be presenting at Scylla Summit 2021, coming this January 12th – 14th, 2021. The event is free and online. Kyle will present his Jepsen test findings in depth, and Konstantin will explain our future plans for Raft. These are just two of many great sessions we have planned at our Summit. Make sure you register today!

REGISTER FOR SCYLLA SUMMIT 2021!

The post Jepsen and Scylla: Putting Consistency to the Test appeared first on ScyllaDB.

ScyllaDB Developer Hackathon: New IMR infrastructure

22 December 2020, 7:38 pm by ScyllaDB

IMR stands for In Memory Representation and in the scope of this blog post we mean the in-memory representation of cells. You might be asking, why is this even something worth talking about? Why not just use the native data model provided by the programming language in use? The answer lies in the programming language in use: C++ is a strongly typed compiled language and as such it needs to know all properties of a type at compile time. Cells on the other hand have various fields depending on the state they are in and even depending on the schema — the CQL type of the value they store. To give the simplest example, while a live cell has a timestamp and a value, a dead cell only has a timestamp and a deletion time. To represent this in C++, one would need something like this:

struct simple_cell {
     timestamp_type ts;
     std::optional<deletion_time_type> deletion_time;
     std::optional<value_type> value;
};

This representation is wasteful, as some of the fields are only used in certain fields. While the amount of waste might be small, there might be millions of cells in memory and this adds up. Using a union or std::variant also doesn’t help, as those allocate space large enough for the largest of their member types.

Now that we established why an IMR is needed, let’s look at why we needed a new IMR infrastructure. Back in the times of old, Scylla had a quite simple IMR format. A cell was a buffer. In the start of this buffer was a bitset, with some flags, from which the state of the cell could be determined. Each state had a specific field layout, storing only fields it needed. Once the state was known, fields could be accessed simply by adding up offsets. This was simple and it worked, but it had one major weakness: it required the buffer to be linearized. (You can read more about key linearization here, and the kinds of behavioral issues it can cause here.)

In Scylla we break up large allocations into fragments of uniform size. This is mainly to reduce memory fragmentation and make the database more resilient. To solve this it would have been enough to just adapt this simple code to work with fragmented buffers. But we had more ambitious plans. If we were going to touch this, we wanted to change it into something that aligns better into our long-term goals: more widespread-usage (migrate more components like collections to also have an IMR format) and JIT compiling.

We wanted something that is not open-coded but generated by the compiler as much as possible, so we ended up with a template-metaprogramming heavy solution. This had the advantage of eliminating much of the boiler-plate of the open-coded solution, as well as allowing nice declarative code on the interface level. But, as it turned out in time, it had some significant disadvantages too: beyond the nice interface was code that proved to be so complicated that very few people could understand it and people started to avoid touching it. Thus it completely backfired in the more widespread adoption regard. Also, it had a horrible debugging experience.

Figure 1: the name of a single function in IMR doesn’t fit into a single screen

For these reasons we decided that we want to go back to the old simpler days of an open-coded IMR format and the Hackathon seemed like the perfect opportunity for this experiment.

Implementation

First we tried refactoring the current IMR format to make it less template-heavy and hence easier to work with, but this proved futile. In the end we ended up effectively dropping the current format and reverting to the old open-coded one. This was not an easy task, as in the years since it was introduced, many improvements have accumulated on top of it. It kinda felt like trying to yank an entire level out of a Jenga tower without it crumbling entirely.

Figure 2: reverting a change from years ago

We also didn’t want to regress in regard to buffer fragmentation, so we had come up with a way to make this work with fragmented buffers, without losing the simplicity of it.

We ended up using a widely used abstraction in C++: iterators. We wrote an iterator for our fragmented buffer type that hides the fragmentation and creates the illusion of contiguity (without promising it). This iterator works well with C++ standard functions like std::copy_n() that we use heavily to extract values from buffers. The drawback is that this iterator is a fat iterator. As the compiler cannot prove that the underlying storage is contiguous it can’t optimize such copies away into simple memcpy() equivalents (or, when the size is known in advance, into single mov instructions). To solve this problem we came up with a function which would double compile our iterator using code:

auto with_iterator_pair(auto&& func) {
     if (is_fragmented()) {
          return func(slow_begin(), slow_end());
          }
     else {
          return func(single_fragment_begin(), single_fragment_end());
          }
     }

This allows those buffers that are small enough to fit into a single fragment (most cells are small enough to fit) to use good old pointers as iterators that are lightning fast, and only fall back to the slower fat iterator for the larger buffers (that are rare).

Next Steps

Obviously a three day hackathon was not enough to get this into a mergeable state. But it was enough for a POC that proves that it can be done. Now only the hardest part remains: polishing and testing. Once done this opens the door for migrating more components to this IMR format. One example is collections. These currently use a serialization format that requires full deserialization for each access and a further serialization pass for each modification. Another example is counters, which also has its own serialization format and requires deserialization for reads.

If you want to learn more about what is going on behind the scenes at ScyllaDB, you’re invited to attend Scylla Summit 2021, this coming January 12th – 14th. It will have two days of technical talks. You can hear directly from many of our engineers, who will present what they’ve been working on this past year, as well as sessions from your industry peers. Plus there’s a free day of training for both app developers as well as DBAs/DevOps.

SIGN UP FOR SCYLLA SUMMIT 2021!

The post ScyllaDB Developer Hackathon: New IMR infrastructure appeared first on ScyllaDB.

Scylla Summit Training Day 2021

17 December 2020, 6:02 pm by ScyllaDB

As successful as our training day was at last year’s user conference, we’re even more excited for our next one. With the virtual format for our upcoming Scylla Summit (January 12th to 14th), our training day will accommodate even more people from around the globe. We’ve got lots of great new courses. What’s more, it’s FREE for all Scylla Summit registrants! Training will take place on Thursday, January 14th 2021, 8:30 AM – 11:30 AM, Pacific Standard Time (PST). Register now and mark your calendars!

Our speakers will cover familiar topics as well as updates and changes to the product. You can choose to follow one of two parallel tracks: The first for application developers and the second for database administrators. You are free to bounce back and forth between tracks and sessions as you see fit. The sessions will be held to the same schedule, so you won’t miss a minute of our great content.

Be sure to bring any questions you have to our live training, as we hope to make the sessions interactive and lively.

After each lesson there will be a quick quiz. Participants who complete all the quizzes will get a certificate, receive Scylla swag and, most important of all, will be conferred special bragging rights.

Agenda and Schedule

Class Details

• Getting Started with Scylla (Basic Concepts, Basic Data Modeling): This session will cover an introduction to Scylla Architecture, basic concepts, and basic data modeling. Some of the topics covered include Scylla Terminology, Scylla Components, Data Replication, Consistency Level, Scylla Write, Read paths + sstable, memtable, Basic Data modeling. Also I’ll show you a quick demo of how easy it is to spin up a cluster with docker.
• Kubernetes Operator: This session will cover our new and exciting K8 Operator for Scylla. We’ll discuss why to use K8s and all its aspects such as pods and statefulsets, Plus learn about Scylla Operator’s features and roadmap. Plus a quick example of how to use the Operator and discuss some best practices.
• How to Write Better Apps: In this class we will touch on Advanced Data Modeling and Data Types, Scylla shard-aware drivers, Materialized Views, Global and Local Secondary Indexes and Compaction Strategies. We will also talk about tips and best practices for writing applications for Scylla.
• Advanced Monitoring and Best Practices: This session will start with an overview of our Scylla Monitoring Stack, which is your first resource for understanding what’s happening in your cluster and your application, and for getting alerts when something goes wrong. Scylla exports thousands of different metrics. We will show how to use Scylla Monitoring to make sense of all that data and provide a full picture of how well your hardware is set up and used, what is the current / historic state of the cluster and how well your app is written.
• Scylla Alternator: Migrations without Compromise: Here we will share details of our Project Alternator DynamoDB-compatible API and how to migrate from Amazon DynamoDB to Scylla, online and without any downtime. We will cover the considerations and strategies of making data migrations at scale.
• Scylla Manager and Cluster Administration: This session will cover how to administer a Scylla Cluster. Topics include: Adding/removing a node, adding/removing data centers, what are repairs and why are they needed, cluster backup (through Scylla Manager), cluster restore options, Scylla security settings, Scylla Manager and how the new Scylla Agent works.

Get Started in Scylla University

It’s been an active year at Scylla University. We’ve added lots of content, including new lessons on Lightweight Transactions, Monitoring, and Using Scylla Drivers… to name just a few.

If you haven’t done so yet, we encourage you to sign up for our free courses at Scylla University and get your bearings on Scylla’s technology and capabilities. A good starting point would be the Scylla Essentials and Mutant Monitoring courses.

Register for Scylla Summit

If you love the training agenda, time to register for our conference! It’s online, it’s free, and it may just change your career.

Register for Scylla Summit 2021 Now!

The post Scylla Summit Training Day 2021 appeared first on ScyllaDB.