Spicing up the Authorization Layer at Quizizz

Introduction

Here at Quizizz, we’ve been continuously improving our product and scaling our operations, serving an ever-growing number of teachers and schools globally. As our platform expanded, so did the complexity of our authorization needs, especially with the addition of more collaborative features and the management of diverse user roles and permissions.

This led us to SpiceDB, an authorization solution inspired by Google’s Zanzibar. SpiceDB provides a robust, scalable, and flexible framework for fine-grained, relationship-based access control, which perfectly suited our requirements.

In this blog post, we’re going to discuss our adoption of SpiceDB, including our reasons for choosing it, how we went about the integration, and the improvements we’ve observed since.

Authentication vs Authorization

Authentication and authorization are layers of an application that serve as barriers of defense to stop actors from performing actions that they shouldn’t be able to.

Authentication is the process of verifying who a user is and establishing their identity. This usually plays out in a “login” experience where the user enters their username/password or uses an SSO (single sign-on) provider in order to prove their identity to you.

On the other hand, authorization kicks in after a user is authenticated. It’s about permissions and determining what an authenticated user is allowed to do. It answers the question “Can this user perform a particular action (like edit or delete) on this resource?”.

In a nutshell, authentication is about confirming identity, while authorization is about granting access based on that identity.

Authorization at Quizizz

Authorization at Quizizz is critical to our collaborative features. Let’s look at how it’s utilized in different scenarios:

• Sharing Quizzes: Teachers often need to share quizzes with colleagues. Our authorization system manages who can access these quizzes and what they can do with them – view, edit, or share.

• Co-Teaching: Classrooms often have more than one teacher instructing students. We want to facilitate collaboration among these teachers, and allow them to work together on creating assignments, assessing their students and making data informed adjustments to their lesson plans. 

• Sharing Reports: Student reports are valuable for collaboration between teachers. Teachers can make data-driven decisions about the pace of instruction and assessment that their students require. Authorization ensures these sensitive data can only be accessed by authorized individuals.

In short, authorization plays a crucial role in enabling collaboration securely and effectively.

The need for change

Our existing system had chugged along quite well for the last few years, but we saw that it was becoming a limiting factor in the kind of capabilities we wanted to build into Quizizz.

A couple of key areas that were proving to be difficult were:

• Many-to-many relationships: Collections are logical groupings (like a folder) of quizzes. A quiz can be part of many collections, and a collection can have many quizzes. Managing permissions between entities which don’t have a strict one-to-one relationship like quizzes and collections was becoming complicated at our scale of data and latency requirements

• Transitive permissions: A Class in Quizizz is a logical grouping of students. Teachers can create and assign assessments directly to a class instead of having to do that to each individual student. We struggled with transitive permissions, such as granting access from a class to a co-teacher and then to other entities that were part of the class. Bringing these together in an efficient and maintainable manner was proving to be difficult.

In addition, as we built out our B2B product, we required a way to model hierarchical structures where every level would have permissions for the level below it (district-school, enterprise-branch etc.)

We started re-thinking our current architecture, debating between other out-of-the-box solutions and building it all ourselves, when we stumbled upon Zanzibar, including a couple of blogs by Airbnb and Carta detailing their experience with in-house implementations of Zanzibar.

Zanzibar

Zanzibar is based on “relation tuples” to define access rules, which it uses to create an entity graph. It then uses set operations to resolve permission checks by traversing the graph of relations. 

These tuples, which could represent entities or other tuples, allow transitive relations. Permissions are defined through userset_rewrite clauses in a tuple, determining access based on whether the entity falls within the permitted tuple set. 

Continuing our research on Zanzibar, we found SpiceDB by AuthZed, an open-source Zanzibar-inspired relation-based access-control service that tries to stay true to the principles outlined in the paper.

On further evaluation, we felt like this was a good fit for us

• The community and the developers behind it were active
  • The discord server has a lot of people talking about how they’re working with SpiceDB in their organizations
  • Their developers gave us a custom patch to help with backfilling relations into SpiceDB at scale, which was incredibly helpful in getting us started
• They had a solid roadmap of features, and fairly frequent release cycles. One thing that was of quite a lot of interest to us was the Tiger cache proposal, which would help solve for ACL-aware filtering (think full-text search across resources that a user has access to) in a performant manner.
• The tooling was pretty mature (SDKs in multiple languages, the playground etc.). 
• We were able to model our current and expected features in the SpiceDB schema DSL in a very simple and straightforward way, making it extremely easy to grok and maintain for people new to the system.

With this, we started tinkering with it and working on a proof-of-concept to see if it would be able to support the scale and latency requirements at Quizizz, which has hundreds of millions of responses every day.

SpiceDB at Quizizz

Authzed has a good series of blogs that break down different components of SpiceDB in quite a lot of detail, which I would encourage you to read! But in this blog, I’ll go through our process of adopting SpiceDB into our stack.

SpiceDB keeps a record of relation tuples in a selected datastore (we used Postgres on RDS). It processes permission checks by dividing the query into subproblems and distributing them across multiple instances. SpiceDB achieves this via kuberesolver, which monitors K8s endpoints for other instances. To streamline this and other SpiceDB tasks, we utilized the SpiceDB Operator to manage our cluster.

We built out a new service that would own the domain of Identity and Access management (IAM) in Quizizz, hooked it up to SpiceDB and started running some tests. After being satisfied with the kind of performance and flexibility it offered us in our controlled load tests, we decided to put this into production in “shadow” mode, i.e, all requests going to our existing authorization system would also be duplicated to the new service, and we would compare the results between the two to verify correctness and performance.

Other services interact with the IAM service to check whether an entity has the permissions to perform some action on a resource that is specific to their domain. 

Results

• We saw a tiny latency increase (as one would expect with adding a couple of network hops), but very little of that was attributable to SpiceDB itself
• We experienced <20-30ms p99 latencies for the CheckPermission API (which tells us whether an accessor has the required permission for a particular resource)
• Due to some optimizations that we had done in our internal layers, not every call to access a quiz goes through to SpiceDB. So the most frequent call was actually to list all the relationships for a particular resource, which had a p99 of <10ms which was excellent.
• When we checked for correctness comparisons across a couple of weeks, we found very few instances where there was a mismatch between our existing system and SpiceDB. In fact, a couple of times SpiceDB was correct when our internal system wasn’t!
• When we tried to build out something new which wasn’t part of our existing authorization model, it took less than a couple of days to have that layer up and running in SpiceDB.

However, we did run into a few issues that arose due to the way some of our existing flows worked and how SpiceDB works internally. We were using the TOUCH operation in SpiceDB, which ensures idempotency and does not throw an error if a relation already exists, to create relations. The implementation of TOUCH (at the time), would delete the older relation (if it already existed), and then create a new one. It would create a transaction, and then perform these operations within that transaction. 

This worked fine under normal load, but in a particular case, when we were on-boarding one of our larger clients (the State of Paraná, Brazil), we were sending one WriteRelationships call for each teacher that we onboarded. This resulted in extremely high contention in these transactions, resulting in a massive spike in our Postgres CPU utilization and SpiceDB API latency (graphs below).

The fix in this case was fairly simple: we modified our flow to batch relationships (which SpiceDB allows as part of a single API call) across the onboarding process and made a single call instead of many. 

Conclusion

We realized we needed to upgrade our authorization system to keep up with Quizizz’s growth and the complexity of our platform. 

The impact of introducing SpiceDB into our systems has been quite significant. 

• In less than a day, we were able to build and deliver a feature that our enterprise client (State of Paraná) requested, which had a strong dependency on authorization.
• We continued building highly collaborative capabilities (like the co-teaching experience) into Quizizz with minimal time spent on modeling the authorization layer, and instead were able to focus all our time on building the best possible experience for our teachers.