Sam Newby
← All writing

Distributed systems

Why CAP Theorem is still important

CAP Theorem is still important in distributed systems, but I think we often talk about it in the wrong way.

If you’ve worked in distributed systems or are learning about them you’ve definitely heard the term CAP Theorem mentioned. It’s one of those fundamental concepts that has existed for a long time and still gets brought up all the time.

And I think it should. I don’t think CAP is one of those things we should leave in the old distributed systems textbooks and move on from. I think it’s still very relevant, but I also think we often talk about it in a way that makes it less useful than it should be.

The classic way people talk about CAP is that you can only pick two. Consistency, availability, and partition tolerance. You get two of them and you have to give one up.

It’s a nice simple explanation, and it’s the one most of us probably heard first. But I think it’s also the bit that makes people misunderstand it.

You don’t really get to ignore partitions

The problem with saying “pick two” is that it makes partition tolerance sound optional. Like you can decide whether or not your system cares about partitions.

But if your system is distributed then partitions can happen. The network can fail, regions can go away, machines can restart, DNS can be weird, queues can back up, replicas can lag, and sometimes one part of the system can talk to something while another part can’t.

That is the reality of building distributed systems. You can make it less likely, you can make it easier to recover from, you can pay very clever people and very expensive cloud providers to hide a lot of it from you, but you can’t pretend the network is perfect.

So for me the useful version of CAP is not “which two do you pick?” It’s more like:

When the network breaks, what do you want to happen?

That’s the bit that matters.

Consistency or availability

When a partition happens you normally have a choice to make. Do you keep answering requests, even if some of those answers might be stale or incomplete? Or do you stop answering some requests because being wrong would be worse?

That’s the trade off.

And I think this is where CAP is still really helpful, because it forces you to be honest about what kind of wrong you can live with.

Sometimes being unavailable is better than being wrong. If you’re dealing with payments, permissions, bookings, balances, or anything where doing the same thing twice would be a disaster, then you probably want to stop and be careful. It might be annoying for the user, but it’s better than corrupting the data or creating a mess you can’t easily unwind.

But sometimes being slightly behind is better than being unavailable. If you’re dealing with a feed, notifications, analytics, metrics, search indexes, or messages that can sync later, then maybe you want the system to keep going. Maybe eventual consistency is completely fine. Maybe the user would rather see something slightly old than see nothing at all.

Neither of these answers is always better. It depends on the system. More importantly, it depends on the specific part of the system.

Different parts need different answers

This is why I don’t really like when people talk about whole products or databases as if they are just CP or AP. It’s not that those labels are useless, but they flatten the conversation too much.

A real system has lots of different bits of data and they don’t all matter in the same way.

Your billing data probably needs a different answer to your notification count. Your admin permissions need a different answer to your activity feed. Your checkout probably needs a different answer to your product recommendation model. Treating all of those things as the same is how you end up either over-engineering everything or being dangerously relaxed about the wrong thing.

I think this is the biggest thing CAP gives you. It makes you ask the question per decision, not just per database.

What happens if this read is stale? What happens if this write is rejected? What happens if two parts of the system both think they are allowed to make progress? What happens when everything reconnects?

These are the questions that actually matter.

Managed services did not make this go away

I think one reason CAP sometimes feels a bit old fashioned is that a lot of us are not building the low level distributed systems ourselves anymore. We use managed databases, managed queues, managed caches, managed everything. Most of the scary words are hidden behind nice dashboards and pricing pages.

That’s a good thing. I don’t want to be writing my own consensus protocol unless I absolutely have to. Most of the time I want to use the boring managed thing and move on.

But the trade offs are still there. They are just in configuration now. They are in read replicas, cache expiry, queue retries, delivery guarantees, idempotency keys, failover settings, regional writes, and all the little choices that seem boring until something goes wrong.

A cache is basically you deciding that stale data is okay for a bit. A queue is you deciding that work can happen later and maybe more than once. A replica is you deciding that reads can be fast but might not be perfectly up to date. None of this is bad, it’s just the trade off showing up in more practical clothes.

This is why I still think CAP is worth knowing. Not because you need to sit there and categorise every tool you use, but because you need to understand what the tool is choosing on your behalf.

The bit I care about

I don’t think CAP should be treated as a trivia question. I don’t think the interesting part is whether you can remember what each letter means, or whether you can label a database in an interview.

The interesting part is that it makes you admit something uncomfortable. Distributed systems fail in ways that single machines don’t. And when they fail you have to decide what kind of failure you prefer.

Do you want to keep answering and clean it up later, or do you want to stop until you know the answer is correct?

That’s not an academic question. That’s a product question, an engineering question, and sometimes a business question.

So yes, it still matters

So yes, I think CAP Theorem is still important. Not because it’s the full answer to distributed systems, because it isn’t. It doesn’t tell you about latency, observability, costs, operations, or the many other things that make real systems hard.

But it does give you a really important starting point. It makes you think about what happens when the network stops doing what you hoped it would do.

And honestly, that question still catches people out. It catches systems out too.