Published on

Scaling: Choosing Under Constraints

Authors

This is Part 4 of Scaling: Orders of Magnitude, the first series in a pillar on scaling systems.

  • Part 1: Reading the Numbers: what a throughput target actually means, and why the raw number lies.
  • Part 2: What Breaks, and When: walking one system up the orders of magnitude and naming what cracks first at each rung.
  • Part 3: The Menu of Levers: the survey of everything you can reach for, on both the API and the client side.
  • Part 4: Choosing Under Constraints: how to actually decide, and how the domain changes the dials.

Part 3 laid out the levers and insisted that none of them is free. Part 4 is about paying honestly. Given a menu where every choice has a bill, how do you decide which bills to pay? The answer is not universal, and that is the point: the right architecture for a payment ledger is the wrong architecture for an ad-serving system, even at identical load. This part is about the forces that force the call, and the domain lens that quietly decides most of it.

Table of Contents

The Forcing Axes

When you choose which levers from Part 3 to pull, four forces do most of the deciding. Every meaningful scaling decision is a trade along at least one of them.

  • Consistency versus availability. When you cannot have both, which do you sacrifice? This is the deepest axis and it deserves its own section.
  • Cost. At scale, the architecture is a spend decision. The elegant design that triples your infrastructure bill can be the wrong design.
  • Operational burden. Every lever you pull is something someone has to run, monitor, and be paged for at 3am. Complexity has an ongoing tax.
  • Team maturity. The right architecture for a team that has run distributed systems for a decade differs from the right one for a team of four shipping their first product. Capability is a real constraint, not an insult.

These axes interact and often conflict. Strong consistency can cost availability and money and operational pain all at once. The job is not to maximise any single axis but to make the trade deliberately, with the numbers from Part 1 telling you how much of each you can afford.

Consistency Versus Availability: CAP and PACELC

The sharpest of the four axes has a precise theory behind it, and learning the vocabulary makes every async decision in the rest of the pillar clearer.

The CAP theorem says that when a network partition happens, when parts of your system cannot talk to each other, you must choose between consistency and availability. You can keep answering requests with data that might be wrong (available, not consistent), or refuse to answer until you are sure the data is right (consistent, not available). You cannot do both, because doing both would require the partitioned halves to agree, and they cannot reach each other. Partitions are not hypothetical at scale; across regions they are routine. So CAP is not a puzzle, it is a recurring operational decision.

PACELC extends CAP with the half people forget. It reads: if there is a Partition, choose between Availability and Consistency; Else, in normal operation, choose between Latency and Consistency. That second clause matters more day to day than the first, because partitions are rare but the latency-versus-consistency trade is paid on every single request. Strong consistency means coordination, coordination means waiting, and waiting means latency. Every time you choose a strongly consistent read you are choosing to be slower, partition or no partition.

This is the theory under the entire sync-versus-async spine of the pillar. Synchronous, strongly consistent request-response buys you correctness and charges you latency and availability. Asynchronous, eventually consistent flows buy you availability and speed and charge you the discomfort of data that is briefly wrong. Neither is better. Which one is correct depends on what your domain refuses to compromise, which is the last and most important idea here.

Cost Is an Architectural Property

At small scale, cost is a rounding error and you should ignore it in favour of shipping. At large scale, cost is one of the primary design forces, and pretending otherwise produces architectures the business cannot afford.

The reason is simple: at 100,000 requests per second, the difference between an efficient design and a wasteful one is not a few servers, it is a budget line the CFO can see. A cache that halves database load halves a large bill. A chatty payload multiplied by millions of requests is real money in bandwidth. Holding a hundred thousand websocket connections has a running cost that a polling design might avoid, or might not, depending on the numbers. The lesson from Part 1 returns here in a new form: once you know your numbers, you can put a price on each lever, and some levers that are technically elegant are economically indefensible. Good scaling is often the art of serving the load for a cost the business can bear, not serving it in the most sophisticated way possible.

Operational Burden and Team Maturity

The costs that do not show up on the infrastructure bill show up in your people, and they are just as real.

Every lever from Part 3 is something to operate. A broker like Kafka is not a line of code, it is a distributed system your team now runs, monitors, tunes, and recovers when it breaks. Sharding is not a config flag, it is a permanent complication in every query and migration for the life of the system. This is why the same technical answer can be right for one team and wrong for another. A team that has operated distributed systems for years can take on async and sharding and absorb the burden. A small team shipping its first product will drown in that same burden and would be far better served by a boring, single-database, synchronous design that they can actually run, even if a whiteboard says it "will not scale." It does not need to scale yet. It needs to survive contact with its own operators.

The discipline is honesty about capability. The best architecture your team cannot operate is worse than a simpler one they can. Choosing under constraints means choosing under the constraint of who is on call.

Premature Optimization Is the Default Failure Mode

With all of that on the table, the single most common mistake in scaling is not under-building. It is building for a scale you do not have.

It is seductive to design for millions of requests per second when you serve hundreds, because the sophisticated architecture feels like competence and the simple one feels like a liability. It is almost always the wrong call. The sophisticated design costs you today, in engineering time you do not have, in operational burden you cannot yet carry, and in flexibility you have not earned, to buy capacity you will not use for years and might never need. Meanwhile the thing that actually kills early systems is not running out of throughput, it is running out of runway before finding product-market fit, and every hour spent sharding a database that serves 400 requests per second is an hour not spent on that.

The ladder from Part 2 is the antidote. Locate yourself honestly, build for the rung you are on plus enough instrumentation to see the next one coming, and raise ceilings when you actually reach them. Scaling is a sequence, not a leap, and trying to leap is how you fall.

The Domain Lens: Same Toolbox, Different Dials

Everything above converges on one idea that has run quietly through the whole series: the mechanics of scaling are universal, but which cost you refuse to pay is set by your domain. The toolbox is the same. The dials are set differently.

Consider the same order of magnitude of load in three domains:

  • Banking. Consistency, durability, and auditability are sacred. A transfer that shows success and then vanishes is a catastrophe, so the domain refuses to trade consistency for availability or latency on the money path. It will pay in speed, in cost, and in operational rigor to keep the ledger correct. Optimistic UI on a transfer is forbidden; strong consistency is non-negotiable even when it hurts.
  • E-commerce and food delivery. Availability and spike absorption win. It is far worse for the site to be down during the dinner rush than for an order's status to be a few seconds stale. So the domain happily embraces eventual consistency, async flows, and optimistic UI, and accepts that a rare edge case will be reconciled after the fact. Being open matters more than being instantly correct on every field.
  • Advertising and analytics. Throughput and cost dominate, and correctness is statistical rather than exact. Losing or double-counting a handful of events out of billions is acceptable; what is not acceptable is failing to ingest the firehose or paying a fortune to do it. The domain optimises ruthlessly for cheap, high-volume, best-effort throughput, and would consider the banking mindset an absurd overpayment.

The illustrative shape of those priorities:

Illustrative NFR Priorities by Domain (Relative, Not Absolute)

Same five properties, three completely different rankings. This is why "how do I scale?" has no domain-free answer. The levers are identical; the domain tells you which bill you are forbidden to pay, and everything else follows from that.

Conclusion, and the Handoff

Choosing under constraints means trading deliberately along four axes, consistency versus availability, cost, operational burden, and team maturity, with CAP and PACELC giving the first axis precise words. It means resisting the pull of premature optimization and building for the rung you are actually on. And it means reading your domain, because the domain sets the dials: the same toolbox builds a bank, a store, and an ad exchange, and the only difference is which cost each one refuses to pay.

That closes the mental model. You can now read a throughput target honestly (Part 1), locate yourself on the ladder (Part 2), survey your levers (Part 3), and choose among them under real constraints (Part 4). The rest of the pillar goes deep on the two halves of the wire. The API / Backend Track takes the sync-versus-async spine all the way down: how far request-response really goes, where a broker earns its place, and how to stay correct once you go asynchronous. The UI / Frontend Track flips the lens to the client, where every async decision made on the backend eventually shows up on someone's screen. This series was the map. The next two are the territory.