Rate Limiting and Backpressure: The Overload Decision That Keeps One Noisy Client From Taking Down Everyone Else

Every API works perfectly until the day a single client misbehaves. A mobile release ships with a retry loop that has no delay, a partner's batch job runs at ten times its normal rate, or a marketing email sends a million users to your login endpoint in the same minute. Nothing was "attacked." Your own traffic took the system down. At AEGONTECH LLC, we see this pattern in post-incident reviews more often than any exotic security failure, and it is almost always preventable with a decision most teams make too late: how your system says "slow down" and what it does when it is told to.

Key Takeaways

• Rate limiting protects a system from callers; backpressure protects it from itself. You need both, and they solve different problems.
• Pick the limiting algorithm for the behavior you want: token bucket for tolerating bursts, sliding window for strict fairness.
• Limit per tenant or API key first, per IP last. Per-IP limits punish offices and mobile carriers that share addresses.
• A rejected request should be cheap, explicit (HTTP 429 with a Retry-After header), and safe for clients to retry with jittered backoff.
• Test the limits under load before production does it for you.

What is the difference between rate limiting and backpressure?

Rate limiting caps how much work a caller may send; backpressure tells callers to slow down because the system cannot keep up. Rate limiting is a policy you set in advance ("100 requests per minute per API key"). Backpressure is a signal produced at runtime by actual saturation: a full queue, a slow database, an exhausted connection pool.

A useful mental model: rate limiting is the speed limit sign, backpressure is the traffic jam. A road with a speed limit can still jam when too many cars arrive, and a road with no limit can flow freely when traffic is light. Production systems need both the posted limit and the ability to react when reality diverges from the plan.

A system that cannot say "no" cheaply will eventually say "no" to everyone at once.

Why do systems collapse under load instead of just slowing down?

Because overload creates more load. When a service slows, clients time out and retry, which multiplies the request volume at exactly the moment capacity is lowest. Engineers call this a retry storm, and it turns a ten-percent slowdown into a total outage.

The arithmetic is unforgiving. If each of your clients retries three times after a timeout, a service that is failing 50 percent of requests can see its effective traffic rise by roughly 2x within seconds. Google's Site Reliability Engineering book describes this cascading failure pattern and recommends capping retries per request and per client for exactly this reason. Research on large-scale incidents consistently finds that a significant share of major outages involve a feedback loop of this kind rather than a single broken component.

The other collapse mechanism is queue growth. An unbounded queue feels safe because nothing is rejected, but a queue holding 10 minutes of work means every new request waits 10 minutes, long after the caller has given up. You are doing real work for nobody. Bounded queues that reject early are almost always the healthier choice.

Which rate-limiting algorithm should you choose?

Choose token bucket if your clients are bursty and you want to allow that, and sliding window if you need strict, predictable fairness. The four common algorithms differ mainly in how they treat bursts:

• Fixed window: counts requests in calendar intervals (for example, per minute). Simple and cheap, but a client can send double its quota by straddling a window boundary.
• Sliding window: counts requests in a rolling interval. Smoother and fairer, at the cost of more memory per key.
• Token bucket: a bucket refills at a steady rate and each request spends a token. Allows short bursts up to the bucket size while enforcing a long-run average. This is the default in most API gateways and a sensible starting point.
• Leaky bucket: requests drain at a constant rate, smoothing traffic into a steady stream. Good for protecting a fragile downstream dependency.

For most B2B APIs, token bucket per API key is the right first choice. It matches how real clients behave: quiet for a while, then a short flurry when a user opens a screen or a batch job starts.

What should you limit on: IP address, user, or API key?

Limit on the authenticated identity first, and use IP address only as a coarse safety net. An IP address is a poor proxy for a customer. A corporate office, a university, or a mobile carrier using carrier-grade NAT can put thousands of legitimate users behind one address, and a strict per-IP limit will block all of them because of one noisy neighbor.

A layered approach, which AEGONTECH recommends as the default for client projects, works well in practice:

1. Global limit to protect total capacity, enforced at the edge or API gateway.
2. Per-tenant or per-API-key limit reflecting what each customer pays for or has been promised.
3. Per-endpoint limit for expensive operations such as search, exports, or report generation, which may cost 50 to 100 times more than a simple read.
4. Per-IP limit on unauthenticated endpoints like login and password reset, where abuse prevention matters most.

Weighting by cost rather than counting every request equally is one of the highest-value refinements. An endpoint that runs a 400-millisecond database aggregation should consume more of a client's budget than a cached lookup that takes 2 milliseconds.

How should a rejected request behave?

It should fail fast, say exactly why, and tell the client when to come back. The standard is HTTP 429 Too Many Requests with a Retry-After header, ideally along with headers that expose the remaining quota so well-behaved clients can pace themselves without guessing.

Three details separate a helpful limiter from a frustrating one:

• Reject before doing work. The check must happen before authentication lookups, database calls, or payload parsing wherever possible. A limiter that costs as much as the request it rejects protects nothing.
• Make it safe to retry. Clients should retry with exponential backoff and jitter (random variation in the delay) so that a thousand rejected clients do not all return at the same instant. Pair this with idempotency keys on write operations so a retry never double-processes a payment or order.
• Document the limits. Publish them in your API docs. Customers who know the budget design around it; customers who discover it in production file support tickets.

How does backpressure work in practice?

Backpressure works by making saturation visible and propagating it upstream, so the slowest component sets the pace instead of silently drowning. There are three practical mechanisms:

• Bounded queues and concurrency limits. Cap how many requests a service handles at once. When the cap is reached, reject immediately instead of queueing indefinitely. This is called load shedding.
• Circuit breakers. When a dependency fails repeatedly, stop calling it for a short period instead of piling more requests onto a struggling service. Libraries and service meshes implement this pattern, and it protects both the caller and the dependency.
• Prioritization. When you must drop work, drop the least valuable first. A health check and a paying customer's checkout should survive a flood that a background analytics sync does not.

In AEGONTECH's own work on products such as Dolfy.ai and Mimicall.app, where bursts of user activity can trigger heavy downstream processing, the guiding rule is that background work must always yield to interactive work. A user waiting on a screen is worth more than a report nobody is watching, and the architecture should encode that preference rather than leaving it to chance.

Where should limits be enforced?

Enforce coarse limits at the edge and precise limits close to the resource they protect. An API gateway, whether managed on AWS, Azure, or GCP, or self-hosted with something like NGINX or Envoy, is ideal for global and per-key limits because it rejects traffic before it touches your application servers.

Fine-grained limits, such as "five concurrent exports per tenant" or "20 writes per second to this database," belong inside the service, since only it knows the true cost. For counters shared across multiple instances, a fast store such as Redis is the common choice, with a deliberate decision about what happens if that store is unreachable. The safer default for most systems is to fail open for general API traffic, so a limiter outage does not become a full outage, while failing closed on sensitive paths like authentication.

How do you know your limits are right?

You find out by testing them under load before production does it for you. At AEGONTECH, we set the initial limit from measured capacity, not intuition: load-test a realistic traffic mix, find the point where latency begins to climb, and set the limit at roughly 70 to 80 percent of that saturation point to leave headroom.

Then watch the limiter itself. Track how many requests are rejected per customer, which endpoints trigger limits most often, and whether 429 rates correlate with customer churn or support tickets. If a legitimate, high-value customer keeps hitting the cap, that is a product signal for a higher tier, not merely an operational nuisance. Dashboards, alerts on sudden spikes in rejections, and a clear owner for limit changes turn rate limiting from a one-time setting into a managed capability.

Rate limiting vs. scaling up: which is the better investment?

They are complements, not alternatives. Adding capacity raises the ceiling; limits and backpressure decide what happens when the ceiling is reached, because eventually it always is. Teams that rely only on autoscaling discover that scaling takes minutes while a traffic spike takes seconds, and that every new instance also multiplies the load on shared dependencies such as the database.

The cost argument is just as clear. Provisioning for the absolute worst-case spike typically means paying for capacity that sits idle more than 90 percent of the time. A good limiter lets you size infrastructure for realistic peaks and degrade gracefully beyond them.

Frequently Asked Questions

What HTTP status code should rate-limited requests return?

Return HTTP 429 Too Many Requests with a Retry-After header indicating how many seconds the client should wait. Avoid returning 503 for limit violations, because that signals a server fault rather than client behavior, and it can trigger the wrong alerts and retry logic.

Should we rate limit internal service-to-service traffic?

Yes, though with different goals. Between internal services, concurrency limits, timeouts, and circuit breakers matter more than per-minute quotas. A single misbehaving internal caller can exhaust a shared database just as effectively as an external attacker.

Is rate limiting enough to stop DDoS attacks?

No. Application-level rate limiting handles abuse and accidental overload, but volumetric DDoS attacks must be absorbed upstream by a CDN or dedicated mitigation service. Treat limiting as one layer in a defense that also includes network-level protection and the practices in the OWASP API Security guidance.

How do we pick the first limit if we have no data?

Start conservative but generous, such as 10 to 20 times your typical customer's observed peak, instrument everything, and tighten based on evidence. It is far easier to lower a documented limit with data than to raise an arbitrary one after customers complain.

Conclusion

Overload is not an exception to plan around; it is a normal operating condition that every growing product eventually meets. The teams that handle it well decided in advance who gets how much capacity, what happens at the limit, and which work is allowed to wait or fail. Those decisions are cheap to make during design and expensive to improvise during an outage.

If you are unsure how your own APIs would behave under a sudden 10x spike, that is a conversation worth having before the spike arrives. The engineering team at AEGONTECH LLC helps companies assess resilience, design sensible limits, and build systems that degrade gracefully instead of falling over. Visit aegontech.dev to book a consultation.