Skip to content
All articles

Your In-Memory Cache Is Lying to You on Serverless

7 min read

You need to remember something between requests — how many times this IP has submitted, the result of an expensive lookup, which webhook IDs you have already handled. On a long-running server you reach for a variable in module scope, and it works, and it is the right answer.

On serverless the same code also appears to work. That is the problem. It does not throw, it does not warn, and in development and light testing it behaves exactly like the thing you meant to build. What it actually is, is something quite different with the same shape.

What module scope means here

A serverless function is not a server. Your platform spins up an instance to handle a request, and — this is the part that misleads — keeps it warm to handle the next one. Module-level code runs once per instance, not once per request, so a Map declared up there really does persist between calls.

It persists across the calls that land on that instance. Which is not the same as persisting, and the gap contains all the bugs:

  • Concurrency spawns instances. Two requests at the same moment mean two instances, each with its own copy of your map, neither aware of the other.
  • Idle instances get reclaimed. Quiet period, cold start, empty map. No event, no log line.
  • A deploy replaces everything. Every warm instance is discarded.
  • Regions are separate. On an edge deployment your visitors may not even be talking to the same continent.

So the honest description of a module-scope Map is: a cache with an eviction policy you do not control, a lifetime you cannot predict, and an unknown number of independent copies. Sometimes that is fine. It is never what the code looks like it is.

The four things people put there

Ranked by how badly each one fails.

Rate limiting. The common case, and the one I have shipped. A map of IP to recent timestamps, rejecting past a threshold. Per instance, this catches a burst from a single source — which is what casual abuse actually looks like — and does nothing at all against a distributed one, or against someone patient enough to spread requests across cold starts. It is worth having. It is not a rate limit, and if you describe it as one to a security reviewer you will have a bad afternoon. I use exactly this on my own contact form, and the comment in the source says all of the above, because the failure mode of this pattern is that it looks like the real thing.

Caching an expensive read. The most defensible use. If the value is identical for everyone and stale is acceptable, a per-instance cache is a pure win — worst case you compute it more often than a shared cache would. Nothing is wrong, you just get a lower hit rate than you think.

Deduplication. Dangerous. Remembering processed webhook IDs in memory means a retry landing on a different instance is not a duplicate as far as that instance knows, and you fulfil the order twice. Deduplication needs a unique constraint in a database, not a set in memory — the whole point is a guarantee, and this offers none.

Counters anyone will look at. Free-tier usage, quota tracking, "how many times has this been called". These do not just drift, they undercount by an unknown factor and reset without warning, so the number is wrong in a direction that makes you feel safe.

The test that tells you which you have

One question: what happens if this value is empty when it should not be?

If the answer is "we do some work again" — a cache miss, a recomputed value — module scope is fine and you should not add infrastructure for it. If the answer is "we let something through that should have been blocked", or "we charge someone twice", you need a store outside the function, and no amount of care in the handler substitutes for that.

The uncomfortable middle is where it mostly is: not a correctness guarantee, but not nothing either. A per-instance rate limit genuinely raises the cost of casual abuse. The right move there is to keep it and write down what it does not do, next to the code, so the next person does not discover the limits during an incident.

Making it honest

Three habits that cost nothing.

Name it for what it is. recentByIpThisInstance is uglier than rateLimiter and stops a reader assuming a guarantee that is not there. The name is where most people form their model of the code.

Bound it. An unbounded map in a warm instance is a memory leak with a long fuse — the instance lives for hours, the map only grows, and eventually the function starts failing for reasons that have nothing to do with the request that triggered them. Evict expired entries on every write. A rate limiter that exhausts memory is its own denial of service.

Do not let it own truth. It can hold a copy, an approximation, a recent result. The moment it becomes the only place a fact lives, that fact is one cold start from gone.

When to reach for a shared store

When correctness depends on it. Redis, or your platform's KV, or just a table in the database you already have — the last option is underrated, because a small table with a unique index and a TTL column solves most of this without a new dependency, a new failure mode and another dashboard.

But do it when you need it. Adding a distributed store on day one for a contact form that receives four messages a month is complexity you carry forever against a threat that has not appeared. The mistake is not choosing module scope. It is choosing it and then describing it, in code and to yourself, as something stronger than it is.