Deep Dive in Caching: Freshness, Failure, and Scale

Caching in system design is easy to explain on the happy path: keep a reusable result somewhere cheaper to access, then avoid repeating the expensive work. The difficult part is deciding when that result is still usable and what the system does when it is missing, old, or unavailable.

I treat a cache as a disposable copy with an explicit freshness and failure policy. If deleting the cache loses an acknowledged business write, that component has taken on a storage responsibility that needs more than a cache label.

This deep dive follows an illustrative incident dashboard. It covers where to cache, how to design keys, which write strategy to choose, why invalidation races happen, and how to keep a cache outage from becoming a database outage. The examples and diagrams are original. The progression from a simple mental model to failure cases takes inspiration from Hello Interview’s caching lesson.

Start with the data contract

Imagine an incident-management product used by several tenants. Engineers repeatedly open the same active incidents. A summary combines the incident title, status, owner, and a count of recent updates. Recomputing that view can involve database queries, aggregation, and serialization.

The expensive work is not necessarily a physical disk read. Databases already have buffers and other reuse mechanisms. An application cache may still save query execution, connection occupancy, network trips, aggregation, or calls to another service. Measure the work you are trying to remove before adding another system.

Different parts of this product need different contracts:

• Incident summary: a briefly old display may be acceptable if the product defines that delay.
• Changing incident status: the write must check the authoritative current state, regardless of what the page displayed.
• Tenant access: a cached summary must not grant access after the caller loses permission.
• Versioned public assets: a content-addressed JavaScript bundle can have a much longer reuse policy than private incident data.

The source of truth owns the authoritative state. A cache hit finds a usable copy. A miss does not. A TTL controls entry lifetime; invalidation deliberately stops reuse; eviction removes entries to manage capacity. None of those words, by itself, describes the product’s consistency guarantee.

Decide whether this result is worth caching

Read-heavy traffic is a reason to investigate caching, not enough to choose it. For each candidate result, answer four questions:

• Cost: which measured query, computation, or remote call will a hit avoid?
• Reuse: will the same correctly scoped result be requested again before it expires or changes?
• Freshness: which old values are acceptable, and which decisions must still validate current state?
• Recovery: can the source survive a cold, slow, or unavailable cache within a bounded fallback policy?

The frequently opened incident summary is a candidate because it reuses meaningful work. A rarely repeated filter combination may cost more to look up and fill than it saves. A status transition still needs authoritative validation. Check the existing query and access path before adding another network dependency; there is no universal read/write ratio that makes a cache worthwhile.

Cache-aside: follow a hit, a miss, and a fill

With cache-aside, the application manages the cache lookup and the source read. It authenticates and authorizes the request, constructs the key, and checks for a usable value. On a miss, it loads from the source, optionally fills the cache, and returns the result.

For the incident dashboard, I would start by caching the expensive summary rather than every database row. A hit then skips meaningful work. Caching an inexpensive indexed lookup may add a network dependency without removing enough cost to justify it.

A useful cache entry can carry a schema identifier, the source version when available, and timestamps used by the freshness policy. The presence of a version does not automatically make the cache current. The reader still needs a way to know which versions are acceptable.

Source-read failures are not cacheable successful results. If a best-effort cache fill fails after a successful source read, returning the loaded value may be reasonable. Record the fill failure because repeated failures turn future requests into misses. Put a deadline and an admission limit around source loading, including the fallback path.

Cache-aside does not inherently provide consistency with the data store. The Microsoft architecture pattern describes its lifecycle and consistency limitations.

Choose the layer that removes the expensive work

A local L1 cache in front of a shared L2 cache can reduce network traffic for hot reads. It also creates more copies to invalidate. A process that misses an invalidation may keep an old L1 value even after L2 has changed.

Keep the hit-ratio denominators explicit. With 10,000 cacheable requests per second, 80 percent L1 hits leave 2,000 L2 lookups. If L2 hits on 90 percent of those lookups, it serves 1,800 requests and sends 200 to the source. The combined hit ratio is 98 percent, not the sum or average of the layer percentages. L2’s hit ratio is conditional on requests reaching L2. Measure its lookup volume as well as its ratio.

Promoting an L2 result into L1 should preserve its freshness metadata. Starting a fresh full lifetime at every layer can repeatedly extend an already old result. Define a freshness deadline for the representation and carry that policy through the hierarchy.

Database query caching and application result caching are different optimizations. The former may avoid database work; the latter can also avoid assembling and serializing the response. Choose the smallest layer that addresses the measured bottleneck and can enforce the required data isolation.

The cache key is part of the correctness model

A key identifies the result that may safely be reused. For a tenant-scoped summary, an illustrative key is incident-summary:v3:tenant=t17:incident=i481:locale=en:view=standard. Here v3 is the representation schema namespace. It is not a claim that the application knows the current database version without reading anything.

Include dimensions that change the result: tenant, resource identity, relevant query parameters, locale, representation schema, and any permission-dependent view variant. Canonicalize filters so equivalent requests do not create unnecessary entries. Avoid placing raw credentials or sensitive personal data in keys that may appear in logs and operational tools.

Key scoping and authorization solve different problems. Even a correctly tenant-scoped key does not establish that this caller currently belongs to that tenant. Perform the required access check before returning cached data. If different roles receive different fields, either cache a safely shared underlying object and apply current redaction, or separate the view variants deliberately.

Schema namespaces make rolling deployments easier: new code reads a format it understands. They also create a new cold keyspace. Plan the resulting source load and clean up obsolete namespaces according to policy. A deployment should not accidentally flush the effective working set at peak traffic.

Negative caching needs a narrower policy

Repeated requests for a genuinely nonexistent incident can be cached as an explicit not-found result for a short period. This is negative caching. It saves repeated lookups, but can hide a newly created resource until that entry expires or is invalidated.

Distinguish authoritative absence from a timeout, overloaded source, or access denial. Do not turn all errors into not-found entries. Scope negative entries correctly, limit their lifetime and cardinality, and avoid letting arbitrary identifiers fill the cache with useless misses.

Read and write strategies are separate choices

Cache-aside and read-through describe who owns a read miss. Write-through and write-behind describe write propagation and acknowledgment timing. They are not mutually exclusive choices for the whole system. A read-through library or service can own loading; an ordinary Redis GET does not know how to query PostgreSQL.

For the incident dashboard, the starting workflow is concrete:

• Read: check access, build the scoped key, and return a usable hit. Otherwise admit a bounded source load, then attempt a best-effort fill under the freshness policy.
• Write: validate the authoritative state and commit the database change, then invalidate affected summaries. Record failed invalidations and repair them through the chosen recovery path.

This does not establish an atomic transaction across the database and cache. A delayed reader can still refill an old value, as the next example shows. If an asynchronous buffer acknowledges a write before durable storage, define what survives a process or node failure.

Write-behind can be useful, but an acknowledged dirty value is not disposable. It needs a durability, replay, ordering, and failure-recovery design. For an incident status change, I would normally keep the authoritative write in the database and evaluate caching around the read path.

Why “write the database, then delete the cache” is incomplete

Deleting a cached summary after a successful database commit is a reasonable starting policy. Deleting before the commit lets a concurrent reader reload the old database value. However, deleting after the commit still leaves another race:

A reader misses the cache and reads incident version 41. A writer commits version 42 and deletes the key. The reader then finishes its delayed fill and stores version 41. The cache is stale again even though the writer invalidated it in the expected order.

A TTL eventually expires that entry under the configured policy, but its clock starts when the entry is written. An old reader can refill late and start a new lifetime. A lagging source replica can also supply an old value. Therefore, “TTL is 30 seconds” is not an unconditional promise that the cached fact is at most 30 seconds behind the source.

Choose a consistency protocol for the actual requirement

• Display with an eventual-consistency contract: invalidate after commit, use a bounded reuse policy, and measure convergence. Return the committed write result to the updating caller when appropriate.
• Version-aware publication: retain a sufficiently long-lived version floor or generation marker, and atomically reject fills older than it. A version field inside a value that was just deleted cannot enforce this by itself.
• Ordered change propagation: use reliable publication and monotonic version handling when updates drive cache changes. Duplicate, delayed, and out-of-order events still need a policy.
• Fresh correctness decision: read or validate against the authoritative consistency boundary. Do not make incident transitions or access revocation depend only on a display cache.

A version floor that can disappear through eviction, expiry, or failover is not an unconditional fence. Define where that ordering state lives and how long an old loader can remain in flight. Delayed double deletion narrows some race windows but a fixed sleep does not prove that every old reader has finished.

An outbox can make a committed change available for reliable downstream invalidation or projection updates. It does not make asynchronous invalidation instantaneous. Likewise, a publication log solves a different problem from proving that every process has discarded its old local copy.

TTL and eviction solve different problems

TTL provides expiration. Eviction manages memory pressure and may remove an entry before its TTL. A cache client must handle both as ordinary misses. In Redis, expiry is checked on access as well as through background cleanup. Delayed physical cleanup does not mean a normal read is entitled to return an expired key. See Redis EXPIRE.

For capacity pressure, selected Redis policies include LRU, which approximates recency, and LFU, which approximates frequency. An allkeys policy considers all eligible keys; a volatile policy considers keys with expirations. With noeviction, writes that need more memory can fail instead of evicting existing entries. Workload, key eligibility, and configuration matter. See Redis eviction policies.

TTL is not a memory budget. Estimate the working set from distinct keys, serialized value sizes, metadata and allocator overhead, replication, and operational headroom. A scan through one-time keys can displace useful entries. Consider admission rules for large or rarely reused results before trying to tune eviction around them.

For a cache of recomputable data, persistence may be optional. Warm restart time and the load required to rebuild the working set still matter. Redis replication is normally asynchronous, so failover can lose recent cache writes or invalidations. Treat that behavior as part of the freshness design rather than assuming a replica makes cached state authoritative. See Redis replication.

Design the miss path before celebrating the hit ratio

For capacity planning, count usable hits that avoid foreground source work, not just found keys. Redis keyspace_hits and keyspace_misses count key lookups. One request may perform several lookups, and a found value can be rejected by the application’s freshness or schema policy. A valid negative hit can avoid a source read. Track actual source loads and refreshes alongside application outcomes. See Redis INFO.

Consider 10,000 cacheable summary requests per second and one source load per miss. At 98 percent usable hits, that path produces 200 source reads per second. At 90 percent, it produces 1,000. An empty or unavailable cache produces 10,000 if every request falls through.

Now suppose this workload has an illustrative total source budget of 500 reads per second, with 100 already used by other requests:

Only 400 source loads per second remain for the cacheable requests. The algebraic requirement is 10,000 × (1 − hit_ratio) ≤ 400, or at least 96 percent usable hits. That floor spends the entire assumed budget; it is not an operating target. The 98 percent case leaves 200 reads per second of headroom.

During a total cache outage, this simplified model can admit at most 400 of the 10,000 cacheable requests per second before exhausting the budget. The rest need explicitly permitted stale responses, degradation, or rejection. A circuit breaker can avoid waiting on a broken cache, but cannot make the database accept the extra work.

These inputs are assumptions, not measured capacity or a benchmark. Refreshes, retries, coalescing, writes, variable query costs, and unrelated traffic change the real budget. Requests per second is not a concurrency limit: source admission must also bound in-flight work, waiting, and deadlines. Expensive misses can dominate tail latency even when most requests are fast. A global hit ratio can also hide one tenant or key’s bad experience.

A slow cache can be worse than a clean miss

If every request waits for a long cache timeout and then starts a database query, latency and concurrency rise together. Put cache access inside the request’s total deadline. A circuit breaker can stop repeatedly waiting on a broken dependency, but it does not create spare database capacity.

Bound source concurrency and queue length. Decide what happens when those limits are reached: return a permitted stale value, reject or degrade the request, or use another explicitly designed path. “Fail open to the database” is safe only within a capacity and correctness policy.

A hot key can overload the cache even when reads hit

If one incident receives 6,000 requests per second and all reads hit the shared cache, the source can be quiet while the cache node serving that key still handles 6,000 lookups. A high hit ratio does not establish spare cache CPU, network capacity, or acceptable latency.

In a stable Redis Cluster, a key maps to one hash slot served by one primary. Adding shards distributes different keys; it does not split that key’s reads automatically. A bounded local L1 can absorb repeated reads, at the cost of more copies to invalidate. Replica reads can spread read load only when their potentially stale results fit the contract. See the Redis Cluster specification.

This is different from a stampede. A hot hit path pressures the cache; an absent hot value can create many source loaders. Measure which component is saturated before choosing between local reuse, shared-cache capacity, coalescing, and source admission.

Prevent a hot-key miss from becoming a stampede

A stampede happens when many callers try to reconstruct the same missing or expired value together. One popular incident can generate hundreds of equivalent summary queries. The cache was meant to remove duplicate work; the expiry boundary has temporarily recreated it.

I would combine several controls, because each addresses a different source of load:

• Local request coalescing: one in-flight load per key inside each process. Followers share the result or wait within their deadlines. Ten processes can still produce ten loads.
• TTL jitter: spread expiration times across keys. It reduces synchronized expiry of a whole dataset; it does not eliminate concurrent requests for one expired hot key.
• Bounded stale serving: when the data contract permits it, serve the previous value while one bounded refresh runs. Keep a separate hard stale deadline.
• Source admission control: limit total concurrent loads and queued work so many distinct misses cannot overwhelm the origin.
• Selective warming: refresh a measured hot working set before it expires. Avoid warming every possible key or launching a deployment-wide refresh storm.

Do not reset the hard stale deadline on every failed refresh. That turns a bounded fallback into indefinitely old data. If the hard limit is reached, the application needs a deliberate error, authoritative read, or degraded-response policy.

A distributed lease reduces duplicate work, but does not prove exclusivity forever

A shared lease can coordinate refresh across processes. A Redis-style implementation uses a unique ownership token and expiration, with an atomic ownership check when releasing it. A plain delete can remove a lease that a newer owner acquired after the first owner’s lease expired.

The loader can also outlive its lease. A second process may acquire it while the first is still running. That is why a refresh lease is an optimization unless downstream writes enforce the required fencing or version checks. Review the assumptions in Redis distributed lock guidance rather than treating a short lock recipe as a consistency protocol.

Cancellation belongs to the coalescing design

If one caller disconnects, removing that caller’s waiter should not automatically cancel a shared load needed by other callers. Give the shared loader its own bounded lifetime. Each waiter keeps its own deadline and cancellation state.

When no waiters remain, decide whether to cancel the load or finish it as an explicitly budgeted cache warm. Clean up the in-flight registry on success, failure, and cancellation. An exception must not leave a key permanently marked as loading. Propagate deadlines into source operations so abandoned requests do not keep consuming database connections indefinitely.

HTTP caching is a separate protocol

An application cache reuses domain data or computation. An HTTP cache reuses a response representation under protocol rules. A Redis TTL does not configure a CDN. A Cache-Control header does not automatically configure an in-process map.

These response directives have distinct meanings in RFC 9111. Vary identifies representation variants; it is not an authorization check. A Set-Cookie header alone does not prohibit HTTP caching. For private incident responses, choose the private-storage and reuse policy explicitly.

ETags can avoid sending an unchanged body

An ETag is an opaque representation validator. A client with a stored body can send If-None-Match. After normal request and access checks, a matching representation can produce a bodyless 304 Not Modified. The client then reuses its corresponding stored body. This illustrative header exchange assumes those access checks succeeded:

GET /api/tenants/t17/incidents/i481 HTTP/1.1
Host: api.example.test
Accept-Language: en
If-None-Match: "incident-i481-v42-standard-en"

HTTP/1.1 304 Not Modified
ETag: "incident-i481-v42-standard-en"
Cache-Control: private, no-cache
Vary: Accept-Language

The validator must correspond to the selected representation, including applicable variants. A matching tag must not let a revoked user bypass authorization. Validation can save response bytes while still requiring an origin request and source work. A 304 is not proof that the database did no work. See RFC 9110.

Stale-while-revalidate is bounded permission

The stale-while-revalidate extension permits limited stale reuse while revalidation is attempted. It does not mean refresh happens on a timer, and it does not permit indefinitely stale responses after repeated failures. stale-if-error is a separate bounded fallback. Check intermediary support and conflicting revalidation requirements. See RFC 5861.

This policy can fit a deliberately public, stale-tolerant status badge. It should not determine whether a caller may read a private incident or perform a current-state transition.

What I would put in the design review

For the incident dashboard, my initial design would be cache-aside for an expensive, tenant-scoped summary, with access checks on every request. I would choose TTL and stale-serving policy from an agreed display contract, coalesce loads locally, and cap source concurrency. Status transitions and sensitive authorization would use their authoritative validation path.

I would add a local L1, distributed refresh coordination, or a more involved version protocol only when measurements or consistency requirements justify them. Each additional copy and coordination mechanism adds another failure mode to explain.

Verify recovery, not just warm-cache latency

Before relying on the design, I would test a cold start, an empty cache after failover, a slow or unavailable cache, a popular key expiring under concurrency, and a broad deployment that changes key namespaces. I would also pause a reader between source read and cache fill to reproduce the stale-refill race.

Correctness tests should include tenant isolation, permission changes, locale and view variants, negative entries followed by creation, and source-version changes during refresh. Failure tests should include cancelled waiters, loader timeouts, rejected fills under memory pressure, and refresh failure beyond the hard stale deadline.

Observe cache and source latency percentiles, hit and miss rates, source-load concurrency, eviction and expiration rates, value sizes, error rates, stale-response age, refresh failures, and the age of invalidation backlog. Avoid putting raw tenant IDs or arbitrary keys into unbounded metric labels.

This article is a design walkthrough, not a measured cache benchmark. The decision I want to defend is specific: which expensive work a hit removes, which old values are acceptable, which operations still require current state, and how much work the source can safely accept when reuse fails.

For related architecture topics, see Kafka vs RabbitMQ, the transactional outbox pattern, and resilient large file uploads. More articles are available in Best Practices.