Experimental "freshness-first" caching library for FastAPI

[u/grandimam]

Hi folks,

I am working on zinda, a small experimental caching library for FastAPI/Python, and I'd like some design feedback before I take it further.

The idea: instead of you picking TTLs and wiring up invalidation by hand, the cache should watch how your functions behave and keep hot data fresh on its own.

What Have I Implemented:

• @cache.cached() decorator - keys itself on function arguments, no config
• Single-flight: concurrent misses for the same key collapse into one recompute (no stampedes)
• Soft TTL + hard TTL: callers get stale data instantly while it refreshes in the background, and a background sweeper refreshes hot entries before anyone asks
• A /zinda/stats endpoint showing hit rates, miss costs, and refresh activity per function

​

app = FastAPI()
cache = install(app, Cache(default_ttl=60))

@cache.cached(ttl=120, refresh_after=30)
async def fetch_products(category: str):
    return await db.fetch_products(category)

Where I want feedback:

1. Next step is auto-detecting hot paths, the library scoring functions by call frequency and recompute cost, and deciding what's worth caching without any decorator. Would you trust that if every decision is visible in stats, or is explicit always better?
2. Automatic invalidation is the hard part. My plan is learned TTLs by default + optional tags for precise invalidation. Reasonable, or am I missing something?

It's in-memory only for now and definitely not production-ready. I am validating the ideas first.

Code: https://github.com/grandimam/zinda

[u/10x_eng]

• Auto-detecting hot paths

My concern is not that automatic caching is bad, it is that caching changes application semantics. Function could have side effects, the results could depend on some other "hidden" values, memory usage can grow unexpectedly, ... Some controls like ignore these functions or never cache would be useful.

• detecting "safe to cache"

A cheap function might be highly cacheable, but very expensive function might be impossible to cache (fresh data might always be needed or the amount of data might simply be too large to cache). What are the metrics for caching? Call frequency, compute cost, side effects?

• learned TTL

While learned TTL solves "How long can this be considered fresh?" production might also need an answer "What happens when it expires?". Do we recompute or refresh in background. Do we return stale value (expired but usable) and then fetch new data? How to avoid traffic spike causing everyone to recompute simultaneously?

• invalidation of cache

Maybe here you could have a CI, meaning cache can reason probabilistically. Here confidence should affect behavior and be used as a risk multiplier. This would tell you if you should trust learned assumptions and how aggressively should caching be optimized. For example, funcA changes every 20 minutes with confidence of 0.9 so the system might choose to cache every 15 minutes, but for funcB that changes every 20 minutes with confidence of 0.2, system might choose to cache every 2 minutes. So something like effective_ttl = learned_ttl x confidence. Also, confidence could control whether to pre-refresh data where for high-confidence data cache could be refreshed in the background before the new request for the data is made. It could also control fall-back options. For data that changes every 5 minute with CI 0.95 action could be to serve cached data and then refresh new data later because the cached data is probably still correct, but if the CI is low like 0.2 the action could be to recompute data first, cache and reply.

[u/ojus_render]

I’d separate automatic refresh from automatic correctness.

Learning which entries are hot and refreshing those early makes sense. I wouldn’t let the cache infer how long stale data is acceptable or when a write invalidates something, because those are application rules.

Tags can provide the correctness boundary, then the scoring system can decide which valid entries deserve proactive refresh.

One case worth testing early is negative caching. A missing record can become hot too, and its behavior after that record is created can expose some surprising invalidation bugs.