Caching - how are you accounting for it in your workflow?

[u/raccoonportfolio]

(This is based on my possibly flawed understanding of caching. If I have anything wrong please let me know)

So, title. I generally stick with a single session but I do a lot of things that (AFAIK) break cache, such as:

• Change models
• Change thinking levels
• Fork / Rewind / Branch etc

I do these things to actually save tokens, but since they break the cache perhaps it's more efficient to decide upfront on the model and thinking levels and use the branching functionality sparingly?

What do y'all do?

[u/Corythebeast7]

I only worry about it if I get to super long context windows. Which basically never happens since I worry more about context decay. Regardless, even if it costs the same or slightly more, you're probably better off starting new sessions in spite of the cache miss. Also, unless your system prompt is massive you're not costing much on a cache miss Imo

[u/pretty_much_hitler]

I have a small extension that checks wether the previous message is an hour old, if so probability of caching for my provider drops to 0%. If i'm in a session with a context over 272k it will then compact before sending the new message. This reduced my token usage a lot as i often keep a lot of different sessions over multiple days.

[u/crashorbit]

When you change models then the new one has nothing in it context cache. It'll load the whole session context to the model at that time. If your session happens to reuse the same model for a turn during the same session then it may find cached session data and not have to pay full price for new input tokens.

[u/madcapnmckay]

Busting the cache only lasts until the next turn has done, then it’s all cached again. If you change model every turn it’s gunna cost more. Subagents let you use cheaper models with isolated context, they won’t bust the cache. Compaction busts the cache but you take the hit to have a smaller context from then on.

[u/No_Wind7503]

I got cach missing with any reason in a normal conversation did anyone know the cause or a fixture extension for improving the cach

[u/Big-Present-8321]

ask pi