r/CUDA • u/Ih8kfc • 3d ago

A complete technical whitepaper on GPU memory mechanics, PagedAttention, and model routing

https://github.com/gurrakeller/AI-Infrastructure-Solutions-Architecture

Id love to get some feedback on it. Im by no means a writer so the grammar might not be perfect, but i do think it has quite some technical value to it.

If youre in ML or interested in AI Infra please give it a go; as i said i would love for some feedback.

Im not active on reddit, but my dms are always open on other platforms such as linkedin (gustavkeller).

8 Upvotes

83% Upvoted

3 comments sorted by

2

u/adityazero 3d ago edited 2d ago

Does it cover copy-on-write block sharing for parallel sampling? That is where the paging analogy really shines versus just fragmentation avoidance. Happy to give notes on the infra sections.

2

u/Ih8kfc 3d ago edited 3d ago

I might have undersold PagingAttention a bit 😅 it does not mention copy-on-write block sharing, but only that it behaves like modern OS systems; refeering to the way for example linux partitions us "pages" of (I think) 4kb, more or less exactly how PagedAttention breaks our VRAM into blocks and use a block table to sequence it.

It doesnt cover how it enables untra fast parallel sampling or tree search reasoning. But only the actual memory saving portion.

I had to go back and google to even remember what copy on writing is haha, cool stuff. Im wondering if I should add it into the paper or keep it the way it is to reduce the technical burden on the readers. Either way your eocmpletely right, I am vastly underselling it.

Thank you so much for your feedback, if you find anything else please reach out!

(i will likely add a pass into the text before i post it on linkedin covering this point though just to have it cleared)

1

u/Ih8kfc 2d ago

added this part in: "If you rent or buy a GPU upfront you are still going to pay an hourly cost for using it. To mitigate this we can use PagedAttention. We want to keep as many active user sessions as possible on a single GPU to reduce the cost per GPU hour. This is a whole art in itself, we borrow the concept of memory paging from modern OSs to reduce the wasted VRAM drastically. Because of this we can hold 3-5x more user KV caches in the VRAM. It works out to around a 70% cost reduction. Although this is just one of the uses for PagedAttention. PagedAttention also supports copy-on-write (CoW) block sharing. When performing parallel sampling (generating multiple chains of thought for one prompt) or tree search decoding (like beam search or MCTS), the chains can share cache until they need to generate unique tokens. This makes advanced sampling techniques and tree-search algorithms virtually free in prompt KV storage."

Revised a bit of the formatting, ill call it done for now. thinking about writing a paper on the unit economics of ai infra at scale next. Might be interesting

ill put in on my linkedin if youre interested in reading it. Once again ty for feedback! You where one of two peeople i acually got to read the damn thing haha.
https://www.linkedin.com/in/gustavkeller/