PagedAttention: Virtual Memory for the KV Cache

KV cache grows linearly with sequence length and at long contexts can eat more GPU memory than the model weights themselves, making it a scarce resource by default. PagedAttention applies virtual memory-style paging to attention kernels to cut this waste.

Open original source ↗Generate a post from this angle

For writing research only. Verify the original source before publishing; market data is not investment advice.