MINI-INFER
LLM Paged KV-Cache & Radix Prefix Studio
Zero-Cold-Start Client-Side CAD

PagedAttention GPU Memory Pool & Zero-Copy Prefix Caching

Simulate dynamic GPU block allocation, inspect request-level logical block tables, observe copy-on-write radix tree prefix reuse, and benchmark speculative decoding acceptance rates in real time.

32
Physical Blocks
< 4%
VRAM Waste
2.8x
Speculative Boost
Physical GPU Memory Pool (32 Blocks • 16 Tokens/Block)
Dynamic allocation eliminates memory fragmentation and avoids contiguous tensor pre-allocation
32
Free Physical Blocks
0
Active Requests
96.2%
Fragmentation Eliminated
GPU Physical Block Pool:
Active Request Block Tables:
Radix Tree Prefix Cache & Copy-on-Write (CoW)
Shared prompt prefixes (system instructions, multi-turn chat) reuse pre-computed KV blocks with 0 recompute

1. Shared System Prompt Prefix (32 Tokens • 2 Blocks)

"You are a helpful coding assistant. Adhere strictly to clean code, defensive type signatures, and PEP 8 standards..."
Shared Block #4 (ref_count: 2)
Shared Block #5 (ref_count: 2)
← Reused by Request A and Request B
Request A (Coding User)
User Prompt: "Implement Dijkstra in Rust"
Logical Block 0 → Physical #4 [Shared, ref=2]
Logical Block 1 → Physical #5 [Shared, ref=2]
Logical Block 2 → Physical #12 [Private]
Request B (Data Science User)
User Prompt: "Explain PCA vs t-SNE mathematically"
Logical Block 0 → Physical #4 [Shared, ref=2]
Logical Block 1 → Physical #5 [Shared, ref=2]
Logical Block 2 → Physical #19 [Private]
✓ 0ms Time-to-First-Token (TTFT): Request B bypassed prompt prefill entirely for the first 32 tokens, achieving instant token streaming because key-value activations were already live in GPU VRAM!
Speculative Decoding Acceptance & Rejection Engine
Draft model proposes K=4 tokens; Target model verifies in a single parallel forward pass
3 / 4
Draft Tokens Accepted
2.35x
Wall-Clock Speedup
4.0 tok
Tokens per Target Step
Token Verification Pipeline:
KV-Cache VRAM Footprint CAD
Formula: 2 × n_layers × n_kv_heads × head_dim × context_len × batch_size × bytes_per_elem
8.39 GB
Total KV Cache VRAM
524 MB
Per-Request Footprint
4,096
16-Token Paged Blocks
Python SDK & CLI Usage
Run continuous-batching inference and benchmark PagedAttention locally
Installation:
pip install mini-infer
Start High-Throughput OpenAI-Compatible Server:
# Spin up local engine with PagedAttention and Radix Prefix Caching mini-infer serve --model Qwen/Qwen2.5-0.5B-Instruct --port 8000 --block-size 16 --enable-prefix-cache
Python API Usage:
from mini_infer.engine import LLMEngine from mini_infer.cache.block_pool import BlockPool from mini_infer.cache.prefix_cache import PrefixCache engine = LLMEngine.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", enable_prefix_caching=True) response = engine.generate("Explain how PagedAttention maps logical blocks to physical memory.") print(response.text)