Skip to content

infersize 📐

PyPI Streamlit App CI License Python Hardware

A deterministic, first-principles analytical capacity planner, VRAM sizing engine, speculative decoding budgeter, and multi-GPU matrix comparator for Large Language Model (LLM) inference.

Why infersize?

Deploying large language models often results in expensive surprises: 1. CUDA Out-of-Memory (OOM) crashes after waiting minutes to download 30GB+ model checkpoints. 2. Massive KV-cache footprint at long context lengths (e.g. 32K or 128K) causing serving latency spikes. 3. Speculative Decoding sizing guesses where target and draft models exceed total GPU VRAM headroom. 4. Cloud TCO over-provisioning by renting expensive 8x H100 clusters when a single quantized 4x L40S or A10G pool was mathematically optimal.

infersize solves this by computing exact weight bytes, SwiGLU expansion dimensions, GQA KV-cache scaling, activation tensors, and CUDA runtime context overhead in under 10 milliseconds on your local CPU — before downloading weights or renting clusters.

⚡ Core Architecture

┌────────────────────────────────────────────────────────────────────────┐
│                              infersize                                 │
│      (Analytical Performance Modeling & VRAM Capacity Planning)        │
│                                                                        │
│  ┌───────────────────────────────────┐ ┌─────────────────────────────┐ │
│  │ 1. Parameter & Weight Sizer       │ │ 2. GQA KV-Cache Engine      │ │
│  │ • Exact SwiGLU 3x gate+up proj    │ │ • Sub-byte FP16/INT8/INT4   │ │
│  │ • MoE active routing (DeepSeek)   │ │ • Context scaling (1K..128K)│ │
│  │ • FP16, INT8, INT4, FP8, BitNet   │ │ • Linear sequence budgeting │ │
│  └─────────────────┬─────────────────┘ └──────────────┬──────────────┘ │
│                    │                                  │                │
│                    └─────────────────┬────────────────┘                │
│                                      ▼                                 │
│  ┌───────────────────────────────────────────────────────────────────┐ │
│  │ 3. Memory & Hardware Matrix Solvers                               │ │
│  │ • Multi-GPU side-by-side comparison (A100, H100, L40S, RTX 3050)  │ │
│  │ • Right-Sized vs Peak Throughput recommendations                  │ │
│  │ • Speculative Decoding Leviathan Speedup & Joint VRAM Planner     │ │
│  │ • Local Physical Hardware Zero-OOM Verifier (NVML / PyTorch)      │ │
│  │ • Cloud TCO (\$ / 1M tokens) across AWS, GCP, Azure, RunPod, Lambda│ │
│  └───────────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┘

📦 Quick Installation

With uv (Recommended & Fastest): 

# Add to existing project
uv add "infersize[ui]"

# Or install in active environment
uv pip install "infersize[ui]"

# Or run instantly without installing
uvx infersize check

With standard pip: 

pip install "infersize[ui]"

From Source with uv: 

git clone https://github.com/Prajwal7-2002/infersize.git
cd infersize
uv sync --extra ui
uv run infersize ui

🚀 Quick Navigation

  • VRAM Analytical Physics — First-principles mathematical derivation of parameter counting, SwiGLU, and GQA KV-cache virtual memory.
  • Speculative Sizer — Leviathan speedup derivations (\(S = \frac{1 + \alpha K}{1 + (K+1)/\gamma}\)) and joint target + draft VRAM budgeting.
  • Hardware Zero-OOM Verifier — Real-time NVML hardware querying to verify zero-OOM safety on physical silicon.
  • Web Studio & Streamlit — Launching and using the interactive browser GUI (infersize ui).
  • CLI Reference — Complete command-line manual (estimate, compare, spec, fit, ui).