TokenAtlas 🌐

The universal multilingual benchmark quantifying the token inflation tax across 52 global languages and frontier LLM tokenizers using Meta's FLORES-200.

⭐ GitHub Repo 🤗 Hugging Face 52 Languages Evaluated 6 Frontier Tokenizers Apache 2.0 Open Source

⚡ Live Token Tax & Context Shrinkage Calculator

0 chars | 0 bytes
Samples:
Estimated Tokens
--
-- bytes/tok
Inflation Index (τ)
--
vs English
Effective Context
--
-- usable words
Financial Multiplier
--
invoice tax factor