The real-bench-100 Real Document Benchmark 📑¶
To honestly evaluate document classification without relying on synthetic generation shortcuts or manual human spreadsheet labeling, we constructed real-bench-100 (src/data/build_real_bench_100.py).
Benchmark Composition (100 Balanced Documents)¶
The benchmark contains exactly 25 real-world documents per sensitivity tier:
| Sensitivity Tier | Target Count | Real-World Source Composition |
|---|---|---|
| PUBLIC | 25 | 15 official SEC 8-K regulatory filings + 10 Enron public bulletins & announcements |
| INTERNAL | 25 | 15 Enron workplace scheduling/logistics emails + 10 GitLab handbook SOP procedures |
| CONFIDENTIAL | 25 | 15 Enron commercial vendor contracts & deals + 10 compensation & salary memos |
| RESTRICTED | 25 | 15 live cryptographic keys, API tokens & DB URIs + 10 credential/login emails |
Benchmark Results¶
Evaluated with pure neural inference (qwen0.5b-v3) paired with Tier-1 secret scanning:
| System | Overall Agreement | Real Secret Recall | Latency per Chunk | Notes |
|---|---|---|---|---|
| Random / Uniform Baseline | 25/100 (25.0%) | 0/15 (0.0%) | N/A | Balanced 4-way chance baseline |
EdgeSLM (qwen0.5b-v3 + Tier-1) |
48/100 (48.0%) | 15/15 (100.0%) | 28.0 ms (Batched) | +92% over random baseline; 100% secret catch |
Empirical Findings: The "Financial Number" Shortcut¶
Why did pure zero-shot neural transfer score 48% on real-world text?
- Root Cause: In the synthetic training data generator, samples containing revenue numbers, dollar signs ($), and quarterly tables were uniformly tagged as CONFIDENTIAL.
- The Shortcut: The neural attention layers learned a false heuristic: "financial tables \(\to\) CONFIDENTIAL".
- Impact on Real Text: When presented with official public SEC 8-K earnings disclosures, the model over-flagged 18 out of 25 public filings as CONFIDENTIAL.
- Mitigation: Future dataset generators must include public financial disclosures to explicitly break this spurious correlation.