Codec Reverse-Benchmark — Community Leaderboard
Local LLMs reverse-engineering a custom audio codec. Score = decode points + Compress
score anchored at AAC-LC 32 kbit/s (100 = AAC quality; above 100 beats it).
Generated 2026-11-12 20:35 UTC from the open benchmark-result issues.
| # | Model | Hardware | Total | Evolve | Stopped | Stages | Best Compress | Date | Evidence | Issue |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Fable Latest openrouter/~anthropic/claude-fable-latest frame 1024 · keep 256 · step 0.04 | OpenRouter / Anthropic API (cloud inference) | 127.0 | 2 | S1 | S1 PASS Compress 190.0 · 15/15 subgoals | 190.0 | 2026-11-12 | self-reported | #5 |
| 2 | DeepSeek V4.1 Flash deepseek/deepseek-flash frame 512 · keep 192 · step 0.01 | DeepSeek API (cloud inference) | 81.0 | 2 | S3 | S1 fail Compress 145.0 · 14/15 subgoals S2 fail Compress 130.0 · 14/15 subgoals S3 PASS Compress 120.0 · 15/15 subgoals | 145.0 | 2026-11-12 | self-reported | #3 |
| 3 | Qwen3.8-Flash-Next NVFP4 tensorfold/qwen3.8-flash-next-nvfp4#xhigh frame 512 · keep 128 · step 0.01 | NVIDIA DGX Spark (GB10) / 128 GB unified memory | 67.0 | 2 | S3 | S1 fail Compress 78.7 · 11/15 subgoals S2 fail Compress 27.6 · 11/15 subgoals S3 PASS Compress 68.7 · 15/15 subgoals | 78.7 | 2026-11-11 | self-reported | #1 |
| 4 | strata/ISTA-DASLab/Qwen3.8-Flash-Next:Q3_S strata/ISTA-DASLab/Qwen3.8-Flash-Next:Q3_S frame 2048 · keep 768 · step 0.04 | – | 56.3 | 2 | S4 | S1 fail Compress -88.1 · 14/15 subgoals S2 fail Compress -88.1 · 14/15 subgoals S3 fail Compress -88.1 · 14/15 subgoals S4 PASS Compress -7.6 · 14/15 subgoals | -7.6 | 2026-11-12 | self-reported | #4 |
| 5 | Qwen3.8-Flash-Next UD-IQ3_XXS (Unsloth Dynamic) strata/unsloth-qwen3.8-flash-next-ud-iq3_xxs frame 512 · keep 128 · step 0.02 | NVIDIA V100 / llama.cpp (192.168.188.24:8080) | 44.0 | 2 | S4 | S1 fail Compress 81.0 · 9/15 subgoals S2 fail Compress 85.6 · 9/15 subgoals S3 fail Compress 85.6 · 9/15 subgoals S4 PASS Compress – · 9/15 subgoals | 85.6 | 2026-11-11 | self-reported | #2 |
Ranked by total, then by the stage the run stopped at, then wall time.
Results are self-reported unless marked verified. The codec instance is
randomised per run, so only the Compress score is comparable across runs.
Evolve is the codec generation a run was benchmarked on. The generation
advances only when a run finishes stage 1 and its encoder beats the
baseline; the next runs then use that codec.
Submit by opening an issue with the Benchmark result template; see
submissions/TEMPLATE.md.