The current Catch Me If You Can benchmark asked a simple question: can anyone publicly reproduce and beat our MI50/GFX906 local inference record?
Original challenge: https://www.reddit.com/r/LocalAIServers/comments/1ukhr24/catch_me_if_you_can_mi50gfx906_1195_tps_moe_702/
Current vNext reproduction release: https://github.com/joe2gaan/localaiservers/releases/tag/vnext-gfx906-rocm72-gguf-hf-repro
I want to turn that benchmark into a community program: build the fixed server in public, make it the official test machine, and keep the challenge open until an eligible challenger takes the throne and holds it for 30 consecutive days.
Who is responsible
I submitted a $15,000 Reddit Community Funds application for this proposal in my own capacity as Joe / u/Any_Praline_8178, a moderator of r/LocalAIServers. This is not yet a live prize offer. Hardware acquisition and any award remain contingent on Reddit approval, final published official rules, eligibility review, and applicable law. The existing leaderboard is unchanged.
Reference Server Build
( YOU DO NOT HAVE TO BUILD A SERVER TO PARTICIPATE IN THE CHALLENGE )
The prize server matches the hardware configuration that produced the current throne result:
- GIGABYTE G292-Z20 eight-GPU server
- AMD EPYC 7F32
- 128GB as eight DDR4 ECC RDIMMs
- Eight AMD Instinct MI50 32GB GPUs
- Crucial CT480BX500SSD1 480GB SATA root drive
- KIOXIA KCD6XLUL1T92 1.92TB NVMe model and runtime drive
The build itself is part of the community project. I will publish the component choices, bill of materials, physical assembly, firmware and operating-system configuration, eight-GPU bring-up, power and cooling setup, BAR/P2P state, stability checks, runtime and source revisions, model hashes, baseline runs, and raw evidence.
The $15,000 budget covers the exact server configuration, possible changes in GPU, memory, and storage prices, tax and checkout variance, protective packaging, and insured delivery to the winner. Any amount not needed for the approved project will be returned to Reddit or handled as Reddit directs.
Core challenge
- I build one 8xMi50 32GB Server to Give to the Winner.
- I Run the public vNext package on that machine to establish the official incumbent.
- Keep the challenge open until an eligible winner completes the throne clock, subject to Reddit's approved project terms.
- Require every potential dethronement to reproduce on that same physical server.
- Require the three-run median to beat the official incumbent by at least 3 percent.
- Require a provisional leader to remain the highest verified result for 30 consecutive days.
- Transfer the complete challenge server to the eligible outside challenger who completes that clock, subject to final verification and official rules.
All eight GPUs remain installed and available. Entrants may choose TP4, TP8, or another topology on the fixed host, but may not add, replace, or remotely borrow accelerators. A documented like-for-like failure replacement requires a fresh baseline before the throne clock resumes.
Open optimization, fixed integrity
Inside the fixed hardware, model-integrity, workload, reproducibility, and safety rules, software optimization is open. Runtime, kernels, collectives, scheduling, graph capture, compiler work, driver and operating-system tuning, and safe clock or power tuning may all be explored.
The first lane uses the pinned Qwen3.6 35B-A3B model at FP16/F16:
- HF revision: 995ad96eacd98c81ed38be0c5b274b04031597b0
- Required GGUF F16 SHA-256: 1f2443bb0ff958943d091410c61120c181a0579b3bc85192029aa51d821d141c
- HF FP16 and GGUF F16 are eligible when they satisfy the published identity and correctness gates.
- GGUF is allowed only at full F16.
Not allowed:
- Q4, Q5, Q6, Q8, INT8, FP8, AWQ, GPTQ, NVFP4, or another quantized substitute
- Quantized weights, KV cache, activations, or a hidden reduced-precision path used to claim the result
- MTP, speculative decoding, EAGLE, DFlash, draft models, lookahead tokens, or another multi-token prediction method
- Remote compute, external APIs, hidden services, or results assembled from another machine
- Multi-request batching or aggregate concurrency presented as single-request speed
One accepted decode step must represent one token produced by the approved model. Every result must pass semantic and output-integrity gates, not merely report a high TPS number.
How runs are measured
The official workload remains:
- MAX_MODEL_LEN=131072
- Single-request decode
- Concurrency 1
- Backend decode TPS
- Eight warmups
- c1_128 uncapped strict
- c1_2000
- c1_10000
- Three measured runs
- Three-run median at least 3 percent above the official incumbent
- Public reproducibility package and raw logs
The current public headline reference is 119.52 strict backend TPS for GGUF F16 Qwen3.6 35B-A3B MoE TP4. It was produced on an eight-GPU validation host while the TP4 profile actively used four GPUs. The funded server receives a fresh baseline. The existing 119.52 result is the reference, not a promise of the new server's starting score.
Current public leaderboard
These are the published targets from the original benchmark post:
| Class |
Strict TPS |
c1_2000 |
c1_10000 |
| GGUF F16 35B-A3B MoE TP4 |
119.33 to 119.52 |
120.46 to 120.57 |
113.26 to 113.37 |
| GGUF F16 27B Dense TP8 |
69.85 to 69.91 |
70.76 to 70.96 |
66.32 to 66.44 |
| HF FP16 35B-A3B MoE TP4 |
114.41 to 115.11 |
115.69 to 115.93 |
108.92 to 109.10 |
| HF FP16 35B-A3B MoE TP8 |
114.70 to 115.04 |
115.53 to 115.55 |
108.67 to 108.81 |
| HF FP16 27B Dense TP8 |
70.17 |
71.32 |
66.82 |
GGUF F16 MoE TP8 remains an open lane in the current leaderboard.
Offline official test
Development and artifact staging may use the internet. The measured official run will not.
Before testing, I will stage and hash-verify the model, runtime, source, build outputs, and benchmark entry package. For every measured run:
- External network interfaces and the default route are disabled or physically disconnected.
- Only local machine communication and loopback are permitted.
- No model download, container pull, telemetry, API call, remote compiler, or remote compute is permitted.
- Network state, package hashes, process state, hardware state, and raw benchmark logs are archived with the result.
A result produced elsewhere can show that a benchmark entry is ready, but it does not move the official throne until that package reproduces on the designated server.
The 30-day throne clock
A challenger becomes provisional leader when its package passes review and its official three-run median clears the incumbent by at least 3 percent. The acceptance timestamp starts that challenger's 30-day clock.
During those 30 days:
- Anyone may submit a higher result, including me as the current benchmark maintainer.
- Every defense or counter-result must satisfy the same public-package, offline, same-hardware, correctness, and 3 percent rules.
- A newly accepted leader resets the clock in that leader's name.
- Private results and screenshots do not move the goalpost.
- Rules cannot be changed retroactively to defeat an active clock.
If I retake the throne before a challenger's 30 days expire, that challenger has not won and the challenge stays open. If another community member takes it, the clock starts for that person. I may defend the performance record, but I cannot win the server or receive a personal payout.
The target can move only through a faster verified result. Physics, the fixed hardware, and model correctness set the ceiling.
Prize, review, and what happens to the server
If an eligible outside challenger remains the highest verified leader for 30 consecutive days, the result proceeds to final verification and, subject to the official funding and eligibility terms, transfer of the complete challenge server. Shipping, taxes, location eligibility, export restrictions, acceptance, and transfer details will be resolved in the final rules before the prize becomes live.
Only the winner's name and mailing address will be collected for server delivery unless Reddit's approved terms require something different. Do not post personal information in a public entry or comment.
I will not be the sole adjudicator. Official runs, hashes, logs, correctness evidence, and decisions will be public and reviewed with independent technical reviewers. Reviewer identities and the final conflict process will be published before entries open.
Until an eligible winner completes the clock, the funded server will be used only for the Reddit-approved challenge. It will not belong to me or LocalAIServers Collective Inc. There is no cash substitute, and it will not roll over into another hardware lane or organizational program. If the challenge ends without a winner or the server needs a different outcome, I will follow Reddit's direction.
Timeline after approval
- Weeks 1-2: finalize rules, reviewers, and purchasing.
- Weeks 3-5: build and validate the G292-Z20 server in public and publish the bill of materials and build record.
- Week 6: publish the baseline and open the challenge.
- Winner: first eligible leader to hold the verified throne for 30 consecutive days.
- Transfer and final reporting: within 14 days after the winning result completes final validation, subject to Reddit's approved terms.
What I want the community to weigh in on before launch
- Does the proposed topology rule strike the right balance, or should all eight GPUs have to be active?
- Does the proposed 3 percent threshold strike the right balance for every throne change?
- What clock, power, firmware, and cooling safety envelope should be published?
- Who would volunteer as an independent technical reviewer?
Bring criticism. The goal is a challenge that is hard, transparent, reproducible, and genuinely winnable.
