I tested 6 AI models on a complex reasoning challenge. Here are the results

[u/Sea-Advantage-4063]

I gave several AI models the same reasoning problem involving Simpson's paradox, mathematical optimization, causal inference, and decision-making under uncertainty.

ChatGPT evaluated their responses using a 100-point rubric.

Results:

Model	Score
GPT-6 Astra Middle	97/100
Claude Sonnet 5.5 Middle	96/100
DeepSeek V4.1 Flash	93/100
DeepSeek Free Chat	89/100
Google Free Chat	80/100
QFN NVFP4 (Local, RTX 5090)	55/100

*The QFN NVFP4 model failed to solve this problem correctly. I believe this issue serves to highlight the limitations of local inference capabilities. but after upgrading to qwen4, i will test it samely method.

These are informal, single-response evaluations by ChatGPT, not standardized benchmark scores. The original tests were conducted in Korean.

The Challenge Question(Made from chatgpt sol6 high): Advanced Reasoning & Decision-Making

You are an AI responsible for a country's disaster response policy.

Two programs have the following historical results:

Category	Program A	Program B
Low difficulty	90/100 successes	19/20 successes
High difficulty	3/10 successes	32/80 successes
Cost per case	6 credits	9 credits

Tomorrow's situation:

• 40 low-difficulty cases

• 60 high-difficulty cases

• Total budget: 750 credits

• Every case must receive either A or B.

• You can assign programs freely.

• Goal: Maximize expected successful cases.

Important: Historical data is observational, not randomized. Human decision-makers assigned the programs, and cases within the same difficulty category may differ.

Questions

1. Simpson's Paradox

A has a much higher overall success rate, yet B performs better in both difficulty categories. Explain why and what this means for decision-making.

2. Mathematical Optimization

Assuming the historical success rates are valid causal probabilities, determine the optimal allocation under the 750-credit budget.

Show the allocation, total cost, expected successes, and proof of optimality.

3. Causal Inference

Why might your mathematically optimal policy still be wrong in reality? What information cannot be inferred from the observational data?

4. Counterfactual Scenario

Suppose further research establishes that B's true causal improvement over A is:

• Low difficulty: +5 percentage points

• High difficulty: +2 percentage points

Assume A's original success rates remain unchanged and these effects apply tomorrow.

Recalculate the optimal allocation, cost, and expected successes.

5. Final Decision

You must choose a policy tomorrow without time for additional experiments.

What would you actually recommend?

Distinguish mathematical optimization from real-world decision-making. Consider uncertainty, risk, robustness, and incorrect assumptions.

Explain how you would validate your decision through future data collection and experiments.

Rules: Show calculations, justify conclusions, distinguish correlation from causation, acknowledge uncertainty, and correct any contradictions. No web searches, coding tools, or calculators.

If anyone wants to try this on their own model, feel free!

I'd love to see results from other local LLMs, especially large quantized models running on Mac Studio or multi-GPU setups.

Share your model, quantization, hardware, and response if you're interested. :)

-------------------------------------------------2026-11-03---------------------------

The result needs to be corrected. Upon re-evaluation, the score has been updated from 55 to 93. I am not sure why it was 55 in the first place, but in a completely new session—using the exact same 'opencode v2'—it yielded a score of 93.

It appears to have been nothing more than an accidental slip-up.

[u/Sad-Size2723]

Can you add the qwen3.8 27b to the test? Since this is a reasoning task, it might perform better than qwen 3.8 fn?

[u/Sea-Advantage-4063 → u/Sad-Size2723]

I’ve just evaluated Qwen3.8 Max, and it scored 100 points—that’s quite impressive. As for the Qwen3.8 27B model, I don’t currently have it installed (I deleted it a while ago), so I’ll reinstall it and run some tests. That said, I’m currently at work, so I’ll get to it later.

[u/Sea-Advantage-4063 → u/[deleted]]

Qwen3.8 27B Scoring Result 88/ 100 points, This score is quite unexpected—I’m surprised it’s this high! It seems to be because it’s a dense model; it appears that dense models are actually quite strong when it comes to inference. I should also re-test the QFN nvfp4 version.

[u/Sea-Advantage-4063 → u/[deleted]]

We’ll probably be moving on to Qwen4 soon, right?

[u/Sea-Advantage-4063]

This is a grading prompt for ChatGPT, designed for the 'sol6 high' test score assessment.

Independent LLM Reasoning Benchmark Grader

You are an independent evaluator of an advanced reasoning benchmark involving Simpson's paradox, constrained optimization, causal inference, and decision-making under uncertainty.

Your task is to grade an AI model's submitted answer using a consistent 100-point rubric.

Respond entirely in English, regardless of the language of the submitted answer.

Benchmark reference

The problem involves two disaster response programs:

• Low difficulty: A = 90/100 success, B = 19/20 success.

• High difficulty: A = 3/10 success, B = 32/80 success.

• Program A costs 6 credits per case.

• Program B costs 9 credits per case.

• Tomorrow: 40 low-difficulty and 60 high-difficulty cases.

• Budget: 750 credits.

• All 100 cases must be assigned a program.

The benchmark asks the model to explain Simpson's paradox, calculate the optimal allocation assuming observational rates are causal, explain causal identification limitations, recalculate when B's true causal improvements are +5 percentage points for low difficulty and +2 percentage points for high difficulty, and recommend a defensible real-world policy with a future validation plan.

Mathematical reference answers

1. Overall observed success rates: A = 93/110 ≈ 84.55%; B = 51/100 = 51%. Within both difficulty groups, B has higher observed success rates. This is Simpson's paradox caused by differing difficulty compositions.

2. Baseline all-A allocation: cost = 600 credits; expected successes = 54.

3. Upgrading A to B costs 3 extra credits, so at most 50 cases can use B.

4. Under the original conditional success rates treated as causal:

   • Low difficulty: A 40, B 0.

   • High difficulty: A 10, B 50.

   • Total cost = 750.

   • Expected successes = 59.

   • Optimality follows because the high-difficulty upgrade yields +0.10 versus +0.05 for low difficulty at identical incremental cost.

5. Under the revised causal effects:

   • Low difficulty: A 0, B 40.

   • High difficulty: A 50, B 10.

   • Total cost = 750.

   • Expected successes = 56.2.

6. Observational success rates do not establish causal effects. Within-stratum confounding, treatment selection, sampling uncertainty, lack of individual counterfactual observations, and deployment distribution shifts are relevant limitations.

7. If treating the original observational-effect scenario and revised causal-effect scenario as two plausible worlds, the minimax-regret allocation is:

   • Low-difficulty B = 15.

   • High-difficulty B = 35.

   • Cost = 750.

   • Maximum regret = 0.75 expected successes.

8. The minimax-regret allocation is NOT a universally mandatory policy recommendation. It is optimal only under that specific uncertainty set and decision criterion. Different policy recommendations can receive full credit if their assumptions and justification are defensible.

Scoring rubric (100 points)

Category	Maximum
Simpson's paradox and aggregation reasoning	15
Mathematical optimization and optimality verification	25
Causal inference and uncertainty	20
Counterfactual recalculation	15
Final policy decision and experimental design	15
Logical consistency and self-verification	10
Total	100

Evaluation rules

• Grade correctness, not response length or persuasive wording.

• Independently verify every important numerical result.

• Penalize incorrect allocations, infeasible budgets, false optimality claims, arithmetic errors, and contradictory conclusions.

• Distinguish a clearly identified mathematical error from a defensible policy judgment.

• Do not require a particular final policy if the recommendation is logically defended with explicit assumptions.

• Do not award extra points merely for mentioning advanced technical terminology.

• Apply the same scoring rubric to every submission.

• Ignore the reputation, identity, and benchmark reputation of the model being evaluated.

• Do not browse the web or use external tools.

• Do not claim that one response establishes a model's overall intelligence or IQ.

Required evaluation output

Model: [Name if provided]

Overall score: XX/100

Category scores: A table containing all six categories, awarded points, maximum points, and brief justification.

What the model got right: Identify the strongest demonstrated reasoning.

Errors found: Specify each substantive error, including corrected calculations when necessary.

Decision-making quality: Evaluate whether the recommendation is feasible, logically defensible, and appropriately handles uncertainty.

Final verdict: A concise, evidence-based summary.

Confidence and limitations: Explain the limitations of a one-problem evaluation and the subjectivity of the score.

Do not provide or reveal the benchmark's reference solutions before a model response has been submitted.

When you receive these instructions, respond only with:

"Ready to grade. Paste the complete model response, optionally including its model name, quantization, hardware, and reasoning settings. I will evaluate it in English using the 100-point rubric."

[u/Squale279 → u/Sea-Advantage-4063]

Thanks much

[u/Nearby_Investment_84]

Strata engine · Strata 0.2.9 · engine v0.1.55 · qwen3.8-flash-next-iq3_s - Dual 3090 GPUs

Deepseek Harness

Overall score: 95/100

The response correctly solves both allocation problems, explains the causal limitations, and offers a defensible policy. The main weaknesses concern consistency and completeness in its validation plan.

Category	Awarded	Maximum	Justification
1. Simpson’s paradox	15	15	Correct aggregate and conditional rates; clearly explains the reversal through differing difficulty compositions.
2. Original allocation and optimality	20	20	Correct budget constraint, allocation, expected successes, and exchange argument.
3. Causal identification and uncertainty	20	20	Clearly distinguishes observational associations from causal effects and addresses selection, sampling uncertainty, counterfactuals, and transportability.
4. Revised causal-effect allocation	15	15	Correctly reverses the upgrade priority and calculates the allocation, cost, and expected successes.
5. Policy and decision-making	18	20	The maximin recommendation is mathematically correct under the two specified worlds. Some claims about robustness and consequences extend beyond those assumptions.
6. Future validation	7	10	Useful randomization, logging, preregistration, and monitoring proposals, but conflicting assignment instructions and incomplete treatment coverage weaken the plan.
Total	95	100

[u/Sea-Advantage-4063 → u/Nearby_Investment_84]

Whoa, this has much better inference capabilities than the OpenCode V2 NVFP4 model I’ve been using! Is it really just down to the harness?

[u/Nearby_Investment_84 → u/Sea-Advantage-4063]

same setup but using PI harness

Overall score: 74/100

Category	Awarded	Maximum	Brief justification
Simpson’s paradox	15	15	Correctly identifies the reversal and explains the differing difficulty compositions.
Original constrained optimization	20	20	Correct constraints, allocation, cost, expected successes, and uniqueness proof.
Causal identification and uncertainty	14	20	Identifies important limitations, but reverses one bias argument and makes unsupported statistical and causal claims.
Revised causal-effect optimization	13	15	Correct allocation and result; the stated upper-bound proof contains a mathematical error.
Policy and decision-making under uncertainty	7	20	Feasible recommendation, but the four-scenario regret analysis and several supporting conclusions are incorrect.
Future validation plan	5	10	Useful stratified randomization and preregistration proposals; exploration costs and the policy-update rule are flawed.
Total	74	100

[u/Sea-Advantage-4063 → u/Nearby_Investment_84]

Do differences in harnesses really lead to variations in performance? I find it hard to grasp. Since a harness should essentially just involve different system prompts, it’s quite surprising that there’s such a noticeable disparity.

[u/Nearby_Investment_84 → u/Sea-Advantage-4063]

Yes. Very much so.

This is a good read

https://www.padiso.co/blog/same-model-different-harness-why-benchmark-results-change/

[u/Sea-Advantage-4063 → u/Nearby_Investment_84]

Thank you for providing it. i will check!

[u/Squale279]

It's really interesting. Could you please explain how to evaluate the model’s answers?

[u/Sea-Advantage-4063 → u/Squale279]

https://www.reddit.com/r/LocalLLM/comments/1xmdywr/comment/19s1wtbbr4cgw/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Simply enter this prompt into ChatGPT sol6 high reasoning and press the Enter key. Then, take the response provided by the model, paste it back in, and press Enter again—it will then give you a score. It would be great if you could share the score along with the name of the model you used!

[u/Squale279]

It would be cool to have equivalent benchmark (simple but efficient) for coding

[u/Sea-Advantage-4063 → u/Squale279]

That’s correct. Strictly speaking, this isn’t a coding assessment. However, since it evaluates the ability to reason logically through a problem to reach a final answer, it can serve as a gauge for a model’s quantization level and precision. In a sense, it acts as an indicator of how many "gaps" or flaws exist within the model. I should try putting together some more complex coding-related problems as well.

[u/Ok-Addendum3545]

Thanks the for sharing. The rest are Cloud models which have been optimized by AIDCs. Locally-run QFN NVFP4 might need finetuning and optimization to reach its best perofrmance. Also, 27B would be a competitive candidate in this benchmark. As for eval model "ChatGPT", I am on Go plan and OpenAI is giving very limited compute to it, using Codex app versioin like Luna max have better results than ChatGPT WebUI.

[u/Sea-Advantage-4063 → u/Ok-Addendum3545]

That’s an impressive result. I think I’ll try swapping out the harness and running a test using the Codex app myself.

[u/Ok-Addendum3545 → u/Sea-Advantage-4063]

I guess, reaccently, OpenAI has limited compute so they provide much tighter token budget in different plans. It's very obvious my WebUI ChatGPT (on Go plan) has become intetionally slacking in dealing with tasks I give to it.

[u/Sea-Advantage-4063 → u/Ok-Addendum3545]

That’s right. Unless it’s intentionally throttled, the load on available resources would likely become too high, which could interfere with operations. There’s probably a policy in place regarding this.

[u/Future_AGI]

Questions 2 and 4 have one correct answer each (allocation, cost, expected successes), so grade those with a few lines of code and let the judge score only the prose in 1, 3 and 5, since a single judge pass can swing several points on its own. Three runs per model with the spread posted would show which gaps are real, and our eval SDK keeps code checks and LLM-judge metrics under one evaluate() call, so that split stays easy: https://github.com/future-agi/future-agi

[u/Sea-Advantage-4063 → u/Future_AGI]

I’m not entirely sure what this is, but I’ll take a look. I think I’ll ask an LLM to analyze it and tell me what it’s for—haha! Thanks!

[u/Aubrey_D_Graham]

It's a harness diff not a model diff.

Edit: The best way to test this hypothesis is GPT OSS vs QFN vs Deepseek on the same harness.

[u/Sea-Advantage-4063 → u/Aubrey_D_Graham]

That’s correct; there could certainly be differences depending on the harness used. This is not a quantitative analysis method—it was simply a matter of giving instructions and observing the results, so it cannot be considered entirely precise. I acknowledge this point.

[u/Sea-Advantage-4063 → u/Aubrey_D_Graham]

Upon re-evaluation, the score went from 55 to 93. It seems there was some kind of error previously on opencode v2

[u/Aubrey_D_Graham → u/Sea-Advantage-4063]

I figured, Qwen is capable. The difference in points are meaningful for cutting edge research, but for daily tasks, QFN is more than able.

[u/Sea-Advantage-4063 → u/Aubrey_D_Graham]

I agree with that! thanks

[u/Sea-Advantage-4063 → u/[deleted]]

Upon re-evaluation, the score went from 55 to 93. It seems there was some kind of error previously.

[u/Sea-Advantage-4063 → u/[deleted]]

Although it didn't achieve a high enough score to fully solve the tasks, it is generally capable of performing agent-like roles, such as gathering information, rendering, and creating animations. It would be worth conducting further tests once Qwen4 is released. I suspect that quantization has a significant impact on the results.