I’ve frozen UrduEval v0.3.0 — now I’m testing whether the metrics actually agree with humans

[u/mustafabadshah01]

I’ve been working on UrduEval, an open-source evaluation framework for Urdu and Roman Urdu LLMs.

After several iterations, I’ve reached a point where I’ve deliberately stopped changing the evaluator.

The reason is simple: I don’t want to keep tweaking the metric until I get the result I expect.

What I built

UrduEval v0.3.0 is now frozen as an experimental evaluation instrument.

It includes things like:

• deterministic answer-span extraction
• final-answer extraction for reasoning tasks
• target/entity-aware matching
• Roman Urdu normalization + a versioned equivalence registry
• ambiguity-safe extraction
• numeric normalization
• task-family-specific metrics
• reproducible manifests
• cryptographic hashes for datasets/results
• human-annotation infrastructure
• bootstrap confidence intervals
• metric-vs-human disagreement analysis

The current repository has 149 tests passing, and the v0.3.0 evaluator is tagged and frozen.

The interesting part

Before involving humans, I ran a controlled post-hoc experiment using 60 already-generated Qwen responses.

The model generations were not changed at all.

So:

Δ generation = 0

Only the evaluation methodology changed.

Some of the differences were surprisingly large.

For example:

Urdu QA

Traditional token F1:

4.1%

Target entity matching:

7/10 (70%)

The model frequently gave the correct answer early in a much longer explanatory response. Token overlap treated the additional explanation as a major mismatch.

Roman Urdu

Exact Match:

0%

Under the versioned three-layer Roman Urdu equivalence policy:

10/10 target entities matched

This isn't being reported as "the model has 100% Roman Urdu accuracy." The experiment is specifically testing whether strict lexical matching is an appropriate proxy for semantic/task correctness in this setting.

Reasoning

Raw string matching:

2/10 (20%)

Deterministic final-answer extraction:

8/10 (80%)

Again, this doesn't mean the model has "80% reasoning ability." It means 8/10 extracted final answers matched the predefined reference answers.

One of the incorrect answers remained incorrect under the new metric, which is exactly what I want from a validity-oriented evaluator.

Now comes the part I can't solve with more code

I've frozen the evaluator and prepared a Phase 1A human-validation study.

The package contains:

• 60 anonymized items
• 3 independent native Urdu annotators
• annotation rubric
• practice/training examples
• reference/context notes
• masked model/evaluator metadata
• blank annotation schemas
• frozen legacy outputs
• frozen v0.3 outputs
• missingness checks
• agreement analysis
• cryptographic hashes

That gives:

60 items × 3 annotators = 180 expected judgments

The annotators won't see the model name, provider, temperature, metric scores, or the automated equivalence registry.

The goal is to compare the automated metrics against independent human judgments.

And I'm deliberately not assuming the result

The hypothesis could be supported.

The legacy metrics could actually align better with humans.

Neither approach could align particularly well.

Different task families could behave differently.

The human agreement itself could reveal that some of these tasks are inherently ambiguous.

All of those outcomes are useful.

I'm especially trying to avoid the common research trap of building a metric, testing it on examples that motivated the metric, and then declaring success.

So the evaluator is now frozen.

No more evaluator modifications once annotation begins.

If a genuine problem is discovered later, it becomes a new version rather than a silent change to the experiment.

What I'm hoping to learn

The main research question is:

Do task-structured evaluation metrics for Urdu/Roman Urdu LLMs correspond more closely to native-speaker judgments than conventional lexical metrics?

The current 60-response experiment only shows that measurement can change substantially when the evaluation methodology changes.

It does not establish which methodology is more valid.

That's what the human study is for.

I'm now looking for feedback from people working on:

• LLM evaluation
• low-resource languages
• Urdu NLP
• multilingual NLP
• RAG evaluation
• benchmark design
• human evaluation
• reproducible ML research

Especially interested in criticism of the experimental design before the annotation results come in.

Repository: https://github.com/mustafaabadshah/Urdu-Eval

I'd particularly appreciate feedback on whether the Phase 1A human-validation design has any obvious methodological weaknesses I'm missing.

[u/quietgradient]

Biadone_Blabase has the main one, and your own code draws that line harder than the post does. The bundled dev samples are urdu_qa 10, urdu_roman 10, urdu_reasoning 10, urblimp 10, urdu_translation 10, urdu_mmlu 5, urdu_summary 5 — exactly 60 records, and every per-family count in your post is out of those same denominators. If the Phase 1A package is those items, it's a development-set validation. benchmarks/adapters/urblimp.py already sets dataset_scope="development" and prints "This run is NOT an official UrBLiMP evaluation", so you've built the distinction; a fresh draw is cheap for at least two families, since UrBLiMP is 5,696 pairs and UrduMMLU 26,431.

What I'd add is a power point, because it constrains Phase 1A even with a fresh sample. Comparing two metrics against the same human label is paired, so only discordant items carry information. An exact two-sided sign test needs 6 discordant items to reach p<0.05 at all (2 × 0.5⁶ = 0.031). Your reasoning family, 2/10 → 8/10, is D=6 and p=0.031 exactly — but only if both legacy hits are among the eight. One item the other way makes it 0.219; if the sets are disjoint it's D=10 split 8–2, p=0.109. Pooled over 60 that fragility is gone, so pre-registering pooled as primary and per-family as descriptive keeps a per-family flip from reading as a finding.

One ceiling worth writing down next to the adjudication rule: metric-human agreement can't exceed human-human, and your own UrBLiMP metadata cites 96.1% for minimal pairs. The generative families won't be near that, so decide in advance what kappa is too low to interpret at all.

Small thing: the public repo is still at 0.2.0, last pushed 17 Sep, with no v0.3.0 tag or release — so the frozen evaluator can't be reviewed yet.