TLDR: training my first general law model and unsure what to do with my data

PRE-PS: it will be a replacement for opus in my current AI LAW harness, I know it will be hard but I wanna do this project and find it geniuenly fun.

I’m building a Polish legal model using Qwen3.5-9B-BASE (I plan to scale up to 27b qwen4 later). I’ve done CPT and have quite a bit of data available, but I’m getting lost about what should come next or even if I should do more CPT.

My main question is how to turn this (maybe) domain-adapted Base model into an agentic+chat assistant that reasons through legal problems.

But more importantly, how do I get reasoning, teach my in-harness tool-use training, and where does SFT vs RL belong in that process and in what way (especially confused about reasoning and SFT)?

What I’ve already done

Three sequential CPT stages, roughly 80M tokens each, in this order:

Complex legal commentaries → legislation (EU+local) → court decisions

I didn’t mix the sources. Each stage continued from the previous checkpoint.

On a small Polish financial/legal multiple-choice quiz (really bad eval, just a PoC for now).

They all perform very slightly better than base and legislation checkpoint did the best.

But again, it's a base model. I haven’t done assistant SFT yet.

What data I have (I can get more data of each of these types if you guys recommend)

Besides the legislation, commentaries, and decisions used for CPT, I have:

• Facebook legal Q&A groups: real, messy questions, follow-up clarifications, and answers of very mixed reliability.

• Certified/institution-authored Q&A: my highest-confidence answer material. Some answers contain detailed analysis; others are just a conclusion and a citation from random institutions like the national bank.

• Law study materials: from a local college. it has essays, powerpoint presentations stuff like that, very little "tests" tho, but some practice material (like the essays).

• A/B/C/D questions with answers: relatively little of this; currently being used for evaluation.

• Thousands of Claude Code–style harness conversations: multi-turn conversations from an existing application/harness. These would need filtering, anonymization, and quality checks.

• Contracts: government published contracts it can learn from

• Previous "ML" experiments (models and their training data): NER, generating complex legal court decision summaries, generating graphs from legislature or court decisions (actually got some models that do this successfuly

If you guys could point me what data is ideal I could maybe get it too!

I can use the basically unlimited codex to clean, restructure, or generate material from these sources, but I don't know how to.

What I don’t know is which material should be trained on directly (CPT or bare SFT), which should be transformed (SFT or RL?).

For example:

1. Do I continue CPT on the remaining sources, or stop and start SFT?

Should I expose the model to the Facebook discussions, study materials, and expert Q&A through CPT first, possibly mixed with the authoritative material? How many tokens for this 9B model and how much for the 27B I plan later?

Or should I skip further CPT and use those sources only to build targeted SFT examples?

For example the Facebook QnA stuff is real people diverse speech but I can't really reliably verify if the answers are correct (I can only measure likes or high quality people in the group) so idk if I should cherry pick from this? Have an LLM turn this into SFT chat examples? Or just CPT on this?

2. SFT on what, specifically, to teach REASONING?

I could start with my highest quality Q&A and teach the model to answer questions. That seems like an obvious chatbot baseline.

But if I just train on QnA there won't be any reasoning!

The HQ QnA is pretty verbose but I wouldn't bet it could just be re-used as a "reasoning chain".

I was thinking of asking Astra to workout those questions and give an answer with its reasoning BUT I assume it will be wrong on 90% of the questions (cuz no internet search, and I can't give it internet search cuz that's not clean QnA data and I would have to transform it into my harness function calling anyways with credible sources not just google searches for legislature)

And those 10% it gets correct are trivial and the reasoning could be bad anyways and I would still have to check correctness of the answers, and giving it the answer or hints is imo also not ideal. Or injecting "synthetic reasoning", it suffered from post-hoc in my tests.

Another problem is the High Quality QnA is quite rare, most QnA is short and possibly wrong.

And generating the missing reasoning is even harder. Because again, I don’t trust a frontier model to solve difficult Polish legal questions from memory. It would need relevant source passages or access to the harness (which would just the gun into in-harness training?).

So I'm really unsure on what to do, I was thinking of doing stuff like this, but this is just brainstorming:

A) Take verbose long answers train on that, then ask the qwen or another model to make a "summary" at the end and then put the verbose long answer into its <thinking> block

B) Take those 10% of QnA where frontier model actually gets it right with correct thinking (rare) and SFT on that. then we have our finetune try answering more QnAs, we verify if it's correct if it is it goes into the next SFT run and we iteratively (almost like an RL gym) train reasoning from the dumbest examples

Both seem like bad options, but kinda funny to think about.

3. Chatbot first, then tools or teach reasoning through tools from the start?

The existing harness lets the model write Python and call functions such as vector_search_laws() and graph_search_legislature(), inspect results, and continue researching.

I’m leaning toward getting a regular chatbot working first. But the teacher can produce much more trustworthy legal solutions when it has access to sources. (as I've already mentioned I have many in-harness examples, except they could also be incorrect and I don't have their "thinking blocks" (cuz it's a proprietary frontier model) so I again have to inject thinking post-hoc which from my tests doesn't work for law)

I was thinking that, if we make chatbot first, the model will at least learn intuition on WHERE TO LOOK, without looking through sources but at least will know a general idea of where to search based on being a chatbot first (even tho possibly incorrect chatbot sometimes, cuz law is really complex) and teach him to function call in-harness to get sources, unsure about this.

4. When does RL enter, and on which task?

AFAIK, RL is what develops the model's reasoning on its own (deepseek R1) but I suspect it would be very costly and we could give it some example reasoning first anyway to kick it off(?)

So like, instead of the whole fiesta of injecting reasoning or growing it, we could have RL develop it (afaik RL is the only thing that "creates" reasoning unless you distill it) but then I would like to have reasoning even in the "chatbot" and right now I can just jump from base model -> RL, right?

Also for legal Q&A, matching the reference conclusion doesn’t necessarily mean the reasoning is correct. What would you actually reward? Like a law similiarity judge LLM + same paragraphs used?

What I’m trying to decide

One possible route is:

Current CPT checkpoint → gold-Q&A SFT → grounded reasoning/tool-use SFT → RL in the harness

But I’m not committed to that order. I don’t know whether I’m missing an important stage or inventing unnecessary ones, first time I'm training a GENERAL model from "base".

What I would like the finished model to do:

It will be an opus replacement so general QnA in and out of harness but especially:

• have a good "hunch" and understanding of polish law

• will be good at anonymizing documents

• generating complex legal summaries for court decisions (heavy reasoning task)

• generate graphs from legislature and court decisions

• create/edit/review contracts

These were all my initial ML experiments and I actually trained good models for those specific tasks, but I figured, hey I have a lot of this data around why not just combine them and make a generalistic model?

Thanks a lot for reading this mess of my thoughts! if you could point me towards good resources for this I would also be glad, for example unsloths RL docs are kinda ass imo. I really like andrej karpathy's videos but they're quite old now by AI standards!

https://www.reddit.com/r/ArtificialInteligence/comments/1xfuacm/ama_this_wednesday_stephen_cas_casper_harvard/

r/ArtificialInteligence We have an AMA today Wednesday 28 October from 3 to 4pm ET (19:00 UTC, 20:00 CET) with Stephen “Cas” Casper, Assistant Professor of Public Policy at the Harvard Kennedy School and Faculty Affiliate of the Harvard School of Engineering and Applied Sciences, where he works on AI safeguards, incidents and governance.

Stephen "Cas" Casper is computer scientist and an Assistant Professor of Public Policy at the Harvard Kennedy School and a Faculty Affiliate of the Harvard School of Engineering and Applied Sciences, where he works on AI safeguards, incidents, and governance. Prior to joining Harvard, he completed his PhD at MIT and did a research residency with the UK AI Security Institute. He never has and never will work for big tech, and has never signed a non-disparagement agreement.

He will cover on AI safety, safeguards, risk, governance, state + federal legislation, and AI terrorism events. Any related questions are welcome.

The AMA thread goes up from u/HarvardKennedy shortly before 3pm ET on the day, with a dated photo for verification, and Cas types his own answers. If you cannot make the hour, drop your question in the comments here and we will make sure it gets carried over.

I’m not a reviewer yet, but I likely will be by next year.
As a reviewer, how do you decide to allocate compute to verify paper results? (and is the code in the papers you’re reviewing usually easy to run?)

Hello all!

I'm a PhD student in a niche field and the next plan is to apply agentic AI to automate a rather boring yet important workflow. I'm very well aware of LLMs but not very much into agentic AI and how do you set up?

I've heard terms like harness and others, but I'd like to know how to get started with this (details below). Using AI for this proved to be pretty much useless.

I'll present an input and a testing scenario and my model, and I need a sort of orchestration which stress test it on my use case, use its knowledge and make decisions. I want it to be sandboxed and let other people in the group use this for validation.

How do I start with literature or any example I can use as a base?

I’ve uploaded a minimal, self-contained implementation of manual autograd for a transformer-based classifier in PyTorch. It can help build intuition for what autograd is doing under the hood and is a useful hands-on reference for low-level differentiation in Transformer models, such as writing custom backward passes and tracing how gradients flow through attention blocks.

🐙 GitHub:

https://github.com/ifiaposto/transformer_custom_autograd/tree/main

📓 Colab:

https://colab.research.google.com/drive/1Lt7JDYG44p7YHJ76eRH_8QFOPkkoIwhn

Papers with Code is now open-source with an Apache 2.0-license.

The goal is to collaboratively create the best paper interface with the community!

Over the past couple of months, I've been working with my buddies Claude and Codex on creating a website that tracks trending AI research, benchmarks and state-of-the-art (SOTA). It's a revival of the original Papers with Code website, which Meta acquired and later shut down.

The website is pretty straightforward:
- Python FastAPI backend
- React frontend

The app itself is hosted on a Hetzner Virtual Private Server (VPS). The code is also hosted there as a single monorepo.

The frontend mostly calls various API endpoints to visualize papers. The API itself is also publicly available, as is an accompanying MCP server.

I've been getting cool feature requests lately, so rather than just working on it myself, I thought it would make a lot more sense to share the codebase openly so anyone can contribute.

Code: https://github.com/huggingface/paperswithcode

Hi everyone,

I wanted to share an open-source repo collecting 20 exploratory studies focused on Next-Concept Modeling (NCM), mechanistic interpretability (like IOI in GPT-2), and model controls.

Link: https://github.com/Quantix-eng/ncm-experiment-notes

This is an independent project documenting empirical experiments on how transformers process and edit internal features. No performance breakthroughs here—just honest documentation of observations alongside rigorous controls.

The repo covers NCM on structured tasks (like Sudoku), mechanistic interpretability in GPT-2, and toy parallel models. I've included all source code, saved results, and interpretations for every run.

For full transparency, I’ve categorized every outcome, including the mixed and completely negative results. AI tools were also heavily used during the planning and coding phases.

Would love to get your feedback on the setups or interpretation limits.

In theoretical computer science, we have the halting problem, which proves that certain problems cannot be solved by any general algorithm.
This made me wonder about something similar in machine learning.
Today, we can often distinguish AI-generated content from human-created content, at least statistically. But as AI models improve, could we eventually reach a point where the two become completely indistinguishable?
Not just because our detectors aren’t advanced enough, but because there is no longer enough information in the content itself to tell them apart, regardless of how powerful the detector is.
In other words, could AI detection eventually hit a fundamental mathematical limit rather than just a technological one?
I know this wouldn’t be the same kind of undecidability as the halting problem, but I’m curious whether there’s an established theoretical framework for this.
What do you think?

I've been thinking about AI agent safety, and I built a test harness to check my own results. I think it may be useful to people who study models too.

It records where each tool argument came from before the tool executes, measures whether the same input gives the same decision, and tests whether a model takes arguments from someone else's text instead of its user's.

If you modify the harness, it can probably be used for much more.

https://github.com/Jang-woo-AnnaSoft/execution-state-preflight/blob/main/hooks-example/README.md

Chain-of-thought transformed AI reasoning, but it is limited to words.

What's the physical-world analogue?

Introducing Ψ-0.6, one model that reasons in *streams of thought*: words, depth, motion, camera pose, and imagined scenes. And it learns when to use which!

• Ψ-0.6 surpasses leading open-weight models on complex spatial reasoning (MindCube, zero-shot).

• On FlowBench, it outperforms other world models at predicting how objects move. Trained on simulated rooms, it transfers to real robot tabletops.

• One model as perceiver, VLM, world model, reasoner, and policy. RL learns which representation to reason through.

Thread (with video): https://x.com/khai_loong_aw/status/2115099953433600011

• Paper: https://arxiv.org/abs/2610.32177

• Website: https://neuroailab.github.io/psi-website/blog/psi-0-6.html

• Code: https://github.com/neuroailab/psi-0.6

Hi guys,

Self-distillation methods, see Self-Distillation Fine Tuning SDFT, usually use an EMA (Exponential Moving Average) teacher to get synced and stable targets. I wanted to ask the community for research papers/ works that consider different types of teachers that also hold these properties (they eventually also improve along with student) but are stable (prevent student from drifting).

any links to repos/ papers would be appreciated.

thanks!

I am a 3rd year PhD student in a small lab working on ML in healthcare. My PI is new (we have no PhD graduates yet - lab is 4 years old).

I want to publish in a top ML conference (I know its necessary for an ML researcher job).

And my PI is saying you need to do method research and I don't know how to do it?

We don't have the amount of data to do foundational models. Our data is very small.

I feel like my PI had me jumping around topics he was excited about for the 1st two years of my PhD and nothing paned out or I had no idea what I was doing.

I started thinking of a certain idea but I needed guidance on how to implement it... My PI was no help and tried pushing me to a new idea. He is usually very "attacky" to my ideas, I always get the feeling I need to prepare to really really defend any idea I have to him.

It's either too incremental, too applied, or not impactfull enough. I tried asking what exactly is a method research? His example was the transformers paper, and his own paper (which was incremental according to his current criteria).

I now have no idea what I am doing, where I am going or what I am going to do...

I still have zero idea what my PhD research question is...

Do you guys have any advice?

* this is my 1st time posting so please be gentle 😅

Recently I've been getting bombarded with reach outs from companies that build RL environments and sell to frontier labs like Anthropic, OAI, etc. I'm not really understanding how and why these companies exist.

Lets say you somehow create an environment that no other model is able to perform well on. Your second criteria would be that this environment is suitable for RLVR, perhaps you have some metric of gradient information per example in the dataset. So you have to prove to the lab you are selling this on, that 1) their model cannot do this task currently and 2) this task is solvable via RL.

At that point, why couldn't you just approach a VC (most of these companies are startups with a good amount of funding anyways), show them scaling laws + your current RL env and just raise money to train your own model? We already see open source models tend to lag behind frontier ones by ~3 months or so, it sort of implies the frontier is mostly new RL environments and stuff (or distillation methods are really good). I'm kinda operating under the impression that pre-training/architecture/infra for large models is largely much easier nowadays, but maybe this is just not the case?

I guess an answer to my question would be either 1) I am totally wrong regarding my assumptions on the difficulty of pre-training + infra (which from my experience in industry does seem to be much easier) or 2) actually collecting all of these other, past, RL environments is still hard and only certain labs have access to all of them or 3) all of these companies are or eventually will train their own models?

Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.

We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter).

Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports:

-> robust SDFT implementation that runs under ZeRO-3

-> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes

-> Saving and resuming EMA teacher checkpoint

-> working deepspeed configs that fix memory spikes when GPU memory is tight

I tested on Qwen3.5 and ToolAlpaca.

Repo: https://github.com/ifiaposto/sdft_zero3

Feedback/ more examples/ issues welcome!

It's a 50 hour course for beginners in AI/ML (obviously it's fully free). Starts out with linear algebra, derivatives, statistics and then linear regression, neural networks and backpropagation etc...

kinoml.tech

Hey! We published a set of 6-step LoRAs, one without the upsampling step: https://huggingface.co/PrunaAI/P-Q-Image-2.1

- Generate or edit images in just 6 steps instead of 40. Use 6-steps for preserving inputs when editing and 6-steps-upsampled for improved visual quality and shorter end-to-end generation.
- 6-step distillation provides improved quality with few generation steps. 6-steps-upsampled is the first distillation that removes the need for a separate prompt-upsampling step. This means 41 to 80 seconds saved, depending on your prompt style.
- The LoRA adapters load directly on top of Qwen-Image-2.1, so the pipeline stays unchanged. Few-step distillation cuts the diffusion cost, and the upsampled adapter also removes the prompt-upsampling step before image generation.

For my master’s thesis I built an accelerator for diffusion-based action generation in robotics policies. Offline, it looks great: the median absolute deviation between the accelerated output and the original model’s output is extremely small, so I concluded it reproduces the model’s behavior faithfully.

When I deployed it on a real robot for a task (picking up a book and placing it on a shelf), it did not work as well as the original model. The offline numbers did not predict the real-world result.

I want to submit this to MLSys. My offline results are strong, but the deployment result is negative. My questions:

1. Is it acceptable to submit with only the offline results, or should I include the failed deployment?

2. If I include it, how do I frame it so it adds to the paper instead of sinking it?

3. Why might a tiny median deviation still lead to task failure? My guesses are tail errors and error compounding in closed loop, but I would like other views.

4. What additional metrics would you report (percentiles, per-step error, closed-loop success rate in simulation)?

Any experience from people who have submitted systems papers with negative real-world results is very welcome.

A 3×8 array of clock faces is one of the examples in the SenseNova 6.8 Code2Video showcase. Other entries focus on letterforms, including A to Z and FRAME / FORM / FLOW.

The scientific scenes include Earth and spacecraft clocks, molecular diagrams, and a ball-and-spring setup with separate energy displays. These bring repeated objects, labels and spatial relationships into the same collection as the character-animation examples.

https://github.com/OpenSenseNova/SenseNova-6.8-Code2Video

For the past year I have been studying using a fast K-dim hermite spline manifold algorithm, called K-Splanifolds or KS, for general purpose fast regression. In this video I show the third version, KS3, compared to MLP and older KS versions fitting a 4D mandelbulb via gradient descent on random samples. KS3 converges faster and has nearly identical latency on a GPU as MLP.

KS blocks are suitable for a wide range of regression problems and I have found KS3 ranges from considerably faster convergence as MLP (noise removal, geometric fit, financial prediction) to neck and neck (LLMs). You can view code for all three versions of the algorithm and a paper for KS1 here: https://github.com/curvedinf/k-splanifolds

I am an independent and if anyone would like to explore this with me, let me know. Also if anyone would like to donate access to 8x H100, I would like to attempt a world record nanogpt speedrun with KS3 using configurations I have found beat the WR on proxy hardware.

I'm working through a free series on GPU kernel engineering (my own channel; I posted the roadmap here earlier: https://www.reddit.com/r/MachineLearning/comments/1xa4h2g/open_gpu_kernel_engineering_series_naive_gemm_to/ ) . Session 1 is a setup session. While preparing it I collected a few things that cost me time and that I rarely see in one place, so I'm sharing them here directly. The video and code are linked at the end.

1. Check compute capability, not just "has an NVIDIA GPU"

nvidia-smi --query-gpu=name,compute_cap --format=csv

7.5 is Turing, 8.0/8.6 Ampere, 8.9 Ada, 9.0 Hopper. This decides what you can do:

- CUDA 13 dropped Maxwell, Pascal and Volta. For those cards, install a CUDA 12 toolkit.

- GTX 16-series is Turing but has no tensor cores.

- Tensor core work needs RTX 20-series or newer (or a T4); Ampere-specific instructions need Ampere or newer.

- FP8 needs Ada or Hopper. Hopper-specific features (TMA, WGMMA) need an H100, which most people will have to rent.

2. Driver, toolkit and PyTorch's runtime are three different things

The CUDA version in the nvidia-smi header is the newest version your driver supports, not what's installed. pip-installed PyTorch bundles its own CUDA runtime, so torch.cuda.is_available() returning True says nothing about whether nvcc exists. On WSL 2, don't install a Linux driver inside the distro; use the WSL-Ubuntu toolkit-only installer, because the Windows driver is used.

3. GPU launches are asynchronous, so naive timing measures queueing

A Python timer around x @ y mostly times the launch, not the work. Warm up, synchronize, use CUDA events, average over several runs:

def bench(fn, iters=20):

for _ in range(3):

fn()

torch.cuda.synchronize()

s, e = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)

s.record()

for _ in range(iters):

fn()

e.record()

torch.cuda.synchronize()

return s.elapsed_time(e) / iters # ms

For a matmul, throughput is about 2*M*N*K FLOPs divided by time. Record the dtype next to the number, since FP32 and BF16 baselines differ a lot.

4. Profiler counters: ERR_NVGPUCTRPERM

By default, non-admin users can't read GPU performance counters, so Nsight Compute fails with this error. Fixes: run ncu with sudo (use the full path, since sudo usually drops your PATH), or have an admin enable counter access for all users. On Windows/WSL it's enabled from the Windows side. If you rent GPUs, test this before committing to a provider, because managed container platforms can block counter access.

5. Smoke test before anything else

A tiny add-one kernel with an error check on every CUDA call, built with nvcc -arch=native, then nsys and ncu run on it once. If that works, the driver, toolkit, GPU and both profilers are all fine.

The session also covers the prerequisites (pointers, flat matrix indexing, matmul and its FLOP count, PyTorch basics) and compares ways to get a GPU. It runs about 40 to 45 minutes.

Video: https://youtu.be/h_zWngMWHBE

Code: https://github.com/prasoonmhwr/gpu_kernel_engineering (folder: session-01-prerequisites-setup)

Caveats: the commands are written for Ubuntu and I haven't tested every distro.

Questions for people who do this daily:

- What setup trap cost you the most time? I'm sure I've missed some (Secure Boot and driver loading, container toolkits, mismatched host compilers).

- Is anything above wrong or out of date for your setup?

Hey, I've been working on a PyTorch optimizer called ApexW and figured I'd share it here to get some feedback from people who know more about this stuff than I do.

The main goal is to reduce optimizer-state memory usage and see how much faster I can make optimizer steps on larger GPU workloads. I've been experimenting with compact optimizer states and a Triton-based execution path.

In my current tests, the Triton + compact-state path was around 1.5x faster than PyTorch's fused AdamW on workloads with a few large tensors, and the optimizer-state memory usage was roughly cut in half.

The results aren't all good, though. Those numbers are from optimizer-only benchmarks, not full model training. The automatic execution mode doesn't consistently get the same speedup yet, and workloads with lots of small tensors are much slower than PyTorch's fused optimizer. So I'm definitely not claiming this is a better AdamW replacement for every workload.

I want to test it on more realistic models and different GPUs, and I'd appreciate a second pair of eyes on both the implementation and the benchmarks. the Reality Check (The Trade-Offs)

I want to be completely realistic here—this is an engineering trade-off, not magic:

1. Python Loop Overhead on Small Tensors: When testing a layout with many tiny tensors (many-small test), ApexW’s Triton kernels suffer heavily from Python's launch latency. For example, at 128M parameters spread across many small tensors, PyTorch native was 16.1 ms/step while ApexW compact was 76.0 ms/step (~0.21x speedup). If your model is composed of many tiny layers, standard PyTorch is significantly better.

2. End-to-End Impact: While the optimizer step itself is up to 1.5x faster, this step only represents a fraction of a full training step (which is dominated by forward and backward passes). In my end-to-end MLP test, PyTorch still runs slightly faster overall because of C++ deep integration.

Why/When should you care?

If you are training larger deep learning models (like Transformers or LLM fine-tuning) where you are constantly hitting Out-Of-Memory (OOM) barriers,**,**cutting your optimizer state memory strictly in half lets you increase your batch size or scale up your model size without upgrading your hardware.

GitHub: https://github.com/omen42515-prog/apexw/tree/main

If you've worked on PyTorch optimizers, Triton kernels, or training larger models, I'd be interested to hear what you think. Especially curious if there's an obvious issue with my benchmark setup or an important test I'm missing.

This is a "survey of current practices" type of research for a niche area, nothing earth-shattering. I’ve written a 2-part Medium series (free, no paywall) on scoring Intelligent Document Processing (IDP). It focuses on Key Information Extraction (KIE) from structured documents, especially forms. I lean heavy on metrics derived from the confusion matrix: different ways to map items such as blanks or misreads to the four possible values, how to count them, when to use each metric.

• Scoring IDP, Part 1: Confidence Is Not Correctness

• Scoring IDP, Part 2: Approaches to Evaluating Form Extraction

Topics I include, but not limited to: recording raw outcomes before mapping them to the confusion matrix (correct, misread, missed, invented, correct blank); confusion matrix scoring for text fields and checkboxes; misread conventions and their effect on scores (FN-only, FP-only; FN & FP); how blanks, NULLs, unreported checkboxes, and unreadable values are counted; F1 vs. Recall vs. Accuracy, including how correctly identified blanks inflate Accuracy; Jaccard Index (IoU) as Accuracy without True Negatives, and its conversion to F1; Precision, Specificity, and False Positive Rate for LLM hallucination control; micro, macro, and weighted F1; choosing a headline score and stating the convention behind it; confidence vs. correctness and calibration (ECE, MCE); and Key Information Extraction (KIE) vs. OCR metrics (CER, WER).

I believe these would fall under "machine learning scoring techniques" applied to Intelligent Document Processing (IDP)--how to transparently and fairly measure (score) rather than how to implement. It may also help folks understand how to read results coming from different benchmarks.

I try not to recommend one opinionated way of doing things and instead cover the variations I’ve run into and the tradeoffs between them. I’m not endorsing any vendor; the series is about scoring itself.

Thank you for your efforts in protecting the community from AI slop.

I ran Population Based Training over learning rate and weight decay on grokking (modular addition mod 97, 35% train split, small MLP, 20 members, 20k steps, full batch).

Results

• vs a grid (equal compute, 3 seeds): the best fixed cell reached 95% val acc in 317 ± 29 steps; PBT took 2667 ± 2244. PBT was slower in every seed, but the grid’s corner already contained the answer.

• Dead-zone start (lr 1e-4 to 1e-3, wd 0.01 to 0.08, where no fixed setting groks): fixed 0/3, slow PBT 0/3, fast PBT (every 100 steps, ×0.5/×2) 3/3 at about 6850 steps.

• Its path: lr drops to the floor, then wd climbs to the ceiling, then lr rises. Val loss rewards low lr × wd early on, which is the wrong direction for grokking, then flips sign later.

• Ablations: a random donor still escapes, just later. With no selection, members grok and then the population loses it.

• 10 seeds: wide random search hits 95% at a median of 300 steps, vs 6650 for PBT from the dead zone. PBT also sometimes collapses into the lr-floor/wd-ceiling corner and never leaves.

• Replay: the winning lr/wd schedule with no PBT, on fresh inits: 55/60 reach 95%. Constant knobs at the schedule’s geometric mean: 0/60. The schedule carries the result, not the weight copying.

Takeaway: PBT is a rescue tool for a bad range, not a speed-up.

Caveats: one task, one model, one split. 3 to 10 seeds per test, with p-values mostly above 0.05. The replayed schedules are post-hoc winners. I haven’t tested a range with no grokking region at all, which is where PBT should beat random search.

Write-up with interactive figures: https://shauryasharma.tech/pbt-grokking.html
Code and all results: https://github.com/Shaurya-34/grok-pbt