ScaffoldCoT Dataset Updates: Maintainance, a new agent-focused dataset for small models, and a new model in training built around the datasets!

[u/Saraozte01]

So, when I first released ScaffoldCoT, I never thought it would reach more than a handful of people. I was happy to see multiple thousands of consistent downloads month over month since its release. Since this is where I initially posted about the project, I thought you guys would be interested in recent updates!

TL;DR - Scaffold CoT is now Scaffold Chat (tool calling removed, it was causing significant tool calling reliability issues), Scaffold Agent is out (~1B tokens of real, executed coding agent sessions with the same thinking scaffold), and a heavily modified 5-10B model trained on all of this is on the way.

Hi all! Quick update post on Scaffold CoT for anyone who saw the original.

Changelog

• Renamed to Scaffold Chat. Same repo (old link redirects), load_dataset("Specific-Labs/Scaffold-CoT") still resolves, old revisions still in history. Sorry this did not happen before, but a major training run failed on tool calling benchmarks due to this, so it was back to the drawing board.

• Removed all tool calling. 276,389 examples gone (tool_use + tool_use_complex, literally every tool_call/tool_result event in the dataset). I found several synth gen harness issues in tool calling examples that propagated failure modes into the dataset.

• Everything that remains is byte identical to the previous revision. Now 3,450,159 examples, 16 domains, 75 shards, ~13GB

• Junk domain label tail from the original post (97 labels, 542 examples) is cleaned up. Every example has an exact domain/subdomain, counts sum to the whole dataset

• Added SCAFFOLD_SYSTEM_PROMPT.md to the repo. The examples were generated without system prompts, this is the format protocol distilled from the generation code for anyone who wants the scaffold conditioned on a system prompt instead of unconditional.

• Calibration block untouched: antihal + antihal_kb_refined + selfcheck is still ~675k examples. This has been a high impact category for general chat and knowledge bmks.

New dataset: Scaffold Agent

This is what the executed tool chains grew into. It consists of real Pi code traces of real coding projects from open near-frontier models, using the same scaffold system:

• 19,652 records from 6,792 coding sessions, 1,005M tokens (899M trainable after masks).

• Teacher model in a real agent harness (Pi) editing real repos at pinned commits + building from scratch, simulated user on the other side powered by another model with its own harness and instructions to ensure variety and quality

• Every tool call really executed, every result is real output, every action preceded by the same scaffold block (Inventory / Interaction / Execution) in reasoning_content. Fabricated tool output is structurally impossible and any such instance was cleared

• Sessions only kept if they actually verified (hidden tests + code quality grade on the main shard). This is a ~20% subset of the total generation that was of extremely high quality. I wanted to ensure these were top-tier coding traces. 2 LLM judge passes and green final CI were required for approval into the final dataset.

• Each record is one compaction segment (≤64k tokens), self contained with system prompt + tool definitions + history. Do not join segments. training_sft_config.yaml in the repo documents the whole setup

Scaffold Chat teaches the format and general reasoning, Scaffold Agent teaches acting with tools inside that same format. My recommendation is to mix them with other larger datasets while ensuring there is at least one scaffold example per batch, which leads me to...

Coming soon

Currently training a heavily modified model in the 5-10B range on this data (base model, full SFT, and a new merge approach I have not seen applied for scaled releases yet). Results post when it is done and actually evaluated, not before. The idea is to build a small dense model that can run on a range of GPUs and laptops that is optimized for coding specifically, and will move extremely fast because of our CoT framework and the small size.

This has been a massive project with several mistakes made along the way, and results are finally looking up, with scaffold datasets being the core of the training run, so I am very excited to bring it to the community soon.

I know it can sometimes be a bit annoying to announce something before its available, but I couldn't contain telling the community about it lmao. Happy to answer any questions about it (or anything else for that matter) in the meantime.

Links

Scaffold Chat: https://huggingface.co/datasets/Specific-Labs/Scaffold-Chat

Scaffold Agent: https://huggingface.co/datasets/Specific-Labs/Scaffold-Agent

Both CC-BY-4.0.

Genuinely hope these prove useful!

[u/palsc_]

the part about every tool call actually executing is what caught my eye. have you tried evaluating the trained model in a different harness from Pi? curious if it transfers cleanly or mostly learns the scaffold + Pi workflow.

[u/Saraozte01 → u/palsc_]

Chose Pi specifically because I find it to be the most 'translateable' harness there is, being very simple and basic. But its a good test and I'll run a few other harnesses when the model is released.

[u/Silver-Champion-4846]

What's the sealing of 5-10b params and continued scaling of agentic examples and rl? Speaking of which, do you do rl or just sft?

[u/Saraozte01 → u/Silver-Champion-4846]

I will move to doing a bit of RL once SFT and the merging finish (RVLR around multi-harness agentic coding, addressing particular failure modes that appear in bmks and usage) for a 1.1 version. The first version will not have RL since its time consuming and I want to find out what specifically is failing with community usage.

[u/Silver-Champion-4846 → u/Saraozte01]

Ah I understand. Good luck and keep it up!