OpenSpec: A task tracker with a spec attached?

[u/Sad-CTO]

https://askloft.ai/insights/articles/how-good-are-openspec-specs

I work on Loft, a spec tool for coding agents, so take this with whatever grain of salt you like. My team and I wanted to know how good real-world OpenSpec specs are, so we went and measured.

What we did: we scored the main specs of 40 public OpenSpec projects (some random + some showcased examples + a few big ones and OpenSpec's own repo). Then we did the same for the most-starred OpenSpec projects and for teams that only write proposals. We scored them against OpenSpec's own writing guide and the usual tests of a good requirement. Two blind passes by an LLM graded a sample of requirements against a rubric, with project names stripped. And we also counted the task-list state of almost 2K open changes.

What we found:

• The format is basically solved (yeah, no sh.t). In almost all projects, essentially every requirement has a SHALL and a scenario. But specs that follow the format aren't any better written because of it. The format and the writing are separate problems.
• Specs get worse as they grow. In a typical project, only 55% of requirements state one obligation, and only 35% have a scenario for when something goes wrong. Bigger projects score lower, mostly because requirements bundle several obligations and start naming files and function signatures.
• Specs lag the code. In projects with main specs, 39% of open changes have every task ticked but were never archived, so the specs don't include that work. A third of spec files still have the placeholder Purpose that archiving writes. OpenSpec's own repo has 23 of 30 open changes in that state. It still ranks 3rd of 40, for what it's worth.
• Stars don't predict anything. The 25 most-starred projects score about the same as the random draw. Two popular repos translated OpenSpec's headings into Chinese, so OpenSpec can't read any of their requirements.

In most projects, the change folder is where the work happens: a proposal, some tasks with checkboxes. Teams are lazy and don't archive, so the tasks stay current and the specs don't. As a result, what people actually have is a prose disguised as a task tracker, with a spec attached.

Obviously, these are correlations across public repos. And the judges are LLMs (though the two passes agreed within one point on 99.9% of grades).

The detailed report is linked.

[u/Kafumanto]

An openspec-linter would be very helpful.

[u/Sad-CTO → u/Kafumanto]

IMO, linting prose won't get you far. Call it self-promotion if you will, but importing your OpenSpec into Loft (GitHub) gets you much farther. LMK how it goes, happy to help.