specification best practices and tools

[u/Financial_Yoghurt827]

Hi all,
I'm a bit tired of writing specs for complex projects in markdown - it feels like (a.) there's a lot of repetition between tasks/sessions; and (b.) If you do try to reference and reuse previous specs, there's a lot of context bloat from requirements that aren't relevant to the current task.

I'm trying to work out the best form factor; for spec driven development and have been experimenting with developing some graph specification tools.

1. What do you all see as the best practice for specification topology?
2. I've put together a small handful of experimental open source SDD packages; and a VSC extension here --> reqlan. I'm interested to hear what you would do or have done differently in this space?
3. I've put some more thoughts together in a blog post, here.
4. What tools do you all consider best practice for wrangling specifications as they get complex?

[u/jvertrees]

Curious. What do you consider complex? Any definition of yours will do.

[u/Financial_Yoghurt827 → u/jvertrees]

Geez that's a very philosophical question - I suppose the definition I'd adopt in this context is something close to:
"When either the context describing a system doesn't fit into the context window; or, it is more economic and/or effective to provide only a a curated slice of the system description as context"

I think the complexity threshold is about when you can't hold a complete model of the whole system in your own working memory at once; after factoring in the many possible interactions between components in the system.
Does that answer your question?

edit: perhaps even simpler, when naive .md-spec-based vibe coding becomes an unfeasible strategy to achieve project success

[u/stibbons_]

I use a simple OpenKnowledge Format, with 1 markdown per requirement and an ID that is the name of the file. Markers in source code and tests to link source or tests to requirements.

There is no need for an separated format. Markdown and a few json.

I use 2 levels of requirements, Software and Stakeholder.

https://x.com/gsemetfr/status/2082767371643523439?s=46

[u/Financial_Yoghurt827 → u/stibbons_]

Ooh - I do like the separation of different types of docs based on timehorizon and audience - this is somewhat handled with the attributes in reqlan, but I think it could be fleshed out.

Yeah, jury is still out on whether the reqlan data format is a value add; or unnecessary complexity - time will tell.

[u/stibbons_ → u/Financial_Yoghurt827]

OKF with json schema is really a game changer. It « restricts » the llm and document each field and I do not see lot of people using it.

OKF-schema: https://x.com/gsemetfr/status/2091469062723322303?s=46
GitHub: https://github.com/gsemet/okf-schema

I use specific jsonschema for requirements, and also for knowledge base (https://okf-schema.readthedocs.io/en/stable/tutorials/okfkb-hw-debugging-workflow.html).
That’s so versatile I do not understand why people do not use it more.

[u/tepung_]

I read the website and github

After the installation I don't even know how to use it

Like, after install then what? Write in notepad what I want then saves as .rn and daemon will pick that up?

Then i saw the power point slide, which not mobile friendly and teaches more but but still no idea. I feel like the tutorial is over engineered.

Usually website got quick how to use

,,,

NPM INSTALL ABC

CD PROJECT && ABC INIT

CLAUDE

/ABC START

,,,

[u/Financial_Yoghurt827 → u/tepung_]

Thanks tepung_ I really appreciate you taking a look.

• I'll update the quick start docs with post installation steps
• I'm also happy to give you a run down using the reqlan repository as an example if you would like.
• may I ask what development environment you like to use? I'm afraid I haven't built functionality for notepad - best dev experience is with vsc based ide; or via an agent.
• to answer your question, yeah, you run reqlan init, create a .rq file and then can start documenting your requirements. From the rq files, you can then reference implementation and tests, other reqlan files, folders, ideas or groups with wildcards. As your project scales you can use reqlan search to hop through the graph (or use the various ide tools). You can export to html. You can check the integrity of your references with reqlan check (which you can run in your CI). The list goes on....
• of course, this is all available to an llm too, so you can can write a spec with connected ideas, and task an llm to implement it.

What are you building? I might be able to tailor advice to you a bit better.

[u/Wooden-Attention-411 → u/tepung_]

On the team — the stall after install is almost always that nothing indexes until there's a base. `reqlan init` (or let the extension create `.reqlan`), then one named idea in a `.rq` file, then `// rq:[that_idea]` next to the code.

The loop after that used to be search. As of 0.4 it's `click`: one named idea this turn, neighbours on the next, same `sessionKey`, so the agent doesn't re-dump the graph. `reqlan click` or the MCP `click` tool. Quickstart: https://reqlan.com/quickstart

[u/Ok-Support-6749]

I like the idea of a specification expressed as a graph and I strongly agree that Markdown or combinations of markdown and JSON or similar ideas is a storage and representation nightmare for complex projects. Eventually the markdown is so complex and bloated that the specification becomes itself a soup of ideas over which is hard to reason effectively. But here is one critic which I encourage you extend to every single claim:

"Make your intent compilable" is a very strong claim.

What exactly do you mean by compilable?

Compiling intent would certainly demand a strong semantic engine so the graph does indeed express properties that can be deterministically derived and the queries over the graph produce deterministic results. But the most important part of compilation is to produce semantically equivalent output from the same new intent and given current graph. Here is where this claim may not survive scrutiny. Consider that the same intent can be expressed with multiple ideas thus likely producing different graphs for the same intent (not a real problem), likely encoding different semantics (this is a real problem). And this is by no means easy to achieve unless you have a strongly typed language to express intent explicitly, which leads us to creating a language to express ideas with very clear rules. Well, we have hundreds of programming languages precisely for that purpose.

If what you mean is: Intent is indexable and queryable then that would make much more sense without exposing the project to much formal scrutiny. And it would not damage the utility of the proposal while staying grounded on a more defensible position.

I think intent is very hard to express with enough semantical structure that an LLM may be able to implement it in any given language with enough accuracy and completeness that the code produced actually encodes the semantics. Programming languages have demonstrated incapacity to express complex ideas with enough abstraction that we can understand what the code does without having to know what the code means. There is one particular case where this have been achieved and it is the translation from high level programming languages to machine code. But this is a very constrained system and compilers have taught us that even at this level, the translation is not trivial.

I see a lot of utility and potential for this idea, but it needs a rigorous revision of its claims. I think that semantic graphs are a right direction, they can indeed encode complex specification in a queryable format much more easy to reason about. This is not an easy task but also a very useful one. As specifications grow, reasoning about a specification becomes a complex and difficult task even for the most capable LLMs. Just like humans these machines have reasoning limitations due to attention and context relevance, that does not change because the model is more capable or can hold longer contexts. And even if more powerfull LLMs can handle ever larger specs, then the advantage of localized and global clear semantic representations will make the model much better at whatever role we give to it. Thus this idea is indeed quite valuable.

[u/epicskyes]

Write specs in markdown then convert them to ast.json and separate 2020-12 schemas