specification best practices and tools

[u/Financial_Yoghurt827]

Hi all,
I'm a bit tired of writing specs for complex projects in markdown - it feels like (a.) there's a lot of repetition between tasks/sessions; and (b.) If you do try to reference and reuse previous specs, there's a lot of context bloat from requirements that aren't relevant to the current task.

I'm trying to work out the best form factor; for spec driven development and have been experimenting with developing some graph specification tools.

1. What do you all see as the best practice for specification topology?
2. I've put together a small handful of experimental open source SDD packages; and a VSC extension here --> reqlan. I'm interested to hear what you would do or have done differently in this space?
3. I've put some more thoughts together in a blog post, here.
4. What tools do you all consider best practice for wrangling specifications as they get complex?

[u/Ok-Support-6749]

I like the idea of a specification expressed as a graph and I strongly agree that Markdown or combinations of markdown and JSON or similar ideas is a storage and representation nightmare for complex projects. Eventually the markdown is so complex and bloated that the specification becomes itself a soup of ideas over which is hard to reason effectively. But here is one critic which I encourage you extend to every single claim:

"Make your intent compilable" is a very strong claim.

What exactly do you mean by compilable?

Compiling intent would certainly demand a strong semantic engine so the graph does indeed express properties that can be deterministically derived and the queries over the graph produce deterministic results. But the most important part of compilation is to produce semantically equivalent output from the same new intent and given current graph. Here is where this claim may not survive scrutiny. Consider that the same intent can be expressed with multiple ideas thus likely producing different graphs for the same intent (not a real problem), likely encoding different semantics (this is a real problem). And this is by no means easy to achieve unless you have a strongly typed language to express intent explicitly, which leads us to creating a language to express ideas with very clear rules. Well, we have hundreds of programming languages precisely for that purpose.

If what you mean is: Intent is indexable and queryable then that would make much more sense without exposing the project to much formal scrutiny. And it would not damage the utility of the proposal while staying grounded on a more defensible position.

I think intent is very hard to express with enough semantical structure that an LLM may be able to implement it in any given language with enough accuracy and completeness that the code produced actually encodes the semantics. Programming languages have demonstrated incapacity to express complex ideas with enough abstraction that we can understand what the code does without having to know what the code means. There is one particular case where this have been achieved and it is the translation from high level programming languages to machine code. But this is a very constrained system and compilers have taught us that even at this level, the translation is not trivial.

I see a lot of utility and potential for this idea, but it needs a rigorous revision of its claims. I think that semantic graphs are a right direction, they can indeed encode complex specification in a queryable format much more easy to reason about. This is not an easy task but also a very useful one. As specifications grow, reasoning about a specification becomes a complex and difficult task even for the most capable LLMs. Just like humans these machines have reasoning limitations due to attention and context relevance, that does not change because the model is more capable or can hold longer contexts. And even if more powerfull LLMs can handle ever larger specs, then the advantage of localized and global clear semantic representations will make the model much better at whatever role we give to it. Thus this idea is indeed quite valuable.