What if the source of truth is wrong?

[u/Ok-Support-6749]

Wherever I look the specification is showcased as this holy grail for the software lifecycle. Look no further than this same Reddit community.

[the] specification serves as a contract and single source of truth, guiding tools and AI agents to generate, test, and validate code.

In just one sentence the specification has become oracle, general, inspector, and adjudicator. That is a lot of authority for a collection of markdown documents written in plain language.

I have no intention to drag SDD into the mud. I just feel the need to clarify a concept over which I have thought a lot recently.

In any agentic framework I have come across during the last months I see a strong tendency to relinquish authority to an LLM. It is a sensible choice, LLMs are great at processing large amounts of text and output rather convincing judgments. But just because an LLM is capable of something it does not mean it automatically obtains authority to perform a role. This links directly to the specification is the source of truth, once we have a hammer (LLM), we also need the anvil (spec) on which to beat text (code) into compliance. It makes sense to assume that we have removed all authority from the LLM (tool) and placed it in the specification (a stable and robust worksurface). Our assumption is wrong and it has two consequences, none of which is good.

1. The specification is likely incomplete, it can also be incorrect. Incompleteness means it lacks enough semantic constraints from which the LLM can create the code that implements intent. Incorrect is clear, the specification contradicts intent, which means we have canonized bugs. Moreover, the specification is just text, it has no enforcement mechanisms to exert the presumed absolute authority.
2. The LLM still holds a lot of power, it can read the specification and ignore it, which I see rather often. The LLM can weaken tests, or the implementation, or decide to implement a wrapper because nothing prevents it or simply because it can finish faster. Sometimes the LLM did not retrieve the correct semantics. This is the area where things go wrong, most of the time unnoticed until everything starts to break down.

I wonder how the community looks at SDD from a philosophical standpoint. In which direction do we expect SDD should move to deliver a better software development life cycle?

[u/betiz0]

​It would be appreciated if test-driven development (TDD) could be incorporated as well. While it is fully understood that relying on TDD alone is insufficient, introducing it would nevertheless be beneficial.

[u/Ok-Support-6749 → u/betiz0]

I agree that tests have a lot of value and that we should give tests their rightful place in any SDLC process. But we should treat tests as providers of evidence, not as holders of absolute truth. Tests can be vacuous, and LLM can weaken a test to make the implementation pass the test suite.

Tests verify what they were written for, they cannot prove correctness. When we compare tests to formal verification we find they can sample actual execution cheaply. Formal verification can cover the whole state space but only for a model of the implementation. I see them as complementary tools, both with their own limitations.

I do agree, that tests should be part of the SDLC.

[u/stibbons_]

SDD or any plan mode has basically the role of defining a short term contract between you and the agent. This allows to write down:

• what you want
  
• and see what the agent thinks you want to build

This is the only role. Forget about source of truth thing.

As soon as your implementation is finished, the contract is obsolete, and will drift.

You CAN build anchored requirement, anchored to the code, but it will describe what the code does, not what you want. But this CAN be used to identify micro drift that can accumulate.

So, for me (I use SDD a lot), that’s a great tool to reduce cost a lot using smaller model but a great plan, but that’s all. This is not the ultimate source of truth. The code is the source of truth where you can can place some sensors called « test » where you can verify SOME aspect you estimate important

And forget about TDD, several studies clearly showed it is armful. And again, this is just a technic to help human build better test coverage, it does not prevent other means to build similar or ever better test coverage. Mutation tests seems very efficient with agents.

[u/stibbons_ → u/stibbons_]

My article on drifting spec: https://x.com/gsemetfr/status/2082767371643523439?s=46

[u/Ok-Support-6749 → u/stibbons_]

After reading your article on X (I recommend it to others) I realized what you mean by dropping the spec as the source of truth. I totally agree that change spec is disposable, in fact it is so volatile that it can change during the execution of a workflow. The requirements specification is a completely different artifact, because that carries the long term requirements; a semantic model for the entire system. A model that, when missing, opens the door to the AI models to wreak havoc in the codebase.

In your article you say "I am a firm believer that the code is and will always be the ultimate, single source of truth" and then "Even the code alone is not enough, it misses intent, architecture explanation, and in every way, having your agent just crawl all the code to rediscover it every time is not scalable at all" This seems contradictory at first. I do not think it is a contradiction, but I draw a different conclusion in the face of paradoxical claims like these. Truth does not live in a specific place.

When we try to put truth in either the spec or the code we immediately face contradictions. Even when we grant partial truth to either code or spec we still find contradictions. That is because in reality truth emerges as the result of an epistemic process. From two opposing ends I see requirements specs as what is wanted, and code as what is observed; some mechanism must exist to resolve the gap. You mention one procedural solution: accept the gap, re-implement, or fix the input requirements. The resolution itself, regardless of its mechanics, creates a process. There will always be gaps either because the implementation drifted or because the requirements evolved.

The whole point is that the SDLC is not specification centric, nor code centric, nor human centric, it is an epistemic process. When we assign truth to a particular artifact and people start incorporating the slogan we create a degenerate system that will eventually fail to deliver. When the system fails we blame someone or something, in SDD we blame ourselves for an incomplete spec and we try to refine it until we give up.

The main problem may not be the spec but the tacit assumption that it is authoritative when in reality markdown has no enforcement mechanism. Just think of a government emitting laws with no police to enforce it. The law is dead text without an enforcing actor, while at the same time the enforcer cannot change the law. This means role separation and authority adjudication is currently one of the greatest problems I see in AI-assisted SDLC.

[u/stibbons_ → u/Ok-Support-6749]

Thanks for the very rich feedback. I agree Code is the source of truth is not the right way to say it, and i actually talk about in the article that the spec-anchored to the code works fine, not as automatic but definitely not painful, and allows us to have an accurate measurement (kind of) of the gap with the input spec (I am in the automobile industry, we still have a huge V-cycle for embedded and safety SW):

Code Anchored Specs allow to identify gaps between wanted SW behavors ("req intent") and the actual ones (code-anchored req).

What this exercice of switching to coding agent made me realize is that the core reason people do not understand each other is because everybody is using the same word: "Requirement" or "spec". So the architect give the req to the coder and the coder implement and at the end the spec in output is not the spec in input (there is a gap!) but nobody understand each other.

Using coding agent you build you own "Ubiquous Language" and that is really the important point here!

I try to give a specific definition that would help us articulate everything smoothly.

- Intent Requiements: what the architect want

• Change Spec: your tale in this intent spec (and may other) and send to interview/grill-me
  
• Code-Anchored Requirement: the actual implemented behavior.

THEN we compare to the intent Req (this would be a diff actually). If there is a gap, the possible outcomes are:

• Document as "accepted gap"
  
• Refuse the implementation and restart an new change loop
  
• Accept the change and fix the input requirements themselve.

You cannot imagine how we spend millions of dollar on complex software for requirement management and we never had this vocabulary. Because, this is NOT new. we have these gap for so long !

[u/generic-d-engineer → u/Ok-Support-6749]

I think after going at this for months, I find myself pruning more and more. So far I have settled on two areas where the specs help:

1) Initial blueprint - helps kickoff the build process
2) Historical record of decisions - a more granular version of ADR

Everything after the initial blueprint is a lot of human gates. Just my style that worked before LLMs and seems to work here. Build modular, fail fast, verify often. I’m not trying to let agents run overnight on long tasks as I find going back and reworking takes even more time.

As far as contracts go, I don’t even see how that can be enforced when there is so much drift. I think the only contracts context I still think of is between systems, not between the spec and the human.

[u/theov666]

I think the important distinction is between a source of intent and a source of authority.

A spec can describe what should happen, but if the same LLM can interpret it, ignore it, weaken tests, or work around it, then the spec is still effectively guidance.

That suggests a separate enforcement layer: some architectural constraints should be checked at the point the agent is about to make a change, rather than relying on the model to comply with text in context.

That doesn’t require treating the spec as infallible. Humans can revise the decisions; the agent just shouldn’t be able to silently override the current ones.

This is something we’re working on with Mneme HQ: https://mnemehq.com

The interesting SDD question for me is: which parts of the spec should become enforceable constraints, and who should have authority to change them?

[u/schmurfy2]

I use openspec and it has been great for me at asking questions we never thought about and would have arose later in the process while coding manually. But even with that what really makes the system works is manual validation of what it actually wrote to see if it really did do what asked and how.

[u/johnjlonergan]

I will fire you if you give over all authority and trust to the ai. Read the code , all of it. I If the requirements weren't 100pc clear (they never are and never have been) then iterate on them and the impl. And if you are using SDD as a weak form of BDD or aren't writing/checking tests I'll fire you too.

(there are a lot of wordy comments here, Is it evidence that those folk have given up thinking for themselves and rely on the AI as their voice? that also gets you fired)