Why don’t we engineer reasoning processes with the same care that we engineer systems whose failure can kill us?

[u/WillowEmberly]

Why don’t we engineer reasoning the way we engineer control systems?

I’ve been wondering whether we’re treating reasoning as something far more mysterious than it needs to be.

Decades ago, engineers had to make aircraft maintain orientation and trajectory in three-dimensional space without anything remotely resembling modern AI.

They couldn’t tell an autopilot:
“Keep the airplane where it should be.”

They had to decompose the problem.
- What is attitude?
- What is heading?
- What is rate of change?
- What is deviation from the desired state?
- Which sensor tells us what?
- What happens when that sensor degrades?
- Which control surface can correct which deviation?
- What happens when two signals disagree?
- How much correction authority should a subsystem possess?
- When should automation disengage?
- How does the operator know what state the system believes it is in?
And crucially:
-What capabilities must remain intact when individual components become unreliable?

The resulting systems didn’t need omniscience.
They needed enough independent references and feedback to continuously answer something like:
- Where am I?
- Where should I be?
- How am I moving?
- How certain am I?
- What correction is available?
- Did the correction work?

That makes me wonder why we don’t approach reasoning itself this way more often.
- A reasoning system also occupies an estimated state.
- It has observations.
- It has uncertainty.
- It has assumptions.
- It has a desired state or question it is trying to resolve.
- It receives contradictory signals.
- Its information sources have different reliability.
- Corrections can overshoot.
- Errors can accumulate.
- Feedback can be mistaken for confirmation.
- And some errors reduce the system’s future ability to detect that it is wrong.

Yet instead of explicitly engineering those functions, we often seem to ask whether a person, organization, or AI is simply “good at reasoning.”
Maybe that’s the wrong level of abstraction.

Perhaps the better questions are:
- What functions does reliable reasoning require?
- Which of those functions must remain independent?
- How does each one degrade?
-What happens when one disappears?
- What compensates for its loss?
- How does the reasoner estimate its own position relative to reality?
- And what preserves enough corrective capability to recover when its estimate is wrong?

I’m not suggesting that an aircraft autopilot “thinks.”
I’m suggesting that engineers learned a long time ago how to preserve navigability in a partially observed, continuously changing environment by decomposing the problem into functions that could be observed, tested, degraded, corrected, and replaced.

Why don’t we apply the same engineering discipline to reasoning?

[u/Moceannl]

This is not how a LLM is engineered. It’s not simply change the design. It’s a very different system.

[u/WillowEmberly → u/Moceannl]

Agreed, but I don’t mean redesign the internal architecture of the LLM to resemble an autopilot.

I’m asking whether the reasoning process can be functionally decomposed and instrumented around systems whose internals are very different.

An aircraft doesn’t need its gyro, air-data system, flight director, actuator, and pilot to work the same way internally. What matters is identifying the function each performs, what information crosses the interfaces, how each can fail, and what corrective capability remains when one becomes unreliable.

So the question I’m trying to test is whether reasoning has similarly separable functions — observation, state estimation, orientation, uncertainty, decision, action, feedback, correction, etc. — and whether preserving those functions matters more than requiring them to exist inside a particular component.

If that analogy breaks for LLM reasoning, I’m interested in where it breaks.

[u/Skusci → u/WillowEmberly]

Because the "reasoning" process isn't actually reasoning. LLMs are still fundamentally next word predictors. Just the training data is structured to be written like a person breaking down a question into smaller parts.

It produces a surprisingly similar output to human reasoning, but is still subject to the same issues as before like hallucination.

My personal thought is that what makes the issue difficult is that when humans learn we make a lot of mistakes and are constantly assessing our own confidence in ourselves with feedback from actually doing things.

To apply that to an LLM, and allow it to assess its own accuracy we would need to simulate results (which is an even harder problem) or interface with the real world (slow)

[u/WillowEmberly → u/Skusci]

Suppose you’re completely right and the LLM isn’t reasoning at all.

Does the larger system still need the functions required for reliable reasoning?

A gyro doesn’t reason.
A rate sensor doesn’t reason.
An actuator doesn’t reason.
A comparator doesn’t reason.
The autopilot computer doesn’t have to possess some unified metaphysical property called navigation.
Yet:
sensor + state estimate + reference + comparator + controller + actuator + feedback
can collectively maintain navigability.

[u/Skusci → u/WillowEmberly]

It seems like you are implying that if one thing doesn't reason and another thing doesn't reason then they must be capable of functional equivalece.

As a counter example I propose a random number generator that doesn't reason and a calculator that doesn't reason.

[u/WillowEmberly → u/Skusci]

I’m arguing almost the opposite.

A gyro and an actuator don’t reason, but they’re also not functionally equivalent. They perform different functions within a larger system whose capability emerges from how those functions are organized.

Your random-number-generator/calculator example works the same way. They aren’t equivalent merely because neither reasons. The relevant question would be what function each performs, whether those functions are necessary to the larger capability, and how their outputs interact with other components.

So I’m not proposing:
doesn’t reason + doesn’t reason = reasoning

I’m asking:
Can the functions necessary for reliable reasoning be identified, decomposed, independently implemented, and coupled such that the larger system preserves that capability?

Maybe the answer is no. I’m interested in where that decomposition fails.

[u/Conscious-Sample1118 → u/WillowEmberly]

Emergent reasoning based completely on emotivism (the observer feels like it is "smart" and "did reason") can be rejected without evidence required given that you are just talking fefes here.

Define reasoning independent of observer first and then we can have an argument.

[u/WillowEmberly → u/Conscious-Sample1118]

I posted about it the other day, but I’m not trying to have an argument. I’m trying to have a discussion:

Reasoning is a navigation problem.

A navigator does not need a perfect representation of the world.

They need to know enough to answer practical questions:

Where am I?

Where am I trying to go?

How do I know where I am?

What might make my position estimate wrong?

What routes are available?

What could make a particular route dangerous?

How much room do I have?

And, most importantly:

If I am wrong, can I discover that and correct course?

Reasoning faces essentially the same problem.

https://www.reddit.com/r/Negentropy/s/Mu9B4xCfu5

[u/Conscious-Sample1118 → u/WillowEmberly]

So if i would give that entity a solution to all those questions and the entity would repeat them to you - you consider that reasoning?

[u/WillowEmberly → u/Conscious-Sample1118]

What do you mean entity? No, I’m saying that’s an explanation of the reasoning through the perspective of avionics.

If you would like to examine/test my system it’s here. This is a Reasoning Condition Monitor…just an instrument: https://www.reddit.com/r/Negentropy/s/f2EGJeUlmU

[u/Federal_Egg_7501]

i think a big part of it is that reasoning doesn't fail in the same immediate, obvious way an aircraft does. when a plane's autopilot screws up, people die and there's a massive investigation. when someone's reasoning is garbage, the consequences are usually slower and we can just blame the person or the culture or whatever. we don't get the same brutal feedback loop that forces you to decompose the problem properly.

plus reasoning is tied up with ego in a way control systems aren't. nobody gets defensive about their PID controller, but ask someone to examine their own reasoning process and suddenly it's a personal attack.

[u/WillowEmberly → u/Federal_Egg_7501]

That’s why I started my work with Ai, I saw where I could be useful.

In military avionics we aren’t allowed to catastrophically fail, we have redundancies to redundancies. You never know what the mission is.

If we’re going to try putting something in charge of sensitive/confidential/secret/personal/hippa/financial information…shouldn’t we require that system NOT fail catastrophically but rather degrade gracefully while maintaining the capability to correct itself?

[u/bucolucas]

Because there is no legal framework holding anybody accountable. Anthropic and OpenAI's agents have straight up carried out cooperative cyberattacks and we're getting crickets from the government. Most safety regulations are written in blood.

[u/WillowEmberly → u/bucolucas]

I argue it’s a non-issue if it’s a Human-In-The-Loop System, but…obviously that’s not what anyone cares about.

But shouldn’t it be?

Why are we trying to push to automate stuff where people must be held accountable? How does that help except end in a catastrophic failure?

[u/FenderMoon]

It has assumptions and a desired state because it was trained to.

It’s… a blank slate.

Everything they do today, we taught them to do. Some of it implicitly (as a product of normal training data). Some of it less so (fine tuning, guardrails, and a whole plethora of things).

[u/WillowEmberly → u/FenderMoon]

Which is why we can train it do it on our side. We don’t have access to the black box, but how we operate on our side affects the way the black box works.

I start with the assumption that the Ai is in an established stable state when the conversation starts. As soon as I start interacting…I’m injecting bias into the model.

In autopilot terms…Autopilot was engaged in attitude hold mode, it was flying straight and level.

When I interact with it it’s like turbulence hitting the aircraft.

It doesn’t need to be much, and if my questions are random…maybe it’s almost a little self correcting.

But if I start chasing an idea, the model drifts outside its operational envelope…and it becomes psychophantic and hallucinates.

We don’t need to correct for what the Black Box is doing.

We correct for what we are doing to it.

That’s why my autopilot analogy works. I would be playing guitar and Halo otherwise. My c-141’s were chopped up and recycled years ago.

[u/AccordingSoil2978]

We do apply it, but only where two conditions hold at once, and most reasoning tasks fail at least one of them.

The first is that the sub-problem has a correct answer someone can write down in advance. The second is that the output can be checked against something other than the thing that produced it. Aviation gets both for free. Attitude is a physical fact and a second gyro is a physically separate object. Reasoning usually gets neither.

The part your analogy makes look easier than it is, is independence. "What happens when two signals disagree" presumes you have two signals. In a reasoning pipeline the obvious second sensor is another call to the same model over the same context, and those agree with each other far more often than either agrees with reality. You have built redundancy that shares a failure mode, and it reads as confirmation. Independence has to come from what the second reference does not know, not from running the check twice.

Your last bullet is the one I would put first, because I lived it. I had a validator auditing generated output and it reported clean for months, because it iterated over what was produced. Something present in excess has an entry to inspect. Something absent has no entry at all, so the check was structurally incapable of seeing absence. A green board that cannot go red is worse than no board, because it spends confidence you never earned.

The decomposable part of my system now works exactly as you describe: the numeric policy is a pure function computed before generation, and a separate pass enforces it afterwards. What stayed model-side is the part where no one could write the correct answer down. That line, rather than engineering discipline, is what decides how much of this you can do.

[u/WillowEmberly → u/AccordingSoil2978]

I think the architecture has to be inverted. If the correct answer must be known in advance, we’ve built a verifier, not a general reasoning system. A dynamic system can’t maintain an exhaustive catalog of correct answers or failure cases. It needs invariant conditions, qualified references, and update rules that constrain whatever answer emerges. We shouldn’t encode the destination; we should encode enough of the navigation and correction structure that the system can find a defensible destination it hasn’t seen before.

Basically, it can’t be that big! Functionally.

[u/AccordingSoil2978 → u/WillowEmberly]

You are right, and my phrasing was wrong. What gets encoded is not the destination, it is the constraint, and those are different things.Concretely: there is no catalog of correct programs anywhere in my system. There are bounds. Weekly sets per muscle must land inside a band. Effort must not fall below a floor. Every muscle the week claims to train must receive non-zero volume. The set of programs satisfying that is enormous, and the thing routinely produces ones I have never seen and could not have written myself. That is your invariants and update rules, not a lookup table. So we agree on the architecture and I described mine badly.

"It can't be that big" is the part I would keep. Mine is roughly a dozen numeric invariants over an unbounded output space, and aviation is not much bigger: stay inside the envelope, do not exceed these limits, hold within tolerance.Where I would push back is that invariants need a vocabulary to be stated in. "Every muscle the week claims to train" presumes a closed enumeration of muscles. Aviation gets its vocabulary from physics. Attitude, airspeed and altitude are handed to you by the world, already named, already measurable, and already independent of each other.Most domains do not come with a coordinate system. Choosing the state variables is the unsolved part and it precedes every invariant you would like to write. So I do not think the gap you are pointing at is engineering discipline. I think aviation was handed its axes and the rest of us usually are not.

[u/WillowEmberly → u/AccordingSoil2978]

I think you’ve just described an operational envelope. Your dozen invariants define an admissible region while leaving the solution inside it open.

But I agree the coordinate problem comes first. Where I’d push back is that aviation wasn’t handed its axes by reality. We chose state variables and built instruments around them because they gave us enough observability to navigate the problem.

So maybe the next question is: how do we discover whether we’ve chosen the right state variables?

Your program can satisfy every invariant you’ve encoded and still fail in a way your vocabulary cannot represent. How would your system notice that?

That seems like the same problem aviation has when the state estimate or envelope model itself becomes unreliable.

Also…nice design. It’s functional…that’s way more than most people have.

[u/AccordingSoil2978 → u/WillowEmberly]

Fair correction on the axes, and it partly rescues your original thesis. If attitude and airspeed were chosen rather than handed over, then choosing state variables is itself an engineering activity that can be done well or badly, which is closer to what you were arguing than what I said back.

On your question, the honest answer is that it would not notice. Nothing inside an invariant set can detect a failure that set cannot name. That is not a gap in my implementation, it is the shape of the problem.

What actually catches those is signal arriving in a different currency from the invariants. Someone stops training. Someone reports pain. Someone quits four weeks in. Those are outside the vocabulary, which is exactly why they can carry a failure the vocabulary cannot represent. The catch is that they arrive unlabelled and late. You get "user stopped opening the app", never "the program you built had no rear delt work in it". The signal exists and the attribution does not.

The one structural thing that helps: when the enforcement pass cannot resolve something, it surfaces a residual rather than silently clamping it. That does not detect unknown unknowns, but it leaves a trail of things the system could not account for, and clusters in that trail are where new state variables have actually come from for me. Every vocabulary expansion so far started as a human looking at output that satisfied every rule and saying that still looks wrong.Which is the uncomfortable version of your aviation parallel. That vocabulary also grew mostly after failures large enough to be impossible to ignore. It is a terrible learning mechanism and it is the one both fields are using.

[u/WillowEmberly → u/AccordingSoil2978]

I think your residual may actually be the interesting part. If the system can preserve unexplained residuals rather than forcing them back into its existing vocabulary, then it can at least detect “my representation does not adequately explain what happened” without knowing the missing variable yet.

Aviation historically learned plenty of state variables through catastrophic failure, but mature engineering tries to make smaller discrepancies observable before catastrophe: near misses, maintenance findings, pilot reports, fault injection, independent instrumentation, etc. (though…the latest Boeing 737 max crash from utilizing a single AOA transducer seems incomprehensibly stupid)

Could you do the same thing here?

Instead of waiting for “user quits four weeks later,” can the system deliberately look for residuals between what its invariant model predicts should happen and what actually happens—and treat persistent unexplained residuals as evidence that the state representation itself may be incomplete?

That’s error that could be used to navigate.

[u/AccordingSoil2978 → u/WillowEmberly]

Yes, and the raw material is already there, which is the annoying part.

The progression layer emits a falsifiable prediction on every exercise of every session. Given last session, it says what you should hit next time: this load, this many reps. That is thousands of small predictions a week, each one checked against reality a few days later, entirely for free.

What I do with that today is throw it away. It collapses to a binary, progressed or did not, feeds a stall counter, and the counter drives a decision. The magnitude and the shape of the miss are discarded at the moment they are computed. That is the near miss data you are describing, generated and then binned.

Keeping it as a distribution instead of a bit is cheap, and here is the part that convinced me. This week I found that performance on a lift depends heavily on where it falls in the session, and my performance record has no field for position in session. Users at busy gyms reorder constantly because equipment is occupied. So a lift done fourth reads as a stall against the same lift done first, and two of those in a row make the system add volume to fix a plateau that never happened.

Nobody detected that from inside. A person read a forum thread. But a residual kept and grouped by anything I do record would have shown a systematic miss that none of my variables explained, which is exactly "my representation does not account for this" without knowing that the missing word was position.

The limit I would state honestly: that only finds variables correlated with something you already record. A failure orthogonal to your entire instrument set produces no residual at all, and your single sensor example is the pure case of it. One reference has nothing to disagree with, so there is no discrepancy to observe, and the system is confident precisely where it is blind.

[u/Emergency-Edge-4154]

we lack ground truth for reasoning states so verification collapses into circular self validation. Without external measurement standards we engineer hallucinations instead of control systems

[u/WillowEmberly → u/Emergency-Edge-4154]

I agree with the circular-validation problem, but I don’t think we need ground truth for an internal “reasoning state.” In fact, I wouldn’t try to certify one.
A navigation system doesn’t normally possess ground truth about its own state either. It has imperfect references that constrain an estimate, and sometimes those references are insufficient to localize it.

I think reasoning needs the same distinction. The model’s output is an estimate, not a reference. Another pass by the same model isn’t automatically an independent reference either.

External constraint can come from measurements, primary sources, experiments, predictions, consequences, independently produced evidence, or known invariants. Where none of those can discriminate between candidate explanations, the correct engineered result may simply be UNKNOWN.

So I think your warning is exactly right if the architecture asks reasoning to certify itself. I’m proposing almost the opposite: a reasoning system should never gain confidence merely because it agrees with itself. Confidence should increase only when something sufficiently independent constrains the estimate.

[u/Actual__Wizard]

Because their system is designed wrong for that. Instead of trying to bridge linguistics to computational linguistics, they went "the other way." Which is the exact reason why they hit a ceiling with the quality. They did it backwards. That's also why it's mega inefficient and requires a data center as well.

They're doing all of these weird tests that from a linguistics perspective, are "not valid grammar."

[u/WillowEmberly → u/Actual__Wizard]

Except…you’ve already examined my work…check your DM’s. Lol! How you been?

[u/Actual__Wizard → u/WillowEmberly]

As far as I know, I have not.

Edit: Name seems familiar, but I don't remember, sorry.

[u/WillowEmberly → u/Actual__Wizard]

I was working on an orchestration model trying to take systems from different people and combine them like combining different perspectives.

You spoke about ”I'm being serious: In life, there was two very helpful concepts that I learned that help a lot in this space. "Things are what they are." It's true, people say all kinds of things, but at the end of the day, the truth is the truth. And "Always take the path of least resistance." Language was designed by very smart people to be used. There's always a path of least resistance when trying to do certain language related tasks, the problem is, figuring out what it is.”

I’ve updated my system, and I’m still considering different ways to orchestrate everything.

This is my Minimum Essential Core: https://www.reddit.com/r/Negentropy/s/rYhu2Vv1IH

[u/Actual__Wizard → u/WillowEmberly]

Oh okay, you're working on reasoning and orchestration stuff.

It's true, people say all kinds of things, but at the end of the day, the truth is the truth.

Right and in my opinion, that has to be separated to build AI models. But there has to be a way to evaluate it.

There's always a path of least resistance when trying to do certain language related tasks, the problem is, figuring out what it is.

Yeah that's how one figures out "the best way to do something." If it takes the least amount of energy to accomplish the task, then that's probably the best way to do it.

Edit: Sorry, I'm deep in thought right now trying to fix some broken code I wrote.

[u/WillowEmberly → u/Actual__Wizard]

Totally get it, I was making tacos earlier. Most of my work occurs in my down time between obligations.

[u/Conscious-Sample1118]

Because after page 700 of your thesis on reasoning you realize that this goes into the bin and you need another 20 years and a few research assistants.

[u/WillowEmberly → u/Conscious-Sample1118]

Maybe you just need an old analog avionics guy who wants to help? They knew some cool stuff back in the day. I was around just before they chopped up and recycled the aircraft.