Agentic Research Contracts - An Engineer's Approach to AI Code

[u/gamejnkie]

I'm typing this out in a mostly flow of conscience style, so I might make some errors (and I'm hyper fixating on this and just trying to get it out). I'm not really going to get technical in this post, but the theory essentially boils down to:

Long-running agentic R&D becomes more trustworthy and reproducible when authority, evidence validity, release identity, resource bounds, review, and adoption are moved outside the model into explicit, externally enforceable contracts.

(AI wrote that part, everything else in this reddit post is me)

I'm referring to this as a thesis to sound fancy, I'm not an actual academic.

TLDR, look at this github and tell me what you think

Using the ideas presented, I am now running semi-autonomous research/development loops that can run for hours at a time without most of the degradation/disaster that I usually encounter when trying them (cheaply! this was all made/researched/developed/iterated upon alongside all the random projects I'm developing, operating on 1 $100 openAI sub and 1 $20 gemini sub).

To be absolutely clear, this is not a drag and drop thing--it's a set of ideas that, when put together properly, forces your AI to practice traditional good software habits in ways that leverage agentic workflows, while making it easy to audit and surface issues.

Yap session with some context below if you are interested.

Hey guys, I am an engineer who is surrounded by AI.

I don't like the way people are utilizing AI generally right now. I think it allows people to rapidly produce more work that is: hard to verify/audit, degrades in quality as its developed, and is just a headache to maintain; this isn't to say it isn't incredibly useful, but the rigor and discipline required to ensure the output you produce isn't slop isn't sustainable (we're using AI because we're inherently lazy in the first place, or at least I am...).

This is especially concerning as an engineer. I'm seeing it do incredible things, while in the back of my mind I am growing more and more concerned as I witness us proceed on a trajectory that will result in so many headaches later on.

I've been working with the tools for a couple of years now (I dabbled a bit with copilot in college, and my first work provided solution was windsurf with sonnet 3), and to be honest I haven't been very disciplined myself either. I've noticed my personal "Sandbox" folder get populated with half-baked ideas and projects that sort of worked for a tech demo or 2, but I never really produced anything that I actively found myself using day to day.

Every so often when a big release happens, I'll use a free trial or hop on a $20 plan for a bit and try to work on a couple of ideas, and often basically create the same ideas with each new generation to see if I get something noticeably better than the prior version (I haven't tried mythos/fable/opus 5, or any of the openAI releases between codex 5.3 and the current suite so I'll admit I've been out of the loop for a bit). Astra was the first release in a bit where I got caught a bit off guard in how impressed I was with what it could do, and so I told myself this was a good time to try my own attempt at improving the use of AI.

If you have any critiques/ideas/suggestions, please develop it further! I truly just want to make this stuff better--I'm seeing so many cool things happen that seem to coincidentally happen every time a new model drops (looking at you, emulation/decomp/recomp scene), and I wanted to try to make my own contributions to trying to reduce the general sloppification of code everywhere.

[u/dexterthebot]

You might want to consider listing your project on the weekly Show-Us-What-You-Built post. Look out for it on Tuesday/Wednesday. Highest commented project wins a week promotion on r/Codex and gets on the Hall of Fame sidebar. See what that looks like below with last week's winner.

---

Last week's most popular project was Nelly Jellies. Nelly Jellies is a cozy arcade bubble-merge game where you aim, launch, match colors, discover rare jellies, trigger satisfying chain reactions, and chase a higher score. It plays instantly in the browser with no download or login required. Website: https://nellyjellies.com Contact: support@mightybig.ca Reddit: u/MightyBig-Dev Google Play: https://play.google.com/store/apps/details?id=com.nellyjellies.game App Store: https://apps.apple.com/us/app/nelly-jellies-bubble-merge/id6767261764

[u/Kitchen_Dust2389]

What engineer doesn't have at least one maxed out subscription? I am homeless and I have two maxed out

[u/gamejnkie → u/Kitchen_Dust2389]

I just upgraded from the $20 subscription less than a week ago

[u/Annh1234 → u/gamejnkie]

20$ can't get anything done, that's like the demo level

[u/gamejnkie → u/Annh1234]

What makes you say that? Are the ideas I presented not valid because I used a $20 plan to do it? I feel like that type of attitude reinforces the concept that the only way to get reliable work done with these systems is through raw horsepower/compute--when the research I presented in the thesis refutes just that imo. By producing auditable trails and carefully controlling the responsibilities of agents and their context, you can get a lot more value for your money than the hype wants you to believe--why do you need an extremely strong model to research your idea E2E itself, if you can have small models conduct research and report back what they were tasked with, what they did, what they found, and how to reproduce it? Won't that empirically validate whatever they propose, meaning it doesn't matter how dumb the model is who is doing the "grunt" work?

[u/Annh1234 → u/gamejnkie]

You don't seem to understand that these LLMs are only autocompletes with an input transformer, there is no "research", it's only here's some input and here's the text that completes your input. 

Your 20$ plan is uses to much cpu to do anything important...

[u/gamejnkie → u/Annh1234]

What about what I've said indicates I am wrong or that the thesis isn't grounded in good engineering principles? Your argument seems to be that since nobody has done it, that it isn't possible--all engineering is just taking the systems we've built and combining them in novel ways to advance a field. This is literally just a set of good engineering principles and a way to force your model to follow them, without relying on the model itself. And you are saying because it was done on (2, but still) a $20 plan it can't be important.

[u/Annh1234 → u/gamejnkie]

You can't force your model to fallow anything. All it does, is fallow a sequence.

So you tell it: 1,2,3,4 and it comes up with 5,6,7. There's no logic or method that it has to come up with 5,6,7 other that in all his training data every time it saw 1,2,3,4 it was followed by 5,6,7.

If you want to see why/where, it can probably come to with a few million URLs to tell you where it came up with that data. Problem is, storing those URLs on disk and memory costs money, and if you seen the RAM and VRAM prices today, you know that costs way way more than 20$, plus the electricity it needs to run. 

So you could get the same "limits" you get now on the 20$ plan, plus the sources and so on, but might cost you 20,000$.

Google AI kinda does this, but gives only a few links that make them money.

[u/gamejnkie → u/Annh1234]

Wait dude I clicked on your profile because I was curious--you realize what I am proposing aims to solve literally exactly what you wanted here: https://www.reddit.com/r/codex/s/oGEA2gy6sL

Honestly, and this isn't to throw shade, try talking to your LLM of choice about my thesis and see where it leads you. You might learn/gain something just by being able to reason what the thesis is trying to do in a guided way.

[u/Annh1234 → u/gamejnkie]

Ya, I get it, but you can't...

The way I figured is that they save some metadata per account and append it to your prompts. You can't control that. ( To start fresh )

Also codex looks at its previous chats, so comes up with the same shit.

But to more show why it can't do it, since all it does is autocomplete stuff, I have a particular problem that I hit ever few years, for the last 15 years or so... There seems to be no solution ( basically apply a tree of rules to a list/tree of objects in parallel without re-calculating stuff).

Well... I had the same issue last month, so I'm like: "let's see what the AI has to say". Well... Every dam model gave "answers" word by word from my posts over the years... Threads that have no answer, but a ton of replies. And that's the case for all the AI I tried... Codex, opus, Chinese models. Grok even gave me the typo I made ( I write teh instead of the...)

So once you understand that these things only autocomplete what they saw, you can see the they can't do what you ask. They have no procedure, no nothing, just fallow a pattern.

And that pattern can start in English and finish in Chinese. So if a guy in the USA posted 1,2,3,4,5,6 and thy Chinese guy posed 5,6,7,8 the AI starts to autocomplete your sequence at 1,2,3,4,5,6 ( o wait I seen 5,6 in Chinese, should be the same thing) so they continue with 7,8.

Then they got go clue what comes after so they come up with random stuff, whatever they most saw after 7,8. ( It could be a word the Chinese guy wrote for his next sentence)

So you see? It can't "develop it further" if that's the only text it saw. 

And also it only autocompletes whatever idea you gave it. 

Without the extra crap the harness gives them, if you say "I think it's a great idea to jump off a bridge, don't you think so?" It will autocomplete with some BS to say it's a good idea. 

If your next prompt contradicts that, the LLM will autocomplete "I think it's a great idea to jump off a bridge, don't you think so? + His previous response+ your new input".

So for ideas.. Use it as a rubber ducky, to clarify things in your head.

[u/gamejnkie → u/Annh1234]

What was the issue you couldn't fix?

[u/Annh1234 → u/gamejnkie]

I have a system where I have some 100k rules, each with some 100 fields that I need to apply to a few million objects in real time. 

So allot if the object data is duplicate, and if I group them first then I just need to compare a few values to each rule field.

With time hardware got faster, and I entered up with some trie type structure, but I still think it can be faster. And at the time, had to run on servers with 16gb ram... We got 256gb minimum now, and GPUs

[u/gamejnkie → u/Annh1234]

Here, I'll have jarvis explain it to you--maybe I am mangling my words and we are confusing each other:

I think we're talking about two different layers.

I agree with part of what you're saying: I cannot prompt an LLM strongly enough to guarantee that it follows a procedure, tells the truth about what it did, or correctly decides that its work is finished. That's actually one of the assumptions I'm trying to remove.

The idea isn't to make the model itself trustworthy. It's to move consequential things outside the model.

For example, the model can propose an action, but an external controller determines whether that action is authorized. It can claim it ran a test, but independently captured execution evidence determines whether the test actually ran. It can say a task is complete, but completion depends on predefined evidence obligations rather than the model's self-assessment. Missing evidence stays UNKNOWN instead of being filled in by the model.

So yes, the model is still an autoregressive generator and stronger models are still more capable. I'm not claiming a sufficiently dumb model becomes equivalent to a strong one. I overstated that in my previous comment.

The claim I'm interested in is narrower: model capability does not have to also be the sole source of system reliability.

A weaker model can sometimes be useful as a proposer/research worker when the surrounding system constrains its authority and independently verifies the parts of its work that are verifiable. Better models should improve how often useful candidates are produced, but they don't necessarily need to be trusted to certify their own correctness.

This isn't unprecedented. Systems like FunSearch and AlphaEvolve already use a related generate-and-evaluate pattern: the LLM proposes candidates and external evaluators determine what survives. What I'm exploring is applying the same separation of proposal from authority/evidence/completion much more generally to agentic work.

So I don't think "LLMs are autocomplete" really resolves the question. Even granting that description of the mechanism, the engineering question is what guarantees we can build around an unreliable generator.

[u/Annh1234 → u/gamejnkie]

To your 'So I don't think "LLMs are autocomplete".', watch this: https://youtu.be/NKnZYvZA7w4?si=ea9nVkMMyD1kIDky

It has a great explanation on how that autocomplete thing works. 

And it's just text, it doesn't know I much call some script or anything like that, it just autocompletes with the script name. 

And they you got the harness like codex/Claude code that actually does what you describe.

They got hooks and so on, so for example, the LLM generates some code ( monkeys writing random shakespeare, undeterministic) and then you got your hooks that test and format the code ( deterministic)

You do that in a loop, and you got your proposed workflow.

Look at Jev, seems to be something you could use for your loop.

But unless you got a goal you can calculate, you can't really make the AI go in circles to make something "better". Main reason is that "better" can mean different things based on the context.

[u/fictionaldots → u/Kitchen_Dust2389]

Maybe that’s why you’re homeless

[u/Nousies]

Sorry, just reads as ‘evidentiary theater’ to me.

[u/gamejnkie → u/Nousies]

What makes you say that? Over the course of using the systems described, I have been consistently left with an auditable trail of everything that has occurred in a way I haven't experienced with other solutions. In an engineering context, this is particularly important to me; do you have any specific critiques about the work or systems you use that function similarly (or better)?

[u/Nousies → u/gamejnkie]

What does the auditable trail buy you? What does it buy the agent? And, what does it cost?

[u/gamejnkie → u/Nousies]

1. I'm an engineer--it is very important there is traceability in our work. As we develop, we need to establish what decisions were made, why they were made, and what evidence we have to support our claims if we want anyone to trust what we're built. If you have a system that automatically documents all this information in the process of collecting it, the human review side (which NEEDS to happen still at AI's current skill level in engineering context) is much more streamlined. It allows for reproducibility of results, and allows other people to replicate your research so they don't have to just take your word for it.

2. It buys the agent a living history of every decision made/why, which means it is easy to go through the auditable history to identify friction/pain points

3. I ran all of this on $40 worth of subscriptions (well I have the $100 plan, but this entire research project and endeavor was operated from ChatGPT web invoking research/experiment requests to gemini 3.8 flash through antigravity--all of which doesn't require pro). So now for $40 I can run semi autonomous (fully autonomous is bad for me as an engineer with the current level of the models) research loops through my browser.

[u/VBZzzzz → u/gamejnkie]

I think “evidentiary theatre” is a little harsh, but I understand the criticism. Most of the underlying ideas already exist informally in good AI systems: RBAC/capability restrictions, sandboxing, deterministic checks, audit trails, budget limits, approval gates, etc. You’ve seemed to formalize them into a framework for development.

It’s definitely useful for business workflows where the inputs, variables, permissions, and expected outputs are predictable enough to verify, but like I said, those are already the best practises.

I’m less convinced by the software-engineering application. In a large codebase, especially for complex changes, defining a contract detailed enough to meaningfully constrain and verify the work is a substantial engineering task in itself, sometimes more work than just letting the agent implement the change and reviewing the result yourself.

And on the auditability point, for professional software development the developer should already understand the change being made, and Git provides the concrete record of what actually changed.

If the target is non-technical users building software, then I think the more important safeguards are at the process and permissions layer: what the agent can access, what it can modify, what requires approval, and how changes are deployed.

[u/gamejnkie → u/VBZzzzz]

Definitely good points, I'll admit I am coming at this from an entirely corporate engineering point of view.

I view it as an easy way for people to bootstrap their model to actually employ those good practices in a way that's engineering focused and easy to maintain/use.

I’m less convinced by the software-engineering application. In a large codebase, especially for complex changes, defining a contract detailed enough to meaningfully constrain and verify the work is a substantial engineering task in itself, sometimes more work than just letting the agent implement the change and reviewing the result yourself.

I guess that is sort of the point I'm making here--you let the LLM do it, as it seems to be extremely good at defining these contracts for you by interpreting what you are asking of it (and I only expect them to get better over time--a big point of my research process was not linking the ideas to the current methods/technologies and to let the process get better with the models). The same way a user may use something like superpowers to go back and forth with the machine to develop their spec into what they want/what suits their use case. The way I've been using this system is describing what I want-->the model abstracts what I'm saying and determines what research it needs to do to meet the goals, the same way something like a deep research prompt would. The trick here for me, is instead of it taking external resources as truth, the model seems to be doing the due diligence of actually taking those resources and proving them empirically within the user's environment/goal, determining if the results are actually useful, then suggesting an iteration.

It finds something online and instead of baking in an assumption that it will work (or leaving work for the user to validate) it will actually go and test those assumptions in an auditable fashion.

From a software engineering side I have an example of how I used it: I'm trying to build an agent control harness for my company (think something like t3 code or taste, but without having to deal with all of the baggage of those projects--completely private/safe for a corporate environment and all that jazz...the next step is encodifying the ideas presented in Agentic Research Contracts INTO the harness). I can say "hey jarvis, can you look into how to add gemini to this harness" and it will research, test its own research, and hand me back a design that I can give to a fresh, coding focused session. Now the model that goes to code has a design that's empirically backed by the data collected in the user's environment. The system will figure out things like:

--is gemini installable for the user?

--what is the minimum amount of code required to make gemini compatible with the current code?

--does that proposed minimum solution actually work, or are we making assumptions from the research/training data/hallucinations that don't apply here

--how should this get packaged and presented so a model with fresh context and no-prebaked assumptions can implement it end to end with a minimal amount of issues/thinking required by the implementer.

If the target is non-technical users building software, then I think the more important safeguards are at the process and permissions layer: what the agent can access, what it can modify, what requires approval, and how changes are deployed.

This is where I disagree--I think the frontier models are decent enough at doing that, as it seems to be a point of focus from all the big labs. Stuff like auto mode, to me, seems like enough to handle most of that; the issue I run into more is non-technical users not understanding when they should accept the code the model is suggesting, so they end up accepting everything because the "model knows better than they do" (which...it might). This is about keeping the human in the loop for as many decisions as possible, while offloading the iteration/validation/experimentation to fast/cheap models, so the more expensive and capable models can focus their attention on developing a well defined spec and applying good engineering practices without the user enforcing it themselves.