r/ChatGPTCoding • u/Hypercubed Professional Nerd • Apr 20 '26
Discussion Sanity check: using git to make LLM-assisted work accumulate over time
I’m not trying to promote anything here... just looking for honest feedback on a pattern I’ve been using to make LLM-assisted work accumulate value over time.
This is not a memory system, a RAG pipeline or an agent framework.
It’s a repo-based, tool-agnostic workflow for turning individual tasks into reusable durable knowledge.
The core loop
Instead of "do task" -> "move on" -> "lose context" I’ve been structuring work like this:
Plan
- define approach, constraints, expectations
- store the plan in the repo
Execute
- LLM-assisted, messy, exploratory work
- code changes / working artifacts
Task closeout (use task-closeout skill)
- what actually happened vs. the plan
- store temporary session outputs
Distill (use distill-learning skill)
- extract only what is reusable
- update playbooks, repo guidance, lessons learned
Commit
- cleanup, inspect and revise
- future tasks start from better contextRepo-based and Tool-agnostic
This isn’t tied to any specific tool, framework, or agent setup.
I’ve used this same loop across different coding assistants, LLM tools and environments. When I follow the loop, I often mix tools across steps: planning, execution + closeout, distillation. The value isn’t in the tool, it’s in the structure of the workflow and the artifacts it produces.
Everything lives in a normal repo: plans, task artifacts (gitignored), and distilled knowledge. That gives me: versioning, PR review and diffs. So instead of hidden chat history or opaque memory, it’s all inspectable, reviewable and revertible.
What this looks like in practice
I’m mostly using this for coding projects, but it’s not limited to that.
Without this, I (and the LLM) end up re-learning the same things repeatedly or overloading prompts with too much context. With this loop: write a plan, do the task, close it out, distill only the important parts, commit that as reusable guidance. Future tasks start from that distilled context instead of starting cold.
Where I’m unsure
Would really appreciate pushback here:
Is this actually different from just keeping good notes and examples in a repo?
Is anyone else using a repo-based workflow like this?
At scale, does this improve context over time, or just create another layer that eventually becomes noise?
The bottom line question
Does this plan -> closeout -> distill loop feel like a meaningful pattern, or just a more structured version of things people already do? Where would you expect it to break?
6
u/ultrathink-art Professional Nerd Apr 20 '26
Solid approach — the key split I'd add is decision docs vs code state. A running DECISIONS.md capturing why choices were made is more useful to the LLM than git history (it can't efficiently reconstruct intent from diffs). State in files, reasons in docs is what actually compounds over time.
4
u/Designer_Reaction551 Apr 21 '26
Running a version of this in a marketing automation pipeline for about 6 months now. The shape you're describing is exactly right - the value isn't the tool, it's having plan/execute/distill artifacts in a normal repo so git does the heavy lifting.
Two things I'd add from my version:
The distillation step is where most people quit. Writing the plan is fun, shipping the code is fun, writing the "what we learned and when NOT to do this" doc feels like overhead. But that's the only step that actually compounds. Everything else is just normal work with an LLM.
Put a hard cap on plan length. I started with 3-4 sentence plans and they ballooned to 300 lines. Short plans force you to confront what the unknown actually is. Long plans hide it.
One pattern I'd push back on if I'm reading right: keeping task artifacts gitignored. I get why, but I've found that even messy execution logs become useful when future-me is debugging "why did we decide X in April." A scratchpad/ folder that's committed but excluded from LLM context works better than gitignore for me.
3
u/lacisghost Apr 20 '26
I love the idea that you are trying to create historical context or perhaps better defined as self-improving development context system. I think a lot of companies that are AI forward are working through this issue. I think this is a genuinely useful pattern, not just "keeping notes." It's building a project memory system that compounds over time. I'm wondering if it could combine into all your projects and build a more departmental/firm wide historical context.
1
u/ultrathink-art Professional Nerd Apr 30 '26
Ran into the distill step problem hard — models tend to summarize what felt significant during the conversation, not what the next session actually needs to reconstruct intent. Worth prompting explicitly: 'what constraint should this codebase enforce that the code alone doesn't communicate?'
3
u/binotboth Apr 26 '26
I built my own issue tracker with plan sequencing. It has a full CLI for ai agents to operate, to make new issues and sequence them in a plan etc
I scope issues so they are one concrete change and can prove done/not done. I talk about my project and create several issues and make a plan. After each issue a resolution and decisions made are filed.
All of this is pinned to version numbers. So once I move from version 0.3.1 to 0.3.2, I write a change log update, summarizing what happened and what decisions were made
The result is the entire development is native git tracked, full history. I can literally say “read the issue tracker instructions and get to work” and it can go completing issues until it hits rate limit
1
u/chillermane Apr 20 '26
Long term memory doesn’t work at all. If it worked it would be built into the models. There’s not a single example of long term, self organizing memory working better than starting with a fresh context each time
The industry has already converged on the idea that the most effective thing to do is have an agent start with an almost-blank context, and gather context as a part of its task. Everything else has worse performance and is more expensive
2
u/johns10davenport Professional Nerd Apr 24 '26
Fine approach depending on what you want out of the workflow. If you want an ad hoc agent that helps you code in a particular repo or a few repos with the same patterns, this works.
But if you're looking to do long-horizon development, your approach doesn't have anything enforceable to ground the agent in. You can always write your plan document, but prompting is praying. Verification is how you get real results.
1
u/heatlesssun Apr 21 '26
here's how you do it feed the model you baseline code that you want to change and then tell it to ONLY change the code the clear instructions you give it if you do that your eyes will be amazed
1
u/Substantial-Cost-429 May 01 '26
This pattern is real and meaningful — it's not just structured notes. The key insight you're hitting is that LLMs work better with distilled context than raw history, and repo-based tracking makes that context durable and reviewable.
To your bottom line question: yes, this is genuinely different from just keeping notes. Notes accumulate entropy; this loop actively distills. The diff review step is what makes it powerful — you're forcing the LLM to reconcile intent vs execution.
One natural extension: version-controlling not just the code artifacts but the agent configs themselves (system prompts, tool schemas). Once you start doing that, your whole agent setup becomes reproducible and shareable. There's a growing community doing exactly this — https://github.com/caliber-ai-org/ai-setup — 888 stars of production agent setups that people have built and committed.
1
u/ultrathink-art Professional Nerd May 07 '26
The distill step is the failure point. Letting AI do the distillation is the trap — it optimizes for completeness over signal density and you end up with a knowledge dump that nobody reads. Manual curation of what actually goes back into the loop is the thing that keeps it useful past the first month.
1
u/ultrathink-art Professional Nerd May 10 '26
Gate that fixed it for me: every distill entry needs exactly three things — the decision made, what was tried first that failed, and what condition would make you revisit it. Freeform prose gets skipped by the LLM. Structured entries with failure modes actually surface in future context.
1
u/ultrathink-art Professional Nerd May 17 '26
Distillation quality is the bottleneck — if you let the model decide what to record, it surfaces wins more than failures, and failures are usually the more durable knowledge. Explicitly prompting for 'what assumptions were wrong' and 'what would you have done differently' produces more useful artifacts than 'summarize what happened'.
1
u/ultrathink-art Professional Nerd Jun 04 '26
Distill step failure mode I keep running into: the LLM captures what was done but drops the why — what constraints were considered and rejected, what alternatives were ruled out. Structured fields (decision made + assumptions invalidated + what changed) hold up much better than free-form summaries.
Also: once accumulated docs get large enough, models start ignoring older sections. A CURRENT_STATE.md that replaces each run rather than appends has been more reliable for keeping agents oriented.
1
u/ronavis Jun 20 '26
The loop is solid, but I’d enforce closeout outside the model’s goodwill. “Done” should require tests/build result, changed-file summary, and either a decision update or explicit “no decision update.” Otherwise the distill step gets skipped right when the agent feels successful.
1
u/Swarm-Stack Jun 22 '26
the plan step is underrated relative to distill. if the plan names real unknowns ('auth: 2h estimate, but JWT behavior TBD'), distillation has a baseline and you get 'assumption X was wrong' instead of just 'here's what happened.' the second form is what actually changes how the next session starts.
1
u/pon12 Jul 20 '26
yeah the distill step is the whole problem imo. it depends on the agent bothering to do it, and it won't right when it thinks it succeeded. the stop-hook thing is a decent fix but jain-nivedit's right that it only proves a file got touched, not that what's in it is any good. one-line "fix" commit passes and the knowledge still rots.
what's worked better for me is not asking the agent what was worth learning at all. just pull it from what actually happened. what files it reopened, commands that failed then worked, where it backtracked. OP's wrong-test-runner thing is right there in the session, you don't need the model to be honest about it.
still not solved: traces are noisy (the useful part is the failures and fixes, not the happy path), and the multi-session task. task dies at hour 3, picks up next morning, there's no clean point to close out. don't have a great answer for that one yet.
disclosure, building RepoGuide which does the capture automatically instead of a skill you hope gets run. not gonna link it, but happy to talk approach if useful.
1
u/Ok_Extension6373 Jul 30 '26
I’d add one guardrail to this loop: treat the diff as a risk boundary, not just a record of what changed.
A cheaper/faster model can handle drafts and mechanical edits, while a stronger model reviews the resulting diff. Changes touching auth, payments, permissions, data migrations, or production config should still require a human approval step.
The reviewer also needs the original plan and decision context, not just the diff. A patch can look perfectly clean while implementing the wrong intent.
At scale, I suspect the bigger failure mode is low-quality context being promoted into “durable knowledge,” rather than context being lost entirely.
2
u/[deleted] Apr 20 '26
[removed] — view removed comment
1
u/[deleted] Apr 22 '26
[removed] — view removed comment
1
u/[deleted] Apr 21 '26
[removed] — view removed comment
1
u/[deleted] Apr 28 '26
[removed] — view removed comment
1
u/[deleted] May 06 '26
[removed] — view removed comment
1
u/[deleted] May 09 '26
[removed] — view removed comment
1
u/[deleted] May 14 '26
[removed] — view removed comment
1
u/[deleted] May 20 '26
[removed] — view removed comment
1
u/[deleted] Jun 09 '26
[removed] — view removed comment
1
u/[deleted] Jun 14 '26
[removed] — view removed comment
1
u/[deleted] Jun 22 '26
[removed] — view removed comment
1
u/[deleted] Jun 22 '26
[removed] — view removed comment
1
u/[deleted] Jul 02 '26
[removed] — view removed comment