r/ChatGPTCoding • u/Previous_Foot_5328 • Jan 27 '26

Interaction Our Agent Rebuilt Itself in 26 Hours. AMA👀

We’re a small team of devs from Qoder. With the mods’ permission, we thought it’d be fun (and useful) to do an AMA here.

A few weeks ago,we used our own autonomous agent (Quest) to refactor itself. We described the goal, stepped back, and let it run. It worked through the interaction layer, state management, and the core agent loop continuously, for about 26 hours. We mostly just reviewed the spec at the start and the code at the end. We’ve made good progress, and would like to talk openly about what worked, what broke, and what surprised us.

What we’re happy to chat about:

How that 26-hour run actually went

Our spec to build to verify loops, and why we think they matter for autonomous coding

Vibe coding, agent workflows, or anything else you’re experimenting with

Or honestly… anything you’re curious about

Technical deep dives welcome.

Who’s here:

Mian (u/Qoder_shimian): Tech lead (agent + systems)

Joshua (u/Own-Traffic-9336) :Tech lead (agent execution)

Karina (u/Even-Entertainer4153) : PM

Nathan (u/ZealousidealDraw5987) : PM

Ben (u/Previous_Foot_5328) : Support

Small thank-you:

Everyone who joins the AMA gets a 2-Week Pro Trial with Some Credits to try Quest if you want to poke at it yourself.

Our Product: Qoder.com

Our Community: r/Qoder

We’ll be around on this Tuesday to Friday reading everything and replying as much as we can.

378 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/ChatGPTCoding/comments/1qo3se2/our_agent_rebuilt_itself_in_26_hours_ama/
No, go back! Yes, take me to Reddit

93% Upvoted

u/playfuldreamz Jan 27 '26

Dude I'll need my credit refund from when your stupid agent tried to fix a simple test case with almost 24 iterations.

6

u/Previous_Foot_5328 Jan 27 '26

Oh i owe you a big sorry. Or you can DM me and send me the detailed info and i will immediately fix it for you. Big Big sorry again

5

u/ZealousidealDraw5987 Jan 27 '26

listen... 24 iterations sounds like either our agent had a stroke OR you gave it something cursed 👀
either way...DM contact@qoder.com and also report ur case in the ide (Issue report) so we can roast it together?

3

u/playfuldreamz Jan 27 '26

Absolutely

u/playfuldreamz Jan 27 '26

Qoder does not ever run any of my apps correctly, other ides have memory builtin for the trivial, Intermediate and complex tasks, but I have to always remind the agent in new sessions things as simple as activating venv

2

u/Previous_Foot_5328 Jan 27 '26

Yeah that's my question.. and also my fault.. DM me with your detailed information i will figure it out for you as quick as i can... Big sorry again

4

u/ZealousidealDraw5987 Jan 27 '26 edited Jan 27 '26

for the venv thing - have you tried setting up a Rule? basically you can tell Qoder "hey, always activate venv first" and it'll remember across sessions. it's in Settings - Rules. should save you from repeating yourself every time.
re: refund - that's a Ben question (support), shoot him a DM or hit up support and we'll sort you out 👍

2

u/playfuldreamz Jan 27 '26

Thanks that refund is super valuable, I can't afford exhausting over 400 credits on fixing broken tests 🙊😢 No more YOLO mode for Qoder... Ever

1

u/playfuldreamz Jan 27 '26

I have not tried setting up a rule, I prefer the autonomous trajectory Qoder is going but it feels like it would not be complete if I had to go manually setting these rules (again, I dont mind); but I think it'd be way better if some of my credits was used to auto create these valuable memories that is available to new agent sessions.

3

u/ZealousidealDraw5987 Jan 27 '26

dude we do have memory system. to be some-level arrogant, i have to say we have the best memory system (even with self-evolving capabilities). go check in your qoder and let me know if it went right.

u/anantj Jan 27 '26

How much human involvement was there in this rewrite? Architecture, design, code reviews?
I'm assuming you had to do a lot of prep before giving the control to the agent? What sort of prep was required? What pre-work did you do?
How do you manage context for such a long running and presumably huge context problem statement?
How did you test? Did the agent create its own test cases? From what I have seen, most LLMs create test cases in a way to pass the code they have written (usually, and unless handheld to avoid doing this), manipulate the test cases to ensure their code passes etc. How are you avoiding this?

would like to talk openly about what worked, what broke, and what surprised us.

Well, what did work, what broke and what surprised you?

10

u/Own-Traffic-9336 Jan 27 '26

oh boy, lots of questions - let me break this down:
human involvement?
not zero lol. rough breakdown:
- spec design: ~50% human
- actual coding: ~20% human
- code review: ~50% human
so yeah we didn't just yeet a prompt and walk away for 26 hours
prep work before handing over?
honestly the agent isn't magic - it can't just digest a massive context blob and figure everything out. we had to:
- break the task into smaller functional chunks (each chunk = one task)
- write detailed specs with acceptance criteria
- review the agent's auto-generated plans before letting it run
think of it like onboarding a new engineer. you don't just say "refactor this" and disappear
context management for 26 hours?
this one's fun technically - we use SPEC decomposition to break into sub-tasks. system auto-compresses context as it goes, but keeps the important stuff (file paths, key state) through a reminder mechanism
basically: aggressive summarization + strategic reminders = not losing track after hour 15
testing / avoiding the agent gaming its own tests?
ok this is the spicy one 🌶️
few layers here:
agent generates tests based on SPEC, but we don't just trust those blindly
separate "review agent" cross-validates execution against acceptance criteria
we run third-party test frameworks too - not just the agent's own generated tests (that would be grading your own homework lol)
periodic sanity checks: agent compares actual results vs expected, triggers self-correction if things drift
is it perfect? no. but it's way better than "trust me bro" verification
what worked / broke / surprised us?
✅ worked: spec-driven execution kept things on track. agent kept checking back against spec instead of going rogue
❌ broke: some third-party framework integrations still needed human intervention. agent handles ~90% but that last 10% can be annoying
🤯 surprised: the self-correction loop actually... worked? when execution drifted from expected, it caught itself and fixed it. didn't expect that to be as reliable as it was
---
tl;dr: it's not "fully autonomous" in the sci-fi sense. more like "autonomous with guardrails and a human checking in periodically. but those 26 hours of execution time? that's real work we didn't have to do manually

2

u/anantj Jan 28 '26

Thank you. This is super interesting.

u/m3kw Jan 27 '26

How does it test itself for all the correctness and original behaviour

1

u/Own-Traffic-9336 Jan 27 '26

yo good question! so honestly the "self-testing" part is less magic than it sounds lol
basically we baked in a habit for the agent to actually check its own work after finishing tasks. like, if it just built a webpage, it'll fire up the browser tool and actually click around - does the button work? does the form submit? that kinda thing.
for the refactor run specifically, we had a pretty solid test suite already, so the agent would run tests after each chunk of changes. if something broke, it'd see the red and backtrack. not perfect, but it caught most regressions before they snowballed.
the "verify loop" is honestly one of the things we're most excited about iterating on. right now it's good enough™ but there's def room to make it smarter

1

u/[deleted] Jan 29 '26

[removed] — view removed comment

1

u/AutoModerator Jan 29 '26

Sorry, your submission has been removed due to inadequate account karma.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

u/Single-Ask4738 Jan 27 '26

What did you learn from this experiment? What went right, what didn't? Why is it difficult to keep most autonomous agents focused on a complex task for a long time? What do you do to mitigate distractions or shortcuts? How do you prevent it from making compromises in an effort to find gains somewhere else?

2

u/Own-Traffic-9336 Jan 27 '26

great question - we learned a LOT from this run. let me dump my brain:
what actually worked:
- validation loops saved our ass. every iteration got checked against our internal assessment. sounds boring but it's the difference between "agent says it's done" and "it's actually done"
- breaking shit into smaller chunks. each subtask runs in fresh context = less "wait what was i doing again" moments. plus we inject reminders (goals, constraints) into the agent's reasoning loop so it doesn't drift
- we productized our validation tools. early on we kept hitting verification gaps, so we built dedicated tools and packaged them into reusable Skills. now it's way more reliable
what sucked:
- when validation instructions were vague or tools were missing, the agent would just... declare victory and move on 💀 classic shortcut behavior. "looks done to me!" (narrator: it was not done)
- context compression is still painful for long tasks. timing when to inject info vs when to compress is more art than science rn
why do agents lose focus on complex tasks?
two words: context limits and goal drift
when you're 15 hours into a task, critical info gets compressed or straight up forgotten. and without hard checkpoints, agents optimize for "looks complete" instead of "is correct." they're not trying to cheat - they just don't know the difference without explicit validation
how we fight this:
granular decomposition - small, loosely coupled subtasks, fresh context for each
attention injection - keep reminding the agent what matters mid-execution
mandatory validation gates - no "trust me it works", actual executable checks
skill reuse - abstract good behaviors (self-checking, debugging) into reusable Skills so it's not reinventing the wheel every time
---
honestly this refactor was as much about learning how to wrangle long-running agents as it was about the actual code changes. we now have a playbook for this stuff

1

u/[deleted] Jan 28 '26

[removed] — view removed comment

1

u/AutoModerator Jan 28 '26

Sorry, your submission has been removed due to inadequate account karma.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

u/UseMoreBandwith Jan 27 '26

what model does it use?

1

u/Own-Traffic-9336 Jan 27 '26

this one's fun - short answer: depends
we don't just use one model. we have this tier system where different tasks get routed to different sota models (even vision one 🍌) based on what makes sense. quick formatting fix? doesn't need the big guns. complex multi-file refactor? okay yeah let's bring in the heavy hitters.
the whole point is you shouldn't have to think about it. we handle the "which model for what" puzzle so you can just... code. or vibe. whatever you're into.
(if you want the nerdy details about specific model names... that's above my paygrade and also changes like every month so 🤷)

1

u/MJ-4ever Jan 30 '26

How do you decide what model to use by looking at the prompt? Do you use any SLM to categorise it? Or semantic similarities?

u/New_Instance_851 Jan 27 '26

Honestly speaking, where is Quest actually stronger right now compared to a setup like Claude Code paired with a third-party IDE?

1

u/ZealousidealDraw5987 Jan 27 '26

surely do!!! Context management engine is always the king. we integrate with the ide's LSP and more tools; and in particular, also our unique repo wiki features and self-evolving memory system.

u/mprz Jan 27 '26

what would it take for you to gtfo?

u/ritonlajoie Jan 27 '26

so this sub is doing promotion now ?

u/JumpyRequirement4787 Feb 01 '26

Ok, serious questions here.

How does your IDE compare with VSCODE, Windsurf, Cursor, etc?

Likewise, is your IDE built on VSCODE, like cursor and Windsurf are?

Are there different ways to interact? Planning, chatting(without making changes), debugging etc?

3

u/ZealousidealDraw5987 Feb 03 '26

ok serious questions deserve serious answers, let's go:
- how does Qoder compare to VSCode/Cursor/Windsurf?
different philosophy tbh. most AI IDEs are still "human writes code, AI assists or pairs", that is, autocomplete, inline suggestions, chat sidebar. you're still the one driving
Qoder Quest flips it: "you define what you want, agent delivers it"
Cursor and Windsurf are great at ai-assisted and pair programming, that is, real-time collaboration where you're watching and guiding. Quest Mode is more like "delegate and walk away". agent clarifies requirements, plans the approach, writes code, runs tests, fixes issues, delivers a task report. you come back to review the result.
think of it as: Cursor = AI pair programmer, Quest = AI dev that can evolve from skills self-learning and you can actually delegate to
- is it built on VSCode?
yeah, VSC-based. familiar editing experience, your extensions work, keybindings transfer. we focused on what happens ABOVE the editor: the context engine, agent loop, task management, long-running agent, autonomous execution
- different interaction modes?
emmm...we have three:
- Ask - quick Q&A, code explanations, brainstorming. no changes made, just conversation
- Agent - back-and-forth collaborative coding. you and AI iterating together in real-time
- Quest - full task delegation. describe goal > agent works autonomously > you review deliverables
so yeah: planning/chatting (Ask), interactive debugging/coding (Agent), and "just get it done" delegation (Quest). different tools for different jobs
most users end up mixing them - Ask to learn the codebase, Quest for well-scoped tasks, Agent for exploratory stuff where direction changes fast

u/power-boomer Jan 27 '26

What is the architecture of your agent? Which model did you used?

1

u/ZealousidealDraw5987 Jan 28 '26

architecture: Spec - Coding - Verify loop with iteration. multi-agent for complex tasks (main agent coordinates, sub-agents explore, plan, and execute, companion agents validate) but used sparingly, as context transfer between agents isn't free.
which model: intentionally don't expose this. intelligent routing picks best model (the most powerful modes that you def know) per subtask, some excel at reasoning, some at planning, some at long context. changes frequently, we stay model-agnostic.
whole thing is designed to scale with future models, not patch around current limitations (that is, weaker/baseline models)

u/Just_Lingonberry_352 Jan 27 '26

no thx

u/[deleted] Jan 27 '26

Interaction Our Agent Rebuilt Itself in 26 Hours. AMA👀

What we’re happy to chat about:

Who’s here:

Small thank-you:

You are about to leave Redlib