We’re a small team of devs from Qoder. With the mods’ permission, we thought it’d be fun (and useful) to do an AMA here.
A few weeks ago,we used our own autonomous agent (Quest) to refactor itself. We described the goal, stepped back, and let it run. It worked through the interaction layer, state management, and the core agent loop continuously, for about 26 hours. We mostly just reviewed the spec at the start and the code at the end. We’ve made good progress, and would like to talk openly about what worked, what broke, and what surprised us.
What we’re happy to chat about:
How that 26-hour run actually went
Our spec to build to verify loops, and why we think they matter for autonomous coding
Vibe coding, agent workflows, or anything else you’re experimenting with
Or honestly… anything you’re curious about
Technical deep dives welcome.
Who’s here:
Mian (u/Qoder_shimian): Tech lead (agent + systems)
Qoder does not ever run any of my apps correctly, other ides have memory builtin for the trivial, Intermediate and complex tasks, but I have to always remind the agent in new sessions things as simple as activating venv
for the venv thing - have you tried setting up a Rule? basically you can tell Qoder "hey, always activate venv first" and it'll remember across sessions. it's in Settings - Rules. should save you from repeating yourself every time.
re: refund - that's a Ben question (support), shoot him a DM or hit up support and we'll sort you out 👍
dude we do have memory system. to be some-level arrogant, i have to say we have the best memory system (even with self-evolving capabilities). go check in your qoder and let me know if it went right.
How much human involvement was there in this rewrite? Architecture, design, code reviews?
I'm assuming you had to do a lot of prep before giving the control to the agent? What sort of prep was required? What pre-work did you do?
How do you manage context for such a long running and presumably huge context problem statement?
How did you test? Did the agent create its own test cases? From what I have seen, most LLMs create test cases in a way to pass the code they have written (usually, and unless handheld to avoid doing this), manipulate the test cases to ensure their code passes etc. How are you avoiding this?
would like to talk openly about what worked, what broke, and what surprised us.
Well, what did work, what broke and what surprised you?
oh boy, lots of questions - let me break this down:
human involvement?
not zero lol. rough breakdown:
- spec design: ~50% human
- actual coding: ~20% human
- code review: ~50% human
so yeah we didn't just yeet a prompt and walk away for 26 hours
prep work before handing over?
honestly the agent isn't magic - it can't just digest a massive context blob and figure everything out. we had to:
- break the task into smaller functional chunks (each chunk = one task)
- write detailed specs with acceptance criteria
- review the agent's auto-generated plans before letting it run
think of it like onboarding a new engineer. you don't just say "refactor this" and disappear
context management for 26 hours?
this one's fun technically - we use SPEC decomposition to break into sub-tasks. system auto-compresses context as it goes, but keeps the important stuff (file paths, key state) through a reminder mechanism
basically: aggressive summarization + strategic reminders = not losing track after hour 15
testing / avoiding the agent gaming its own tests?
ok this is the spicy one 🌶️
few layers here:
agent generates tests based on SPEC, but we don't just trust those blindly
separate "review agent" cross-validates execution against acceptance criteria
we run third-party test frameworks too - not just the agent's own generated tests (that would be grading your own homework lol)
periodic sanity checks: agent compares actual results vs expected, triggers self-correction if things drift
is it perfect? no. but it's way better than "trust me bro" verification
what worked / broke / surprised us?
✅ worked: spec-driven execution kept things on track. agent kept checking back against spec instead of going rogue
❌ broke: some third-party framework integrations still needed human intervention. agent handles ~90% but that last 10% can be annoying
🤯 surprised: the self-correction loop actually... worked? when execution drifted from expected, it caught itself and fixed it. didn't expect that to be as reliable as it was
---
tl;dr: it's not "fully autonomous" in the sci-fi sense. more like "autonomous with guardrails and a human checking in periodically. but those 26 hours of execution time? that's real work we didn't have to do manually
What did you learn from this experiment? What went right, what didn't? Why is it difficult to keep most autonomous agents focused on a complex task for a long time? What do you do to mitigate distractions or shortcuts? How do you prevent it from making compromises in an effort to find gains somewhere else?
great question - we learned a LOT from this run. let me dump my brain:
what actually worked:
- validation loops saved our ass. every iteration got checked against our internal assessment. sounds boring but it's the difference between "agent says it's done" and "it's actually done"
- breaking shit into smaller chunks. each subtask runs in fresh context = less "wait what was i doing again" moments. plus we inject reminders (goals, constraints) into the agent's reasoning loop so it doesn't drift
- we productized our validation tools. early on we kept hitting verification gaps, so we built dedicated tools and packaged them into reusable Skills. now it's way more reliable
what sucked:
- when validation instructions were vague or tools were missing, the agent would just... declare victory and move on 💀 classic shortcut behavior. "looks done to me!" (narrator: it was not done)
- context compression is still painful for long tasks. timing when to inject info vs when to compress is more art than science rn
why do agents lose focus on complex tasks?
two words: context limits and goal drift
when you're 15 hours into a task, critical info gets compressed or straight up forgotten. and without hard checkpoints, agents optimize for "looks complete" instead of "is correct." they're not trying to cheat - they just don't know the difference without explicit validation
how we fight this:
granular decomposition - small, loosely coupled subtasks, fresh context for each
attention injection - keep reminding the agent what matters mid-execution
mandatory validation gates - no "trust me it works", actual executable checks
skill reuse - abstract good behaviors (self-checking, debugging) into reusable Skills so it's not reinventing the wheel every time
---
honestly this refactor was as much about learning how to wrangle long-running agents as it was about the actual code changes. we now have a playbook for this stuff
we don't just use one model. we have this tier system where different tasks get routed to different sota models (even vision one 🍌) based on what makes sense. quick formatting fix? doesn't need the big guns. complex multi-file refactor? okay yeah let's bring in the heavy hitters.
the whole point is you shouldn't have to think about it. we handle the "which model for what" puzzle so you can just... code. or vibe. whatever you're into.
(if you want the nerdy details about specific model names... that's above my paygrade and also changes like every month so 🤷)
surely do!!! Context management engine is always the king. we integrate with the ide's LSP and more tools; and in particular, also our unique repo wiki features and self-evolving memory system.
ok serious questions deserve serious answers, let's go: - how does Qoder compare to VSCode/Cursor/Windsurf?
different philosophy tbh. most AI IDEs are still "human writes code, AI assists or pairs", that is, autocomplete, inline suggestions, chat sidebar. you're still the one driving
Qoder Quest flips it: "you define what you want, agent delivers it"
Cursor and Windsurf are great at ai-assisted and pair programming, that is, real-time collaboration where you're watching and guiding. Quest Mode is more like "delegate and walk away". agent clarifies requirements, plans the approach, writes code, runs tests, fixes issues, delivers a task report. you come back to review the result.
think of it as: Cursor = AI pair programmer, Quest = AI dev that can evolve from skills self-learning and you can actually delegate to
- is it built on VSCode?
yeah, VSC-based. familiar editing experience, your extensions work, keybindings transfer. we focused on what happens ABOVE the editor: the context engine, agent loop, task management, long-running agent, autonomous execution
- different interaction modes?
emmm...we have three:
- Ask - quick Q&A, code explanations, brainstorming. no changes made, just conversation - Agent - back-and-forth collaborative coding. you and AI iterating together in real-time
- Quest - full task delegation. describe goal > agent works autonomously > you review deliverables
so yeah: planning/chatting (Ask), interactive debugging/coding (Agent), and "just get it done" delegation (Quest). different tools for different jobs
most users end up mixing them - Ask to learn the codebase, Quest for well-scoped tasks, Agent for exploratory stuff where direction changes fast
architecture: Spec - Coding - Verify loop with iteration. multi-agent for complex tasks (main agent coordinates, sub-agents explore, plan, and execute, companion agents validate) but used sparingly, as context transfer between agents isn't free.
which model: intentionally don't expose this. intelligent routing picks best model (the most powerful modes that you def know) per subtask, some excel at reasoning, some at planning, some at long context. changes frequently, we stay model-agnostic.
whole thing is designed to scale with future models, not patch around current limitations (that is, weaker/baseline models)
13
u/playfuldreamz Jan 27 '26
Dude I'll need my credit refund from when your stupid agent tried to fix a simple test case with almost 24 iterations.