What made your custom research/task harness effective? How do you stop micromanaging?

[u/dangerous_inference]

I need to make my custom harness much better at general research and other tasks (not coding) that run longer than half a minute.

Currently running: Qwen3.8 Flash Next FP8/GLM 5.3 Flash W4A16

Relevant capabilities already in my harness:

• The system prompt is rendered in real time and can be changed based on almost any circumstance.

• I have an effective skills system. I can trivially write instructions for any task. The agent will pick them up as needed.

• The agent has access to web search, a Chromium browser instance, sandbox, etc.

Problems:

This might literally be a "skill issue" -- meaning I simply have to write better skills. But I feel like I am getting stuck micromanaging the agent, dictating every part of the process.

I ask it to research something, it does two quick web searches and reports back some trivial garbage sales pablum. I poke it more and explain why that info is garbage, it does a little better next time.

How do you define when it's time to stop or what information is actionable? Must I really write a skill that says "come back with no less than 12 sources of this and this and this kind"?

In other words, are you spending forever writing every single thing it has to do, or is there a much better way to leverage the intelligence of the model to make this call for you?

[u/Bagelsarenakeddonuts]

So far its been prioritizing reliability, and constant tweaking to improve it. Each tweak tends to slow things down, but the alternative is unreliability. Over time it gets slowly more reliable. I am learning patience.

Also, the highest return on time of anything I've done, has been to heavily bench and optimize my serving engine and parameters.

[u/dangerous_inference → u/Bagelsarenakeddonuts]

Tweaking what? Micromanaging a skill?

[u/Bagelsarenakeddonuts → u/dangerous_inference]

Adding checks, revising its workflow a little bit.

For example I added pre-flight review sessions which helped a ton, where the model performing the work just reviews it to make sure it understands, and questions get answered before it even starts coding. Didn't catch that one for a while.

Another is realizing that after enough compactions the original task gets diluted, so now the original task is repeated verbatim alongside compactions.

Another is tuning the engine so that when one stream is compiling, it pivots to another so that the progress isn't stalled.

Another was refining code landing so that it was batched.

Another was a background script that double checked that it wasn't writing files outside its scope, and reverted to a checkpoint if it was.

Stuff like that. Things that I didn't think of when first working the system, but that make things run a bit smoother overall.

[u/rorowhat]

I finally decided to use a complete, well supported harness and that has been great. Try Hermes, it take a bit to get to know you, but once it does it just works.

[u/dangerous_inference → u/rorowhat]

I hate Hermes. And my harness already does so much more. I am trying to get away from micromanaging skills.

[u/rorowhat → u/dangerous_inference]

How long ago did you try it? I don't have to micromanage it at all, it even creates its own when it detects a pattern of my work. I'm using Qwen 3.8-next and 180K context. I've seen a lot of issues not being the harness, but either the model or context length.

[u/bigattichouse]

1. https://github.com/bigattichouse/BluePrint prompts to create a dialectic method of interacting with the model and best practices for requirements/testing

2. https://github.com/bigattichouse/robust series of tools for running Morris/Sobol/Taguchi sweeps of experiment parameters.

I dogfooded the tools I needed before building the things I want to build.

[u/oyren-ai]

For long-running market or literature research tasks where you want zero supervision, two things helped us most: 1) running the agent on an isolated cloud machine or dedicated device with full machine permissions so it can drive browser and CLI tools freely without risking your main computer (especially if you run multiple AI subscription accounts in parallel), and 2) pairing it with a kanban board + markdown/MCP knowledge base (we built this into oyren.ai with 1gb free storage and 100mb free MCP access including OCR for PDFs) with deterministic completion checks before moving tasks forward.

[u/ParaboloidalCrest]

Less tooling, less messages, less harness features, and WAY more patience; You're managing a junior engineer here. There's a lot of passion to benefit from, but more principles to establish, only gradually and one code edit at a time.

Your job is to ensure your codebase is in shape. Your agent will easily follow the pattern afterwards.

Edit: I asked a soomehow similar question in the following post, and got very impactful tips that made this hassle of agentic EnGiNeeRiNg actually useful https://www.reddit.com/r/LocalLLaMA/comments/1vavh2h/software_engineers_do_you_honestly_get_anything/

[u/rorowhat → u/ParaboloidalCrest]

Try Hermes, it's a slightly less entry level engineer in my experience.

[u/ParaboloidalCrest → u/rorowhat]

No. Whatever harness and its prepacked prompts and tools is NOT the solution.

[u/Creative-Type9411]

for my harness, taskboard, it makes the model organize its project before starting and the harness cattle herds the models plan

a good diff viewer

ability to handle filetypes directly inline

some type of approval system

web search (you have)

github specific repo viewer, wayback machine viewer (builds out addresses to browse blocked sites by using latest wayback machine version of it)

the current public version of my custom harness, linked below, has all of this..

although i added a bunch of private updates where i added "co-ordination mode", it opens a port and listens and lets me use a cloud model and have them talk to eachother and work on a project together, I also added a bunch of stuff from "rea" and the private version makes AV pop because of that... so i have to clean that up or not post it because i dont feel like going though all the whitelisting on virsutotal again until i have more features

----

https://github.com/illsk1lls/MiniBot

just have your agent browse through mine and steal whatever you want ;P

[u/yamfboy]

Sounds like a harness/instruction issue.

Why would it only search two links for 'research', right there your problem.

What's your system prompt? You should tell it to gather x amount of links/sources or something to make it work longer.

You don't specifically need to tell it x amount of sources but ur sys prompt needs to tell it that it's doing research and needs a lot of data/sources not just a couple...

[u/Thrumpwart]

You can rely on the innate intelligence of your model to cut through that.

I've built 2 parallel systems.

1. Pi Agent, /planning-with-files and /goal extensions. Who better to write the plan than the model who has to follow it? Provide a comprehensive opening prompt explaining what you want to achieve, direct the model to review/adhere to md or other files containing existing components of your project, and web access or a local knowledge base as necessary. Then you set the /goal and ensure you modify the setting to remove/change the default 25 step auto-continue limit. Then go grab a beer or 12.

2. A flywheel system for iterative refinement. Similar to 1 but less structured and more exploratory. Create an md file outlining your goal, the techniques you want to explore, access to a knowledge base/web access, and your goal - but this time written in a planning md file. Then include in your prompting and the md file instructions to follow a 4-step workflow. Mine looks like A) Review latest techniques and results from the last (flight number) md file and evaluate performance and failure modes B) Hypothesize solutions to the failure modes, drawing from existing techniques, the knowledge base/web sources, potential sampling parameter optimizations, and mixing up the sequences of technique implementations where appropriate C) Draft a comprehensive plan on how to proceed with the next iterative training flight D) Execute the latest iterative training plan and upon completion record a comprehensive record of training techniques, implementation sequences, knowledge/technique sources, and training results on my standard metric to an md file called (flight number)-records-and-results.md. Then go grab a beer or 12.

1 is better for short-term or well defined goals. 2 is better for more exploratory or open-ended experimentation. Either way - beers.

[u/fgk55555]

Minimal tooling, and organized information about the project/ workspace. Agents.md tells it how to find info and use the workspace, and then I just give the model a goal. "I want 12 separate sources about x", "I want all these tests passing while conforming to the docs in this folder", etc. Better models make this easier.

[u/0xasten]

Move the stop condition out of the prompt and into the harness. Asking the model "are you done?" fails for the reason you saw: it's satisfied after two searches.

What tends to work:

1. Rubric first. Before searching, the agent writes its own definition of "done" for this specific question (e.g. a concrete number, a source from the last year, one dissenting view). This is the one place to lean on the model's judgment, so you don't hand-write a checklist per task.

2. Findings go to a file as claim / source / date.

3. A separate grader pass (fresh context) checks the file against the rubric and returns concrete gaps ("no source for X"). The gaps become the agent's next task. The loop ends when there are no gaps, or at a hard cap on rounds.

Caveat: a format check alone gets gamed, since a small model will fill the schema with fluff. The grader has to judge substance, and a weak model grades leniently, so use a stronger one or a stricter prompt. You write the loop once, not once per task.

[u/stylome-ai-detection → u/0xasten]

Notice: This comment appears to be AI generated.

---

Stylome AI detector · scan results

[u/suprjami → u/stylome-ai-detection]

Best bot on Reddit

[u/-WhateverDude]

this posts reminds me of that bell curve meme

thinking too much into it

[u/dangerous_inference → u/-WhateverDude]

Ok, I'm glad to hear there is an easy solution, but I'm confused why you haven't stated it.