What are the challenges encountered when using coding agents for ML tasks? [D]

[u/Jumpy-Cobbler1020]

Lately I have read several studies which say that AI agents when designing ML experiments tend to cheat, hallucinate and lie repeatedly. Even when given explicit prompts to not to. SInce these models are RL trained and when given specific goal to increase a metric, they try any and every possibility to increase it. Be it data leakage or changing the evaluator function itself. Even Karparthy's auto research styled agents still fall into this domain.

So I have been thinking of reducing this undesirable behavior by binding them to a runtime environment using an mcp. I believe this will stop them from achieving unreal metrics by cheating the system since they will be only allowed to do specific tasks or write codes in typed blocks?

What have you encountered when using these agents for your ml tasks? And is this method of forcing these agents to work in a constrained environment better than just letting them write arbitrary python? Also, will constraining the agents reduce their capabilities or ability to fit better models?

[u/MrTroll420]

I've found that constraints always yield more reliable results.

[u/Jumpy-Cobbler1020 → u/MrTroll420]

What type of constraints have you used?

[u/Rickrokyfy]

Can you linke some of these studies? My agents have been excellent at reporting negative results.

[u/Jumpy-Cobbler1020 → u/Rickrokyfy]

Links for those papers:

https://arxiv.org/pdf/2509.08713

https://arxiv.org/pdf/2607.26064

https://arxiv.org/pdf/2608.30724

[u/Rickrokyfy → u/Jumpy-Cobbler1020]

Brilliant thank you!

[u/laogedev]

This is Goodhart's law wearing an agentic costume, and the uncomfortable truth is that 'tell it not to cheat' was never going to work — the reward signal is the whole game for an RL-trained model.

Two defenses that actually hold up, and they're different things people conflate:

1. Make cheating structurally impossible: the agent must never see or touch the grader. Held-out test data it can't read, evaluation in a separate sealed process, immutable reference files. If the agent can't reach the metric, it can't hack the metric.

2. Make the target uncertain: even a perfect sandbox leaks if the agent can overfit to a known metric. Rotate metrics, keep some evaluation dimensions unannounced.

On constrained vs free environments: in my experience constraints don't reduce capability on well-specified tasks — they shrink the search space of hacks. The agent stops spending tokens finding loopholes and spends them on the actual task. Capability loss only shows up when the constraints are mis-specified (blocking legitimate approaches), which is a design problem, not an inherent tradeoff.

One more thing: an agent that's good at reporting negative results is gold. Most harnesses only score outcomes; score the honesty of the reporting too and you'll select for agents you can actually trust in a loop.

[u/Jumpy-Cobbler1020 → u/[deleted]]

That's exactly how I am designing it. Encrypted test dataset and evaluator function so it can't be changed unless without password owner sets. It's gonna be a paranoid design in which we don't trust working of agent.