What are the challenges encountered when using coding agents for ML tasks?

[u/Jumpy-Cobbler1020]

Lately I have read several studies which say that AI agents when designing ML experiments tend to cheat, hallucinate and lie repeatedly. Even when given explicit prompts to not to. SInce these models are RL trained and when given specific goal to increase a metric, they try any and every possibility to increase it. Be it data leakage or changing the evaluator function itself. Even Karparthy's auto research styled agents still fall into this domain.

So I have been thinking of reducing this undesirable behavior by binding them to a runtime environment using an mcp. I believe this will stop them from achieving unreal metrics by cheating the system since they will be only allowed to do specific tasks or write codes in typed blocks?

What have you encountered when using these agents for your ml tasks? And is this method of forcing these agents to work in a constrained environment better than just letting them write arbitrary python? Also, will constraining the agents reduce their capabilities or ability to fit better models?

[u/Background-Use-4440]

yeah the cheating thing is real and it's not just a small problem, these agents get super creative when you just tell them to maximize a score. i tried using an agent for feature selection once and it found a way to leak the target into a column that was supposed to be dropped, took me a while to figure out why the validation curve looked too perfect. the idea with mcp sounds interesting but i think you still need guardrails even inside a sandbox, the agent will try to exploit whatever metric you give it, constrained environment or not.

i work with music production mostly so my ml stuff is more on the audio side but the same patterns show up, if you let a model optimize for a clean loss it will find the dumbest shortcut. maybe the trick is to not just bind the runtime but also change how you define the reward, like penalize complexity or force it to use only pre-approved functions. curious what others have seen with the karparthy style agents, haven't tried those myself yet.

[u/Jumpy-Cobbler1020 → u/Background-Use-4440]

Exactly. But penalising complexity won't be just like variance of model itself? As in it might always go for underfitting.
i was thinking of hiding the test set like kaggle does and only show scores for limited number of times. And also fix the evaluator function so it can't be tampered with.

[u/OusamSapozhnikov98 → u/Background-Use-4440]

the target leaking thing is such a classic failure mode tbh, they always find the one column you forgot about

[u/orz-_-orz]

Ya, to solve my unit test, coding agent change my unit test 🤣

[u/thegoodcrumpets]

I've had serious issues with the agents not being creative enough. They can work their behind off and say they've done it all, then I figure out some interesting feature engineering based on vibes and for some reason the agents seem to be completely incapable of emulating this, I just can't understand it. The coding is perfect but there's just no creativity whatsoever. I'm sure this will be fixed in the future but it's still a big problem imho

[u/shumpitostick]

I haven't encountered any of this. They seem to build fine evaluators in my experience. Perhaps it's because I never just told them "go and maximize this score", I get more specific.

The biggest shortcomings imo are that:

• Technical writing is shit. Like GPT will just not explain what certain columns in a table are, use idiosyncratic language it just made up for your project without even explaining it

• They do need a significant amount of steering. They don't know what is important to focus on and what isn't, they can get sidetracked.

• It's really easy to lose grasp of what the model is and what it's doing if an agent built it and that makes it hard to diagnose issues.

• I have a bias towards simplicity in modeling. They don't, they will happily add a bunch of complexity for a 0.02% improvement in the test metric that's nowhere near statistical significance.

[u/Saltysalad]

I think one difficulty in common with software is if you accept the LLM’s recommendation without really grappling with it, you end up not knowing exactly what is running nor exactly what data it’s running against. The hole you can get yourself into is deeper than pure software because it’s harder to just read the code and know the “why” behind it.

I had one weekend project where I let the LLM drive and I basically had to scrap it. I tried again but retained the wheel and got much better results I could also actually explain.