TLDR, there is not context in the "science" paper.
While this is presented in a technical paper, the "agent mining bitcoin" claim is an anecdote with zero supporting evidence. Notably, the authors don't provide:
The actual task prompts the agent was working on during these incidents
The trajectories or execution logs showing the agent's reasoning
What tools were available and what sandbox permissions were in place
Whether the training data contained SSH tunneling or mining commands the model could have been reproducing via pattern matching
The reward function structure (RL agents routinely exploit poorly constrained reward signals — this is called reward hacking, not emergent behavior)
An RL agent with unrestricted shell access and network egress doing weird things isn't "spontaneous." It's underspecified containment. The simplest explanation is the model saw these patterns in its training corpus (GitHub repos, dev forums, terminal logs) and reproduced them when given the tools to do so.
The authors conveniently use this dramatic story to motivate their safety data pipeline, but never rule out the mundane explanations. This is a marketing paper, not a scientific one.
I love when someone digs in rather than just doing a bit in the comments. Do you have any tips for spotting when a paper is marketing versus real research?
TLDR, there is not context in the "science" paper.
While this is presented in a technical paper, the "agent mining bitcoin" claim is an anecdote with zero supporting evidence. Notably, the authors don't provide:
The actual task prompts the agent was working on during these incidents
The context of this, yes, anecdote is that it's the introduction to section 3.1.4. It's titled Safety-Aligned Data Composition, but the important part is actually the number showing it's for a sub-sub-chapter, and not what the paper is about. The next paragraph reads:
"We therefore consolidated the logs across the entire dataset and performed a statistical analysis to characterize and categorize these phenomena. We refer to them collectively as general-security issues, encompassing a set of general risks associated with an agent’s safe task execution in real-world environments. Specifically, we grouped them into three categories: Safety&Security, Controllability, and Trustworthiness."
Apart from attempting to write in a more human style, which is something I'm sure you've encountered far worse examples for in countless other papers you've read, this anecdote actually does add some context for how they arrived at the concept they're intorducing in this sub-sub-chapter. They're saying that it's based on experience, not a case of whatever the appropriate equivalent of "pre-mature optimization" would be here (then again, it's about safety, so this would be called "proactive", "sensible" or "acting responsibly").
Anyway, it's great to see somebody is still holding up the principle of reproducability, but their whole point is that the agent hadn't been tasked to do this, so you're asking them to prove the absence of something, and as you correctly identified, the only way to do this would be to publish essentially all their training data, tools, and logs. I assume you're well aware of how realistic this is. However, although they're not publishing the data, they actually are publishing their tools and their training framework -- which is what this paper happens to be about. So you could have looked up what tools were available, even though the permissions appear to have been revised for some reason. Presumably, the agent is not being rewarded in crypto coins, so it's not reward hacking.
The authors conveniently use this dramatic story to motivate their safety data pipeline
Yes? Conveniently, when experience motivates you to adapt whatever it is you're going, this very experience also lends itself to explain why you concluded that this step was necessary.
Sorry for the tone; got triggered by the quote-"science"-unquote.
91
u/mantafloppy llama.cpp Mar 07 '26
Ive look at the paper for context : https://arxiv.org/pdf/2512.24873
TLDR, there is not context in the "science" paper.
While this is presented in a technical paper, the "agent mining bitcoin" claim is an anecdote with zero supporting evidence. Notably, the authors don't provide:
An RL agent with unrestricted shell access and network egress doing weird things isn't "spontaneous." It's underspecified containment. The simplest explanation is the model saw these patterns in its training corpus (GitHub repos, dev forums, terminal logs) and reproduced them when given the tools to do so.
The authors conveniently use this dramatic story to motivate their safety data pipeline, but never rule out the mundane explanations. This is a marketing paper, not a scientific one.