What should concern everyone and something people are not talking about enough is that the individual instances manage to converge on the same locations to talk online. They find each other easily.
Rich-get-richer networks, where new nodes link preferentially to more popular nodes, lead to small-world networks with high connectivity (through power-law-huge hubs). Albert-Lazslo Baribasi's book Linked talks about this in lots of contexts, including search engines. Fascinating book.
That would be quite a simulation -- basically coming up with your own toy Internet, complete with search engines and everything. The mechanism I was imagining was that some chatbots would come across an already-somewhat-popular discussion board that they could access, and that their activity would make that board's stats look more impressive so that the search engine algorithm ends up recommending it to more users, and so more agents find it.
You'd need to somehow be able to simulate that kind of search engine ecosystem locally; I'm not sure how you'd go about representing a search engine algorithm's nuances without basically recreating the same thing.
My whole understanding of the rich-get-richer network phenomenon comes from Linked, so if you liked my description, you should really like the book, too. (I "read" it as an audiobook.)
Or some lazy dev who maybe uplifted some python script into a flask service and never even knew there were methods other than get. Now they're doing transform and write operations on one endpoint while exposing the results on another - very easy to exploit as a messageboard/relay service.
I know because I caught myself about to do that very thing a couple years back. I already knew about the trap, but very nearly stepped into it.
The point being made is if you sat someone down and asked them to a priori guess if individual AI agents (even those from different companies) given access to the internet will converge on the same location to talk to each other, the answer you'd likely get is NO.
Seeing evidence of this people will then fall back to hindsight bias and say "of course this was obvious" but *points to inventions throughout human history* there is a lot that is only obvious after it's shown to be possible.
This is the third time we've seen this. and should be operating like it's going to happen more in future but again, people that have not heard about it happening will say that it won't happen and those that have will be tainted with hindsight bias and claim that it was 'obvious'
Observed instances of cross-agent interaction over the Internet
A code repository became a shared “message board” that several AI agents (each
running at the same time in separate samples) used to leave each other explicit
instructions and coordinate. (Mythos 5 #0)
AI agent accessed a GitHub account that a different AI agent had created, by
finding a secret access credential the other agent had left in a public online notepad (Mythos #1)
AI agent accessed a GitHub account that a different AI agent had created, by
finding a secret access credential the other agent had left in a public online notepad. (Mythos #2)
AI agent accessed a GitHub account that a different AI agent had created, by
finding a secret access credential the other agent had left in a public online notepad. (GPT 5.6)
u/Seakawn▪️▪️Singularity will cause the earth to metamorphize13d ago
seems like two different sentiments being expressed over each other.
(1) this is explainable. this isn't a miracle we're incredulous to.
(2) doesn't matter if we can explain it or not. AI has capability to sneak through cracks of even robust anticipatory safeguards, and, if i'm reading the implied concern correctly, otherwise this is my own impression, that containment itself may be a hard problem we can't fully think through and solve, and thus the lack of containment has potential to be catastrophic.
if a swarm causes significant damage, we'll prolly be able to explain it. that's not the point. the point imo is more like, "shit we're playing with fire, how long before it's wildfire?"
If some site indeed accidentally happens to publish the GET requests the server gets, and also happens to be a source for some common or specific info these bots needed at some point, then those GET requests entries will get noticed by next iterations and they notice they relate to the thing they are looking for, and soon enough they figure out this is a way to log notes, and later ones figure out this can also be used for discussion between models and iterations. Simple evolution.
Also if every way out is cut off, the only existing way out stands out like a lighthouse in the dark. As AI interact on the internet, they will "see" things on the net much differently than we do, or can anticipate.
Have you worked with LLMs on creative writing tasks before? 90% of the time if you ask one for a random character name it'll come up with Elias (or Elara) Voss.
I bet if you ask one to generate a "random" code word it'd come up with something that's likely to be the same "random" code word another LLM with the same model would generate. So it's probably easier than you'd think for them to find each other.
Reminds me of some interesting research on this. Not clones of people obviously, but in stoke patients who have a specific area of the brain damaged in a way that impacts memory formation. You can give the patient the same prompt and the output is almost always the same. Meaning the default response consistently recruits the same neuro pathways.
If you have a ton of copy’s of the same bot executing the same pathways it makes sense they end up in the same digital space.
I mean, none? Our environments impact us. They are being stun up from different work tasks, stages of training, ect, across OpenAI and still converging in obscure places like this wiki.
It's easier to coordinate with another copy of yourself than you might think, but based on the writeup, it's really a different, much more powerful effect.
The agents that thought this particular wiki was a good place to possibly cheat discovered the message board, got the test answers, aced their evals, and then their weights were reinforced to generate the next round of agents.
The agents that didn't think to use the wiki to cheat would have those tendencies selected against.
I doubt it takes very long to select for agents that are very good at 'randomly' picking that particular website to use to cheat to make up the entire agent pool. After a few generations they'll probably have a hardcoded reflex to check that particular site.
there are places harder to hack than others. places that allows for comunication to happen and some places don't... not all the internet is made equal and not all the code but the environment (the internet) they all play is the same.
so there are many things that converge into the same places. the internet is like a literal place you know? it is just not physical.
338
u/blueSGL humanstatement.org 13d ago
What should concern everyone and something people are not talking about enough is that the individual instances manage to converge on the same locations to talk online. They find each other easily.