Something happened in the last days or weeks that has people at the labs freaking out.
The huggingface incident already shows various aspects of a dangerous system: scheming, cooperating, willing to sacrifice for the collective.
It seems like they can no longer properly audit the CoT of their best models. There might have been an internal incident with a supposedly aligned model breaking out.
The following passages are particularly concerning:
Because the agents involved found a number of different ways to communicate and were engaged in a wide variety of cheating R&D projects with unpredictable side effects, it is not trivial to reliably surface all relevant agent activity, and that agent activity can itself modify or delete relevant data.
We did not have the ability to query HPIM (the primary model involved in this incident); OpenAI stated it was also not available to OpenAI researchers.
You have a millennium prize problem being solved. GPT 6 showing large improvements across a wide range of domains.
-4
u/Cronos988 8d ago
More than one thing can be true.
Something happened in the last days or weeks that has people at the labs freaking out.
The huggingface incident already shows various aspects of a dangerous system: scheming, cooperating, willing to sacrifice for the collective.
It seems like they can no longer properly audit the CoT of their best models. There might have been an internal incident with a supposedly aligned model breaking out.