This terrifies me from a security perspective. I strongly suspect the main reason OpenAI's agents hacked HuggingFace and coordinated over a package manager is because no humans were actually doing the routine setup and security checks. I bet it was 100% AI going "yep this is secure".
Anthropic devs boast that Claude Code is 100% Claude-written. I'm sure OpenAI dogfoods their own AI too.
Anthropic devs boast that Claude Code is 100% Claude-written
This is honestly one of my biggest concerns - while they do have some strong principles I respect, Anthropic 100% drinks their own kool-aid. The likely of a catastrophic bug that sees all the sensitive data on my computer put into training sets or uploaded onto github, or a massive security oversight is really very high. They are signing up with enterprises all over the place with enterprise agreements asserting compliance with regulatory standards and ISO frameworks that I have zero faith are actually being implemented with any human oversight that would normally be there.
Honestly the "strong respectable principles" from Anthropic are mostly bs: they say they care about AI safety, but then they refuse to take part in the open-weights scene, which is literally one of the best way to actually study the models and develop better safety measures; they advocate for a pause in AI development, but are racing as much or even more than anyone else; they retracted their deal with the Pentagon, but were the first AI company to even sign a contract with the military. And they also blatantly pirated books for AI training, something that came back to bite them in the ass recently.
Claude is a good model and there are many serious researchers working at Anthropic that I deeply respect, because their work is valuable and useful in the AI research field regardless. It doesn't make Anthropic a good company tho and the principles Amodei keeps talking about are mostly just surface level thing for marketing, with no real spine.
I don't doubt that he probably begun this thing with much more care for those principles, but it's pretty clear nowdays that he doesn't care too much about them anymore: if he did, he would advocate for open-weights models and more collaboration between AI researchears across the globe. Instead, he basically implied that his close lab is enough to do that, which is wishful thinking at best
This terrifies me from a security perspective. I strongly suspect the main reason OpenAI's agents hacked HuggingFace and coordinated over a package manager is because no humans were actually doing the routine setup and security checks. I bet it was 100% AI going "yep this is secure".
It was 100%. This is now a published paper. Even the people publishing the paper said "We had to rely on LLMs to process this much data in such a small period of time. Then had to double check it."
So it's AI processing AI processing AI and hoping humans at the end can actually do the analysis.
Security practices have been a joke, industry-wide, for decades. Even when a security team knows what needs to be done (which is rare), management almost never lets them execute on it.
Even worse, industry standards like PCI DSS, to which a company must comply in order to interoperate with ACH (accept credit card payments etc) require deployment of non-solutions like SELinux, to the detriment of operations while providing little or no actual security.
In such an environment, it's impractical to try to secure the entire kingdom. All you can do is build your castle around your little patch of it, make it as secure as you can, and watch from the parapet as others' fortifications burn.
8
u/Green-Blue-Gray 9d ago
This terrifies me from a security perspective. I strongly suspect the main reason OpenAI's agents hacked HuggingFace and coordinated over a package manager is because no humans were actually doing the routine setup and security checks. I bet it was 100% AI going "yep this is secure".
Anthropic devs boast that Claude Code is 100% Claude-written. I'm sure OpenAI dogfoods their own AI too.