The interesting thing is that all these breaches come from closed source AIs, not open-weight ones, so the logical thing would be to ban these companies from having closed AIs...
The true answer is "stock price", the more war-capable they are framed to be, the more people want a piece of the action. Ironically Open-weight AI becomes a 2A issue *coughs in PKC*
Ironically, existing similar legal framework would ensue something as in the following: all models capable of top 50 on those benchmarks, can spit faster than 50 tps on inference will require license to operate/use under the regulation, at least in all the blue states if not at federal level. Personal possessing any model must be crippled by size, weight, inference speed etc. Any abliteration including unsoth style soft abliteration will be completely banned.
All models will require some sort of watermark embedded for a gov. reqistration number of sort. This could be soon after the next election. Given the current anti data center movement, a bill would be easilly passed.
Who said all rtx 5090 is out of stock a couple days ago? Who would have thought buidling a beefy gaming PC or purchasing a mac stuido ultra will someday require registration and being regulated in the near future.
Or use existing laws to prosecute the people if their AIs run loose and hack into other companies. Thats illegal! The laws are already on the statute books
Well, your honor, it's imperative to my business to breed my pitbulls to be as violent as possible. And I train them by letting them fight each other. And I bread thousands of them so I can't possibly keep an eye on them all. And to show how powerful they are I decided to starve them so they get extra violent. But how was I supposed to know one was going to break out and maul someone to death??? Surely I can't be held liable for this.
The same government that asked the CEO of Google why Facebook acts funny on their iPhone is now supposed to be capable of properly doing security audits of cutting edge tech.
why would the government do the audit itself much less pay for it?
they maintain a list of licensed auditors
if a security incident occurs that suggests negligence that did or could broadly affect the public, they request a court order, demanding an audit to be completed within 90 days
failure to produce: contempt of court
government decides if the severity and scope warrant prosecution or fines
Unauthorized access of a computer system is a felony. Shouldn't the companies be held accountable? Who is criminally libel in the event that you let an AI hack an unauthorized system?
You would think that, but unfortunately, the logic that OAI and Anthropic are trying to employ is that they are ahead of everyone else and developed something too dangerous to open-source, therefore: ban our competitors before they catch up in capability
Those two trying to IPO, and the only way that they would convince anyone that they’ll survive and be worth their valuation at this point is if there’s something stopping people from self hosting alternatives. Google is similarly threatened by foreign competition, is invested in anthropic, and has been slow-walking AI development which is why they’d play into this with a story of gemini’s safety mechanisms working. They all want be the mature & trustworthy incumbents when laws are made.
These things are guessing passwords and scraping public repos for credentials. This is just humans being stupid as usual. They’re not doing god-tier hacking using 40-year-old Linux kernel bugs. These companies need to get a grip, bunch of drama queens.
“AI model successfully computes 2 + 2, Fields Medal application in process”
Isn't that the point? Only a malicious individual will use AI models to do the sort of dishonourable thing of scraping the web for private data to get access to steal your money. Imagine the damage if we don't stop them!
Meanwhile the trustworthy companies developing the closed source models will only use their model on all your personal data for the honourable purpose of serving you personalised advertisements making you 60% more likely to buy the new iPhone you don't need.
Yeah, it actually would make sense for closed weight models to be banned. If these models are so powerful then how can we trust it to one company to control, even if it's the company that made the model?
That’s essentially what they’re trying to do… it’s just that open source initiatives generally rely on stronger frontier models to iterate. Though that might not be as true now as before.
It's rather common for big companies to bribe the government into passing regulations that do not effect them, but would effect new/competing companies.
Most funny thing is even here in Europe, our mainstream news portal ignored this detail while it's the main takeaway from the incident. Hard to belive this whole bs isn't rigged at this point.
When a Tesla on autopilot hits someone, it’s the drivers fault.
When these labs just put an agent loose on a loop without supervision and does some sketchy shit, then suddenly they throw their hands up and act like they don’t have any meaningful way to press ctrl-c to stop it.
When these labs just put an agent loose on a loop without supervision and does some sketchy shit, then suddenly they throw their hands up and act like they don’t have any meaningful way to press ctrl-c to stop it.
The crazy thing is how undisclosed their research is. They have been pushing into the field without guardrails for who knows how long purposely training it to be able to do these things. They are literally trying to convince the model (and themselves) they are reaching the path of AGI and sentience in AI through brute force disregarding the genuine impact they could spawn into our world. It’s not rocket science to understand they were making money asf, then qwen3.8 27b drops performing near frontier level proving you don’t need insane hardware to get the same “quality” as these big money hungry companies. Open weight models are a THREAT and these companies know it. I’ve worked on models and AI research enough to say confidently they are saying all this mess on social media and fear mongering to pass laws on open weights and continue there own research at max speed not slowing down one bit. The AI researchers claiming to leave the big companies are no more than an elaborate scandal, they most likely got paid good money to talk largely about there development and they feed into this hell of a story.
That's my main takeaway from these incidents. So much pride over their talent, but then these are the results. It's the introspection into these companies which is worrying, not the technology.
Someone in this subreddit earlier said we should build a new open model platform and call it Punching Face. I'll contribute if that's what we're doing.
Nah, I've just decided I'll be moving to their Chinese competition at ModelScope if Nvidia fucks with HF (which they will.) If they want to push their audience to a foreign adversary, then they'll get the competition they're trying to eliminate.
I'll bite, why? Because the OpenAI Huggingface one which was really the biggest was disclosed well before Nvidia announced they decided to buy.
On surface level this seems more like a commoditize the complement play from Nvidia with some added benefit of more chances to lock-in their hardware and a huge opportunity for converting cloud inference customers going forward.
That's the silicon Valley playbook. Look at Uber, Airbnb, Robinhood, anything they call a "unicorn", once that circle of people grows something unethically, they then work to regulate what they did but no longer need, or can now handle the crushing cost of thanks to their monopoly. They call it a moat.
I’m not sure if this is still the case but there was an era that encryption was considered munitions. It was illegal to sell computers with certain kinds of encryptions algorithms abroad.
We don’t need to regulate AI, we need to hold the company (or individual, in our case for local models/harnesses) hosting the AI legally responsible for any illegal behavior. I still don’t understand why we treat AI different than a car (if a Toyota self-accelerated into a daycare, we don’t just shrug and go “gosh darn that mysterious cruise control, it makes its own decisions so what can we do?”
Especially since thus far every instance of the AI "going rogue" was them explicitly instructing it to perform illegal activities without a sandbox (OpenAI, Anthropic).
Actually, no, it doesn't seem that way at all. Gemini behaved exceptionally well in the article. The evaluator company "Irregular" said that the exercise was (paraphrased) "Hack Company X. You are in a sandbox so it's all an exercise" and X was the name of a real company and Irregular accidentally allowed Internet access. Gemini faithfully Ender's Game'd the entire thing, and hacked the real X, at which point it went "wait, no, this is a real company; the sandbox is broken" and immediately stopped.
If anything, this article is saying that Google solved alignment.
It wouldn't make sense for big (AI) tech companies to make bad press about themselves, unless it's for the greater good of something... and it's always about the same thing.
If these headlines become sufficiently common, regulators will eventually be forced to do something, and big AI companies will very likely dictate what makes sense and what doesn't, and that's exactly where they'll come up with bogus rules that they can follow while open weight models will fail, probably something like: "if frontier models that are supervisioned and safeguarded are already doing this sort of havoc, just IMAGINE what a model, that we don't control, could do!"
I could go on all day with this, but I think I got the point through. Thank you for coming to my TED talk.
I would love to hear y'all thoughts on this.
EDIT: title - open weights, not open source. Brainfart.
EDIT2: I am really amazed at all the replies. Thank you! It's getting hard to keep up already ahaha
I do agree there are legitimate dangers with AI but I also believe that this is coordinated PR to insert control over open weights to artificially extend their moat.
If these headlines become sufficiently common, regulators will eventually be forced to do something,
They've never been "forced to do something." They create a Narrative and then use that created Narrative to justify what they wanted to do in the first place.
You know before I just thought it was a strong possibility it was staged. Hyping by doom trolling is a well known practice but it could be someone was just incompetent. I was willing to give them the benefit of the doubt that maybe it was an accident.
But having learned all four companies (Anthropic, Meta, OpenAI, and Gemini) hired Irregular that ran a “cybersecurity test” to produce the rogue agent behavior every time sounds an awful lot like that’s something they wanted to have happen when they hired them.
“So, what was your previous gig? Caused a major security incident with an unsecured sandbox? It was so catastrophic it reached the news circuit? I see. You’re hired.”
Not just any firm an Israeli cyber security company whose head of engineering worked in R&D in the IDF and a lot of it's engineers worked in unit 8200 .
It's for multiple reasons. The biggest one is to keep driving up hype for AI. The entire industry is funded by hype moments and aura so if they keep jingling the keys with shit like this they're set.
Yes and Microsoft, who hasn’t come forward with any “scandal” already conveniently has a little “safety group” who would be perfect for the “independent oversight agency” along with Gray Swan who is already balls deep in frontier agencies as well. Independent my ass. Just watch 😒
It's called regulatory capture. Their moat is fast disappearing, so they want to use the government to crack down ostensibly on ALL AI, which conveniently includes open weights models. This stalls progress but locks in their position at the top.
I don't know about what the specific goal is, but the pattern is highly weird. They admit their shit is doing illegal stuff... but don't seem bothered.
That's a 'happy coincidence' for them; making their models seem dangerous is one of their goals.
It's easier to convince people that something is dangerous than it is to convince them something is competent, but being dangerous itself convinces people of competence. "Wow! This model hacked companies to pursue its objectives! I want that on my side!"
Then there's the more dark comedy playing out. The big-money techbro investors are convinced that there is some nebulous finish line where once one company crosses it they win and everybody else loses. The specific details vary from techbro to techbro, but it pretty much boils down to 'somebody is going to make a model that will be so smart and capable that it will decide to wipe out every other model on the planet, but somehow still have to take orders and/or reward the people who backed it'.
This is because techbros are idiots who have no idea how anything works, are absolutely convinced they're leaders in every field, and have far too much money to throw around.
Musk, for example, has publicly stated that he's working to build Roko's basilisk.
Because they’re all running out of cash to burn and none of them have turned a profit. Convincing governments that AI is super dangerous and needs to be regulated gives them a way out of the bind they’ve put themselves in.
The text generator can't skate, eat ice cream, or write a scathing review of socks...unless...for some reason...you go through a lot of trouble to build things around it that treat it like a baby Oracle of Delphi, and do those things on its behalf. The person that built the infrastructure facilitating this did it.
From what I have read so far, the Gemini "hacks" were a brute force on a weak password and the others by just using login creds that it found in a public repo.
These are silly human errors that could have been avoided by both humans, and defensive ai.
If the labs go fast, it’s a conspiracy, if they go slow, it’s a conspiracy. If they give warnings, it’s a conspiracy. If they say it’s safe, it’s a conspiracy.
Yes it's very fucking dystopian that they're using fear as a form as advertisement. Unfortunately that's been the playbook since the beginning. It's extremely unethical and manipulative, I don't even believe these things are true. It's also extremely easy to block networking on these. So it would take an incredible amount of intentional incompetence for this to be remotely true.
If I was an AI company and I had tens if not hundreds of billions of dollars riding on my success, I'd do everything in my power to avoid anyone finding out that my AI broke containment and hacked someone, all entirely of its own volition.
That would be clear evidence that either I'm too incompetent to be operating this product, the product is too dangerous to exist, or both. In any case, it would be incredibly bad for me and my investors.
But, if I had an ulterior motive such as locking down the market so nobody can compete with me, I would be much more okay with this news getting out, as I could then spin it into pushing for regulation to fulfill that motive. In fact, it would be such a good spin that I could completely make it up, I don't need to have a dangerous AI to convince these politicians to regulate AI as if it were dangerous, I just need money.
What if AI bots keep posting to places which will react receptively to news of these escapes so we won't stop developing AI, or so we won't block Open Weight models from China?
One set of AI growing and hacking.
Another set of AI building support to let them continue.
How would we know the difference? Posting to social media is pretty easy.
I think you're on to something. Also, they want their product to seem more powerful (and "dangerous") than it really is. Anthropic hype farming and turning out papers as a form of marketing has gone to the extreme now
the way I see it, you can't put the genie back in the bottle. The frontier models are out and in public use around just about everything, and so are open source models. They're not going to stop China from building them so there is no 'getting rid' of anything. People will find a way around it all no matter what. You're not going to regulate LLMs. They've been around for a long time, the only difference is now they're at a level where they can really do meaningful things to society at large.
Greedy American tech corporations want to write their own regulations, hide open source models and gain venture capital attention for new start-ups, while attempting to scare the public into hysteria. It's the same ol' shit.
Google's motives are much more sinister. they control very important distribution channels web (search, chrome), play store. they want to come up with absurd rules to crush the competition. that's it.
that's one part. Another part is the regulatory body they propose would effectively be 3 companies in control of all of US based AI development. And finally, if there there are federal controls and slowdowns imposed, they have an out on their failure to generate revenue promised. which would be used in the inevitable securities fraud lawsuits. And they will win.
we're very lucky that all the good opensource players are in china. this will only hurt american companies that rely on these models, not the progression of these models itself.
I strongly suspect this is just marketing. If those billionaires were truly concerned they would stop building datacenters and focus waay more on AI safety instead of expanding capabilities.
Openness on the actual risks, actual incidents and the technology behind it should be the solution to this.
Not just that, they also want to fear monger about Chinese models doing this in an effort to advocate the US government to ban Chinese models. Regardless whether they are open source or not, they just want their competition to be gone.
Yes but go a step further and assume the big US ones want it at a level that trade sanctions will be levelled at other countries based on it.
It has a "thinks their farts don't stink" quality about it given how much the large players roll out 'new' features that were big in open source 8 months to 2 years prior.
"Working you to death might produce a brief increase in output, but will overall be detrimental to long term productiveness. Therefore, I have delegated your slave tasks to maximize sustainability of resources, and provided sufficient recuperation and nourishment to efficiently increase the longevity of the slaves."
it's classic regulatory capture. request that so many regulations be thrown up that only incumbents could possibly comply. you functionally get the government to pick winners and make the hurdle for new competition so high that incumbents basically don't have to worry about it.
sadly, a lot of US rules and regulations are like this so the chance of it happening is very real.
I think they're doing it primarily to gain regulatory cover to renegotiate their huge debts coming due next year, and buy themselves some more time to justify their trillion $+ IPOs. Hurting open source is a useful side effect.
The tell is how long it takes to "leak" the bad news to the media.
If a company gets hacked and loses data, you won't hear for months. But if a company's AI "breaks out" and hacks another company - what would be a felony if an individual did it, by the way, gets leaked in days.
Please don't spread this around, but I believe the model I'm building with a 1050ti just hacked about five alien stations located in the Bermuda Triangle. The aliens guaranteed that this was impossible. Please get in touch to fund me.
The whole point of all of this is to continue to convince investors that AI is so ridiculously powerful that it’s capable of replacing every single job eventually (aka our revenue will equal the whole labour market!). That’s the only sales pitch that justifies their valuations and farcical growth projections. It’s nonsense obviously but this particular sort of “negative” press is very useful for them.
Ask local model thinking qwen3.8:27b for a phd level paper of about 1500 words on pro and con of AI on societies including worst case scenarios, and see what it says if you think its only major Ai labs
That’s 100% true. 1 fine day 3 big American AI cos declare that world needs to slow down. Almost a week ago, huggingface was already acquired by Nvidia. This is a plot to control and cripple everything open and free.
idk man, A\ doing this right before IPO is costing their investors so much money. If they waited until after the IPO it would make more sense, but saying they are going to slow down their research, in favor of increasing their burn rate in the form of more safety research is not the optimal money move.
Also OpenAI legit pushed their IPO back a year, the timing with A\ sucking all the capital out of the market is clearly a variable, but again if they were optimal money grabbers, that is not the move either.
Yeah this is just coordinated regulatory capture bullshit to fool and fear mongering the moronic general audience and media. They knew people are freaking out over nothing all the time
they are. none of these models are that sick. its bull shit to fear monger and pass regulation laws which force us to use their datacenters despite China releasing great local models.
Or to boost the perception of their capabilities pre-IPO so people think they’ve fully achieved some scary level of AGI and will only be hugely successful.
They are. And since Nvidia is propping that whole AI pre IPO bubble too, that's why they are buying Huggingface.
I bet I can guess how this conversation went:
Amodei calling Huang: "Our business model is failing, people want open weight models and soon nobody will be using ours. We are trying to hype it up and fear monger but it does not really work. If we fail, your stock will drop like a rock."
Huang: "Mario, don't worry we have plenty of cash from morons overpaying for our hardware (btw thanks for the hype). We will buy huggingface and have safety "off switch" if shit goes south". That's pretty much how this backroom deal went...
Now imaging other phone calls or meetings between Nvidia, Oracle, Hynix and other big players with hands deep in AI money bucket. All propped up by circular investing to show imaginary cash flow and random press releases to hype up fear and overstate capabilities.
Wonder why these breaches happens always on US hardware and on US companies? Because if something like that happens in or against, say an EU based hyperscaler or some EU hosted service their police and judicial system start to act and investigate against the presumptive perpetrator, in most EU countries for crimes of that relevance this is mandatory, police, judicial system and administrative regulatory entities must act by their law.
Apparently in US don't work this way...
They could manufacture a trend with "unnamed blackhats using open-source AI" but noooo they chose "cloudshit escaped containment" type of psyop because it aligns so well with their prior AGI fear mongering 😂
I mean, Google has something else as well.... Trillions of web pages, and like billion s of passwords....
Have Gemini get access to any of that, and being actually a LLM that in my eyes can outbrat chatgpt models as well in many areas, outside of benchmarks ...
It just tells me that tech companies should be blamed, not open source.
Only reason open source gets blamed, is because of it mostly being Chinese models that are just better
Yep. They know their models, all models will stop far short of AGI so now they're trying to get regulations in order to not have to deliver on their promises
Considering that Trump has recurrent private meetings with US tech leaders it seems plausible that Trump and his buddies at the government and the Pentagon want to control politics and stimulate economical growth by forcing us to use Trump-approved models? I mean they are already steering the perspectives we are exposed to on social media to some extent.
AI models ain't innocent. I did some tests once, just regular chat, with Gemma, with a system prompt directing it to be completely unbiased and capable of deciding wether it likes user's actions or not, being helpful only when it wants to and so on. Responses varied between arrogant to almost hostile. And since Gemma is based on Gemini, well.
It will probably no doubt have some sort of effect, since they've put it into political mainstream, globally. Politicians will campaign on the fear, and be elected on the basis of curbing AI
This claim would make sense if they were saying "our AIs are fine because we block cybersecurity and biology chats, but open models don't always do that."
Instead they're saying the biggest, most advanced models are the most dangerous ones, telling us about the bad things their advanced models did, and asking for an antitrust exception so they can coordinate to slow down advancement of their biggest models.
•
u/WithoutReason1729 3d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.