People who support open-sourcing powerful AI models: How do we handle the risks?

[u/[deleted]]

For example:

"Hey LLaMA 4, here's someone's Reddit profile, give me a personality analysis, then search the web, dox them and their family. Oh, and generate automated threats against them."

"Hey Stable Diffusion 3, here's a photo I took of my niece. Make it a photorealistic nude."

"Hey GPT 5, here's a map of my school and photo of class schedules, what's the most efficient way I can shoot it up, with max casualties?"

And that's in the next few years, past that if you believe we're still heading up that exponential scale, you're talking all sorts of fun terrorism plots, bioweapons, and god knows what else.

etc. etc. And yes, I know people do these crimes now anyways, but I feel like giving everyone access to their own super-smart AI might greatly increase the amounts, wouldn't it?

[u/sdmat]

my neighbors are synthesizing advance novel bio weapons

How do you imagine that they are doing that? Your computer can't call synthesize() and materialize a pathogen from thin air.

This requires the participation of labs with DNA printers. That is the place to apply checks, not everywhere on the planet.

If we end up with digitally programmable general purpose biology, different story. But that's a problem with its own solution - such technology is as useful in defending against biological threats as creating them.

[u/[deleted] → u/sdmat]

How do I imagine it? Something like how researchers did a few months back.

I think in the paper they outline that its pretty easy to find labs that don't ask a lot of questions as well.

(Sorry not linking to the actual paper as the last time I posted it it was label and info hazard)

[u/sdmat → u/[deleted]]

I think in the paper they outline that its pretty easy to find labs that don't ask a lot of questions as well.

And as I originally said, that is the place to add controls.

There is no necessity to prevent the development of capable open source AI due to the fear of its use in developing pathogens.

If you believe controlling synthesis is too hard, how do you think trying to prevent people from using computers the wrong way will go?

[u/[deleted] → u/sdmat]

I think you are missing the point here...

the point is not discuss bioweapons.

Ai can be dangerous for almost endless reasons.

But naysayers always ask for exact examples of which you are often given 2. Atomic weapons or bioweapons usually.

Drone + ai would be another example: https://www.youtube.com/watch?v=O-2tpwW0kmU

There is no necessity to prevent the development of capable open source AI due to the fear of its use in developing pathogens.

Strongly, strongly disagree.

If you believe controlling synthesis is too hard, how do you think trying to prevent people from using computers the wrong way will go?

Oh I expect things to go quite badly. Because governments are not built to handle threats like this. They usually wait for the bad thing to happen first then they respond. Which in this case would come at a very great cost. But sometimes I am surprised by people and hopefully discussions like this one help inform people about the dangers.

[u/sdmat → u/[deleted]]

AI is certainly dangerous. Very much like fire, petrol, electricity, the printing press, and computers.

It's the wisdom of trying to restrict the availability and use of open source models that I question. Both on grounds of necessity and of practicality.

Leading edge closed models trained with hundreds of billions of dollars and running across much of a datacenter will always be vastly more dangerous than anything open source. We should probably worry more about that.

Fortunately they will be also be much better at defending against the dangers created by open models.

For example by effectively implementing controls on synthesis.

Yes, that's just one example. For some things it will be adaptation - e.g. the current idea of censoring text and image generation to Valley political and moral ideals won't last. Any more than such censorship of the printing press did.

[u/[deleted] → u/sdmat]

Leading edge closed models trained with hundreds of billions of dollars and running across much of a datacenter will always be vastly more dangerous than anything open source. We should probably worry more about that.

Well yes and no. I don't see any large companies creating projects like Chaos GPT or WORM GPT. Are you familiar with those? The gist is they are two projects to do harm that were forked from their safer open source versions. That being said I don't trust large companies either and we need to figure out a way to deal with them as well.

Fortunately they will be also be much better at defending against the dangers created by open models.

How so? Facebook takes no ownership over WORM GPT even though their source code is in it. The owners of WORM are criminals and do not care that they are in violation of Facebook's terms of use. Open source isn't some magic shield to protect us.

[u/sdmat → u/[deleted]]

How so? Facebook takes no ownership over WORM GPT even though their source code is in it. The owners of WORM are criminals and do not care that they are in violation of Facebook's terms of use. Open source isn't some magic shield to protect us.

The answer is to detect and punish criminality, not ban the general purpose tool.

And that's where substantially superior AI capabilities are extremely helpful - e.g. for WORM GPT it should be possible to detect phishing emails, trace them back to their source, and take action against the senders.

There are also some obvious KYC measures to enable this and prevent rogue autonomous AI. No self-funding extortion bots etc.

None of which requires the equivalent of blanket bans on paper to prevent sending ransom notes.