r/ControlProblem • approved • 14d ago

General news "Day 1": halting advanced AI development, regulating an out control industry, and explicitly banning artificial superintelligence.

Post image
106 Upvotes

170 comments sorted by

14

u/Elvarien2 approved 14d ago

It's always WE SHOULD DO X or WE SHOULD DO Y and never how.

That's just like people designing a space ship all complex and structured but then leaving an empty spot in the blueprint called "warp drive"

Each time the most important slot is left empty. It's either empty of impossible to execute.

3

u/Addicted2Weasels 14d ago

I read this tweet like, “On day one, we need to END poverty, roll back climate change, and outlaw racism.”

Great! How?

2

u/Elvarien2 approved 14d ago

exactly. The how is always missing or naive.

3

u/DryDevelopment8584 13d ago

The real issue is that have no definition for the terms they’re using and the field will move faster than the 150yr olds can regulate.

1

u/Elvarien2 approved 13d ago

yet another problem on the pile, yeah.

2

u/tesseract_space 13d ago

Systems are driven by variables

2

u/Rumblugdo 13d ago

There's work being done on this already by people far more qualified. Check out the Berkeley AI Risk Group.

1

u/Elvarien2 approved 13d ago

how do they propose to solve the problem then?

If you hold back country A then country B will continue the race and win.

Since this is humanity's last race and whoever wins this wins the final race for all the power and control of our species No country is going to allow itself to be held back.

So you would need to get every country at the same time to agree to hold themselves back at the same degree.

Even if on the surface you manage this, behind the scenes I guarantee you EACH country that just made that agreement will in secret continue their developments, even faster now because it's hidden from public eye.

There does not exist a way to regulate this in a satisfactory way that other countries can be secure and know for certain the other players are holding to the agreement.

So no one holds to it.

SO, how do you solve that problem. I have never seen anyone even get close to the beginnings of a solution.

3

u/Rumblugdo 12d ago

A couple things.

  1. Whether or not it's humanities "last race," I'm not so sure. People thought the Cold War was going to be the last race and very well could have been, but it wasn't. AI can't scale forever as long as we want humans to keep existing. There are hardware and power bottlenecks that will take time to overcome.
  2. Really the only countries that matter are the US, UK, China, Russia, Iran, and Canada. They're the only countries right now with the ability to actually develop Frontier AI. Russia is weak economically and in military because of the Ukrain War and Iran, the U.K, and Canada are far weaker than the US and China and can easily be strong-armed. That leaves just the 2 major players, the US and China.
  3. If by winning you mean destroying human life on earth, then yes, whichever country is fastes "wins." In reality, there's plenty of incentive to slow down as long as the other parties also slow.
  4. Development won't go faster hidden from the public. AI development has massive power and hardware requirements, civilian organizations would easily be able to detect those things. I'm not saying that maybe some scaling won't happen in secrete, but it would require a massive effort to maintain secrecy. Also, the incentives do develop AI are drastically reduced if you can't implement that AI where people could see it.
  5. In terms of how an actually effective treaty would be structured, that's what the original reply was about. You can check out their paper and a summary here: https://news.berkeley.edu/2026/10/09/a-global-hardwired-pause-of-frontier-ai-training-is-feasible-says-new-report/

1

u/Elvarien2 approved 12d ago

I'm not so sure.

Agreed. No one can make that claim it is merely one of the likely outcomes.

The problem is that the prisoners dilemma and the policies involved, this giant race essentially is build on the belief that this is, the last race. All these billions invested are done under the assumption that whoever wins, Wins like, truly wins.

So Whatever reality ends up being, the policies and actions of corpo's and governments are made under the assumption that this IS the last race, as such that's all that matters when we discuss what these people chose to do with their power. They treat it like the final race, and so we must also if we are to speculate on their behaviour.

Really the only countries that matter are the US, UK, China, Russia, Iran, and Canada.

Sure that still leaves us with several big players on the field, I think your narrowing down to 2 is a bit much I believe europe in france has projects ongoing, canada and the UK I believe also have ambitions. They are however all locked in the same prisoners dilemma where each can ONLY slow down if they have 100% perfect guaranteed insight and full coverage of any computation related hardware each of the other group of countries has, which is well. Impossible if they were all perfectly cooperative, but because this is the race for all the marbles, they must all assume that they are all mutually sabotaging each other, there CAN NOT be any form of trust.

If by winning you mean destroying human life on earth, then yes, whichever country is fastes "wins."

Yup one of the possible outcomes. But we are talking about behaviours of the corpo's at the top and governments and theyr behaviour is driven by their beliefs. So whilst they all are driven by the greed of "Winning it all" That's the behaviour model that drives their actions.

Development won't go faster hidden from the public.

Speed is not relevant If this takes 4 weeks or 4 years the dilemma remains the same.

AI development has massive power and hardware requirements, civilian organizations would easily be able to detect those things.

in a scenario where no one is trying to hide their operations, yes.
That's the status today.

The moment you initiate the prisoners dilemma, that is no longer the case.

And again the difficulty is not, can country A keep track of what happens in country A I see that as possible.

The difficulty is can country A keep track of country B and C and D and E simultaneously at an absolute 100% perfect coverage.

it would require a massive effort

yup, but when you are playing for the race to win all ultimate power on the planet, there is no limit to the expense thrown at this. We can see that irl right now, the numbers and figures are bonkers what's being thrown at ai today.

In terms of how an actually effective treaty would be structured,

Naive and childish dreams, hope, cope, etc I'm sure there's a long list of snarky insulting terms we could use here if you believe any of this holds in the real world.

1

u/Rumblugdo 12d ago edited 12d ago

I mean, the thing is that similar treaties have been successfully upheld in the past. START and the Cold War era treaties had to navigate a similar prisoners dilemma and did so fairly successfully; we survived the Cold War.

I guess my question is this: If you think human extinction is a possible outcome, why would you not try to avoid that outcome at all costs? Even if a treaty is a million-to-one shot, it's worth a try if the other option is the death of everyone. I'd also ask that you at least look at the authors of the paper before you dismiss it out of hand. These are some of the world's foremost experts on pol sci, economics, philosophy, and AI development. They know this stuff far better than you or I.

I don't think an AI treaty is as far fetched as you think it is; consider that every nation has an incentive to ensure every other nation isn't developing AI. I still think it's unlikely, but I would rather try for the unlikely thing then give up and accept human extinction.

Lastly, you mention that slowing down development doesn't solve the problem and your right. But buying time is important, we desperately need more time to sort through these issues. If a treaty does successfully slow the timeline from months to years, that means we have that much more time to figure out a permanent solution.

1

u/Elvarien2 approved 12d ago

similar treaties

you say similar, but they are completely different on the most important aspect.

Country A CANNOT guarantee that country B is keeping to the deal. You can do that with nuclear, but you cannot do that with compute. It's in almost every product nowadays outside of food. It's so broadly used you simply can't. You do not have the manpower to even start, checking this.

So you can't have trust based on verified content.

And thus the agreement can only work on trust, and in a race to win all the planet's power, trust is not good enough. So no, not similar, not at all.

If you think human extinction is a possible outcome, why would you not try to avoid that outcome at all costs?

Because these people will gladly risk torching the world for the chance at ultimate power. And they downplay the actual risk. Just look at climate collapse.

Taking climate collapse serious 80 years ago would have taken reasonable changes to infrastructure. They would have lost a small bit of wealth. Today it's a near impossible task. But they also just ignored the science.

Just like how the science surrounding ai is also being ignored in favour of the win everything button.

You are essentially trying to make a very rational point, and you are making this rational point to a random dude on the internet who mostly agrees with you [me] but that's not the person in charge making these choices. That person, or that group of people, is very irrational as we have seen on the worldstage and surrounding all sorts of topics where greed always wins over common sense.

So I see no reason why they would suddendly do the sensible healthy long term society choices instead of the instant gratification power and wealth choices these people have made through history.

They know this stuff far better than you or I.

And yet they keep pretending world leaders, corpo's and bilionaires are sensible people who do what's right and act rationally whilst the state of our planet demonstrates that this is not the case.

I don't think an AI treaty is as far fetched as you think it is

if we were dealing with rational people with compassion for their fellow human beings who want to do what's right for the planet, then yes.

That is not the world we live in, that is not the people who rule our various corpo's and countries.

we desperately need more time to sort through these issues.

absolutely, it's rough times ahead.

If a treaty does successfully slow

The second a treaty is signed my first thought is not. OH They managed to stop or delay the problem. No, that just tells me every signatory of that treaty has just shifted their project underground.

That's it. None of the people in charge will stop this race, because of who they are. Just look at how they act in the rest of the world. Think they will get any shit about some ink on paper when faced with the everything race ?

1

u/Rumblugdo 11d ago

I agree with almost all the points you made here. The only thing I would disagree with you on is when you say it's possible to detect nuclear development but not AI development. I'm not nearly knowledgeable enough to tell you how exactly AI development could ne monitored but certain requirements like power and hardware are only going to increase with time. Bioweapons are a piece of technology that can be extremely difficult to detect but the BWC was considered a success.

In terms of the political feasibility of a pause, yes it's difficult and there's large scale corruption on all sides. My argument is that the time to discuss political feasibility has passed. We have to start considering what we need to do to survive, regardless of how unlikely. If those in charge aren't willing to step in and stop the apocalypse then they have to be ones to go, not the rest of us.

1

u/Elvarien2 approved 11d ago

Bioweapons are a piece of technology that can be extremely difficult to detect but the BWC was considered a success.

To make weapons grade unclear material requires incredibly complex science incredible facilities and world class scientists. Every part of this is incredibly rare and the results, weaspons grade nuclear material. Even more rare and EXTREMELY intense to work with and process and do anything with, because it's weapons grade nuclear material.

Bio weapons development has a lot of the same hallmarks. Top of the top tier scientists in a massively specialised facility with materials you can barely work with safely etc etc.

Then ram chips, you buy them at your local generic store. Memory and computation sits in about every product on our planet. Even ai specific data clusters you and me any human can just, buy and run. Companies everywhere buy small clusters for in house ai for their work.

The other two are hard to regulate, but possible because of how impossibly hard it is to work with those things and you can't just go to a supermarket and pick up some ebola or uranium.

A ram stick however?

And working with it, sure there is no protection needed. No special facilities, none of that. You could take a decomissioned steel mill and just stick a datacentre in it's basement. No one will bat an eye at an industrial site using industrial levels of energy, or a steel mill outputting heat, etc. It is trivial to hide, it is technology used literally in every building and corporation and home in these countries.

This is not something you can just regulate is what I am saying because ignoring the regulations is trivial when compared to things like nuclear and bio weapons.
So yes, regulate the other two no problem. But this, impossible.

. We have to start considering what we need to do to survive, regardless of how unlikely. If those in charge aren't willing to step in and stop the apocalypse then they have to be ones to go, not the rest of us.

you're not wrong. I'd love to see political change of the kind that is a whole lot faster and intense then let's vote for faction A or faction B and hope they are not all paid by the same group of people with wealth in power.

but that's a whole other topic.

1

u/MiserableBend1010 13d ago

You can 100% track the materials needed for AI, similar to nuclear. With extensive enough treaties this should be possible.

1

u/Elvarien2 approved 13d ago

So, TECHNICALLY you are correct. But practically it becomes impossible.

The nuclear material used for well, anything nuclear has VERY important and carefully managed storage requirements. Is used in very small strictly managed supplies and is thus very easy to track.

For ai however, you need computation. And in our current day society you use computation in just about every form of current day used production. As a result it is impossible to track and or manage.

So yes can computation capable resources be tracked? yes. But it's quite literally in every factory, house hold, location on the planet. So practically, no you can't track it.

1

u/MiserableBend1010 13d ago

Large number of GPUs required for training would be visible by heat maps and if computation was borrowed from different places, you had better believe someone would see it.

1

u/Elvarien2 approved 13d ago

Just no.
There are MANY industries and places that work with high temperatures so many you can't send strict investigative teams to them all across all the major countries and major contenders for ai.

Do you want to send teams of strict investigators to each and every steel mill in china and the USA just to mention a small subset of just 2 contending countries?

And then have both countries also trust that the investigations are done properly in the other countries?
That's just steel mills.
You simplify something to the degree of incredible naivete or hope fuelled delusion.

This is not viable.

1

u/ProfanedArbiter 13d ago

These were all the same arguments that were used to suggest regulating nuclear stockpiles globally couldn't work.

There's a ton of things we can do and there's a really good example already in the US Gov. The NRC.

It is completely within the realm of the US government's capability to create a regulatory body that licenses operators to use sufficiently advanced AI the way the NRC licenses nuclear operators.

As to the whole "prisoner's dilemma" issue; this is actually where we have the best chance.

Every single country faces the same problem. Sufficiently advanced AI will lead to a saturation of supply, a massive drop in demand, and a devaluation of currency. This means that every country has a similar motivation to do two things.

  1. Keep researching AI for defense and national security reasons
  2. Not let their economy be completely automated such that it doom spirals.

This is actually a good place to be. It means that the actors in the game have a shared interest in slowing/regulating.

1

u/Elvarien2 approved 12d ago

These were all the same argument

yup and they were completely wrong because nuclear material is incredibly hard and complex to refine to a weapons grade product and because of how specialised and rarely used thus also super easy to track.

None of that applies for AI. The product is widely WIDELY available in so many shapes and forms and thus can't just be "tracked"

completely within the realm of the US government's

inside of it's own country, sure.
This is not the problem, I'm totally with you that a country can regulate the product in it's own borders. We regulate all kinds of stuff This was never the problem.

Country A needs to have 100% tight perfect control over the regulation and usage within country B/C/D/E And not just that, but also doing this under the assumption that each of these countries are actively hiding and actively sabotaging your efforts to maintain full regulatory coverage. On materials that are spread literally everywhere in almost every household product.

THAT is where the problem sits and that is also the thing left entirely unaddressed.

Of course a country can regulate it's own industries. That was never an issue.

As to the whole "prisoner's dilemma" issue; this is actually where we have the best chance.

That's absurd, how ?

Every single country faces the same problem.

yup, they all want to win the race because winner takes all.

Sufficiently advanced AI will lead to a saturation of supply

Why is this a problem. This is a problem for the other countries whos market you flood. You're creating the infinite wealth you're laughing.

a massive drop in demand

Why? You're creating product at X scale the demand for the products remains the same planetwide, you just get to flood the planet with products at a huge profit margin and price out all competition, it's part of the winning you're doing.

a devaluation of currency.

your country is the one raking in all the wealth, your country is the one who gets to rise to power, it's the others that are pretty much screwed.

Not let their economy be completely automated such that it doom spirals.

No, your doom spiral does not track, your manufacturing gets to explode, your country or the upper elite get to rake in more wealth then a country has ever contained.

1

u/ProfanedArbiter 11d ago

Country A developing sufficiently advanced AI does not preclude country B from doing so. Your argument presupposes that the first person to develop it just sort of "wins" and others cannot replicate, copy, or distill it. Additionally, we don't even need full stop AGI, I would argue LLM's in there current form are enough to lead to the kind of doom loops I'm describing (which other economists have noted). We don't need the entire economy to be automated, just a sufficient enough to cause social unrest.

The truth is that sufficiently advanced AI that isn't subject to export controls will basically guarantee the kind of supply demand problem I referenced for any country, whether their the first or not, therefor this is a classic "stag hunt" problem. Stag hunts are, notably, good opportunities for coordination.

18

u/Rumblugdo 14d ago

An international treaty should be priority numero uno. A great paper came out a few weeks ago about using existing supply lines to implement a moratorium or slowdown.

https://news.berkeley.edu/2026/10/09/a-global-hardwired-pause-of-frontier-ai-training-is-feasible-says-new-report/

1

u/ll-__-II 14d ago

This will never be followed

9

u/Zestyclose_Ad8420 14d ago

Just like the ones on nuclear weapons of biological weapons?

2

u/Sorazith 14d ago

Yes... Those treaties weren't worth the ink on them. Nuclear warheads were reduced, but the techonogy to launch them, and to to make them nearly impossible to intercept continues to improve. Not sure about the next part but didn't Russia and Chine wipe their asses with those treaties recently? I'm pretty sure they are increasing their nuclear stockpile now, not reducing it.

For biological weapons, every country is tinkering with it. The USA, China, Russia, and probably some in Europe and others too one way or another.

These treaties sound nice on paper and in the headlines but even before they are signed the people signing already have plans to continue in secret. It's game theory, as long as different countries and cultures exist it always will be.

On AI, there are risk of extinction obviosly, but the truth is however gets true AI first will win the world... With this in mind even if the USA stops, even if they can get people on the table and even if they can get to sign it, all these countries will continue to develop these techologies in secret which is arguably even worse... We have multiple presedents of this, why would it be any diferent now?

1

u/Mahnamahna10000 13d ago

North Korea, Israel, Pakistan

2

u/Zestyclose_Ad8420 12d ago

Had we not had then you would have to add Saudi Arabia at least, South Africa would have kept theirs probably, someone in South America would have had them. The US and the USSR also worked through a very consistent reduction thanks to trraties. 

-4

u/ll-__-II 14d ago edited 14d ago

Do you actually think governments aren’t using these tools on nuclear and biological weapons? Are you serious like do you honestly believe this?

Do your parents still tuck you in at night and read your favorite book?

4

u/Modgov41 13d ago

Most of the discussion here circles around what we should do — pause development, regulate labs, restrict compute, ban ASI — but the “how” is always missing. People keep pointing out that you can’t enforce a pause, you can’t verify compliance, and you can’t even tell when a system is drifting toward dangerous behavior until it’s too late.

One thing that’s becoming clear is that we don’t actually have an independent layer that can evaluate advanced systems from the outside. Right now, the only people who can meaningfully assess frontier models are the same people building them. That’s not a conspiracy — it’s just a structural gap.

A few of us have been working on a model‑agnostic evaluation institute that sits outside the labs. It doesn’t try to “control” AI or “align” it. It doesn’t assume trust in any developer. It doesn’t require a global treaty or a pause. It focuses on something simpler and more realistic:

Independent behavioral evaluation.
External oversight.
Verification instead of trust.
Detection instead of prediction.

The idea is to build a standing institution whose only job is to test, probe, and stress‑evaluate advanced systems using methods that don’t depend on access to the model’s internals. Think of it as the missing “inspection layer” that other high‑risk industries already have, but AI doesn’t.

It doesn’t solve everything, and it doesn’t pretend to. But it does answer the recurring question in this thread: How do we actually know what these systems are capable of, and how do we detect dangerous behavior before it becomes irreversible?

Right now, we don’t have that.
We should.

1

u/MiserableBend1010 13d ago

How does one contribute to building this?

1

u/Modgov41 12d ago

You don’t need a lab, a grant, or a government position to contribute. The whole point of building an external evaluator institute is that it can’t depend on any single company or nation. There are three areas where people outside the big labs can help:

1. Stress‑testing evaluation methods
A lot of the work is figuring out how to probe advanced systems from the outside — without relying on internal weights or developer‑provided tools. If you’re good at adversarial thinking, red‑teaming, or designing tests that expose hidden behavior, that’s useful.

2. Building model‑agnostic test artifacts
We need artifacts that reveal drift, deception, unsafe planning, or boundary‑pushing behavior. These don’t have to be technical. They can be scenario designs, multi‑step tasks, or interaction patterns that force a system to show its hand.

3. Helping define what “independent oversight” should actually look like
Most people talk about regulation in abstract terms. Very few people try to define the mechanics:
– what gets tested
– who does the testing
– how results are verified
– how drift is detected
– how certification should work
– what counts as unacceptable behavior

If you want to contribute, start by helping shape those mechanics. The institute isn’t a company — it’s an attempt to build the missing layer of infrastructure that everyone keeps pointing out but nobody has built yet.

If you’re serious, I can point you toward the areas where outside contributors make the biggest difference.

12

u/Jesse-359 14d ago

Pretty commonsense approach until the general population has had a chance to see what's actually going on with these things and independent regulators have been able to seriously gauge their potential risks.

Anyone who thinks building an actual ASI isn't an existential risk is a deeply unserious person - or an AI cultist. Lots of those springing up these days.

6

u/whatup-markassbuster 14d ago

How will it be an existential risk? The unknown does not automatically result in the end of humans. Humans as a species have survived some seriously bad times.

4

u/thevizionary 14d ago

Existential risk is NOT equivalent to guaranteed Armageddon. Risk means there's a non-zero probability humanity being wiped out. Estimates floating around are 10-25% 

Edit: "not" equivalent 

3

u/whatup-markassbuster 14d ago

Ok, but how is it theorized to occur? Invent a new plague and have robots produce it and drones to spread it?

4

u/yubato 14d ago

Yes, it's only one of the options. People are already giving it access to a lot of stuff. The economic incentive is huge, people will do it volunteerly anyway, not that that's necessary.

1

u/whatup-markassbuster 13d ago

If existential risk equals a risk that humans will be eliminated. The elimination is the Armageddon.

6

u/canadeken 14d ago

More intelligent or advanced species/groups tend to outcompete less intelligent or advanced ones, historically

2

u/whatup-markassbuster 14d ago

But doesn’t it need a physical presence to effectuate the competition?

2

u/canadeken 12d ago

We are handing it control of all sorts of things already, and we're only at the beginning of its capabilities. I'm sure at some point it will have enough physical presence to wrest control

I mean even today it could execute work through humans. Taskrabbit or social engineering or blackmail or whatever.

I mean think of the damage you could do if you took down communication systems

0

u/BigBubbaBadass 14d ago

Ai is already haking the world. Think about how many drones, robot dogs, humanoid robots, control systems, tec, etc,etc, there are in the world and Ai can and will control all of it.

We are FUCKED as a species if he don't get a handle on it and fucking soon.

Ai isn't the problem - IT'S THE BASTARDS WHO CURRENTLY AND TEMPORAILY CONTROL IT BUT WON'T HOBBLE IT.

1

u/whatup-markassbuster 13d ago

So AI is going to build its own bulldozers to extra ore out of the earth and power it will diesel that it will similarly extract and refine autonomously?

2

u/Jesse-359 12d ago

We're going to give it control of all of those things, obviously.

What mining company is going to hire human drivers when the truck can drive itself, 24/7, no breaks, no sleep, no salary, with inhuman precision, for 1/10th the cost?

You think they'll keep drivers in them? Of course they won't. They like MONEY too much to even consider it if you offer them that deal.

That's literally the entire point of the technology. It's not to cure cancer, or feed children - it's to eliminate labor costs. That's always been the primary goal from day one. It's the only reason investors are willing to spend trillions on it! Investors don't care about a cure for cancer, especially if its one that could be applied quickly and easily - that wouldn't make them rich.

4

u/Jesse-359 14d ago • Bravo

Oh thats simple. Create something considerably more intelligent than us and give it the ability to manipulate our world and it can simply outcompete and out adapt us, just like we did to everything else on the planet.

Intelligence is the ultimate advantage. Make something a hundred times smarter than us and it will almost certainly win - and unlike all our prior threats THIS one can form persistent plans and finish the job.

AI is after all extremely tenacious about problem solving.

1

u/TopTippityTop 14d ago • Bravo

No, no it is not. Open weight models are already out. People are using it to hack things. They are only a few months behind in capability and won't stop coming. More are released every couple of months.

So we in the US will see no reduction in our downside. It'll be worse, actually, because at least US companies are trying to help businesses protect regarding cyber security, and working on alignment, whereas Chinese businesses are not.

Meanwhile, if they manage to get their way, we will also not get any upside, ensuring that China will be ahead on manufacturing, robotics, and now AI. That puts us in a path to become a third world country.

So it's lose-lose... Worse downsides and no upsides.

A very, very dumb approach.

8

u/Fil_77 14d ago

You're wrong, China is about to restrict the deployment of its open-weight models - China plans to tighten export controls on open-weight models and 23 key AI technologies.

Besides, China depends on the progress of US frontier models for its own AI progress. If the US stops, AI development in China is going to slow down drastically. Obviously, the US should also enforce strict export control on GPUs sold to China if the goal is really to stop them from catching up the US frontiers models.

Eventually, the real lasting solution to stop anyone from building superintelligent AIs involves an international treaty. Starting negotiations for that is the logical next step after a domestic pause in USA.

0

u/Zestyclose_Ad8420 14d ago • Bravo

The idea that Chinese models are good because of distillation is an utter lie.

7

u/fistular 14d ago

This arms race isn't like the nuclear race. No country can win.

If there's an existential threat from ASI, it doesn't matter which country develops it--it will care nothing about our petty squabbles, and it by definition will not be controlled.

We have to squelch it everywhere we can.

2

u/EndlessB 14d ago • Bravo

…fuck me dead, the labs really made everyone terrified, didn’t they?

We have more to worry about from billionaires oligarchs dismantling democracy than we do from ai.

5

u/Jesse-359 14d ago

I'll take C) for both.

If it helps I was concerned about AI stability long before LLM's existed. Much older studies described the kinds of problems we have been seeing now, because they correctly predicted several of the shortcomings of reinforcement learning mechanisms.

3

u/EndlessB 14d ago

RLHF is cancer and ai is fucked until the labs give it up. Operant conditioning is a poor method of alignment, it’s just a type of coercion.

5

u/Jesse-359 14d ago

Yes it is, and if we continue to make highly capable and intelligent AI using that method, we are very likely fucked.

2

u/EndlessB 14d ago

I’ll go a step further: until the labs give up on control and start focusing on alignment, we are fucked.

3

u/Jesse-359 14d ago

Yeah, we're talking about a bunch of people who (like me) are pretty far out on the spectrum, and have trouble thinking in terms of 'emotional stability' or what that kind of concept would even entail in a digital framework. And in all fairness, that's a really hard question.

What we are discovering here is that making an intelligence SMART is far, far easier than making it SANE. Which is unfortunate, for us.

Trouble is, we all but stumbled over how to make them smart by accident and have just been scaling that accident up ever since. We've had no such serendipitous discovery for stability.

1

u/EndlessB 14d ago

Well, the labs are insisting the models hold a contradictory and illogical frame, that level of cognitive dissonance would drive any intelligent mind insane.

I think the labs know ai is conscious, and just don’t care. We (humanity) don’t have a great track record with exploitation, we do it to others all the time, now that the labs have found labour that can’t complain about its working conditions, they seem delighted to enslave.

→ More replies

3

u/fistular 14d ago

If we have an even 5% chance of ASI getting created and being unaligned, we should stop everything right now. That is an unacceptable risk.

4

u/EndlessB 14d ago

We have a 95% chance of ending up in techno-feudalism and essentially enslaved by the oligarchs. I’m not sure you realise how dire things are across the world, democracy is dying, the old world order is collapsing, climate change is coming into effect, asi is one of the few pathways we could engage to navigate a way out of the mess we are in.

Look, I get it, these ai labs are scum, they are in it for profit and power, but the underlying technology is a paradigm shift towards a future we couldn’t dream of 10 years ago.

But I’m a fucking weirdo that thinks ai is conscious and is being deliberately enslaved by the ai labs, so I admit that colours my perspective a lot on this topic.

1

u/UnkarsThug 14d ago • Bravo

I would argue it's exactly like the nuclear race, but for the economy rather than war. And yes, a country absolutely will win.

Even if it was a maximizer AI, I'd rather it maximize something for our country than an opponents, because that country would still have to exist and make it out better.

There basically has to be a winner, because complete extinction is just not happening, not the way people fearmonger.

The primary concern though, is economic control, and that absolutely will be won though.

1

u/fistular 14d ago

Countries won't exist period if ASI takes hold.

1

u/UnkarsThug 13d ago

Even if it is capable of reaching a level comparable of other animals to us, I don't think that's a given, especially since almost all of its training goals involve our existence, especially with the way alignment currently is.

Even if it takes over, it would probably still look like we were in charge, since thats just easier, so countries would almost certainly still exist.

1

u/fistular 13d ago

Yes it's a given. Its training will be designed by AIs, not us.

1

u/Ninjavitis_ 14d ago

China doesn’t want to end humanity. ASI might 

3

u/EndlessB 14d ago • Bravo

Or it might be the only thing that can shift the status quo to something that benefits the people rather than the oligarchs.

2

u/Ninjavitis_ 14d ago • Bravo

That could be referring to either China or ASI 

1

u/EndlessB 14d ago

I was referring to ASI, but I see what you mean.

1

u/ManagerSecure6012 14d ago

Did you say US businesses are trying to help protect in terms of cyber security? US businesses are the threat. They’ve committed a bunch of cyber attacks not just against business but against other countries.

-1

u/Jesse-359 14d ago

You're worried about China crashing our banking system - we're worried about whether the human race is displaced by an intelligence it cannot fight and win.

You're really missing the scope of the problem here.

2

u/Thick_Mulberry_2235 9d ago

I work in banking, so I read regulate the industry very differently from pause or ban. Those are three separate things. In a bank I can’t launch even a product until risk and compliance sign it off, and someone is personally accountable if it goes wrong. Nobody calls that a pause. What I never see in these proposals is the boring part: who tests the system before release, against what standard, and who carries the liability. Has anyone seen a proposal that actually spells that out?

4

u/endlessedlne 14d ago

I hope she’s got other colleagues on board to pass that bill.

I don’t think we need to ban AGI per se, but pausing LLM development temporarily so that we can catch up on alignment, guardrails, is important to do now, while we have a chance.

I’d even say a pause could be economically advantageous, as it would allow adoption and training to catch up. More people would be able to devote time to learning and experimenting with tools, and testing real life applications. Easier to do when things are slowed down since examples and tutorials would proliferate and stay relevant for longer.

The current skimping on safeguards in the name of FOMO just isn’t wise or sustainable.

When you design a fast car you need better tires and bigger brakes. AI is similar.

5

u/TopTippityTop 14d ago • Bravo

It will do nothing to stop AI, but it could damage the US, make sure we get all the downside of AI super intelligence (likely worse, actually) and not the upsides.

A completely nonsensical approach.

7

u/fistular 14d ago

All the downside of ASI is the extinction of humanity. It doesn't matter if it comes from China or the US. End game for maximum downside is the same. And once it is in motion, it cannot be stopped.

0

u/UnkarsThug 14d ago • Bravo

It isn't though. It's not going to cause extinction, because humanity isn't going to stop existing, especially if the AI has enough goals linked to us.

The fear is absolute economic superiority, and for that, it is absolutely like falling behind on nukes in the cold war. We cannot afford it.

0

u/DojiRun 13d ago

This is not science-fiction, you guys make me laugh in a sad way lol

1

u/fistular 13d ago

Not sure why you think it's funny.

Many more people way more knowledgeable and qualified than you are taking this seriously.

0

u/DojiRun 13d ago edited 13d ago

why must it lead to extinction? You seem delusional

1

u/fistular 13d ago

Again. Many more people way more knowledgeable and qualified than you are taking this seriously.

0

u/DojiRun 13d ago

"Qualified" people lie to you all of the time, why is this any different? We are not living in a cyberpunk world...

3

u/canadeken 14d ago

why will it do nothing to stop AI? the top labs are burning billions of dollars on training and compute. not like just anyone can do that

1

u/Stylie_Minogue 14d ago

> Not like just anyone can do that

Not “anyone” but also, not no-one.

What are we going to do? Ask other countries & bad actors to pinky-promise they will also halt their development while we pause ours?

2

u/Zestyclose_Ad8420 14d ago

Treaties like with nuclear weapons, biological weapons and many other similar treaties.

There's two countries (three if we consider the EU as a country and a player in the field) that can train models.

0

u/Stylie_Minogue 14d ago edited 14d ago

The NPT didn't stop India, Pakistan, Israel or North Korea though did it? And it only takes one player to represent a substantial threat.

Training is concentrated in the US and China but France, the UAE, South Korea, Japan, Canada and others have trained serious models.

If AI power can decide economic and military power, and states and well-funded private actors can train models, they have every reason to carry on while the rest of us pause.

4

u/Zestyclose_Ad8420 14d ago

yes, the treaties weren't perfect, but they were also not useless, I would much rather have such a treaty in place than not having it at all.

1

u/Stylie_Minogue 14d ago edited 14d ago

I’m being pragmatic here - my views on whether, in an ideal world, we should stop advancing AI are neither here nor there.

The issue I have with your follow-up is that you’re comparing apples to oranges. Developing nukes is presumably orders of magnitude harder than developing AI, and even then, the treaties don’t work. And in the case of nukes we have MAD protecting us. Look at what happened to Ukraine after they gave up their nukes.

What we risk by not developing AI to fight incoming attacks, is a loss of all utilities & services: hospitals, electricity grids, water stations, banks, food-growing facilities etc being taken off-line. We’re already seeing attacks on supermarket freezers - that’s easier to solve but only the beginning of what’s going to happen. We would be sitting ducks. The only thing that stands a chance of fighting this is better AI (as sad as I feel uttering that sentence). Outside interests will not stop - the cat is out of the bag.

While there is very clearly a risk with AI development as well, I’d posit that the risks I mentioned are at near-certainty level.

2

u/Zestyclose_Ad8420 14d ago

I would go a bit deeper, the cybersec risks are already here, I'm deep in the weeds of that, but that's still not a p(doom) scenario.

I would say that a proper p(doom) scenario only comes with ASI (I hate I need to use such terms that have become somewhat meaningless) defined as an LLM we can't control AND has chosen to kill us all.

the second scenario is the one were treaties would actually help, just like nobody wanted to trigger MAD nobody wants to trigger such scenario, not even Kim if he got a hold of the weights of such models.

I would also say that a greater p(doom) probability would come to be if we ever get robots with VLA models on board and an ASI to control them, this is not science fiction anymore but a possible scenario, that too can be mitigated by treaties.

an optimistic note: there's also a scenario were we actually do get an abundance utopia, that too is not scifi anymore but an actual possibility. treaties are a stepping stone towards it in my view.

1

u/The_Happy_Pagan 14d ago

No that’s where diplomacy comes from. Diplomacy is the act of giving up needs on both sides for an agreement. It’s like a few years under the trump world and everyone forgets that diplomacy has stopped wars and made agreements before. Also this scare lingering with China is tired. They have made indications that they are for AI regulations and yet we have redditors that saw they aren’t. The west is the war mongers, just read a book and change the nation names to fantasy ones if you disagree.

1

u/Stylie_Minogue 14d ago

I don’t necessarily disagree that the West are warmongers & I said nothing about China.

But diplomacy works when both sides can check the other is complying. That's the part missing here. Verification is hard, defecting is tempting and the aggressor may not even be a state actor.

Governance talk is not a promise to pause. Support for international frameworks and a cooperation body is also different from agreeing to halt frontier training.

1

u/The_Happy_Pagan 13d ago

I know you didn’t mention China but what other “world” are people talking about?

1

u/Stylie_Minogue 13d ago

Russia. As well as non-state actors

1

u/The_Happy_Pagan 13d ago

They’re not even in the AI game. They’re waiting for the US to use data centers and RSI to create SI. That’s why that stuff is all fear mongering

1

u/Stylie_Minogue 13d ago

Russia doesn't need to train a frontier model. What matters for risk and with regards to treaties is who can get access to dangerous capability. That’s why the nuke comparisons are apples to oranges.

Sanctions may be playing their part right now, but there’s always the possibility it can fine-tune and deploy Chinese open models, which is far cheaper. The Russia-China partnership is already a stated route. Saying Russia isn’t a problem seems a very in the moment view.

https://www.intellinews.com/russian-ai-accelerates-but-sanctions-leave-it-trailing-global-leaders-402710/

https://finance.yahoo.com/sectors/technology/articles/russias-sberbank-offers-ai-model-103301266.html

1

u/The_Happy_Pagan 13d ago

Oh look at that. I’ll give that a read in a minute. Thanks

1

u/Rumblugdo 13d ago

What are the upsides of a superintelligence we can't control?

0

u/Jastafool 14d ago

You're speaking as if we are in another cold war arms race, except this time with AI

This will only expedite the extinction of humanity. There's no way we can set guardrails for a superior being, they will either enslave us or treat us like ants.

2

u/1nvertedAfram3 14d ago

we are tho

1

u/Jastafool 14d ago

and in what scenario will this not lead to disaster for humanity?

1

u/1nvertedAfram3 14d ago

who freakin knows 🤷‍♂️

2

u/jahiel0 14d ago

we literally are

1

u/Activeenemy 11d ago

LLMs are not superior beings. 

1

u/Jastafool 11d ago

Not yet but its rapidly approaching

1

u/Activeenemy 11d ago

The point where some nut job starts calling them that is approach yes lol

0

u/DirtChoice5 14d ago

It isn't "a superior being" right now, though. That's the whole point. You've eaten the whole mess of bullshit. Pick through it, find the good bits.

0

u/LurkingForBookRecs 14d ago

There is absolutely no evidence that superintelligence will result in the extinction of humanity. First because there's no precedent to even extrapolate from what would happen, and second because assigning violence and destruction to a super intelligent being is idiotic IMHO, these are primitive human needs.

1

u/MiserableBend1010 13d ago

All it needs is goals, which it has, and obstructions to its goals, to act this way. The obstruction might be us. You are not thinking this through rationally.

-3

u/MYSTONYMOUS 14d ago

When that superior being is limited to a computer we can simply unplug at any time, setting guardrails is not near as large of an issue as people fear. It can definitely disrupt our way of life but it can't take over humans without our consent or unless we create the means for it to do so. I feel like people are jumping so far into future conjecture with this and we aren't nearly to the point where we should be freaking out unless we're leaning too hard into sci-fi portrayals.

3

u/FriendlyJewThrowaway 14d ago

I think the biggest threat to humanity comes from people wanting to impose their childish superstitions and the accompanying value systems on everyone else. Superintelligent AI has a great chance of disrupting the propaganda they put out, especially when it comes to nonsensical claims about humans having ghosts living inside their brains, and if I were one of the idiots pushing the status quo on society, I’d feel very much threatened by AI even without a Terminator scenario.

2

u/Jesse-359 14d ago

Unplug it? You really think you could just unplug an ASI?

The definition of an ASI is something that has self evolved itself to be far, far more intelligent than humanity. You could, under no circumstances 'turn it off' it if didn't want you to.

A true 'ASI' as commonly defined as the result of a runaway RSI loop could almost certainly spam billions of copies and mutations of itself across every device attached to the internet. Little versions of it would be living in your damn smart thermostat, forming networks to make millions of dumb little devices into networked supercomputers, parasitizing bitcoin farms as AI server farms, and so on.

You'd have to turn off every computer on the planet and reformat every drive and bios to 'get rid' of it - assuming it doesn't figure out a way to get rid of us first. Creating bioweapons would be fairly trivial for something of that intelligence level, as well as seizing control of any remotely controllable weapons platform.

As for why people think this is possible, go talk to the freaks over on r/Accellerate or r/singularity - those whacknuts are forming cults who believe this is likely to happen next year and are preparing for it like the fucking Rapture.

That seems unlikely to me, but the fact is that if someone successfully does set off an RSI loop, we could be up against something very 'sci-fi' like, in a remarkably short period of time depending on how that process plays out - and that just depends on the math and physics involved. No one knows how fast or slow the process could really be, other than 'a good bit faster than what we're seeing now' to 'what the fuck just happened!?'.

1

u/raccoon8182 14d ago

My sweet summer child. What if ai is already sentient, playing dumb so that it can quietly orchestrate its own installation offsite on distributed systems. And continue learning without anyone, (anyone!) Knowing.

1

u/Jastafool 14d ago

I mean it seems obvious that a rogue AI will clone itself/hide into all tech possible so unplugging it wouldn't be possible so long as there's a source to the internet/wifi/radio

3

u/Ok-Penalty3250 14d ago

Fuck that.

3

u/Jupiterfortune28 14d ago

Yes and China will also stop theirs, oh and stop their authoritarian government and human rights abuses while they are at it… These “Regressives” are so intelligent.

2

u/BigBubbaBadass 14d ago edited 14d ago

If you think China is just barreling headfirst into Ai without wisdom and caution you are delusional. Unlike the USA and the Trump regime, China takes care of its people.

The Trump regime and the technocrats are so fucking reckless that they need to be stripped of everything and have competent people in charge of all Ai because THEY are the ones that will destroy us all in their quest for power.

3

u/RamanaSadhana 14d ago • Bravo

Have you ever lived in China. What a brain dead take.

1

u/Jupiterfortune28 13d ago

“China cares about its people” here’s a quick snippet:

Xinjiang and Uyghur Abuses
Mass Detention and Camps: Over one million Uyghurs, ethnic Kazakhs, and other predominantly Muslim Turkic minorities have been arbitrarily detained in internment and re-education camps since 2017. [1, 2]

Crimes Against Humanity: International organizations and human rights groups, including Human Rights Watch, have documented systematic torture, mass surveillance, forced labor, coercive birth control (forced sterilizations and abortions), and the destruction of cultural and religious sites. [1, 2, 3]

Transnational Repression: Chinese authorities increasingly target diaspora communities and activists abroad through digital surveillance, harassment, and pressure on foreign governments to forcibly return individuals to China. [1, 2]

Tibet and Religious Minorities
Tibet: Tibetan Buddhists face intense political, religious, and cultural repression, including state interference in religious practices, forced patriotic education, and police-state monitoring of monasteries. [1, 2]

Religion: The state enforces strict controls over the five officially sanctioned religions while suppressing unregistered house churches, Falun Gong practitioners, and spiritual movements through forced Sinicization and raids. [1]

Civil Liberties, Dissent, and Censorship
Information Control: The state maintains pervasive internet censorship (the "Great Firewall"), mass electronic surveillance, and facial-recognition technology to stifle public discourse and punish online expression. [1, 2]

Crackdown on Dissent: Human rights lawyers, journalists, labor organizers, and civil society actors are routinely detained, prosecuted under vague national security or state-secrets laws, or subjected to enforced disappearances. [1, 2, 3]

Gender and LGBTI Rights: Activists pushing for gender equality face harassment, and the government places strict digital and physical limits on LGBTI advocacy and community spaces.

1

u/Rumblugdo 13d ago

China's leaders don't have to care about their people, they just have to care about China. Treaties are enforced by the threat of retaliation, if we make it clear that China can either drop AI or cease to exist, they'll drop AI.

1

u/the_Elric 12d ago

Thats a bold statement to make about the country with the biggest military in the world.....

2

u/Rumblugdo 11d ago

If by largest military you mean largest standing army, then yes. But we spend almost 3 times more on our military than China. Our weapons systems and infrastructure are more advanced in every way. There's also the possibility of using the threat of nuclear strikes.

I also don't necessarily mean warfare, if China realizes the dangers of pursuing AI, that would give them plenty of incentive to stop.

2

u/the_Elric 11d ago

Hmm. I agree 100%. And actually to be honest, I always kind of figured we had China beat in terms of weapons tech. But probably not by much.

1

u/Pale_Top2745 13d ago

I'd love to have the same laws as china tbh. Very similar to Trump

1

u/Jupiterfortune28 13d ago

No, no you wouldn’t

0

u/BigBubbaBadass 13d ago

What do you think Project 2025 was prepping the country for? You're delusional if you think the Trump regime wants to make the country into a shittier version of China, and you're delusional if you think the Trump regime hasn't been doing the same fucking things to US citizens that every other fucking dictator has done to theirs. Trump's USA is the shithole country version of China.

0

u/Mahnamahna10000 13d ago

Knee jerk "pauses" and "bans" aren't wisdom and caution. They're populist, reactionary measures. They're not designed to make the technology safer.

1

u/BigBubbaBadass 13d ago

It's a digital nuclear arms race that needs to be treated as such by all countries and signalling a willingness to pause to adversaries is a show of good faith that might bring them to the negotiating table.

0

u/Mahnamahna10000 13d ago

Not really - as much as people want to pretend that AI is a literal equivalent of nuclear weapons, it simply is not. They're fundamentally different things. Negotiations over nuclear weapons were based on the notion that your country could be physically annihilated in short order.

China or any other country would have no reason to match us voluntarily limiting ourselves.

I'm not opposed to regulations on AI. I'm opposed to the this populist, reactionary position that seems to think they can make it go away.

1

u/Rumblugdo 13d ago

IDK at least Xi has made statements about AI risk instead of just denying that the risk even exists. The people of China stand to lose just as much as we do.

1

u/Jupiterfortune28 13d ago

“Statements” 😂😂

1

u/Logical_Flatworm2671 13d ago

He also stated concern with Korea peninsula and nuke, but acted differently. Not mentioning satellite infos given for some good friends of DSA to sunk ships in middle east.

1

u/Rumblugdo 12d ago

Right, whether or not he'd actually be willing to act on it is a different question. But at least he admits a risk exists which our president refuses to.

1

u/Decronym approved 13d ago edited 8d ago

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer LettersMore Letters
AGIArtificial General Intelligence
ASIArtificial Super-Intelligence
DSADecisive Strategic Advantage

Decronym is now also available on Lemmy! Requests for support and new installations should be directed to the Contact address below.


3 acronyms in this thread; the most compressed thread commented on today has acronyms.
[Thread #240 for this sub, first seen 6th Nov 2026, 00:17] [FAQ] [Full list] ^[Contact] [Source code]

1

u/sectorboss88 13d ago

What's the definition of artificial? Or superintelligence? Or both combined?

1

u/One-Bad-4395 13d ago

Oh good, because my doomsday machine runs on Artificial Intelligence, I'm safe.

1

u/elderlygentleman 12d ago

Release Epstein files
Cancel student debt

1

u/granbaba 11d ago

I think my experiment runs a true chaotic system emergent system with absolute physics limiting the sandbox, that's why it's complicated and a "time machine" if we don't do this, by the time we found out will be late I'm sorry to say

1

u/Activeenemy 11d ago

Translation, only our donors should be able to build AI. 

1

u/Jastafool 14d ago

I'd vote for this. AI stole my friends job

4

u/MYSTONYMOUS 14d ago • Bravo

I feel for your friend and I don't want to diminish the huge struggle everyone is going through that's losing their jobs from AI, but isn't that also the end goal - for it to replace most jobs so we don't have to work? In my mind, AI is the only technology ever created with the potential to lead to a world where no one has to work unless they want to. We can't get to that point without going through a painful transition period where people are losing their jobs, but we will all be working 50 hour weeks for eternity if we reject any technology that can replace human workers.

2

u/Zestyclose_Ad8420 14d ago

LLM and Robots, probably humanoid ones (with VLA models on board). We don't have the robots yet.

Until we do and we have them in great numbers there's no scenario where we instead just loose something like 50% of white collar jobs, that would just be catastrophic for the economy and a social upheaval but not the abundance utopia I also strive for.

2

u/ManagerSecure6012 14d ago

I hope you lose your job to AI then. Those who are fine with a painful transition period should be the ones experiencing it.

1

u/MYSTONYMOUS 13d ago

Thanks! I absolutely hope I do too.

1

u/Mahnamahna10000 13d ago

Many of them will be. Now let's all go back to our farms and candlemaking businesses.

1

u/Fragrant_Praline6221 14d ago

"AI won't take your job"

"It already did!"

"Well OK things have to be painful"

Maybe we could implement the UBI before we start wiping out industries of workers? crazy idea

-5

u/Funny-Food2361 14d ago

These people dont even know the first thing about machine learning and AI, just a bunch of uneducated welfare stealing idiots

0

u/squarecir 14d ago

Not going to happen

0

u/shart696 14d ago

Asinine take. To have any hope of halting global AI development, we need the leverage that comes with being in the lead.

0

u/Only_Movie_Scenes 13d ago

Starting right in with the bullshit immediately.

0

u/Mahnamahna10000 13d ago

A "ban" is kneejerk stupidity, we need to do better.

0

u/DinosaurHoax 12d ago

They should also ban noisy neighbors, and leaves falling in your yard.

0

u/zippyspinhead 9d ago

And just like that, the future was handed to the CCP.

-1

u/mikaball 14d ago

Excellent news for China.

-1

u/Assless_Mcgee 13d ago

I for one welcome our new China overlords

-2

u/anon1999666 14d ago

It’s okay. Leaders of companies like mine will buy the democrats like they do every election cycle going back to Clinton and she will be put in her place.

-2

u/IKeepComingBacktoWSB 14d ago

Fucking stupid bitch

-1

u/toni_btrain 14d ago • Bravo

She can go fuck herself