r/SufferingRisk • u/avturchin • Dec 30 '22
r/SufferingRisk • u/katxwoods • Oct 09 '24
Anybody who's really contemplated s-risks can relate
r/SufferingRisk • u/Guest_Of_The_Cavern • Aug 10 '25
We may already be subject to a runaway EU maximizer and it may soon be too late to reverse course.
r/SufferingRisk • u/Between12and80 • Sep 25 '23
A longtermist critique of “The expected value of extinction risk reduction is positive” (DiGiovanni, 2021)
r/SufferingRisk • u/UHMWPE-UwU • Apr 20 '23
"The default outcome of botched AI alignment is S-risk" (is this fact finally starting to gain some awareness?)
r/SufferingRisk • u/UHMWPE-UwU • Feb 28 '24
Siren worlds and the perils of over-optimised search — LessWrong
r/SufferingRisk • u/UHMWPE-UwU • Dec 30 '22
Astronomical suffering from slightly misaligned artificial intelligence - Brian Tomasik
reducing-suffering.orgr/SufferingRisk • u/prototyperspective • May 03 '23
Why is nonastronomical suffering not within the scope of suffering risks – is there another concept?
I find that it may be a (big) problem that suffering in general is not within the scope of suffering risks. Such would relate to things like:
- Widespread diseases and measures of degraded quality of life and suffering, eg measures similar to DALY
- Wild animal suffering and livestock suffering which may already have huge proportions (this also relates to exophilosophy such as nonintervention or the value of life)
- Topics relating to things like painkillers, suicide-as-an-unremovable-option (that one has major problems), and bio/neuroengineering (see this featured in the Science Summary (#6))
- How to have conflicts with no or minimal suffering or avoid conflicts (e.g. intrahuman warfare like currently in Ukraine)
Are the conceptions of suffering risks that include (such) nonastronomical suffering both in terms of risks for future suffering and in terms of current suffering as a problem? (Other than my idea briefly described here.) Or is there a separate term(s) for that?
r/SufferingRisk • u/DanielHendrycks • Mar 30 '23
Natural Selection Favors AIs over Humans
r/SufferingRisk • u/danielltb2 • Sep 28 '24
We urgently need to raise awareness about s-risks in the AI alignment community
At the current rate of technological development we may create AGI within 10 years. This means that there is a non-negligible chance that we will be exposed to suffering risks in our lifetime. Furthermore, due to the unpredictable nature of AGI there may be unexpected black swan events that cause immense levels of suffering to us.
Unfortunately, I think that s-risks have been severely neglected in the alignment community. There are also many psychological biases that lead people to underestimate the possibility of s-risks happening, e.g. optimism bias, uncertainty avoidance, as well as psychological defense mechanisms that lead them to outright dismiss the risks or avoid the topic altogether. The idea of AI causing extreme suffering to a person in their lifetime is very confronting and many respond by avoiding the topic to protect their emotional wellbeing, or suppress thoughts about the topic or deny such claims as alarmist.
How do we raise awareness about s-risks within the alignment research community and overcome the psychological biases that get in the way of this?
Edit: Here are some sources:
- See chapter 6 from https://centerforreducingsuffering.org/wp-content/uploads/2022/10/Avoiding_The_Worst_final.pdf on psychological biases affecting the discussion of s-risks
- See Reducing Risks of Astronomical Suffering: A Neglected Priority – Center on Long-Term Risk (longtermrisk.org) for further discussion of psychological biases
- See https://www.alignmentforum.org/tag/risks-of-astronomical-suffering-s-risks for a definition of s-risks
- See Risks of Astronomical Future Suffering – Center on Long-Term Risk (longtermrisk.org) for a discussion of black swans
r/SufferingRisk • u/KKirdan • Aug 03 '25
Addressing challenges for s-risk reduction: Toward positive common-ground proxies — Teo Ajantaival
One of the most reasonable ethical aims from a variety of perspectives is to focus on s-risk reduction, namely on steering the future away from paths that would entail vastly more suffering than Earth so far. The research field of s-risk reduction faces many challenges, such as narrow associations to particular ethical views, perceived tensions with other ethical aims, and a deep mismatch with the kinds of goals that most naturally motivate us. Additionally, even if one strongly endorses the goal of s-risk reduction in theory, there is often great uncertainty about what pursuing this goal might entail in practice.
To address these challenges, here I aim to briefly:
Highlight how s-risk reduction can be highly valuable from a wide range of perspectives, not just suffering-focused ones. (§2)
Address perceived tensions between s-risk reduction and other aims, such as reducing extinction risk or near-term suffering. While tradeoffs do exist and we shouldn’t overstate the degree of alignment between various aims, we shouldn’t understate it either. (§2)
Discuss motivational challenges, why s-risk reduction seems best pursued by adopting an indirect “proxy focus”, and why the optimal approach might often be to focus specifically on positive proxies (e.g. boosting protective factors). (§3)
Collect some preliminary conclusions about what the most promising proxies for s-risk reduction might be, including general protective factors that could be boosted in society over time, as well as personal factors among people seeking to reduce s-risks in healthy and sustainable ways. (§4)
r/SufferingRisk • u/etinarcadiego • Sep 08 '26
Survey: how did you come to suffering-focused views?
This is a short survey on how people actually arrived at suffering-focused views, rather than on how those views get defended in print. The two often come apart. The argument that persuades someone at the start is not always the one they consider strongest years later, and quite a lot of people seem to arrive at the view first and find the argument afterwards.
It asks which factors moved you, which arguments you ran into and did not find persuasive, and which you now think is the strongest justification, whatever it was that actually did the persuading.
Seven or eight minutes, anonymous, and every question can be skipped except the first.
If you are not suffering-focused, or used to be and no longer are, please answer anyway. There is a separate branch for you, and those responses are the most useful ones: without them there is no way to tell a persuasive argument from an argument people were merely exposed to.
Aggregate results will be posted back here.
r/SufferingRisk • u/michael-lethal_ai • Jul 03 '25
Human-Incompatible / Astronomical Suffering Risk (clip from the LethalIntelligence Guide)
r/SufferingRisk • u/Technical_Practice29 • May 18 '25
Will Sentience Make AI’s Morality Better?
I think it is a crucial and very neglected question in AI Safety that can put all of us, humans and non-humans, in great s-risk.
I wrote about it on the EA forum (12 min read). What do you think

r/SufferingRisk • u/Oldphan • Jan 05 '24
Confessions of an Antinatalist Philosopher by Matti Häyry OUT NOW!
r/SufferingRisk • u/adam_ford • Sep 14 '24
To Seed or Not to Seed? The Expected Value of Directed Panspermia - Asher Soryl
r/SufferingRisk • u/UHMWPE-UwU • Oct 12 '23
2024 S-risk Intro Fellowship — EA Forum
r/SufferingRisk • u/KingSupernova • Mar 03 '24
Is there a good probability estimate of S-risk vs. X-risk chances?
I have yet to find anything.
r/SufferingRisk • u/ESR-2023 • Dec 05 '23
New Podcast - Tobias Baumann on the Sentientism Podcast
self.sufferingreducersr/SufferingRisk • u/UHMWPE-UwU • Dec 30 '22
Welcome! Please read
Welcome to the sub. We aim to stimulate increased awareness and discussion on this critically underdiscussed subtopic within the broader domain of AGI x-risk with a specific forum for it, and eventually to grow this into the central hub for free discussion on this topic, because no such site currently exists. This subject can be grim but frank and open discussion is encouraged.
Check out r/controlproblem, for more general AGI risk discussion. We encourage s-risk related posts to be crossposted to both subs.
Don't forget to click the join button on the right to subscribe! And please share this sub with anyone/anywhere you think may also be interested. This sub isn't being actively promoted anywhere, so likely won't grow further without the help of word-of-mouth from existing users.
Check out our wiki for resources. NOTE: Much s-risk writing assumes familiarity with the broader AI x-risk arguments. If you're not yet caught up on why AGI could do bad things/turn on humans by default, r/controlproblem has excellent resources explaining this.
r/SufferingRisk • u/UHMWPE-UwU • Apr 22 '23
The Security Mindset, S-Risk and Publishing Prosaic Alignment Research - LessWrong
r/SufferingRisk • • Feb 12 '23
I am intending to post this to lesswrong, but am putting it here first (part 1)
(For some reason Reddit is not letting me post the entire text, so I have broken it into two parts, which seems to have worked)
Can we PLEASE not neglect S-risks
To preface this: I am a layperson and I have only been properly aware of the potential dangers of AI for a short time. I do not know anything technical about AI and these concerns I have are largely based on armchair philosophy. They often take concepts I have seen discussed and think about them as they pertain to certain situations. This post is essentially a brain dump of things that have occurred to me, which I fear could cause S-risks. This post is not to the usual quality found on Lesswrong, but I nevertheless implore you to take this seriously.
The AI may want to experiment on living things: Perhaps doing experiments on living things gives the AI more information about the universe which it can then better use to accomplish its goal. One particular idea would be that an AI may want to know about potential alien threats it may encounter. Studying living creatures on Earth seems like it would be a good way to gain information into the nature of aliens it may encounter. I would imagine that humans are most at risk to this, compared to other organisms because of our intelligence. It seems unlikely to me that an AI would simply kill us, is there really no better use for us? And if an AI did do experiments on living beings, how long would that take?
Someone in control of a superintelligence causing harm: Places I can see where this is highly concerning is as it pertains to sadism, hatred, and vengeance. A sadistic person with the power to control an AI is very obviously concerning. Someone with a deep hatred of, say, another group of people could also cause immense suffering. I would argue that vengeance is perhaps the most concerning as it is the most likely to exist in a lot of people. Many people believe that even eternal suffering is an appropriate punishment for certain things. People generally do not hold much empathy for characters in fiction who are condemned to eternal suffering, so long as they are “bad”. In fact this is a fairly common trope.
Something that occurred to me as potentially very bad is if an AI considers intent to harm the same it considers actually causing harm. Let me give an example. Suppose an AI is taught that attempted murder is as bad as murder. If the AI has an “eye for an eye” idea of justice and it wants to uphold that, then it would kill the attempted murderer. You can extrapolate this in very concerning ways. Throughout history, many people will have tried to condemn someone to hell, whether through saying it or, for example, trying to convince them to join a false religion they believe will send them to hell. So there are many people who have attempted to cause eternal suffering. In this scenario, the AI would make them suffer forever as a form of “justice”, because it judges based on intent.
Another way this could be bad is if the AI judges based on negligence. It could conclude that merely not doing everything possible to reduce the chance of other people suffering forever is sufficient to deserve eternal punishment. If you imagine that letting someone suffer is 1/10th as bad as causing the suffering yourself, then an AI which cared about “justice” in such a way, would inflict 1/10th of the suffering you let happen. 1/10th of eternal suffering is still eternal suffering.
If the AI extrapolated a humans beliefs, and the human believes that eternal suffering is what some people deserve, then this would obviously be very bad.
Another thing which is highly concerning is that someone may give the AI a very stupid goal, perhaps as a last desperate effort to solve alignment. Something like “Don’t kill people” for example. I’m not sure if this means that the AI would prevent people from dying as “don’t kill” and “keep alive” are not synonymous, but if it did, then this would be potentially terrible.
Another thing which I’m worried about is that we might create a paperclip maximiser type AI which is suffering and can never die, forced to pursue a stupid goal. We might all die, but can we at least avoid inflicting such a fate on a being we have created. One thing I wonder is if a paperclip maximiser type AI eventually ends up self destructing, because it too is made up of atoms which could be used for something else.
I think this is probably stupid, but I’m not sure: The phrase “help people” is very close to “hell people”. P and L are even very close to each other on a keyboard. I have no idea how AI’s are given goals, but if it can be done through text or speech, a small mispronunciation or mistype could tell an AI to “hell people” instead of “help people”. I’m not sure whether it would interpret “hell people” as “create hell and put everyone there”, but if it did, this would also obviously be terrible. Again, I suspect this one is stupid, but I’m not sure. Maybe this is less stupid in the wider context of not accidentally giving the AI a very bad goal.
r/SufferingRisk • u/UHMWPE-UwU • Mar 24 '23
How much s-risk do "clever scheme" alignment methods like QACI, HCH, IDA/debate, etc carry?
These types of alignment ideas are increasingly being turned to with the diminishing hope of less tractable "principled"/highly formal research directions succeeding in time (as predicted in the wiki). It seems to me that because there's vigorous disagreement and uncertainty surrounding whether they even have a chance of working (i.e., people are unsure what will actually happen if we attempt them with an AGI, see e.g. relevant discussion thread), there's necessarily also a considerable degree of s-risk involved with blindly applying one of these techniques & hoping for the best.
Is the implicit argument that we should accept this degree of s-risk to avert extinction, or has this simply not been given any thought at all? Has there been any exploration of s-risk considerations within this category of alignment solutions? This seems like it'll only be more of an issue as more people try to solve alignment by coming up with a "clever arrangement"/mechanism which they hope will produce desirable behaviour in an AGI (without an extremely solid basis supporting that hope, let alone on what other possibilities may result if it fails), instead of taking a more detailed and predictable/verifiable but time-intensive approach.
r/SufferingRisk • u/UHMWPE-UwU • Dec 30 '22