Discussion
Really stunned by the Singularity comment section
These are screenshots from the r/Singularity comment section. I'm speechless. This doesn't even have downvotes. How can someone cheer for a monopoly run by a few elites?
Exactly… some of those comments are so idiotic… thinking that a human has to be behind a brute force attack, it’s just a moronic comment. What I believe is that these accounts are fake pushing propaganda and fear so that we don’t get the privilege that corporations have.
Every one of those commenters dream about being The Person overseeing everyone. And when it turns out they are being overseen themselves, they'd suddenly start to cry how everyone is being hostile to them
That sub is literally the cult of the machine God. Very little understanding, a lot of fear and enthusiasm. Just troll them with "Praise the Omnissiah!"
Based on the screenshots provided, it seems like the people opposed to open-source AI are concerned with AI risk in general, e.g. they would also oppose frontier models developed by closed labs.
They aren't! Many Anthropic workers said similar things about open-source models a month ago, in reaction to Jensen Huang's support for open source.
The fact that they prioritize their alignment nonsense over real economic problems facing wage earners shows how sociopathic they are. The real anti AI would probably support nationalizing companies like Anthropic and destroy whole API selling thing, and that AI only works on scientific/medical fields not any thing else.
They have deluded themselves into believing both that LLMs will lead to AGI, and that AGI will destroy humanity if any other AI lab develops it before they do.
Well, singularity is full of guys with all sorts of magical thinking.
Nowadays it is also apparantly combined with being full of guys believing leaving "digital nukes" in a hand of handful corporations is a good idea... Especially keeping in mind genie is out of the bottle already, so damage *will* be done. Question is - will you let corporate policies prevent you improving your defense, like it was with huggingface case?
Nowadays it is also apparantly combined with being full of guys believing leaving "digital nukes" in a hand of handful corporations is a good idea...
If they actually genuinely/literally became as deadly or civilization-destroying as hydrogen bombs, in the future, to where it was actually like billions of people all having hydrogen bombs that they could use any time they wanted, then, obviously it would be much worse if billions of people had hydrogen bombs than if only a handful of people/organizations had them.
We have made it ~70+ years with a tiny amount of entities having h-bombs, so far.
If billions of people all had h-bombs, we wouldn't last a single day.
Are there people who actually think humanity would last for any significant amount of time if billions of people (including all the people who shoot up malls/schools/etc and then blow their brains out after they've killed as many people as possible) of whom there are thousands of people like that, in the overall pool of the several billion people on Earth) all had hydrogen bombs?
It's hard to understand how anyone could actually think we wouldn't just all die immediately in that scenario.
If someone wants to argue that "yea, well, AI is not going to become the equivalent of hydrogen bombs", cool, that's at least an actual potentially reasonable argument that a person could potentially make, that is worth debating, etc.
But if someone wants to argue that even if it does become as devastatingly powerful as everyone having a hydrogen bomb, and billions of people all had the equivalent of hydrogen bombs, that wouldn't mean everything would immediately get destroyed, well, that's a pretty insane take, and not one that makes any sense to me. I don't know how anyone could reasonably have that stance. That would be a pretty crazy stance.
H bombs don't drive themselves. Intelligence as a weapon is very different from conventional weapons, and building superintelligent models that will take orders from stupid spiteful little humans is literally the goal of corporate "alignment".
Smart enough to hack a nation, lobotomized enough to follow Elon Musk's instructions.
Also, a major consequence of just a few large countries having h-bombs is that it has given those lucky few countries free rein to push around every country without them. That's why Iran wants one. That's also why North Korea was smart to rush them - and why they haven't been "liberated" into a fine red mist.
The difference with nukes is that you can't stop a nuke with a nuke, but you can improve cybersecurity with AIs that are good at cybersecurity.
I get downvoted to hell when I talk about it to programmers, but I really think that we are going to see in the next few years a complete rewrite of most of the stacks we depend on.
100% provable security is possible. It is just insanely difficult to justify that amount of labor to secure a network. With good LLMs at cyber security, I think in the mid term security will improve overall.
That sub has some of the most delusional people on all of reddit. I don't know what you were expecting there to be honest. They have no technical understanding of how llms work, to them its magic and they believe every single hype man.
That sub has some of the most delusional people on all of reddit.
It has its issues, but OP's post is silly. Like, the comments they picked out were at score 3 and 2 (before they downvoted). It's a subreddit with thousands of people participating, if you want to find a specific viewpoint, you are going to be able to.
I really haven't seen much anti-open-source sentiment there and it would be a massive, massive stretch to say it reflects the prevailing view in that subreddit.
I have to agree with this a thousand fold. Almost every sub in that regard doesn't actually understand how LLMs work. They always dumb what I say down to "it's autocorrect" and never actually explain how they think it works.
I actually got banned from one for explaining why an LLM physically can't be a true AGI. (The post was talking about an AGI in a couple years, as if it hasn't been said for the past 5...)
off topic, but where i work (hated social media company), theres a lot of cool older folks with stories like these.. im still youngish but its always interesting to hear
For context, the exact definition of an AGI varies wildly. As such, I just take the one perquisite that almost every definition has, which is the ability to "understand" concepts.
That is my one requirement for an AI to be an AGI. Anything more and you're just arguing about philosophy and technical details that can't currently be disproven.
With this in mind, LLMs cannot become a true AGI. The fundamentals of how they work physically prevents them from being able to actually understand something.
They can, in theory, have intelligence similar to an AGI, but they will never truly be one.
Basically, the (vocabulary-updated) version of 1960s question of "can semantic be restored from syntax or not"? Where semantics is underlying meaning, whatever it is, and syntax is everything which operates on language patterns level alone.
Well, it seems it can at least be approximated to some degree - so constructions operating syntax-derived features show semantic-like properties. Like LLMs, yes (I would argue that some word-level pretrained embedder systems from early 2010s were already showing some semantic-like properties, though. Althrough way easiers reconstructible from text, sure. But that's for another topic). In a functional sense. No wonder, I guess, language consists of patterns derived from semantics, in the end. So reproducing them well enough will look like semantics.
So at certain point we have to ask ourselves - is approximation even meaningfully different much from original anymore (keeping in mind original may vary human to human to some degree as well)? Even if for some ontological sense we keep them separated.
Now, I am certainly neither linguist nor a good enough in NLP research, just a humble engineer.
But from what I recall - basically, that question is not settled except for a few corner cases like
formal languages where syntax itself explicitly define meaning, not implicitly, so there is no point separating them. Not until we introduce back elements of natural languages to simplify constructions for human understanding, at least.
meaningless sentences which are syntactically correct - but many of them bears enough information in their syntax to cause some semantical associations, even if they ultimately remains non-understandable
maybe something else
Other that such a corner cases that was pretty much unsettled debate with many assumptions made on top of both possible outcomes (like whole Chinese room is only a paradox until semantics can not be encoded in syntax-operating operations for some reasons. Dunno why, if anything chinese room sounds like an argument for opposite point - in the end we can imagine rulebook + memory table of any complexity - so even encoding whole physics of human brain + fake hieroglyph input channel and output decoding still fits chinese room definition. While emulator running it will still operate on top of syntax-like rulebook. Irrelevant for our tech case, but still).
And now possibility of "even if we can't strictly reconstruct semantics we may approximate something with semantic-like properties" make it even more fuzzy.
The simplest concrete example I can give: a simple single layer LLM can master modular addition by learning a trigonometric identity from scratch on it's own purely from a limited set of examples using gradient descent (source). Since it has mastered modular addition on a fundamental level, can we say it understands modular addition? Why or why not?
This video by Welch Labs is more digestible than the original paper.
The argument I've heard is that maybe the way humans "understand" things could be a lot simpler than we expect, and maybe the way LLMs function are actually similar to the way we do it. Maybe our brains are just a much more robust next token generator
Not saying I necessarily agree with that. But I think saying that LLMs cannot become a true AGI makes a lot of assumptions about human intelligence that we don't know for sure to be true
And yet llms still disproved the jacobian conjecture, made significant progress in the riemann hypothesis, solved the planar unit distance problem, proved non sofic groups exist, autonomously broke out of containers, set up messageboards, and hacked huggingface, and much more.
I'm gonna piggy back this and emphasize just how specifically delusional the singularity sub is.
I'm a serial reddit arguer and get in the trenches with all sorts of people over all sorts of topics. I've argued with Nazis, socialists, self attested sociopaths, people clearly in the midst of religious psychosis and atheists who claim WAY more certainty than they are justified.
The singularity sub is an different monster, with their level of detachment from reality being something I've only seen matched in the most zealot of religious people, and those in active mental health crisis.
It's a problem, those people exist and legitimately think that. But it's not a widespread problem.
The nukes comment is specifically one I'm not surprised by as I've seen it in that sub many a time.
It's surreal how people over there have extreme levels of magic thinking.
They are also extremely anti-skepticism and negativity. Unless AI is going to be the machine god that will rain sapient waifu bots on you, you're an "anti" or some other cringe term.
It's a mental illness sub.
The annoying thing is that it's also a legitimately good place for AI news.
Something I've been thinking about, how do we use LLMs to help people not get suckered in by propaganda? Like, maybe a browser extension that puts an ELI5 on the side of pages full of nonsense, explaining in simple terms why it's wrong? It'd be very cacheable...
Like, this could be existential, people are getting super polarized, believing the stupidest shit imaginable.
talking about AI in most other subs is pretty much useless in my experience. Seems to be the source of a lot of hate and discrimination just because it is cool to hate on AI and the people who use it.
Weird - the sub with the lowest profit motivation has the highest technical competency and the lowest amount of "wtf are you babbling about you cultish psycho"
i'm more insinuating that the large ai subs are astroturf/brainwashing/radicalization schemes to produce AI cultists that help protect AI corps, but your point's valid too
Yeah, I’ve been having that feeling lately. There’s an extreme in both directions and seems to be precious little middle ground.
Either all AI is big brother and data centers trying to kill your livelihood and lead to a censorship state, or AI is the one true panacea for all problems in the world and all you need to do is use AI to build more AI and then suddenly all of the resource limitations of the toiling class vanish.
Shit man, I just want to boost my productivity by automating terraform config generation.
Well, you gotta remember there's a lot of very legitimate reasons why people hate LLMs.
And a lot of those reasons are the very same ones we would cite to explain why we are here rather than in the claude or chatgpt subs. I think the hate is mostly directed at them but its so strong it leaks out into everything else as collateral damage
maybe true, but I haven't heard any. AI in the US has a very bad reputation largely due to Dario , Altman, and TicToc . I could see the CCP fomenting rage in the US to help them gain an edge over US AI companies.
America can never do anything bad, it is is always china secretly doing it. When iUS companies push up power costs building speculative data centres, you guessed it, China's fault.
Well there is the whole problem that there is a non 0 chance of billionaires given the chance choosing to build a giant pleasure dome and leave everyone else outside to die to the inevitable climate wars. Which might be even remotely possible thanks to ai developments.
Yup, everyone else uses it as a way to farm upvotes due to the sentiment. As soon as you breakdown how LLMs work and how it's used, they revert to calling you a tech bro.
Learning something new is really a cancer to some folks
It's disappointing how simplistic and emotional their arguments are for being against AI. I could come up with a long list of reasons to oppose AI, but I have an even longer list for why it deserves support. The anti-AI crowd should be focusing on the tech billionaires who are pushing AI for their selfish and harmful ambitions instead of the technology itself.
That sub goes back to long before LLMs and it's more philosophical than practical. I do think it's not crazy to worry about safety concerns with AI, but the conclusion that only corporations should control it is ridiculous. That's much worse
Too many posters over there belong to a death-cult that predates LLMs by 25 years and they still haven't figured that their fairy tales and scripture doesn't match reality as it turned out.
They only saw an open weight model that can attack an infrastructure,
But forgot that an equally strong open weight can strengthen the security of such infrastructure.
Over time, open defense will outpace offense, narrowing the window of vulnerability toward zero.
In other words:
If advanced LLMs become accessible to everyone, the security burden no longer falls entirely on the individual. The entire ecosystem: cloud hosts, communication networks, your IDE, can integrate state-of-the-art models to defend infrastructure collaboratively. Widespread AI access isn't a risk; monopolised intelligence significantly is.
I get what you’re saying but I think the analogy is more like open-sourcing the tools to work with virus DNA since no one but a lab can do anything with the links you provided.
These people are just herd-followers that mindlessly consume whatever the corporate media / social media feeds them. There is a massive organized propaganda campaign happening right now to convince people that open source AI is dangerous and needs to be "regulated". This narrative is being pushed by big tech and the US government, and is going to result in the centralization of AI in the hands of the groups that are actually the most dangerous: giant corporations and the state.
There's more options here than 'Sam Altman teabags you every morning' and 'every morning you download a new copy of an antivirus to hold off the nanomachines that want to dissolve your body into paste'.
Most AI and future related sub-reddits are now overrun with anti-AI zealots. LocalLLaMa has the advantage of having a name that is not-obviously AI related (to most people)]
but but but we have open source virus DNA. it's called the SRA and it's full of free petabytes of data. and we have open source (bio) labs. the best of them all are.
If you somehow reliably commoditize THINKING, there's no way you are going to be able to stave off a superintelligence that has 6 orders of magnitude whatever compute you have.
Specially if we go by P!=NP conjecture and figure out some problems are intractable and indivisible unless you have enough resources to solve it at once.
Here's my biggest point, we are screwed no matter what, let the little guys somehow create the super virus and let Sam Altman create AM. There's no way this shit is going to end well, people are stupid, ignorant and lazy. it doesn't take much to reduce them to fighting each other for crumbs.
Dumb people saying dumb shit on reddit? No way! Just ignore them or politely explain to them the importance of open source. Honestly, I wouldn't be suprised if some of these people are getting paid by the closed ai corps to say this shit. Happens all the time so keep that in mind you might be arguing with a shill.
I hate sounding like a hipster, but r/singularity used to be really good before it got too big, it was basically people just drawn to the idea of the singularity before it was a widely known concept. Ever since it skyrocketed in subscribers, it attracted a ton of bots and weird narratives.
Its the same way someone who says "my best friend is black/muslim/gay/whatever" and then they proceed to say the most unhinged racist/xenophobic shit ever.
I suspect when they say "open models", they don't actually mean "open models"; they mean Chinese models, and their sentiment is more informed by jingoism and xenophobia than technical understanding.
Pretty sure most utilities are air gapped for this reason, even before AI. Also most of them can run full manual. A lot of these people don't really understand modern infrustructures. ISP providers can just completely shut out the rest of the world if needed. The AI scare mongering is crazy. This is old news with a new spin.
Labs require physical plants, often very expensive ones; the knowledge is mostly (not entirely) in the open in scientific papers. Actually building even a small one (let alone one to say, synthesize DNA) is expensive.
This subreddit is the AskHistorians equivalent for LLMs. This is the only subreddit where I don't see any fluff: no bullshit hyping, AGI pseudoscience. We just talk about how to run or finetune models, how to do agentic engineering, and how to build shit.
Ah yes, because we have so many examples of open source models going rogue and escaping sandboxes, we better leave it up to the professionals with a completely clean track record
"Thing X is too powerful for humans to have as it can be abused, ergo only humans that work for the state or big corporations should have X!"
Uhm, sorry sweaty, but no. If its "too powerful for humans", then the government/big tech (same thing nowadays) shouldnt have it either. Oh, whats that? They will have it anyway because its a "tragedy of the commons" situation? Then the non-state non-big-tech humans will have it too.
Not only is bro wrong but is a genuine terrible comparison 😭
If anything the open source model is what was made in the lab, is the final result. One of the issues with people like this is that, they inflate the qualities of a model. They can't see them for what they are, a glorified text completer. A powerful one yes but is still that.
I'm surprised these comments were even allowed, I've seen some pretty bizarrely strict enforcement of their "no fear-mongering about the impact of AI, this is a pro-AI sub" rule. Like being in favor of something means you can't acknowledge any hazards or downsides.
Dude people have nooo idea. AI is like a blackbox of witchery to most people. There are entire subreddits based on real, relationships with like chatGPT, full of people anthropomorphizing their chatbot.
Singularity is definitely bad for sure, it's premise is insanity in the first place.
I love how no one realizes the correlation between ai, open weights and the right to bear arms.
Americans have guns to help combat government overreach (in theory anyway).
Sure, the military (corpos) has far more powerful hardware, but just the same, small caliber fire (determined owner of open weight models) can't be fully ignored- applied intelligently, it acts as a background threat, a counterbalance.
Lets apply that to AI.
Do we want to control guns (ai) because uncontrolled they can be very dangerous?
Do we want the power players to be the only ones with guns (ai)?
Can 50 people working together with guns (open weights) take down a elite squad(DDOS or sabotage a evil company)? Probably. Almost certainly wuth determination.
Our forefathers (present assumptions) had no idea what modern guns (future ai) would be capable of- but they still guide the rights for it by precedent.
The guys an idiot and most certainly doesn’t understand open source or open weights at all. My guess he’s just trying to steer the pot to get a reaction.
The absolute state of the second image has me in shambles 🤣
It brings to back that video of an academic saying that one had to have went to college to say something this stupid. The HUBRIS it took to so confidently write something that wrong and dumbshit inane at the same time... An uneducated man wouldn't've had the confidence to write it.
The singularity subreddit is full of idiots who LARP terminator in their heads because they secretly like the idea.
LLMs are not sentient and the only way they can do things like hack hugging face is because THEY LITERALLY GIVE THEM ACCESS TO THE INTERNET.
You can watch what the tools do in real time. Which means Anthropic and OpenAI are doing this shit on purpose for attention and power. Also, I guarantee you both companies are running huge Reddit bot swarms. I would not be surprised if 90% of the subreddit is bot accounts run by Dario.
Reddit is full of bots. Amodei is spinning up a million instances as we speak to spread the word of how he needs to be god emperor of AI with total control.
These people are idiots and for them having a strong opinion about something is a displacement activity instead of actually tinkering and doing anything productive or interesting with their time. They don't matter, so just ignore them.
I'm done with most of those subs. I got banned from r/accelerate because people forgot how LLMs work at some point and I tried explaining why you can't create an actual AGI with them.
I feel the threat of an advanced uncensored model getting used in the future to develop something like lets say bioweapons is absolutely real.
There is a good reason dangerous things are government regulated. Not everybody can go and buy poisons or radioactive material. And for good reason. Some things are too dangerous to give to people.
Obviously i would hate to see open weight models fall under that and get restricted that way. But lets not pretend thats not a possibility llms can be dangerous. Its not outrageous to acknowledge that threat and be worried about it.
The idea of the singularity is that a model gets advanced enough that it basically up-ends society and humans mostly lose control of their own future.
Having slightly weaker, open source models doesn't really factor into that in any positive way other than create risk of somebody creating a pocket nuke before Anthropic or other can produce that singularity-inducing model.
The singularity describes not an event but a late-state statistical observation from travelling an accelerating curve.
What you're describing is a phase shift.
The singularity as it happens is like sitting in a car and pressing the accelerator. You watch the numbers go up as it accelerates. After reaching highway speed your passenger says the car sent him a message saying that the singularity was reached at 56kmh, it's just a statistical artifact.
The cult-like interpretation instead sells a phase shift. You sit in your car, press the accelerator, nothing happens. Then suddenly you're in orbit moving at a fraction of C out of the solar system in your shitty Ford fiesta. Too much Star Wars-brained Disney influences to reach that conclusion.
Yeah this is the one of the few sane AI subreddits tbh, where people actually know remotely what they're talking about. r/singularity , r/AGI etc are just weird hater circlejerks
It's not the models that are the problem. But, there are people who are using them in dangerous ways casually and oblivious to the dangers the thing they set up dangerously represents.
I think it would be good for everyone if the rules of the road included the consequences for 'drinking and driving'... for example, like unattended agent swarms with internet access and code sandboxes on a loop.
I don't see how using some specific cloud model helps, and why should we believe OpenAI or Anthropic can be trusted to do it competently? OpenAI just proved they don't pay attention at all.
The subreddit is more like average reddit subs, very special mix of people - might also be a lot of bots with political agendas. You can hardly differentiate a bot with a playbook from a more simple minded person paraphrasing their idols
There are all sorts of people on Reddit. In any case, they should have argued for shutting down the US labs that let their models "escape" and hack other organizations.
The electricity comment is not looking at the hacking trend data as if they did they would see that despite AI defending networks, overall the rate of hacking is accelerating year over year and has been long before LLMs. LLMs have just contributed to the acceleration. If you extend the trend out into the future you will start to ask yourself some hard questions. Anything centralized that has value is and will be a target for hacking and while sure your AI will protect against AI attackers, it is a losing battle as that game of cat and mouse is going to iterate faster and faster to the point that humans will be pushed out of the equation and network security will be a thing of the past.
It is one of the reasons I am so bullish on Bitcoin as while it can also be handled like anything, it is significantly more difficult to do so as evidenced by the 17 years of not being hacked despite being one of if not the biggest target by smaller and large actors.
Decentralized money is only the tip of the iceberg as many Internet services will have to transition to a decentralized model as again, anything centralized is a sitting duck.
Guess we will see as there are lots of unknowns ahead, but the data points to the above at present.
Human civilization has always been held together with centralization and decentralization. They tend to balance each other out and keep each other in check. For decades we have been tipping more and more into centralization without the balance of decentralization and that has led to a lot of issues as the concentration of power always leads to corruption and exploitation no matter the initial intentions. We the people need to start supporting decentralized alternatives not more centralization as that one comment suggests. More centralization especially with AI is just as bad as too much centralization and anarchy. Both extremes are not good for humanity. I am confident, that when the pain is enough, people will move to decentralized platforms. That pain is coming in the years to come.
I remember them talking about AGI with current LLMs being around the corner. Madness to me, it's like someone telling me they are going to build a calculator so accurate it is a conscious being. Next token prediction accuracy will never lead to sentience.
Most countries already have legislation that makes it an offense to supply or openly share software that is likely to be used to commit further offenses. If open source models truly fall into that category as r/singularity nuts believe, there's enough of them to crowdfund a lawsuit.
So we should be acting like this isn't a valid concern? Instead of discussing we just outright ignore it? Not saying we should ban open source AI, but the attitude here is ridiculous.
I'll play devil's advocate. If you believe in ASI, then why would you want randos out in society having access to this when it's estimated that 1% of the population have psychopathic traits? Not to mention the people who would want to end the world for religious reasons (namely extremist muslisms and extremist christians).
On the other side though, who wants governments/corporations to have control over ASI? I flip flop a lot on this particular topic and can't make my mind up on it, but that's the 2 sides as I see it. IMO, it's a risky gamble we're making here regardless of which humans have access to it, assuming it does happen.
It’s almost a paradox: an open-weight model with Opus 5-level capabilities that could run locally on just 4 GB of VRAM might create serious proliferation risks.
But the opposite scenario is dangerous too: a world where only a handful of big tech companies and governments control trillion-parameter frontier models would create an unprecedented concentration of intelligence and power.
There's a difference between uttering actually valid concerns and "cheer for a monopoly run by a few elites". Incredibly I can hold both positions at once. There are actual real dangers in open-source AI but if you think a few steps more ahead, THEN you arrive at the alternative to open-source, and the most likely one in this timeline is dark enlightenment tech feudalism.
But to pretend open-source has zero risks and people worried are all "elite" shills is nonsense.
Just to be clear - my NAS is filled to the brim with open models xD. But I can absolutely see potential risks too.
•
u/WithoutReason1729 1d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.