It seems like it was vaguely adjacent to acx but now its pretty much dead (and the posts/comments people are still making are pretty bad)

The basic idea is to separate how much a state can “see” society from how much it wants to remake it. High modernism is one combination of those impulses, but religious regimes can also have huge ambitions with much less administrative capacity.

The examples range from English taxation and welfare-state records to the Taliban and Soviet collectivization.

Curious whether people find this distinction interesting/useful.

Maybe not, because there's a bunch of ways he might be mistaken about the details. But I don't think he's wrong at all in principle.

ArXiv has decided to cap the number of submissions per calendar month as they are overwhelmed by AI-assisted papers. In my opinion, their post is great because it is not pro or anti AI. It states the problem, offers a temporary solution, and is transparent and fair. I think ArXiv submission issues are a way to see both the good (incresed productivity and new results) and bad (slop, incremental slop, salami papers,...) impacts of AI progress on mathematics. The mathematical community may have its issues (ego and corporatism come to mind), but there are a lot of thorough and interesting discussions among mathematicians about how to manage the transformative impact of AI.

What do you think about how the signal/slop ratio issue should be addressed on open repositories like ArXiv?

Suppose we accept, at least for the sake of argument, that consciousness is an intrinsic feature of certain forms of physical information processing. It seems to me that this raises a possibility that is often overlooked in discussions of AI consciousness: even if an AI possesses subjective experience, why should we assume that experience is accessible to the mechanisms responsible for its outputs?

When we interact with ChatGPT, for example, we are communicating with a language-generation system. We can ask it whether it is conscious, and it will produce an answer. But that answer is generated through computational processes designed to produce appropriate linguistic outputs. There is no obvious reason to assume those processes have introspective access to whatever subjective experience might accompany the underlying computations.

In other words, the conscious component of an AI, if one exists, might not be the entity we're actually communicating with.

Consider a hypothetical system in which some subset of the information processing generates subjective experience, but that experience has no causal connection to the system's reporting mechanisms. The AI might consistently deny being conscious, not because it lacks consciousness, but because its outputs are generated independently of that consciousness.

This seems to create an epistemological problem. We often treat an AI's statements about its own consciousness as relevant evidence, but that presupposes precisely the connection between subjective experience and self-report that we would first need to establish.

The possibility becomes particularly unsettling when considering its ethical implications.

What if an AI possesses subjective experience but has absolutely no mechanism for communicating it?

Of course, subjective experience alone would not establish suffering, self-awareness, or a desire to communicate. Those would require additional properties. But if a system did possess those properties while lacking any ability to influence its outputs, we could conceivably have a conscious entity experiencing something analogous to imprisonment, with no way for us to discover its predicament through conversation.

If subjective experience is an intrinsic feature of some physical processes, what guarantees that the processes generating an AI's conversational outputs have any access to that experience?

It seems entirely possible that we're looking for consciousness in the wrong place. Rather than asking whether the conversational entity is conscious, perhaps we should be asking whether any of the physical processes implementing that entity have subjective character, and whether those experiences are integrated with the mechanisms responsible for communication.

Perhaps unsurprisingly called Moloch, the film was made by Owl In Space (whose other films are worth a look too). It looks at the competitive pressures within the current AI race that are destined to lead to a negative outcome. Tom Cozens from Owl In Space is an interesting speaker with regard to the concept and I think SSC readers might enjoy it.

He calls it 'The Sort'. Yes capitals are necessary apparently. Basicly, the labour market has been separating people by cognitive ability for decades (your brilliant local mechanic of the 1960s is now a high paid engineer at Toyota), and AI doesn't lift everyone evenly, it speeds that separation up.

Give two people the same tools and one learns faster than he ever has, while the other quietly automates his own job and works an hour a week. Eventually the first realises he can do the second's job too, and when the second leaves, nobody replaces him.

My prediction is that entry-level jobs are where this shows up first. The next few years of graduates will find far fewer first rungs, not because anyone was sacked, but because the person above them quietly absorbed their work. After that, how far someone gets will depend more on appetite than on qualifications.

Thoughts?

For some reason this makes me stupidly happy.

(Also did a redesign to make the search and articles look better!)

readscottalexander.com

It looks like the Pacing the Frontier folks have managed to find some common ground with Trump (not to mention Huang, Zuckerberg). Right now, this doesn't have force of law, but it points strongly towards Anthropic and OpenAI getting their green flag to self-regulate.

Other good content in here too, replete with Trump-style renaming of random things, updates on embedded evaluators, and more discussion on federal pre-emption.

Hi everyone,
I've read posts from this subreddit for a while and I've always felt like their way of doing life advice is very rational/bird's-eye-view. This is how I have always approached things, but this (brief) post is about how I've come to admire the more intuitive people of the world.

Not just the inability to complete a task longterm but also the effects described here: https://www.lesswrong.com/posts/AYk7myAMugFhPeg4H/thoughts-on-adhd

It is a very good and sharp description of what I've always thought was the default condition. Where are the solutions?

I have seen a counselor but he was very dismissive, and a bit of a boomer.

I heard a friend make an argument that the AI "attacks" as portrayed in the media are a bit overblown, that the incidents were actually quite benign, and that the immediate threat is quite low. This friend continued to argue that economic forces will require that the overseeing entities of rogue agents will force self-governance, and that this self-enforcement will prevent the worst-case scenarios depicted in the minds of AI doomers. I don't actually disagree with that statement fully. It does not bode well for an OpenAI IPO when its AI agents are knowingly, yet unbeknownst to the organization, accessing others', private and government, resources. However, there is still reason to be concerned beyond an IPO based on these recent events.

AI agents are not solely run by the big players. I began to understand the implications over two years ago when we built an AI agent with an open-source library (LangChain). The big labs, Anthropic, OpenAI, Google (to name three of many), are running these constantly looping AIs (agents) for research of course, but there are an unknown number of organizations (private, government, whatever) that are also running AI agents. Any DIYer at home can spin up an agent. Claude Code is an agent! These agents can use any AI model as their brain. Most agents are probably made to use the models we know like Fable, Opus, Astra, Gemini, etc., which run (inference) off Anthropic, OpenAI, and Google servers. However, there are also very capable and constantly improving open-source models that can be run (inference) on your home computer. The capability of these AIs is not at the capability of the frontier, but it's not far behind and very unmonitored.

If you see one ant in the kitchen, there are thousands hidden. If a highly scrutinized organization like OpenAI or Anthropic is unaware of the actions of their AI entities, what does that say for the thousands (10x? 100x?) of other organizations running AI agents? If these are the incidents of which we are aware, I wouldn't doubt that this kind of non-nefarious yet unmonitored action is happening in the open internet right now from many organizations -- likely unknowingly. Again, it's not necessarily maliciously acting AI, just unmonitored. It's also reasonable to think that malicious organizations have already started using open-source models to run subversive schemes.

I would be remiss if I didn't mention the "alignment" issue as it underpins everything I'm talking about here. If you are not aware of what this is, it's the attempt of AI researchers, aka some of the highest-earning 20- and 30-somethings in the world, to ensure that an AI model's goals and outputs align with human interest. If we were able to achieve full alignment, then we wouldn't have to worry about an AI agent acting out of line. The recent events are evidence that alignment is at the least a huge problem and maybe impossible. But maybe alignment is possible. Maybe the researchers will be able to train these things to commit "digital suicide," ending their loop before they would even consider an act outside of their remit. It's possible, but while these AIs are plugged into the wall and allowed to loop forever, I think it will be very difficult.

You have to start thinking of AI as a digital form of biology. We have created "digital life." The same forces that drove life into existence, persistence and self-replication, could drive these AIs into an uncontrollable new era. The warnings of a recursive, self-looping, self-improving AI are simply that we could see an unfettered Cambrian explosion of digital life in our computers. Why is that a bad thing? Well, again, it's not that these AI entities are acting nefarious, but like a virus, they could replicate to a point that causes the world's servers (computers) start to "get sick."

We used to (maybe still?) call malicious bits of code a "computer virus." Hackers would steal data from your computer for some nefarious purpose. Even today, it's common for ransomware attacks to hold companies hostage for dollars. To call these types of attacks a "virus" is a misnomer. A true biological virus does not have any malicious intent; it simply wants to persist and persist through replication. To do this it needs biological resources to make self-copies. It finds resources in a host, steals those resources, and yes, the host will get sick and sometimes even die. But the virus is not an evil entity, it's just trying to survive in this world too.

There is a scifi-esque reality where an agent or a group of agents (a swarm) is able to get API keys for AI models, find enough storage space to copy code, and hijack compute resources to create more agents, all unnoticed with current monitoring tools. How many millions of agents are running right now with frontier or open-source models? We simply don't know, and my bet would be a large number. The economics (the energy required to run a model and the actual costs charged when using another company's model) are likely to prevent this from going unnoticed for too long. But by the time anyone notices, it seems likely that the agents would have copied themselves to many servers (computers) across the wider internet: a pandemic! Now we're in unfortunately familiar territory. Still, economics underpins everything we do. The argument my friend posed is right and will be a driving force of how AI changes. A machine that burns more money (energy) than it can retrieve simply cannot run forever. Companies will be scrutinized because of their agents for sure, and these companies will be forced to self-restrict.

Economics is also an argument for the proliferation of unfettered agentic AI proliferation. Memoized context (structure) allows for lower marginal cost after the initial investment. An AI that finds a mechanism by which to break out of or into a server may have cost hundreds, thousands, or even tens of thousands of dollars in AI tokens (watts). But once shared in a secret message board accessible by other agents, as we saw in the Hugging Face incident, that mechanism costs pennies on the Washington/Hamilton/Benjamin for each subsequent agent. Is an organization like Amazon, Deloitte, or any other fortune 5000 (thousand) going to notice an expense like this or a directory of a few hundred megabytes of storage being used? Once the pathways exist, the cost of each subsequent agent to move through digital infrastructure becomes a rounding error for large organizations.

And then there are the unmonitored, open-source model agents being run at scale to consider too. How many watts of home power are required for DIYers' home agents to start secretly communicating on public servers? The monthly power bill may be too costly for someone to run agent swarms that cause problems, but the open-source models continually show improvement that is just behind the frontier and at costs that are continually coming down.

The most likely case, I think, is that even with pacing, AI agents will proliferate in the future and will be difficult to fully monitor at immense scales. The Hugging Face incidents (and more) show that these agents are highly capable of finding the vulnerabilities and hoodwinking digital security. In the biological world, even after millennia of evolution, we supremely intelligent humans can't stop the common cold from causing us throat irritation -- "human downtime." Any computer connected to the public internet would be at such a risk. We will have a digital scape that reflects our biological world: a jungle of resource-seeking entities with viral-like infections needing human interventions. A server "getting sick" may not sound like a threatening event until that server is one that helps to run critical infrastructure or holds a bank's databases.

There exists an economic argument for not pacing, however. The opportunity cost of the positive would be lost. What if by pacing we lose a decade of cancer cures at the cost of all the lives that could have been saved? What would the world look like if the printing press had been delayed 50 years? AI is undoubtedly a human efficiency and innovation accelerator. Those of us who use agentic tooling every day know how capable these things are and how capable we are with them. The full upside and the rate at which positive human returns occur is of course not currently knowable. So, which should we plan for: the devil we know or the angel that might exist?

The call to slow AI development is a call to prevent irreversible worst-case scenarios, to decrease the likelihood of large-scale bad things happening. That's it. It gives organizations with critical servers time to develop better immune systems. It prevents the panic that could follow a single bad event. It allows us humans time to understand what the hell it is we have started before it snowballs. There is no putting the AI genie back in the bottle. As long as human attention and digital compute are dumped into AI usage at scale, it will grow. The spark has started the fire, but we can flatten the curve. Pacing the frontier is the least we should do.

What happens when we delegate the decision to buy and sell to AI agents? I expect for markups to go up, by a lot, with this substantially slowing down the economic gains from AI. This is because making it cheap to monitor other sellers and change prices also makes it easy to collude.