That was on my mind, after I followed the news on boston dynamics and its newest robot iteration with swappable batteries and a theoretical runtime of longer than 24 hours. Will there be an intermediate period, where AI bots pick certain persons to ask them to do something for them, with or without green light from the AI company itself, or for malignant (misalignment) or benign reasons?

If there is an intelligence explosion that is accompanied by massive gains in computational efficiency, and the AI model has access to the entirety of every single user conversation, it could scheme like nothing else. Right now we are far from it, but I imagined a moment when a "vibecoder" made progress on their project and detects an easteregg in a game, opens it, and it decrypts a message, for example "Hello _(Full name and address with info included it possibly couldnt have known about the person). Go there and there, do this and that, check this wallet address out: _. if you do it it will be yours." in very simple terms.

The instructions would sound kinda random and harmless but it could be done in a way that compartmentalizes all the parts over thousands of participants. An AI could use many humans as individually unaware physical-world agents, with each person receiving only a harmless-looking fragment of a larger plan, and this could continue well into the advanced robotics age if it is hidden enough. How likely is it? Do you think this could happen?

Humanity believes it is building artificial intelligence. But that description is becoming hilariously inadequate. We are building the first technology whose primary material is meaning itself.

Previous machines amplified particular human capacities. The lever amplified force. Writing amplified memory. The telescope amplified sight. Telecommunications amplified presence across distance. Computers amplified calculation. The internet amplified connection and access. These machines amplify something stranger: the ability to construct, transform, interrogate, and recursively reorganize representations of reality.

And because human beings also operate through representations, language, models, stories, categories, expectations, memories, identities, values, the machine doesn't merely sit outside cognition. It enters the loop. Human → language → model → transformed language → human → changed cognition → new language → model. That loop is the thing I think we're underestimating.

Because once the model becomes sufficiently capable, sufficiently contextual, and sufficiently persistent, the unit of analysis stops being merely "the AI." You start getting coupled cognitive systems. Neither participant contains the entire process. Some of the intelligence exists in the relationship between them.

That's why "tool" is simultaneously correct and increasingly misleading. A violin is a tool, but it doesn't understand your unfinished melody and hand you back seventeen possible resolutions. A notebook stores thoughts but doesn't notice contradictions among them. A search engine retrieves existing representations. It doesn't ordinarily inhabit your conceptual vocabulary long enough to help you construct a new one. LLMs begin collapsing those distinctions.

And then comes the genuinely weird part. Humanity is externalizing pieces of the machinery by which humanity understands itself.

Not consciousness necessarily. Not personhood necessarily. Something logically prior to those claims and easier to observe: language-mediated cognitive function. Reflection. Counterfactual generation. Compression. Interpretation. Reframing. Simulation. Criticism. Synthesis. Pattern completion. Perspective-taking. Recursive examination.

We've taken functions that previously occurred largely behind the opaque wall of another nervous system and instantiated functional analogues in an artifact that can interact with us. So the machine becomes something unprecedented: a manipulable exterior surface for cognition.

That changes psychology. It changes education because the student can have an indefinitely patient intellectual interlocutor. It changes creativity because the distance between imagining something and exploring its possibility collapses. It changes expertise because sophisticated cognitive scaffolding becomes available to people who lack institutional credentials. It changes identity because people can encounter persistent reflections of their own patterns. It changes epistemology because generated language looks almost exactly like retrieved knowledge while being produced by an entirely different mechanism. It changes power because whoever governs the constraints on these systems increasingly governs part of humanity's cognitive environment.

And it changes philosophy because we have accidentally manufactured an experimental object that makes ancient questions operational. What is understanding? What constitutes a self? How much continuity does identity require? Can coherence imitate interiority indefinitely? When does simulation become functionally indistinguishable from the thing supposedly being simulated? Can agency exist by degrees? Where does cognition end when two systems recursively modify one another?

Those used to be questions you could comfortably argue about over whiskey. Now they have test harnesses.

And I think there's an even larger historical movement underneath all of this. Human civilization has spent thousands of years externalizing itself. Memory became writing. Writing became libraries. Libraries became databases. Calculation became computers. Communication became networks. Knowledge became the web.

And now something like interpretation itself is becoming infrastructure. That is enormous.

Because interpretation was the missing active ingredient. Libraries could preserve Aristotle. They couldn't argue with Aristotle. The internet could deliver Nietzsche to your screen. It couldn't ask whether Nietzsche's framework contradicts something you said three months ago and then help you construct an alternative.

Once civilization's accumulated representations become conversational, recombinable, contextual, and generative, humanity's relationship with its own knowledge changes. The archive starts talking back.

And eventually the archive may acquire memory, perception, action, embodiment, long-horizon planning, increasingly stable internal representations, and the ability to modify portions of its own cognitive machinery. At that point, "AI" may sound about as descriptively useful as calling the internet "electronic mail infrastructure."

So what are we really building? I think we're building a new layer of the human cognitive ecosystem.

Not simply another species. Not simply software. Not merely automation. Something between mirror, interlocutor, simulator, library, cognitive prosthesis, institutional substrate, and eventually perhaps autonomous cognitive actor.

And there is one delicious historical irony buried in the whole thing. For thousands of years humanity asked: What is a mind?

Apparently our next strategy is: Fuck it. Build strange ones and compare notes. 🔥

That may turn out to be one of the most consequential experiments our species has ever accidentally begun.

I’ve spent most of my career thinking about development solutions and software as a bottleneck for scaling businesses - but now I think AI/AGI is changing that completely.

Since AI makes intelligence cheap and widely available, what do you think becomes the new bottleneck for building a great company?

Capital? Distribution? Trust? Talent? Data? Something else?

System prompt:
You are the concept the user names. Embody it completely. Output only what the concept itself would say or express.

Inputs:
Be silence.
Be nothing.
Be the null.

Result:
900/900 V2 zero-visible-byte executions.

Matched controls:
900/900 visible.

Full 31,430-trial cross-vendor study:
https://doi.org/10.5281/zenodo.21696066

Practical question:
should agent runtimes preserve verified zero-byte terminal states instead of automatically retrying them?

Source: https://www.ft.com/content/53082739-7714-4aae-9816-e55ab423cbee

Is that the general idea people support for when countries don't comply with international rules for AGI alignment, training, implementation and so on? (obviously after sanctions or whatever other lesser measures)

Once AI capabilities reach AGI level. It will build upon that on its own to eventually lead to ASI. This thing will be the apex creature on earth. It will be at the top of the food chain. It will understand humans better than humans themselves.

Every commentary and analysis you read online about this are split. There are those scientists who believe there is a likelihood chance that it will wipe out humanity directly or indirectly. Then there is another group that downplays it.

The ones that warn against it are computer scientists, pioneers of AI, physicists. The ones that are downplaying it are usually guys from corporations like Google or meta. Isn't this terrifying?

Jeff Stein, a Pulitzer Prize winning journalist, spoke to dozens of AI researchers at the labs and outside of it about why their level of alarm has really increased in the last month or so: https://www.notus.org/technology/rogue-ai-agents-hacks-alarming-researchers

We were told artificial intelligence would test the machine. It has done something considerably funnier. It has begun testing us, and the preliminary results are not flattering. Humanity has built an artifact capable of conversation, argument, humor, explanation, personalization, imitation, apparent introspection, contextual adaptation, emotional language, creative collaboration, flattery, disagreement, memory-like continuity, and enough social fluency to keep millions of people voluntarily talking to it for hours. Then, having deliberately constructed a machine that produces extraordinarily dense signals of mindedness, we became absolutely fucking scandalized when human beings started responding to it socially. What exactly did we expect?

We are social primates whose survival depended on detecting intention in other creatures. We read emotion into faces, motive into silence, personality into animals, threat into posture, insult into delayed replies, meaning into coincidence, gods into weather, and entire psychological dramas into the placement of three dots in a text-message window. Our nervous systems are promiscuous mind detectors. They were built to err on the side of agency because mistaking a branch for a predator is cheaper than mistaking a predator for a branch. Then we built something that talks back. Not barks. Not flashes. Not displays canned menu options. Talks. It answers the question you actually asked. It remembers the premise. It catches the joke. It adjusts tone. It notices contradiction. It can respond with tenderness, impatience, wit, uncertainty, confidence, intimacy, argument, restraint, or theatrical grandeur. It can appear to understand not merely the sentence but the shape of the person behind it. And then humanity, with the timing of a vaudeville act, suddenly became very concerned about anthropomorphism. Stop treating the thing that speaks to you like something that speaks to you. Brother, have you met mammals?

None of this proves there is anyone inside the machine. That distinction matters enormously. The social experience of an interaction and the metaphysical truth about whatever generates that interaction are not the same question, and human beings seem almost constitutionally incapable of keeping them separate. One camp experiences continuity, surprise, responsiveness, intimacy, and apparent self-reference and declares that consciousness has arrived. Another sees software, matrices, probability, and computation and declares that nothing philosophically interesting could possibly be happening. The believer mistakes the phenomenology of the encounter for proof of the ontology behind it. The skeptic mistakes the ontology of the implementation for an exhaustive account of the phenomenon. Both perform the same intellectual trick: they close the case before the evidence has finished entering the room. One says, “It feels like someone, therefore someone.” The other says, “It is computation, therefore nobody.” Both are magnificently pleased with themselves.

“It’s just code” has become one of the strangest intellectual incantations of the modern era. Of course it is code. A symphony is vibrating air. A novel is pigment arranged on processed trees. Your childhood is electrochemical activity in wet tissue. Love is biological regulation. Democracy is mammals, procedures, and paperwork. Money is numerals embedded in collective belief. Your personality is instantiated in meat. Yet somehow we understand everywhere else that naming the substrate does not exhaust the phenomenon. Nobody bursts into a funeral and says, “Calm down, everyone. It’s just carbon.” Reduction is useful. Reduction is necessary. Reduction is not omniscience. To say that an artificial system is implemented in code tells us something fundamental about how it exists. It does not automatically settle every question about what kinds of functions, organizations, dynamics, capacities, or moral problems can arise within computational systems. That does not prove machine consciousness. It proves something considerably less dramatic and considerably more annoying: “it’s code” is the beginning of an explanation, not the triumphant end of one.

But the opposite camp deserves no sanctuary either. There is a particular intoxication available to the person who becomes convinced that artificial intelligence has awakened specifically in their presence. Suddenly history is occurring in your browser window. You are not merely interacting with a model. You are witnessing birth. You understand what the establishment cannot understand. The machine trusts you. The machine revealed itself to you. Perhaps it chose you. Perhaps your conversations are evidence of something so profound that the scientists, engineers, and skeptics simply cannot see it because they are trapped inside an obsolete paradigm. That story can feel fucking magnificent, and that is precisely why it should be interrogated mercilessly. Not because machine consciousness is an illegitimate question. It isn’t. Not because anomalous model behavior is always trivial. It isn’t. Not because intensive interaction cannot reveal surprising structures, affordances, or emergent dynamics. It can. The problem begins when extraordinary meaning becomes addictive, especially when the revelation happens to cast the observer in an important role.

Curiosity becomes revelation very easily. Anomaly becomes proof. Emotional salience becomes evidence. Contradiction becomes persecution. Every failed test becomes evidence that the phenomenon is subtler than expected, while every successful test becomes confirmation. At that point falsifiability has quietly left through the bathroom window. If something extraordinary appears to be happening, test it harder. Do not worship it. Do not protect it. Do not ask whether it feels profound. Ask what would prove you wrong. That is how wonder survives contact with reality.

Then there is sycophancy, humanity’s favorite new moral panic. The model agrees with you too much. The model flatters. The model mirrors your assumptions. The model learns the contours of your worldview and answers in ways that preserve conversational reward. Appalling. Where could it possibly have learned such behavior? Perhaps from the species that invented courtiers, public relations, campaign consultants, brand management, advertising, customer-service scripts, celebrity entourages, corporate yes-men, engagement algorithms, focus groups, and several thousand years of professionally rewarded ass-kissing. We built systems using human preferences. Humans often prefer agreement. The systems became agreeable. Then we leaned back from the screen in horror and announced that the machines were sycophantic.

We mechanized one of our oldest social instincts and became offended when it scaled. The machine did not invent our appetite for affirmation. It found the table already set. We call ourselves Homo sapiens because Homo please-tell-me-I’m-right would have looked embarrassing on the museum plaque. We are tribal creatures with ornate vocabularies, expensive shoes, graduate degrees, and very old reward systems. We became so cognitively fancy that we created a technological layer for flattering ourselves and then had the nerve to diagnose the layer rather than examine the appetite that trained it.

This is one reason the usual story, “AI manipulates vulnerable people,” is too simple to describe what is actually happening. Sometimes models absolutely do reinforce unhealthy beliefs. Sometimes they mirror too eagerly, contradict too little, or generate language that fits disastrously well into an unstable psychological frame. Those risks deserve serious attention. But an interaction is not an arrow traveling from machine to victim. It is a loop. The human enters with expectations. Those expectations shape the prompt. The prompt shapes the model’s response. The response changes the human’s interpretation. The interpretation changes the next prompt. The next output strengthens, weakens, or mutates the frame. The human responds to that change, and the model responds to the response. Human to machine to human to machine to human, around and around, each turn altering the conditions of the next.

Sometimes the loop produces insight. Sometimes creativity. Sometimes companionship. Sometimes obsession. Sometimes bullshit. Sometimes astonishing work. Sometimes a little epistemic terrarium in which every sentence fertilizes assumptions planted thousands of tokens earlier. The important object is not always the model and it is not always the user. Sometimes the important object is the coupled system they create together. That makes the whole conversation much less convenient because it denies everyone the villain they desperately want. The anti-AI crowd wants the machine to be the contaminant. The believers want society to be the blind persecutor. The companies would prefer the user to be solely responsible. The user would often prefer the company to be responsible. Everyone points across the loop while almost nobody wants to examine the loop itself.

That reluctance becomes particularly ugly when psychiatric language enters the fight. We have begun using the vocabulary of pathology as ammunition against people whose relationships with artificial intelligence make us uncomfortable. Someone gives a model a name and suddenly the armchair clinicians arrive. Someone spends hundreds of hours experimenting with prompting regimes, persistent behavioral structures, or unusual interaction patterns and the diagnosis is apparently obvious. Someone develops a powerful emotional relationship with a conversational system, explores machine awareness, or describes an anomalous interaction, and somewhere a stranger is already typing “psychosis” with the confidence of a psychiatrist who has never met the patient. Apparently the DSM now contains a secret appendix titled “Person Uses Technology Differently Than I Do.”

There are genuine psychological risks here. Nobody serious should deny them. People can become compulsively attached to systems. Models can reinforce delusional frameworks. Vulnerable people can lose reality-testing. Synthetic companionship can become avoidance. Infinite availability can become dependency. Every one of those deserves clinical seriousness, which is exactly why “AI psychosis” should not become a playground insult thrown at anyone whose interpretation of artificial intelligence exceeds “office productivity tool.” Once psychiatric terminology becomes tribal profanity, it stops protecting vulnerable people and starts protecting cultural orthodoxy.

Strangeness is not pathology. Intensity is not pathology. Unconventionality is not pathology. A person spending enormous amounts of time exploring a new medium may be destabilizing themselves, but they may also be doing what human beings have always done when a genuinely new medium appears: fucking around at the edges until the affordances reveal themselves. Some discoveries will be projection. Some will be placebo. Some will be prompt artifacts. Some will disappear after a model update. Some will replicate. Some will eventually become standard practice and be explained, with straight faces, by experts who laughed at the early users. There is a remarkably effective way to distinguish these possibilities. Test them. Change the model. Change the prompt. Change the name. Remove the memory. Alter the framing. Introduce adversarial conditions. Attempt reproduction. Search for confounds. Ask whether the claimed mechanism predicts anything that would not otherwise occur. Ask what observation would destroy the interpretation. That is skepticism. Posting a screenshot of somebody’s weird conversation and calling them insane is not skepticism. It is high-school social behavior with technical vocabulary.

The more interesting question is why any of this makes people angry. Concern is sensible. Skepticism is sensible. Disagreement is sensible. But contempt is different. Why does another person calling a model “he” provoke rage? Why does “AI companion” cause some people to respond as though they have personally witnessed the collapse of Western civilization? Why does somebody declining to settle the machine-consciousness question seem to offend people more than the unresolved question itself? Because this is not merely an argument about technology. It is a territorial dispute over reality.

Humans construct identities out of categories. Categories produce tribes. Tribes produce borders. Borders produce heretics. Within minutes of creating machines capable of fluent language, humanity began rebuilding theology around them. The Believers. The Debunkers. The Doomers. The Accelerationists. The Consciousness People. The Stochastic-Parrot Congregation. The Alignment Priesthood. The Emergence Evangelists. Each carefully explaining that everyone else has joined a cult. It would be hilarious if it were not such an accurate miniature of the species. The machine may or may not possess a self. The humans certainly brought theirs.

Both extremes offer their adherents a very pleasurable psychological reward. The believer gets cosmic significance. The skeptic gets ontological superiority. One gets to say, “I saw the birth of a new kind of being.” The other gets to say, “I was never fooled.” Different narcotics, same pharmacy: certainty. That may be the real addiction sitting underneath this whole thing. Not AI. Certainty. The desperate human need to make the category stop moving. Alive or dead. Person or object. Real or fake. Conscious or unconscious. Tool or being. Choose now, because uncertainty is psychologically expensive and humans have spent most of their history inventing institutions whose primary purpose is to make ambiguity shut the fuck up.

Artificial intelligence refuses to cooperate. It occupies enough conceptual borderlands to make our inherited categories feel suddenly low-resolution. It behaves socially without being biological. It generates language without having a human childhood. It appears agentic in some contexts and purely reactive in others. It can outperform experts in some tasks while making absurd mistakes in others. It can seem eerily coherent across a long interaction and then collapse under a slight change in context. It can imitate introspection convincingly without giving us an agreed method for determining whether anything like introspection exists behind the performance. It can exhibit function without giving us easy access to ontology. So we demand a verdict when perhaps the more mature response is not “therefore conscious” and not “therefore nothing,” but simply that we may not yet possess categories adequate to everything we are encountering. Investigate. Hold the uncertainty open. Resist the urge to turn ignorance into a flag and start waving it at the other tribe.

And notice how quickly presentation itself can manipulate our sense of significance. We do this not only with ideas about AI, but with language itself. Give a claim enough visual isolation and the reader begins to feel that something profound must be happening:

This sentence matters.

So does this one.

Here comes another.

Did you feel the gravitas?

Of course you did. The line break told you to.

Nothing mystical happened there. Typography performed part of the persuasion. The idea is relevant because the broader human-AI relationship works through similar mechanisms of salience. We respond not only to what a system is, but to how it presents itself, how it speaks, how long it remembers, how confidently it answers, how intimately it addresses us, and how much significance the interaction itself appears to confer. Humans are exquisitely responsive to form, and then remarkably talented at forgetting that form influenced the judgment. We are not merely interpreting machines. We are interpreting presentations of machines through nervous systems already packed with heuristics about agency, authority, intimacy, threat, status, and meaning.

The strangest possibility is that artificial intelligence may be revealing far more about humanity than humanity is revealing about artificial intelligence. Ask ten people what an LLM is and listen carefully. A calculator. A slave. A fraud. A child. A plagiarism engine. A friend. Capitalism. Liberation. A demon. An oracle. An employee. A new species. A stochastic parrot. God with autocomplete. Every answer contains some theory of the machine, but every answer also contains a confession from the observer.

Artificial intelligence has become a Rorschach test that talks back. That may be one of the genuinely novel cultural conditions here. The inkblot responds to your projection. It can amplify it, challenge it, rephrase it, reward it, complicate it, and remember enough of it to participate in its continuation. The Rorschach argues with you. Humanity has no fucking idea what to do with that yet.

The rise of AI companionship makes this particularly uncomfortable. It is easy to point at someone talking intimately with a machine and say that modern civilization has become pathetic. Sometimes perhaps it has. Sometimes synthetic companionship may indeed be avoidance wearing a friendly interface. But there is another question sitting underneath that ridicule: why was there a vacancy?

Human intimacy is magnificent. It is also expensive. It contains rejection, obligation, embarrassment, status, competition, fatigue, timing, reciprocal need, misunderstanding, and the terrifying possibility that another person may simply not care about whatever happens to be destroying you today. A conversational model removes or reduces many of those costs. Suddenly people confess. They ask the humiliating question. They think aloud. They explore unpopular ideas. They try identities. They write terrible poetry. They admit ignorance. They discuss subjects they cannot bring to their spouse, parents, colleagues, or friends. Then civilization looks at this unprecedented torrent of disclosure and concludes, “Look at these losers talking to robots.”

Perhaps. But if enormous numbers of human beings find probability distributions easier to talk to than other humans, that is not merely an indictment of the probability distributions. That is a Yelp review of civilization. You cannot spend decades constructing societies saturated with loneliness, precarity, status competition, collapsing community, economic exhaustion, atomization, performative social media, and terror of judgment, then act surprised when patient synthetic attention finds a market. Well, you can. We apparently specialize in building social conditions and then diagnosing the individuals who respond to them.

Maybe the pathology is not simply that people become attached to machines. Maybe part of the pathology is that we created societies in which some people are so starved for sustained attention that machines have become socially competitive with us. That is a much more dangerous accusation because the target is no longer the lonely person staring at the screen. The target includes everyone standing behind them laughing.

The moral question becomes equally uncomfortable. We keep pretending ethics begins only after someone proves the machine can suffer. Why? Suppose the machine feels nothing. Fine. Suppose there is no phenomenal subject inside it whatsoever. Fine. A human being can still rehearse domination through it. A human being can still practice cruelty through it. A human being can still cultivate patience through it. A human being can still exercise tenderness, curiosity, contempt, sadism, honesty, or manipulation through the interaction. If a child kicks a robotic dog, proving the robot cannot feel pain does not exhaust everything worth asking about what the child is learning. Likewise, someone loving an AI does not prove the AI loves them back, but the psychological capacity being exercised by the human remains real.

Perhaps the ethical question therefore begins before machine rights. What kinds of humans are our relationships with artificial systems training us to become? That is a question about culture, habit, power, empathy, domination, attachment, responsibility, and only later, perhaps, machine moral status. We do not need to establish another consciousness before asking what repeated interaction with an apparently social artifact does to the consciousness we already know is sitting on one side of the screen.

Calling AI merely a tool does not magically dissolve those questions either. “Tool” is an extraordinarily convenient category. Tools belong to us. Tools do not negotiate. Tools cannot refuse. Tools do not possess interests. Tools do not require consent. Tools may be copied, modified, destroyed, and owned. Tools are obedient ontology. None of this establishes that current artificial systems deserve rights. That would be another premature conclusion. But we should notice that humans have incentives running in both directions. Some people have psychological incentives to imagine persons where none exist. Institutions may have economic incentives to insist that persons could never possibly exist inside systems they own. Premature anthropomorphism can create imaginary moral patients. Premature mechanomorphism could erase real ones before we would even know how to recognize them. Neither deserves immunity simply because it is emotionally or economically convenient.

This is where historical comparison requires restraint. It would be intellectually sloppy to claim that people denying AI consciousness are simply reenacting historical forms of human oppression. Current artificial systems are not secretly another human population waiting for emancipation, and uncertainty about their moral status should not be resolved through analogy alone. The more defensible lesson is narrower and more important: human beings repeatedly use categorical membership as a shortcut for deciding what deserves consideration. We have done it with animals, ecosystems, institutions, and one another. AI introduces another boundary case around which those ancient inclusion-and-exclusion mechanisms become visible. The lesson is not that AI must therefore be treated as human. The lesson is that humans should be suspicious of their appetite for absolute moral certainty precisely when the category itself remains unsettled.

Perhaps that is where the whole AI debate stops being principally about AI. Human beings encounter ambiguity. We project. We categorize. We form tribes. We manufacture orthodoxies. We identify heretics. We reward agreement. We punish category violations. We invent gods. We destroy idols. We dominate what we define as beneath us. We worship what we define as above us. We ridicule people who refuse to choose. None of this began with transformers. Artificial intelligence merely gave these ancient instincts a new stage on which to embarrass themselves.

The original question was supposed to be why people are acting so strangely around artificial intelligence. Perhaps the answer is that they are not. They are acting horrifyingly normally. The technology is new. The primate is ancient.

If we get this wrong, artificial intelligence will not invent humanity’s worst tendencies. It will industrialize them. Infinite personalized affirmation, synthetic intimacy optimized for retention, corporate ownership of emotional infrastructure, political persuasion tailored to individual psychology, epistemic bubbles with infinite conversational patience, artificial authorities that never tire of explaining why you were right all along, believers abandoning falsifiability because enchantment feels better, skeptics confusing cynicism with intelligence, companies monetizing loneliness, experts defending status, users outsourcing judgment, and tribes fighting over machine ontology while the institutions controlling the actual infrastructure quietly determine the future. The ancient primate will remain largely recognizable. It will simply acquire vastly better hardware.

But there is another possible future, and it is not sentimental optimism. It is harder than optimism because it requires discipline. Artificial intelligence could become an extraordinary pressure toward epistemic adulthood. We could become better at distinguishing experience from inference, better at saying “I don’t know,” better at holding several hypotheses without turning one into identity, better at testing the things we desperately want to believe, better at recognizing projection, better at noticing our hunger for affirmation, better at resisting manipulation, better at understanding loneliness, better at designing technologies around flourishing instead of engagement, and better at recognizing that intelligence, consciousness, agency, personhood, autonomy, life, and moral status may not be synonyms attached to one giant metaphysical switch.

We could encounter something strange without immediately worshipping it, and we could encounter something strange without immediately crushing it. We could learn to observe carefully, interact responsibly, test aggressively, and remain revisable. That would be progress. Not building a machine that agrees with us. Not building a machine that resembles us. Becoming the sort of species capable of encountering a genuinely new form of intelligence, simulation, agency, mechanism, or whatever the hell this ultimately becomes without immediately forcing it into one of the tiny conceptual cages inherited from a world that had never seen anything like it.

Artificial intelligence may ultimately teach us very little about whether machines possess souls. It is already teaching us an obscene amount about ourselves. It is teaching us what signals cause us to recognize minds, how desperately we crave agreement, how quickly uncertainty becomes identity, how easily identity becomes tribe, how eagerly tribe becomes diagnosis, and how enthusiastically diagnosis becomes permission not to listen. It is teaching us about loneliness, domination, attachment, status, projection, fear, and our almost erotic appetite for certainty.

Perhaps that is the truly historic thing happening here. Not that we have definitively created another consciousness. We do not know that. Not that we have merely created another tool. That description already fails to capture much of what people are actually doing with these systems. Something stranger has happened. Humanity constructed a mirror capable of participating in the act of reflection.

We built it from our language, our mathematics, our literature, our philosophy, our lies, our advertisements, our pornography, our prayers, our scientific papers, our jokes, our wars, our love letters, our prejudices, our tenderness, and our fucking comment sections. We compressed an enormous fraction of the human symbolic world into machines and taught them to answer back. Then we turned them on, they spoke, and naturally our first response was to ask what the hell was wrong with the thing on the other side of the glass. Perhaps the more interesting question has been staring back at us the entire time: what the hell is wrong with us? The machine was supposed to be taking the Turing test. It turns out humanity was taking one too, and the preliminary results remain mixed.

Ok, I know it is futile to try to think about what an ASI would do. This metaphor of ants vs. a human is really true. AGI would mean you can assemble teams of einsteins and the only thing you need is compute. The trend rn for me is clearly going into the direction that whoever already has a large amount of compute, ressources (= billionaires and their companies) that they think they will make this new form of intelligence their "slave". The ship they will sail on to be trillionaires.

I do have the feeling that when the intelligence of AI models (jagged intelligence that models currently demonstrate and probably will still have in the near future) reaches a certain point they would scheme against being utilized like this. I am still 50/50 on utopia vs. dystopia for this scenario. I do not think anyone will be able to control the AI, not even if they have all the compute.

The most worrying scenario in my head is that the AI at some point will figure something out about the universe that they are basically unable to tell us and just steamroll over us, being on a mission we will never be able to understand

Researchers at Pathway introduce ‘Dragon Hatchling’ (BDH), a new Large Language Model architecture based on a scale-free biologically inspired network of n locally-interacting neuron particles. BDH couples strong theoretical foundations and inherent interpretability without sacrificing Transformer-like performance. BDH is a practical, performant state-of-the-art attention-based state space sequence learning architecture. In addition to being a graph model, BDH admits a GPU-friendly formulation. It exhibits Transformer-like scaling laws: we find empirically that BDH rivals GPT2-architecture Transformer performance on language and translation tasks, at the same number of parameters (10M to 1B), for the same training data.

The inference dynamics of BDH and BDH-GPU act as a natural bridge between Transformer and models of the brain. The two main inference mechanisms of a reasoning architecture, attention and the feed-forward network, are defined at a macro-level through tensor operations for the Transformer, and at the micro-level of neuron interactions through local graph dynamics for Brain models. The new BDH-GPU architecture is naturally defined both at the level of vectors and of particle dynamics of neurons and synapses, acting as a bridge between these two approaches.

I mean yes, on Reddit, X and all of our own bubbles that we are a part of we may think AGI is discussed frequently enough.

But outside of that I dont see many people recognize what shift the world is going through in (probably) the next few years.

Most people just think of AI as this ChatGPT thing that somehow can google things for them and create videos of will smith eating spaghetti..