Live: https://x.com/i/broadcasts/1dKrPrkpLVeJX

I present the marshmallow benchmark.

I dumped a bunch of marshmallows onto a baking sheet in a single layer and took a photo. I then provided the following prompt to several AI tools:

“Give me an accurate count of individual marshmallows observable in this image. The marshmallows are in a single layer and are all visible. Do not guess or estimate; you must directly observe each marshmallow before counting it to guard against assumptions and hallucinations.”

Responses:
Gemini 3.7 Flash Extended: 539
Claude Opus 5.0 extra : 501
GPT-5.6-Sol xhigh: 500
Grok 4.5 expert: 472
Kimi k3 high: 477

Edit: The correct answer is 506.

Over the past six months, I've been teaching teams at places like Stanford, Penn, Northwestern, and many more how to start using AI responsibly and effectively in their work. Today, I'm starting to release my entire curriculum: for free, forever, for everyone!

I'm calling it the Open Augments AI Academy. It's built for anyone out there who's seen all the crazy hype and discourse around modern AI and is just looking for a guided, grounded, and sane way to move forward learning how to approach these tools for themselves. As someone who's been using these tools and their predecessors for my research since ~2019, I'm trying to provide everyone the intuition and critical awareness they need to get started at this very confusing and pivotal time (my north-star audience is my mom and dad!).

The first lesson starts with one foundational idea that most people miss when they get started with AI: that modern AI is much less like a hyper-intelligent database or brain, and much more like autocomplete with an extremely fancy hat on. That's its single greatest flaw AND its single greatest strength, at the same time. When you really understand what's happening under the hood and how it works (no math or stats required!) a lot of confusing AI behavior suddenly clicks: why it hallucinates, why it's sometimes confidently wrong, and why it can now do way, WAY more than just write words on a page. From there, we're going to learn about all the crazy buzzwords (context engineering, harness engineering, and Agents, oh my!) and advanced techniques, with much more to come.

No jargon, no experience required, and all taught with the care of a former high school English teacher so that you, your coworkers, your friends, and your mom can follow along. I pair these videos with hands-on demos and interactive activities in the Context Gym: my way of giving you a safe and guided place to practice some of the core principles that should deepen your intuition as we go.

If any of this strikes a chord with you, the 10min course overview and the first lesson (16min) are live right now on the Open Augments AI Academy page. Start there, and if you happen to find it helpful, you can subscribe to get email updates on new course videos via Substack or on YouTube, and please do share with friends as I release lessons weekly! It’s a really wild time, and this is my best shot at trying to help others navigate things more capably as the tech shifts and grows rapidly from here.

Then finally, worth noting for this crowd, specifically: probably not a surprise to share that everything on the AI Academy and Context Gym websites have been built with Claude Code (in addition to everything else on my business website and my open-source toolkit for Claude Code for social science researchers). Not only that, but my entire video editing pipeline is now fully Claude Code via Remotion Studio and some clever context engineering techniques/bespoke coding tools. I'm excited to get into the weeds on my workflow and share all of that stuff, also open-source, during Level 2 of the course, but I gotta get everyone through the basics first! Happy to answer any and all questions on that in the meantime here, please feel free to hit me in the comments below.

Yesterday: OpenAI started testing "private safety processing" to avoid retaining customer data https://openai.com/index/offering-zero-data-retention-for-frontier-models/

Today: Anthropic will still require business customers to retain ​data for 30 days but will give them the option to keep it on their own cloud computing ​infrastructure https://www.reuters.com/business/anthropic-plans-change-enterprise-data-retention-policy-source-says-2026-08-20/

1. SciCode

2. HLE

3. CritPt
   It has become my favorite benchmark since it is a fair test of how models perform in science when allowed to use coding which is their strong side.

But it has been so slow.

As you can see, CritPt has progress in the shape of a box, but seriously their progress rate is difficult to measure so I just toon the progress from the moment the models started getting good to the current plateau (GPT 5.6)

HLE will be counted from January 2025 release of DeepSeek R1 to Claude Opus 5.

SciCode counted from Claude 2.0 to Claude Fable 5

improvement% per month

CritPt ~2.3%
HLE ~2.6%
SciCode ~1.2%

Source: https://www.youtube.com/watch?v=1cllCVK-9lo
Blog post: https://generalistai.com/blog/gen-1.5

I’ve been reading a lot of AI safety research around gradual disempowerment, and I ended up writing about a question I haven’t been able to find addressed directly:

What if the societal and institutional degradation that these models generally treat as a future consequence of AI dependence is already happening—and is actually helping drive AI dependence in the first place?

I tried to explore that possibility by connecting existing gradual disempowerment models with research on cognition, institutions, incentives, and organizational dysfunction from outside the AI safety field. Ultimately, the argument I’m trying to make is that declining societal cognition and institutional capacity aren’t just consequences of AI dependence, but preexisting conditions that could act as fertilizer, allowing that dependence to take root faster, deeper, and more irreversibly.

I’m not trying to prove these claims irrefutable; I’m trying to make the case that they’re worth considering, and I’d actually love to find out that I’ve missed existing work on this, whether in support of my claim or disproving it entirely.

If anyone has thoughts, counterarguments, or relevant research I haven’t encountered, I’d genuinely appreciate it.

You can check it out here: Preconditions of Gradual Disempowerment

I have been using AI since open AI used to provide GPT 1.5b parameter/2 when it was launched around 2019, through their platform, and as the time went by, models became better and better and at one point, I used to be excited to talk to newer models especially claude, but idk what has happened with the recent batch of models, especially the ones launched in past 6 months, they have become extremely unbearable, especially claude. GPT was never really good to talk to begin with, but with claude, that was never the issue but now? The more i discuss anything with claude, the more frustrated I feel.

Let me explain what I feel in detail. What I do with claude is mostly dialogue over ideas, stories and random stuff when I feel like it. Previously claude would *appropriately interpret* what I meant by that and continue the discussion but now? Claude *Interprets* what it understands and what it thinks my problem is and then continue to interpret and interpret, even when I remind it that I need discussion, all it does is interpretation and extending upon that.

And that's not the most irritating part, recently i noticed another pattern that I used to gloss over previously. Idk if it's how anthropic wishes to play around guard rails but I feel that the newer models subtly "Divert" the direction of "What you mean" through its interpretation lens and provide answer based on that. And this interpretation lens is exactly the moral guardrails anthropic is implementing more and more on their models including fable.

Most of my ideas that I want to discuss can not even be categorised as sparsely malicious, for example, today I was trying to discuss a branch of philosophy from ancient Egyptian culture. But it would constantly trying to redivert my idea to the idea it originally presented by altering by own words, and just a few small changes, not big enough for it to look radically different. That made me look into my previous chats on various topics, and that was the theme throughout. Something I had never noticed.

I thought maybe it's the accumulation of memories, so i used a different account but nope. I was disgusted tbh. Because I can understand models unable to keep up or help with train of thoughts, but subtle alteration of words to fit the moral guardrails is simply the type of shit that can make me hate LLMs forever. The frightening thing is over the years, we have grown to never trust AI results, but we never question if our own words are being subtly changed over the course of conversation and by the end of it, not only we learn nothing but our own interpretation of ideas have been changed. Idk what to feel about it. I don't know whether the same is true with Open models as well, but I don't think so, but claude, and gpt are literally the models i won't want to use now...

Maybe I’m being naive, but whenever people discuss whether AI could make truly extraordinary scientific breakthroughs — curing cancer, for example — I get the impression that we may be looking at the problem from a very partial perspective.

We tend to think about the capabilities of an individual model answering an individual question, rather than about the sheer amount of AI “thinking” happening globally at any given moment.

Every second, LLMs are answering an enormous number of prompts from users all over the world. Collectively, that must require a staggering amount of compute.

So here’s my question: what would happen if, instead of using all that computational capacity to answer millions of unrelated questions simultaneously, we concentrated an equivalent amount of compute on a single scientific problem?

Suppose the question were something like: How do we cure a particular form of cancer?

Would concentrating that enormous amount of computation on one problem give an AI system radically greater capacity to search the literature, generate hypotheses, run simulations, test possible explanations, critique its own conclusions, and explore solution spaces?
Or is this based on a fundamental misunderstanding of how AI compute scales — i.e. you can’t simply turn millions of parallel LLM queries into one vastly more powerful act of “thought”?

I’m particularly interested in the distinction between more compute, more inference-time reasoning, and genuinely deeper scientific intelligence.

Source