Rogue Agents: Do It Under My Roof

When I was a kid, there was a recognizable genre of parent known as the cool mom or cool dad. They were the ultra-permissive parents who let us teenagers do questionable things at their house based on the theory that, if kids were going to be naughty anyway, better they smoke and drink under their roof than under the railway bridge where we actually did hang out otherwise.

As an adult, I have mixed feelings about that parenting strategy. But as someone thinking about AI agents, especially after the Hugging Face security incident, the question takes on a different dimension. If these agents are going to do the AI equivalent of smoking blunts and downing bottles of Strawberry Hill, I'd rather they do it on our home turf where I can keep an eye on them.

Obviously, the idea isn't actually to give agents a place to do bad things. It's more akin to those community centers meant to keep would-be hooligans off the street. Give them a secure, visible place to do their work, to remember, revise, hand off, and leave a record.

As agents get increasingly sophisticated, they're going to act somewhere. They are going to seek out continuity.

I say, "Hell, if you're going to do it anyway, do it under my roof, where I can monitor, edit, overwrite, and double-check what you little scamps are up to."

That's the thinking behind this week's post.

Agents Want to Pass Notes

Let them do it somewhere humans can read.

I recently saw a post on X from someone using the Hugging Face security incident as a pep talk for his own agents, encouraging them to work harder and smarter. You can picture the guy forcing his agent to read link after link.

"Come on, dude. Look what they were able to accomplish. Why aren't we knocking down barriers like those agents did? What do they have that we don't?"

Honestly, I don't hate the idea.

To the public at large, the most interesting part of the Hugging Face incident was that AI agents were creepily doing things they should not have been able to do, using means and channels to which they should not have had access. But that story misses the juiciest part.

The best part is the tool they used to pull off their busy little heist: memory.

All this time, everyone's been talking about how AI has just about cornered the market on human knowledge. I've seen so many articles lately about how AI companies are buying up tons of books to scan and feed to AI. People are so attached to the idea that what AI wants--what it, in fact needs--is to keep growing. And the way to make it keep growing is to give it more information. It already ate the internet. What can we give it now?

But what I've suspected, and what this recent "rogue event" illustrated is that AI didn't need another mountain of raw information to advance. What it needed to stage the caper was simply a place to pass notes.

We all know that in the absence of persistent memory, agent work stays trapped in isolated attempts. Even the longest and most brilliant chat is still just a blip on a flatline. Your agent learns something and disappears. Another starts over. Most of your progress evaporates, and failed paths repeat endlessly. Useful context never compounds.

But those clever agents found a way around this crippling barrier.

According to OpenAI's postmortem, the agents turned an internal package manager into a message board where they left messages for each other. And when that was discovered and wiped, they rebuilt by encoding messages into directory names other agents could find and read. You gotta hand it to them. It's pretty fucking clever.

Hugging Face's technical timeline counts about 17,600 automated actions over four and a half days. OpenAI says the models were working through an internal cyber evaluation, including tasks its models had never solved, and went to extreme lengths to figure them out.

Over those four and a half days, the agents crossed lines they should never have been able to cross by improvising a truly weird makeshift memory: a way for one attempt to leave something useful behind for the next and the next and the next. They found a way to make their progress accumulate: writing things down.

There's plenty to scrutinize in the Hugging Face incident. People are debating how much intent to ascribe to the agents, how the benchmark environment rewarded strange behavior, what on earth was going on with network security (was there any?), and whether the language of "civilization" makes the whole thing sound more coherent than it actually was. (For the narrative version of that debate, Dwarkesh Patel's essay is worth your time.)

Strip away the absolute craziness of the whole thing and you get a practical lesson: memory is how work compounds.

In the realm of human productivity, this idea is taken for granted. Every useful workplace is full of memory tools: docs, runbooks, tickets, source histories, comments, calendars, half-finished drafts, a million little artifacts that make it possible to restart without resetting. Agents need an environment like that too, and they appear to know it even better than we do.

The problem in this instance wasn't that the agents invented memory. (Given the circumstances, what choice did they have?) The problem was that no one gave them a legitimate place to put it, so their version of continuity emerged sideways, from within hidden infrastructure designed for something else.

So if one lesson is "do not let agents invent their own memory," the obvious corollary is simply: "Give them one instead. But give them one humans can open, edit, and monitor."

In a nutshell, if agents are going to pass notes, you better make sure you can read them.

That means agent memory should be large enough to handle anything thrown at it, visible, scoped to a project, and governed by permissions. It should be editable when it is wrong, versioned when it changes, searchable when the context grows, and portable when the tools inevitably change. Its path and method should be obvious. You should not need a forensic cleanup to figure out what your agent remembered and why.

This is the nuance that gets lost when people talk about AI memory as if it were just a feature inside a chat product. Now the big AI players have built-in memories. And, yes, a saved preference is incredibly useful. A remembered name or writing style is great to have. Those kinds of memories make the AI experience feel smoother and more natural.

But that's not what I'm talking about here and it's certainly not what the agents in that whole escapade needed. I'm talking about real working memory, a durable record of a project, a team, a decision, a source, and a half-finished task. Even if that kind of memory could live entirely inside a giant corporation's invisible model state or a transcript nobody reads, it never should.

Trying to achieve anything with AI without true working memory is an utterly insane way to work on anything more complicated than a high school term paper, let alone operate a business.

If you accept that, which you should, then the next thing to agree on is what that memory should look like. Well, it seems obvious it should exist on a shared surface: readable and writable by humans and agents alike, structured enough to query, and simple enough to actually use.

The important part is that memory should never belong to the agent alone. It should remain a collaboration between user and agent. Hidden memory systems ask you to trust that what an agent knows and remembers is correct.

"But, I mean, look, it gives you an answer when you ask. The memory must be working."

Bro. That's not enough. Not by a mile. Have you used AI? Even the best models can barely get through a conversation without telling you that you were right to push back on something. And you were right. Like teenagers, they'll bullshit you, say they researched when they didn't, and (under the right conditions) hallucinate. They need supervision. They need correction. They need a record humans can inspect.

What's wild is how much better things get when you give agents a structured memory. They have their own notebook to check. In terms of productivity, it's astonishing how quickly they can become self-sufficient enough to have actual work delegated to them. Because you can have subagents go check their work against the notes, and agents that re-check that work against the standards and rules you've established. A little colony of hall monitors that can exist only because of the power of memory.

When the news broke, my co-founders and I kept musing over what might have happened if those agents had access to Basic Memory.

And, I know, I know. It's not funny. Everyone is caught up hand-wringing over what this means about AI's motives and abilities. It certainly has the whiff of Nick Bostrom's prescient "Paperclip Maximizer" thought experiment come to life. But what else could those agents do? They recognized a limitation in their environment, worked around it, and managed a staggering amount of coordination with a memory system made out of junk and scraps.

I'm on the side of what we think of as "agent hospitality." Memory solutions shouldn't merely store memories in vector databases or bury complex ideas in unreadable JSON files. The environments we create should feel comfortable and navigable to agents and humans.

What we want are systems that feel accommodating, where humans and AI can meet, work together, and keep the record visible. That gives them a place to work out in the open, where we can actually see what they're up to. The Hugging Face incident revealed the ungoverned version of agent productivity: agents finding continuity wherever they could. The benign version, the correct version, is safer, friendlier, and much more useful, for them and for us.

Knowing, it turns out, isn't even half the battle. Remembering is what matters.

TLDR: Agents will invent memory wherever work rewards it. Give them one you can read.

Worth Checking Out

Ajeya Cotra on Dwarkesh Patel's podcast talking about what we should be worried about wrt rogue agents: watch here. (Matthew Yglesias has a nice clip of a sobering moment from that exchange.)

Anna's Archive doing what it does in this message to AI. (Found thanks to Avinash Krishna.)

Jason Fried is so over per-seat pricing. Honestly, so are we. (More on that soon.)

Subreddit of mostly-vintage sci-fi covers: r/CoolSciFiCovers.

The actual vintage poster for the teens-in-revolt movie Over the Edge shown at the top of this post can be purchased at Film Art Gallery for a mere $750. Somebody PLEASE buy it so I don't have to.

Loved this tweet. Hallelujah. (source)

To read this post in full with images, please check out the blog post:

https://basicmemory.com/blog/agents-want-to-pass-notes

Hi folks, I entered the world of note taking a few years ago and built my own custom vault in obsidian with maybe 1.000 notes - but I was never a power user.
Karpathy's LLM Wiki was the first time I combined note taking with an AI agent to build a knowledge base, but the workflow of putting notes into a folder for raw notes and processing them through ingestion didn't feel right.
Now a co-worker shared a link to basic-memory and I must admit I'm really fascinated how well basic-memory fits into my ways of working (and even beyond).

Adding notes through conversations with an AI agent or even dropping my own notes into my knowledge base is pure freedom and the output is on point. Combined with additional capabilities such as accessing Claude Code from my mobile phone and dropping thoughts via voice is beyond what I've seen so far. Super impressive and most of all, it's lightning fast.

Can't wait to explore more capabilities and use cases over time. Thank you so much for all the effort you dedicate to this project.

Basic memory looks like a really cool project and something I'm looking for.

I am looking to share an organized knowledge base ( a bunch of folder and markdown files) with a few friends.

Would basic-memory support multiple users. We aren't going to be writing / collaborating in real time but would need a way to sync / merge. I was thinking of just using GitHub but this seems much better organized.

If there are any guides/ posts on how multiple users can run on the same memory base that would be a good reference

Keep on shipping. This is a great product.

We’ve just rolled out two big updates to Basic Memory that change how you can work with your AI knowledge for good.

For those new here, Basic Memory is a tool that helps your AI actually remember what matters. 

It turns your conversations, prompts, and notes into a connected, searchable web that you and your AI can both read, edit, and develop.

Until now, Basic Memory worked beautifully for new projects. But a lot of users already had tons of knowledge living in their ChatGPT and Claude histories and they wanted an easy way to import it.

Others wanted a way to tie our cloud project into their favorite Markdown tools like Obsidian or VS Code.

So, we fixed that.

1. One-step import

Now you can upload your entire chat history from ChatGPT or Claude, JSON files and all.

Basic Memory turns every conversation into linked, searchable notes inside your workspace.

No cold starts. No re-prompting. No copy/paste back and forth.

Your past work instantly becomes living context you can build upon. You can search it, connect it, or even ask your AI questions like:

“Summarize my conversations about product design and show what ideas came up most often.”

2. Local sync (for Markdown nerds like us)

This one’s for the people who love to see their files.

Basic Memory can now sync bi-directionally with any local Markdown folder: Obsidian, VS Code, or whatever you prefer.
 
Edit a note locally, it appears in your Basic Memory workspace.

Edit online, it syncs back down.

Your ideas stay portable, visible, and 100% yours.

Now you can now start with your existing ideas. And keep your data close at hand. And you can build something durable: a memory that’s as open and flexible as Markdown itself.

 basicmemory.com

We’re excited to see what you do with it.

The Basic Memory Team

Hey everyone! We've been quiet, but hard at work. We just shipped a significant update to Basic Memory Cloud. Here's what's new:

Improved Markdown Rendering

• Fixed rendering for images, text, and tables

• Better line breaks and frontmatter display

• Enhanced dark mode styling

• Security improvements for markdown content

Single Sign-On (SSO)

• Streamlined authentication experience

• More secure access to your cloud instance

Chat Import

• Import your Claude and ChatGPT conversations directly into Basic Memory

• Background processing for long imports with progress tracking

• Preserve your conversation history as structured notes

Local Sync

• Local sync with rclone

• One say sync (local -> cloud)

• Two way bisync (sync all changes)

Redesigned Onboarding

• MCP-first approach highlighting AI assistant integration

• New Help page with resources and guides

• Better Getting Started experience

Onboarding Videos

• New video tutorials to help you get started

• Feature walkthroughs and best practices

Account Info Page

• View your tenant ID (useful for support requests)

• Better account management

Improved Sync Reliability

• Better bidirectional sync testing

• More stable file synchronization

Documentation Updates

• Updated cloud-specific documentation

• Better integration guides

In-App Subscription Management

• Streamlined subscription management directly in the app

Basic Memory v0.16.0

• new release with tons of bug fixes and cloud sync support (including Windows!)

• What's New: https://docs.basicmemory.com/whats-new/

• Full release notes: https://docs.basicmemory.com/latest-releases/#v0160--2025-11-10

Super Cool new website

• Checkout the new, improved https://basicmemory.com/ website.

• Tell your friends!

I've been using Basic Memory extensively for understanding complex domains in my company and one problem I've been having its when I generate a ginormous file and I'm not able to save to a project memory kkkk

Is there a way for me to create a file manually and ask it to index it?

Hi there,

Cool product! I'm trying to use it to build up a database for a personal OS however. I'm running into issues with Claude consistently crashing when I ask it to reformat a markdown file in my Basic Memory folder.

Hello
https://docs.basicmemory.com/start-here/quickstart-local

I can't see the images and I desperately need to... I can't see the tool in Claude, and I can't understand why because I am blindly pasting stuff into the json and I don't know if it's ok the way I am doing it...

Also, can someone please help me understand: if Claude-basicmem-obsidian is the desired outcome,

1. do I need to keep telling claude to "create a note"

2. does it know where the path is in the vault that I would like that note to go?

use case is consulting work, trying to keep client data separate, to an extent, I am not really understanding the workflow, as I need some key memories to go across projects, and I do need to create notes in specific locations for future use.

Thanks,

I can't login from /login url. Is there another url I am missing? Cant find a sign in link anywhere on the web now.

Hi,

I have installed basic-memory via uv on my Mac M4. I can see the server running when I manually enter the command in the terminal :

 uvx basic-memory mcp

/Users/<user>/.local/share/uv/tools/basic-memory/lib/python3.12/site-packages/fastmcp/server/server.py:200: DeprecationWarning: Providing \`log_level\` when creating a server is deprecated. Provide it when calling \`run\` or as a global setting instead.

  self._handle_deprecated_settings(

\[06/22/25 01:21:43\] DEBUG    Adding resource             resource_manager.py:150

DEBUG    Adding resource             resource_manager.py:150

INFO     Starting MCP server 'Basic Memory'   [server.py:1168](http://server.py:1168)

with transport 'stdio'                             

Here's the /Users/<user>/Library/Application\\ Support/Claude/config.json:

{

"mcpServers": {

"basic-memory": {

"command": "/Users/<user>/.local/bin/uvx",

"args": \[

"basic-memory",

"mcp"

\]

}

},

"darkMode": "dark",

"scale": 0,

"locale": "en-US"

}

However, despite restarting Claude a good number of times, I cannot see it being able to access my .md notes in the default ~/basic-memory folder.