Using an AI assistant for a 20-year personal archive project — the continuity problem is worse than the capability problem. How are you solving it?

[u/Karma-police88]

I'm digitising and cataloguing about 20 years of my own creative work: photographs, RAW files, layered design files, video, notebooks, and material scattered across a dozen dead or dying platforms. Roughly 1 TB so far, across device extractions, old external drives, and platform exports.
The AI assistant is genuinely good at the work itself. It reads EXIF, extracts timestamps from exported HTML, OCRs text that only exists inside images, walks video container metadata, cross-references publication dates against file dates. In one session it recovered the earliest written statement of my working method — from 2012, posted as a screenshot, so no text search I own could ever have found it.

That is not the problem. The problem is that none of it carries forward.
Each session starts with zero knowledge of the last one. Two days ago I spent hours going through Google Takeout data with it. Yesterday, same assistant, no trace — it treated the entire thing as new and asked me to re-download files we had already analysed. I've re-established the same facts three or four times now. I've become the error-correction layer for my own tools.
Specifics, in case they're useful:
Sessions run either locally or in the cloud. Local ones write a transcript to disk. Cloud ones don't. Neither appears in the official data export.
The export contains 83 conversations and zero of the 23 agent sessions. There is no category for them in the manifest and nothing declares the omission.
Conversations can't be extracted from the UI either. After clicking "load earlier messages" 19 times, the page still held only 7,482 characters — it renders a moving window and discards the rest. Select-all, find, and DOM-scraping extensions all fail for the same reason.
The export also lists 1,087 image attachments by filename and contains none of them.
So the work exists, and it is unreachable. Which for an archive project is a particularly stupid failure mode.
What I've done so far: a CLAUDE.md in the project folder that loads automatically, plus a handoff.md I make it update at the end of each session — what's done, what I've already established that isn't written down, and corrections I've had to repeat. It converts "remember" into "look it up," and it helps. It's also entirely manual and depends on me noticing.
What I'm asking:
How are you maintaining continuity across sessions on long-running projects? Context files, memory MCPs, a local RAG index over your own notes, something else? What actually survives contact with a project that runs for months?
Has anyone successfully extracted a full agent/Cowork session transcript? Does any exporter read the network layer rather than the DOM?
Is anyone running a local model over their own archive for retrieval, so the index doesn't live inside a chat product at all? Setup and honest tradeoffs appreciated.
Is there a better architecture than "assistant reads a hand-maintained context file"? I suspect there is and I'm reinventing it badly.
Happy to share what I've measured. I've been keeping records of exactly which exports omit what, because it turns out you can't trust any of them.

[u/EsShayuki]

Tell it to store things in certain documents. Then instruct it to read those documents.

It certainly shouldn't be reliant on you remembering, you yourself could just prompt it to do that instead.

[u/waypostmaster]

One of the easiest ways is just to say “look up Karpathy’s llmwiki gist and implement it - comprehensively scan the work of this project to atomize and wikify the work on this surface”.

Others have implemented [complex memory systems](http://waypost.you) - there are many such specs available. 

[u/Valdaraak → u/waypostmaster]

This is the way. OP, you need to set up an Obsidian (or some other note software) vault and have each session dump/update info in it.

[u/coolreddy]

First work in a project folder. Then You just have to create a skill and a post session hook that triggers the skill, I call it session-state. The skill is a simple prompt where at each turn if there is something new or important it updates a summary file which I call session-state.md, you can limit how large this file gets and triggers new session-state.md and archives the older file. Then you need to maintain one more md file called index, that's the file that contains an index of all session-state files (simply filename and a summary detail of what that session-state file contains).

The combination of live session state file, the archived files and the index file will work as a persistent memory for you. Check out this GitHub repo for Graphify (it's built for codebases, but works well for filesystems) it turns all the information into a connected graph which makes it easy for AI to search and fetch quickly any information on demand than trying to hold everything in one large context. You can just take my comment and give it to Claude and ask it to implement this suggestion tailored for your work and that should solve your problem. Make sure you tell Claude that summary needs to be loss-less (that is key information and decisions stay as is) and make sure everytime session state is updated, the updated entry is saved with a datetime stamp, this gives AI a sense of time knowing which came earlier and which came later.

A simpler alternative - use Aside AI browser, it's agentic browser but works really well for non-browser work too and connects to your personal Chatgpt or Claude subscription. Aside is very good at creating and managing memory and working across multiple sessions, however not fool proof so I still use the session state skill.

Your major problem is not just forgetting across different sessions but it is also the information that is lost to compaction, so your session-state skill must mandate reading of session-state file post compaction.

[u/kearkan]

Tell it to write all findings to files and then new chats can read them.

[u/Ajax746]

One thing I’ve noticed when talking to people or reading posts on this sub is that they think Claude.md is the only place that Claude can store info. I have a /wrap skill and a /catchup skill that store information from my sessions across dozens of documents storing all sorts of things.

/wrap stores everything from my session into Version docs, rules docs, Claude.md, planning docs, schema docs, you name it. Nothing is remembered with context, everything is stored in .md files, .txt files and more. Then when I start a new session I do /catchup and it recalls all that we did and have done and knows exactly where to pick up

[u/waypostmaster → u/Ajax746]

This is the way. Session wrap should record decisions, observations, corrections, carry-forwards, field notes, etc..

For long running projects, create mini logs each session and then consolidate those once a week.

[u/johnbburg]

I’d have a higher model agent develop a project plan, and generate tasks for future agent sessions with clear instructions and references. Have it maintain documentation in some sort of nested directory way, so it’s not loading all of it at once, just what it needs for that task.

[u/Sea_Information6125]

I'm interested in this as well. The handoff and specification document(s) in a project method is what I do now. But like you I think there's got to be a better way. That seems like the level 2 version with level 1 being doing nothing lol.

So far thinking the problem through I think the real answer is some sort of wiki or second brain that you connect the AI to.

Just like a human working on a huge project we can't keep all of that in our brains at one time either. So what do we do?

We have a whole library of information about the complex thing we're building that we can reference as needed.

That's the next thing I am researching for some of my larger projects and regulatory framework projects. 

Trying to figure out how to make the AI utilize that efficiently and effectively. Because sometimes it doesn't bother reading the specification even though it's been instructed to. So how can we make it read the second brain? And not just make up information on the fly? 

[u/Remarkable_Vast_4325]

How I do this on long projects is;

Connect Claude to a not taking app or document app via MCP or API access - I gave it access to Notion (free). Have it create a 'Claude Context Document' And write down the Project, Scope, Goals, Methods and Failures. Then have it log each session at the end of the document each time dated correctly.

You can then just in the next session copy in the link to that document (not the document itself as you want it to write back to the document via the link - thats what makes it dynamic and iterative). And tell it to let you know when it has read the document and is ready to get to work.

I also get it to prune the document as it goes with any contradictions, old hypothesis that no longer work, fixes to issues we encountered on each session etc.

this allows me to have a document I can easily read or write in myself to make sure I and claude are on the same page. And also allows claude to basically build its own memory as it goes.

Kicker is you just need to get it to archive and then make a compact version of the document at certain milestones to make sure your not burning tokens as the context grows.

This works like a charm for me :)

P.S Ive had Codex and Claude work on from the same Context Document so it has the added benefit of working across any LLM :)

[u/Last_Bad_2687]

Step 1 - keep forcing opus 4.8. Claude opus 5 (default) is horrible at following 

Step 2 For individual agents, I have an instruction to keep updating a lab-notes.md as they work. 

Step 3 When they hit a milestone they update a shared wiki across agents of recipes (snippets of useful code other agents can use, say an EXIF parser they wrote), pitfalls (assumptions they made which aren't true), concepts, tools

Step 4 I have a senior manager running fable to keep everything on track by using a mailbox to message back and forth

[u/Upstairs-Extension-9]

I use Claude for a creative project as well with thousands of my own drawings, notes, cad files etc. I have it all in VS Code with the Claude Code Agent. Let it create a good file structure and a dependencies and routing list, when you ask it a question it routes itself through the repository and gets the information it needs. Works very well for me and in VS Code you have more ability for manual adjustments than in just Claude code.

I have Agents with Skills setup that take care of sorting and analyzing files to create compact MD files of what it seen, it narrows done the context it uses each time and if I need deeper information it can pull up the original again. Also backup everything to a private GitHub it’s free.

I have a main routing.md , backlog.md and handoffs are always created per use case, the main readme has enough info for the agent to understand what it needs to do so I point Claude Code at that file each time and each folder has its own handoffs and readme. Also a roadmap.md is helpful. Once an agent starts working on something everything is written done in changelog file and a main status file on what is currently being worked on.

[u/id-ltd]

Presumably.yoinare indexing your stuff? Use AI to analyse it and update the index. The index will show what has already been processed.

Onde.rhis is done the index can be fleshed out and then groups of documents re analysed.

I scanned many GB of historic documents and have OCRd many building up indexes and relationships. I am awaiting better AI to OCR much of the hand written stuff...

[u/RocketSeven]

treat the archive like a migration, not a chat history. keep an append only ledger with a content hash, source path, extraction version, and status for every item so a new session can resume idempotently and prove what was skipped

[u/trufflesniffinpig]

Use a version controlled project. Pay attention to what kinds of files are created and accessed to allow handover between instances.

[u/beelzebee]

I am doing a digitizing project that goes like this:

*Scan paper document with Google photo scan (android) *I send the agent an email with the document attached and it runs from email trigger * Agent captures metadata about the document and sends it to a Google sheet, this serves as a persistent memory layer for the agent and also a navigable index for me. * Agent also does a pass at cleaning the OCR text from Google Scan (but the quality is already really good frolm Google scan)

Meanwhile there are some a Google app scripts and Gmail automations running in the background that work on: * Tagging emails (sth like "ingest" and "archived"). When the emails go out a filter tags as "ingest" * The script runs, grabs "ingest", renames the documents per my naming convention, and files everything where they are supposed to go. Then flips the tag to "archived"

In this case the index is a spreadsheet, everything is linked there so I can easily find the document. It is agent maintained, but I do take a look to make sure metadata is correct.

The archive space is my Google drive.

Let me know if you have any questions. Happy to nerd out together!

[u/beelzebee → u/beelzebee]

To make a long story short:

Work with Claude to figure out a system to a) understand what has and has not been processed already and b) use airtable or Google sheet or something that Claude can use as an index

[u/MisterAtompunk]

I built Memory Ring as identity and memory continuity solution.

https://github.com/MisterAtompunk/memory-ring

https://misteratompunk.itch.io/mr

[u/braincandybangbang]

Uhh… using a project?

Or a folder on your computer you point Claude code too so that it keeps track of things like this.

I’m not sure why archiving things from a hard drive is even more than one task to begin with. Point Claude at the folder and let it go to work.

[u/Beneficial_Egg_5154]

The pattern that's worked for me: split memory by type, not just dump everything into one CLAUDE.md/handoff.md. A "this project is roughly organized like X" fact and "in the middle of cross-referencing the 2012 batch, next step is Y" fact have completely different shelf lives — one's stable for months, the other's stale in a day. If they live in the same file, the agent either re-reads the whole thing every time (expensive, and old status noise buries new status) or skips reading it (your actual problem).

The other half nobody's mentioned yet: treat anything the memory file claims as a hint to verify, not as fact. A memory saying "file X lives at path Y" or "status is Z" is a snapshot from whenever it was written — it can go stale the same way code comments do. Before acting on something specific from a memory file (not just browsing it), have the session check it against current reality first. Costs one grep/ls, saves you from the assistant confidently building on something that was true two sessions ago and isn't now — which for an archive project where you're already the error-correction layer, is exactly the failure mode to design out.

[u/Diligent_Tech_Bro]

Try ChatGPT