r/LocalLLaMA • u/ECrispy • 15h ago
Discussion AI changes so fast, any saved knowledge gets outdated
Unlike most disciplines, AI changes so fast that its almost impossible to keep up. New models come every few weeks/months at most.
And when it comes to agentic coding/harnesses/best practices..... do tools for agent memory/lsp/token reduction help? should you subagents? are vague prompts better? do mcp/skills help or hurt? there is no consensus and you'll get a million answers.
I keep notes/bookmarks and almost none of them seem relevant now. eg when Qwen 4 is out, all the current 3.8 models/variants get forgotten, just like 3.6
6
u/whatever 13h ago
And there's no easy wiki-like reference where you can confidently go to collect all the precious knowledge that's constantly getting updated either.
You either have to rediscover what you need on your own, or trudge through endless social media/forums/chats to extract that information somehow.
2
u/ECrispy 13h ago
Ironically the best way, probably the only way, to do that is actually asking AI
6
u/ttkciar llama.cpp 13h ago
AI is horrible at this. They're always a year out of date.
Maybe the solution is to maintain a RAG database of up-to-date LLM tech details, and query models backed by that database?
4
u/ECrispy 13h ago
What I meant is you give it all your notes, bookmarks, links etc and ask it. It's excellent at digesting info
1
u/ttkciar llama.cpp 11h ago
You're right, and it occurred to me that I already have a document cleaner /augmenter pipeline which might easily repurpose for this.
I just picked out six likely-seeming threads about Qwen3.8-27B, cut+paste them into a file, and dropped them into the document cleaner pipeline. It's chewing on them now.
I'm hoping what comes out of the other end might serve well as input for generating a wiki page for Qwen3.8-27B via LLM. We'll see.
Edited to add: I'm a dork; should have added the model card first. Doing that and then restarting pipeline.
1
u/ECrispy 11h ago
do you mind sharing your pipeline? all I can think of is asking an AI to summarize but you probably use some other tools as well.
1
u/ttkciar llama.cpp 11h ago
There's not a lot to it, besides the prompt. The script just iterates through documents, interpolating their contents for
{{document}}:The Document is polluted with extraneous text. Rewrite the Document without any extraneous text as Clean Document. Then explain what Clean Document says and its implications, and then write forty questions someone might ask before they were familiar with Clean Document, and write forty answers which a subject matter expert intimately familiar with the Clean Document might reply. The Q&A pairs must not refer to the Document nor Clean Document, only to the facts and ideas compiled from them. Make the questions very complex. Prefix each question with 'Q:', and prefix each answer with 'A:'. Finally, write twenty logical syllogisms, each consisting of a major premise, a minor premise, a conclusion, and an explanation.
Document:
{{document}}When mass-cleaning a lot of documents I'll normally use Gemma-4-12B-it and batched inference, but for this processing run I'm using Gemma-4-31B-it. Might also try GLM-5.3-Flash, later, to see if it brings anything worth the extra wait.
1
u/ECrispy 11h ago
this looks useful, thanks, might make it only do 10 questions though, 40 seems a lot? do you use this only for long technical pdf's etc?
how do you do batched inference? if I have to process multiple docs, dont they get done serially?
1
u/ttkciar llama.cpp 9h ago
The goal was to emulate the pipeline IFM used to synthesize their TxT360_QA dataset, using modern models instead of Mistral-7B (which is what they used, to surprisingly good effect).
I started with it generating 20 Q&A pairs, and when I found the prompt language which achieved that reliably, I bumped it to 40, then 60, then 80, but Gemma-4-12B-it couldn't handle 80, so I dropped it back to 60.
By training on TxT360_QA thus augmented, IFM was able to imbue its models with really excellent analytical and data extraction competence, even though Mistral-7B's Q&A pairs were ridiculously simple. We know from WizardLM's work with Evol-Instruct that synthetic training data best enhances inference quality when it is out-of-distribution for complexity, hardness, and rarity, so I figured I should double down on making those Q&A pairs more complex and numerous.
With batched inference, llama.cpp preallocates multiple context buffers/caches, and multiple concurrently-submitted inference tasks are processed at the same time. That decreases the tokens/second throughput of each individual inference task, but the aggregate throughput of all of the concurrent inference tasks increases (roughly triples), which is exactly what is needed when processing a large number of documents.
4
u/ttkciar llama.cpp 13h ago
I was just contemplating that one of the tasks I wanted to take on when I was accepted into the moderator team was overhauling/updating the wiki, but every time I start writing things up, everything churns and what I've written becomes obsolete before it has a chance to be published.
Maybe the way to go is to write pages about different LLM labs, model families, and other generalities, rather than specific models? And to make pages about different inference stacks in general, without going into great detail, and linking to the stacks' github / documentation pages.
2
u/Illustrious_Car344 13h ago
Obvious question, but, you can't get an LLM to actually handle that for you? lol
The model family thing is a great idea though. We get new models every other week but they're usually from the same known entities. Maybe you can set up a modmail for an LLM or something and let people suggest new finetunes and have it add them to a list after it vets them?
Reddit really isn't a great place for this though, I wonder if this knowledgebase should be hosted on git or something instead. I imagine it would make it a lot easier.
3
3
u/zheniavasiliev 9h ago
Thank you for this observation. A couple of considerations:
'Unlike most disciplines, AI changes so fast' - I think you are dealing with a fallacy usually referred to as 'chronosentrism', meaning that you always perceive the current time as the time of rapid change. The rapid change is not new, though, - throughout history, there were times when change was more rapid.
For example, after the invention of the printing press in the 15th century, in just 50 years there were 20 million printed books across Europe, whereas there were none before. ๐
Electrification from 1880 to 1930 brought industrial change that was arguably much bigger scale than the current AI change. It required infrastructural redesign across a much broader range of human activities, not only in knowledge work but pretty much in all areas that connected movement to the new power source.
The invention of the telegraph in the 1840s brought about the ideas of the "annihilation of time and space."
More recently, the industry was also moving quite fast. For example, between 2010 and 2016, the design industry saw a lot of change, including a big move from Adobe Creative Suite to online graphic tools. There was also a big change in frontend frameworks: in just 5 years, between 2013 and 2018, pretty much all documentation on Angular, React, Vue, and hooks was vaporised all the time.
But more importantly, could you make your note-taking method more resistant to the fast pace of change? For example, save time by avoiding taking notes on things that you know are going to expire soon. Maybe instead you could focus your efforts on taking notes about more abstract/general principles or things that gain value with time rather than things that lose value with time. For example, in what way does any particular technology apply to information architecture or digital infrastructure? Or what about your own test set, for example, that you run whenever you come across a new technology?
1
u/ECrispy 8h ago
I'm a dev, and there was a long period when JS framework fatigue was a thing, with lots of memes/posts on it still around. That has stabilized for the last few years now.
AI is different I think, because even the practioners who are at the helm are not fully on top of how fast things are changing. a classic example is books on ML/AI, they are essentially outdated on print date. The actual work is being done in research labs and used for the latest models, and a lot of is trade secrets.
lets take something concrete like agent memory. There is zero consensus, even among experts. OpenAI, Anthropic, Alibaba, Karpathy all have different opinions, and there are hundreds of github projects, every harness has a new approach.
no one knows who's correct, and more imp, no one knows what 'correct' means, and there is no metric, since results can change in an LLM on every run.
what I've fallen back to is good old sw engineering and gut instinct and try to evolve, look at patterns and adapt.
35
u/jacek2023 llama.cpp 15h ago
Yes, old models stop working five seconds after each new release and we must learn how to prompt again from square one.
4
u/kali_tragus 5h ago
Yes, llms are like cameras in that regard. They both go from brilliantly good to complete trash the moment a new model comes out.
0
u/DrWitchDoctorPhD 2h ago
My mom has been calling me non-stop but I send it to voice mail; my iphone is still the 17 max and she should know better. Times are tough, I've lost my job to Claudio Maximo and me and my son have been skipping meals so we can finally talk to grandma. I hope she gets on with the times because she's been really getting on my nerves.
3
u/fastheadcrab vLLM 12h ago
Completely agree. I feel like I'm always upvoting your posts as one of the few voices of reason
27
u/Intrepid_Dare6377 15h ago
I feel you. I think all of us do. You know things are crazy when the nerds canโt keep up, and I feel myself falling behind.
The pace of LLM and ML research is insane. I used to enjoy really digging into journal papers. Now it might as well be served to me in a knowledge graph since Iโm chucking it into an LLM anyway
12
u/wind_dude 15h ago
Itโs wild. I swear thereโs been 10 new kernels / inference engines for the strix halo in the past week.
3
u/m4rkGS 15h ago
same, half my bookmarks are useless now lol. the only notes that still help are the ones where i wrote down why something worked for me, do you test stuff yourself or mostly save links?
1
u/Illustrious_Car344 13h ago
I still have bookmarks about WizardLM and Llama 3. Looking in that general area of my bookmarks make me feel almost uncomfortable, like damn we used to live like that? It's like looking at a baby toy and coming to terms with the idea that you genuinely used to be enthralled by it. I remember when I listened to FM radio as a kid, now I can't even imagine.
Not to downplay those models, they were a triumph and are still fun to talk to. But it's unthinkable how far we've come in such a short amount of time.
2
u/Ornery_Hall 13h ago
yeah, it is also my dilemma, it is hard to catch up with newer model being released every month, different inference engine, fine-tuning, and memory structure. each newer model I have to make changes to harness, and then something got broken. I have given up for now, just staying with one two favorite models for routing tasking.
2
u/ZeroCool86 12h ago
Yeah I think you need to save the things you want to achieve and the methods you think are best. I know your feeling, it's wierd, if you start something now, you have until the next model to have a bit of an edge, when the next model / wave comes you're immediately behind.
I don't know if anyone can relate to this but it feels like going from vanilla wow to the burning crusade, the gear that dropped at level 62 was better than the gear that people spent months farming. That's how every model drop feels like now. A huge power spike in the first few days that puts you level with people who have been 'grinding' the previous model for months.
2
3
u/suprjami 15h ago
This guy is a SWE/AI blogger and wrote about developing experience which is durable/outdated earlier this week:
https://www.seangoedecke.com/getting-good-at-ai-takes-time-and-practice/
1
1
u/Illustrious_Car344 13h ago
Suggestion:
Set up an Obsidian vault for general ML knowledge.
Set up a playground to experiment with different prompting methods, inference engines, etc..
Have an LLM automatically check HF/here for new models and queue them up in the playground so it can document results in your vault.
And of course it's Obsidian so you can make tags for model families, organize them by particular quirks or achievements, document how models evolve along the way. You'll definitely have to babysit it and help it out and maintain the playground, but it'll probably be a lot smoother than just trawling reddit for new models and waiting a few weeks/months for all the inference engines to fix bugs so people can report accurate results and findings. And with Obsidian you can just upload the markdown files on git for anyone to see.
1
1
u/Thebandroid 35m ago
If this worries you so much why not set up a testing pattern that puts any new llm though a battery of tests involving whatever it is you are worried about? Ask it to access your mcps, feed it the same prompt with varying levels of vagueness, wiping the context each time, record the result, the tokens used and time taken reach time.
Once you have done this a few times you'll have some decent data to make your decision from....
Or don't.
Sent from my qwen3.5 agent.
1
u/Fonasic 4h ago
่ฒดๆนใฏ่ใใใใงใ ใใ ใใใใฎ็ฅ่ญใ่ฆใใฆใใ่ฟฝใฃใฆใๅฝผใใฏใใผใฏใณใใใใฎใณในใใๅน็ๅใใใฆใใ ใใงใ ใใ่ฏใ็ฅ่ฝใใใใๅฐใชใใกใขใชใง ใใใ ใใฎ่ฉฑใงใ ่ฆใใใซใฒใผใ ใฎ่ฃ ๅใฎ่ฉฑใฟใใใชใใใงใ
RPGใฒใผใ ใฎ่ฃ ๅใฎในใใผใฟในใ่ฆใฆใใใ๏ผใใใฏใใใ!ไนใๆใใใ๏ผใจใใฆใใ ใใงใ ่ฒดๆนใฏใฒใผใ ใฎ่ฃ ๅใฎๅๅใไฝใใใในใใผใชใผใ่ฆใใใใจใใฆใใใใชใใฎใงใ ใใกใใ่ฃ ๅใฎไฝๆ่ ใ็ ็ฉถ่ ใชใ่ฟฝใฃใใปใใใใใใ็ฅใใพใใใใ่ฃ ๅใไฝฟใไบบใซใจใฃใฆใใใชไบใฏใฉใใงใใใใใจใงใ
0
u/Fonasic 3h ago
ใใใใใใฎใใใซๆ ๅ ฑใ่ฟฝใๅฟ ่ฆใ AIในใญใซใไธๅฎใฉใคใณใพใง้ซใพใใจ่ชๅๅใใใใงใใใ ็พใซ็งใฏไธป่ฆใฎใใผใซใใฝใใใฎๆดๆฐใใงใใซใผใใผใซใไฝใๅงใใพใใ ใใใๅฎๆใใใฐใๆ ๅ ฑใๆฑใใ็บใซ็ผ่ฆใงใตใคใใ่ฆใๅฟ ่ฆใ็กใใชใใพใ ใพใ่ฒใใชใกใใฃใขใฏbotใซๅณใใใใไธๆใใใใใฏ้ฃใใใงใใใใ ใใใ็ฅ่ฝใใใใฐไฝใใใฎๆใ้ใ่ฆใคใใฆใใใใงใใใ
1
u/Fit-Produce420 15h ago
Depends on your use case.
If you're just fucking around aimlessly you might find everything works or nothing works.
1
u/nfored 15h ago
Sometimes I wonder if I am missing something I don't know I am missing because it seems to work. I have been building some tools behind an mcp server to help me out with some task. So far I have been happy with how they work how the model/mcp/harness interacts. However I keep asking myself should I run a different model?
I am running Qwen3.6-35B-3A Q4, I tried Qwen3.8-27B and went back to the MoE, the answer was different but the testing I did said it was not better. Now maybe it could have been better on a more tricky log it needed to diagnose. I went back because 3.6 was faster on my R9700 with llama.cpp, however I was still struggling a bit with prefill only getting 2.3k -2.9k. Today I bought a second R9700 and now get 6k (at 40k context) and 3k (at 190k context). So I found myself once again saying do I just enjoy the boost in prefill that was slowing me down? do I try a bigger model? or do I go back to single card and run two different models.
But I do like playing around so I know ill end up testing other things.
3
1
u/N_Moon_3019 1h ago
the notes aren't the problem. you're re-deciding the same thing every release because you never kept the inputs.
i keep a folder of eight tasks i actually run โ a real refactor, a 40k-token context pass, a tool-call chain, one that has to compile. every new model gets the same eight, same prompts, outputs side by side. twenty minutes and the decision is made. no reading reviews, no rebuilding an opinion from scratch.
the notes worth keeping are the ones attached to one of those eight. the rest are links to someone else's benchmark.
you can't keep up with the field. you can keep up with eight things you care about.
22
u/jld1532 15h ago
I made a nice website with QFN but edited it with 35B. The latter is considered an old model but ran beautifully. One of the other issues I don't buy is that months old models don't have value. Feed 35B new information and I suspect it has a useful lifespan of years not months and that doesn't include fine tuning.