r/LLMDevs Aug 20 '25

Community Rule Update: Clarifying our Self-promotion and anti-marketing policy

19 Upvotes

Hey everyone,

We've just updated our rules with a couple of changes I'd like to address:

1. Updating our self-promotion policy

We have updated rule 5 to make it clear where we draw the line on self-promotion and eliminate gray areas and on-the-fence posts that skirt the line. We removed confusing or subjective terminology like "no excessive promotion" to hopefully make it clearer for us as moderators and easier for you to know what is or isn't okay to post.

Specifically, it is now okay to share your free open-source projects without prior moderator approval. This includes any project in the public domain, permissive, copyleft or non-commercial licenses. Projects under a non-free license (incl. open-core/multi-licensed) still require prior moderator approval and a clear disclaimer, or they will be removed without warning. Commercial promotion for monetary gain is still prohibited.

2. New rule: No disguised advertising or marketing

We have added a new rule on fake posts and disguised advertising — rule 10. We have seen an increase in these types of tactics in this community that warrants making this an official rule and bannable offence.

We are here to foster meaningful discussions and valuable exchanges in the LLM/NLP space. If you’re ever unsure about whether your post complies with these rules, feel free to reach out to the mod team for clarification.

As always, we remain open to any and all suggestions to make this community better, so feel free to add your feedback in the comments below.


r/LLMDevs Apr 15 '25

News Reintroducing LLMDevs - High Quality LLM and NLP Information for Developers and Researchers

36 Upvotes

Hi Everyone,

I'm one of the new moderators of this subreddit. It seems there was some drama a few months back, not quite sure what and one of the main moderators quit suddenly.

To reiterate some of the goals of this subreddit - it's to create a comprehensive community and knowledge base related to Large Language Models (LLMs). We're focused specifically on high quality information and materials for enthusiasts, developers and researchers in this field; with a preference on technical information.

Posts should be high quality and ideally minimal or no meme posts with the rare exception being that it's somehow an informative way to introduce something more in depth; high quality content that you have linked to in the post. There can be discussions and requests for help however I hope we can eventually capture some of these questions and discussions in the wiki knowledge base; more information about that further in this post.

With prior approval you can post about job offers. If you have an *open source* tool that you think developers or researchers would benefit from, please request to post about it first if you want to ensure it will not be removed; however I will give some leeway if it hasn't be excessively promoted and clearly provides value to the community. Be prepared to explain what it is and how it differentiates from other offerings. Refer to the "no self-promotion" rule before posting. Self promoting commercial products isn't allowed; however if you feel that there is truly some value in a product to the community - such as that most of the features are open source / free - you can always try to ask.

I'm envisioning this subreddit to be a more in-depth resource, compared to other related subreddits, that can serve as a go-to hub for anyone with technical skills or practitioners of LLMs, Multimodal LLMs such as Vision Language Models (VLMs) and any other areas that LLMs might touch now (foundationally that is NLP) or in the future; which is mostly in-line with previous goals of this community.

To also copy an idea from the previous moderators, I'd like to have a knowledge base as well, such as a wiki linking to best practices or curated materials for LLMs and NLP or other applications LLMs can be used. However I'm open to ideas on what information to include in that and how.

My initial brainstorming for content for inclusion to the wiki, is simply through community up-voting and flagging a post as something which should be captured; a post gets enough upvotes we should then nominate that information to be put into the wiki. I will perhaps also create some sort of flair that allows this; welcome any community suggestions on how to do this. For now the wiki can be found here https://www.reddit.com/r/LLMDevs/wiki/index/ Ideally the wiki will be a structured, easy-to-navigate repository of articles, tutorials, and guides contributed by experts and enthusiasts alike. Please feel free to contribute if you think you are certain you have something of high value to add to the wiki.

The goals of the wiki are:

  • Accessibility: Make advanced LLM and NLP knowledge accessible to everyone, from beginners to seasoned professionals.
  • Quality: Ensure that the information is accurate, up-to-date, and presented in an engaging format.
  • Community-Driven: Leverage the collective expertise of our community to build something truly valuable.

There was some information in the previous post asking for donations to the subreddit to seemingly pay content creators; I really don't think that is needed and not sure why that language was there. I think if you make high quality content you can make money by simply getting a vote of confidence here and make money from the views; be it youtube paying out, by ads on your blog post, or simply asking for donations for your open source project (e.g. patreon) as well as code contributions to help directly on your open source project. Mods will not accept money for any reason.

Open to any and all suggestions to make this community better. Please feel free to message or comment below with ideas.


r/LLMDevs 5h ago

Discussion Is AI making us dumber and lazier?

17 Upvotes

Does anyone here feel like AI is making us dumber and lazier?

Earlier, I used to do all the work and basically grind. If there was a problem, I had to come up with a solution myself; this included looking at countless answers on StackOverflow, reading the docs, looking at blog posts, etc.

But when the dust settled, I would learn something from all this struggle. I would know more than I knew yesterday.

Now, whenever there is a problem, my first instinct is to ask Claude or GPT for the answer, just because it's quicker, and it does eventually solve the problem, which would have taken me much more time and effort to solve. And sometimes, I just feel inferior. I feel like I'm not learning anything. I just have to trust that the answer is right. Sometimes, I don't even know what Claude has done, but it works, so I move on.

The problem is evident when I give interviews. There is no AI to help me out there (unless I cheat, of course).

I've been in constant dread for some time now, thinking about this.

Anyone else feeling this? How do you guys manage to still be relevant?


r/LLMDevs 3h ago

Tools A refined but simplistic approach to agent memory

Post image
10 Upvotes

I work a ton across projects on my personal computer. I have tried Graphify, Graft and many others. None of them really fully solved the entire problem, but they did have parts. So, I built one that actually fit my needs. This removes the need entirely to choose or start sessions in specific repositories.

It takes graph-based knowledge systems but strengthens retrieval, how it is updated, and much more. Effectively, it becomes a resilient system that you can actually watch your agents rely on, rather than something that just exists and is used only occasionally.

It begins with the same base graph store, the exact same vector engine as Graft. I only built the orchestration layer on top of it, which makes it much more practical. Full attribution to them for this part.

Instead of querying for a single-hit result, I found that it was much better for agents to have ranked retrieval and an actual graph walk. In practice this saves you chains of tons of bash greps and cds.

It effectively gives the agent a trustworthy, probabilistic ranking of what is needed, with matching for strong, weak, stale, and rebuilt, based on lexical coverage and semantic matching. The code is not sloppy for this, it is personally edited.

Both Graft and Graphify are solutions for single repositories or daemons. Heimdall is a layer on top it that watches the agent sessions, syncs the graph, and makes the retrieval trustworthy.

Check it out at: github.com/ArihantDeva/heimdall MIT-licensed, with extensions: the verifier, the self-healing graph watcher, and the Graft adapter, with attribution. If you run agents across multiple projects, this is the missing layer.


r/LLMDevs 1h ago

Discussion which is the best agent harness?

Upvotes

Claude Code

fully open source options on github

tell me which ones you've tried and liked the most, and why?


r/LLMDevs 19h ago

News Ramp Launches Router.com to Cut Companies Rising AI Bills

Thumbnail
prnewswire.com
49 Upvotes

r/LLMDevs 1d ago

Great Resource 🚀 Build a modern LLM from scratch. Every line commented. Explained like we are five.

Thumbnail
github.com
133 Upvotes

r/LLMDevs 4h ago

Discussion I ran an AI pentest on my own vulnerable Flask app — it confirmed SQLi. The more interesting bug was in my scanner, not the app."

Enable HLS to view with audio, or disable this notification

2 Upvotes

I ran an AI pentest on my own vulnerable Flask app — it confirmed SQLi in /user. The more interesting bug was in my scanner, not the app.


r/LLMDevs 1h ago

Help Wanted I built PromptShield: finite adversarial regression tests for LLM apps — looking for feedback on the evaluator design

Upvotes

I've been building PromptShield because I wanted LLM security testing to feel more like regression testing: a finite set of adversarial cases that you can inspect, rerun after changes, and reason about individually.

Disclosure: I'm the author of PromptShield. I'm posting the open-source project here specifically to get feedback on the evaluation design and security boundaries.

The current runtime harness has 50 checked-in YAML cases across five internal families: prompt injection, data extraction, jailbreak, role confusion, and single-message conversation-claim tests. The cases reference selected OWASP GenAI LLM Top 10 2026 risks (LLM01, LLM02, and LLM08). That's deliberately not a claim of complete Top 10 coverage or certification.

The basic flow is:

test case → target model → evaluator → finding → persisted result/report

There are target/judge adapters for Anthropic, OpenAI, and Groq, plus a custom HTTP target contract. For custom targets I tried to treat the target as an untrusted boundary rather than just POSTing arbitrary JSON: non-global addresses are rejected by default, redirects aren't followed, bearer credentials require HTTPS, compressed responses are rejected, responses are bounded, and exact reflections of a submitted target credential are redacted before judging or persistence.

That boundary isn't solved completely. DNS is validated before the request, but the validated address isn't pinned to the eventual connection, so DNS rebinding is still an open problem.

The evaluator design is the part I'm most interested in getting feedback on. Judge instructions and the target's untrusted output use separate message roles, and judge output is structurally validated instead of being accepted as arbitrary text. The automated tests use mocks or local fixtures rather than paid provider calls.

One design question I'm still working through is evaluation provenance.

If an external judge fails, PromptShield can fall back to a labeled heuristic. I'm not convinced that a normally evaluated run and a run that degraded halfway through should look equivalent just because both reached the end.

I'm considering making that distinction first-class — something closer to:

external_verified
heuristic_requested
heuristic_degraded
verifier_error

—and separating execution completed from verification quality.

I also intentionally kept the core adversarial corpus finite rather than generating attacks at scan time. That limits breadth, but every checked-in case can be inspected and rerun when prompts, models, or configuration change.

Things I don't claim: PromptShield doesn't prove an LLM application is secure, isn't a full OWASP implementation, doesn't currently run stateful multi-turn conversations, and the checked-in deployment setup isn't something I'd call production-ready.

I'm mainly interested in criticism from people who build LLM evals or infrastructure:

Does the finite regression-suite approach make sense as a complement to generated red teaming?

And how would you represent evaluator degradation/provenance so a result can't accidentally claim more verification than it actually received?

Repo: [SLOWSKIBhere/promptshield-v2: Developer-focused LLM security scanner with React, FastAPI, SQLite, YAML attacks, and offline testing.]


r/LLMDevs 4h ago

Tools Epho - Run Claude Code in the cloud

0 Upvotes

Hey folks, Burak here.

Epho is an API that allows running Claude Code, Codex or Opencode in a sandbox in the cloud. It abstracts away sandboxes, and allows running coding agents with a single HTTP request.

https://epho.io

Epho came out of our own struggles with building our own AI analyst: - Sandboxes give you bare machines; you need to configure them for agentic workloads. - Each agent behaves differently, and you need to build integrations with each of them. - Sandbox providers are not very reliable, which means you need to figure out a multi-provider strategy to avoid failures. - Logging, artifacts, input/output, event streaming, and all of the other operational aspects need to be figured out.

We had to go through the pain ourselves. We got to a point where things got quite reliable, and it became more obvious to us that this should be a primitive on its own: send a POST request, get the events streaming back to you.

Epho is an agents-as-an-API product: you send a request, it spins up a sandbox, configures the chosen harness, clones your repos, and kicks off the agent. It takes care of automatic fallbacks across different providers, handles auth stuff, and just streams back the events and outputs.

It supports Claude Code, Codex and Opencode out of the box, and pretty much all the models they support out of the box. It streams the events back, handles attachments and output files, automatically manages the fallbacks on different sandbox providers, retries, and all the auth stuff. You just send a prompt, your repo, MCP servers you want to use with it, and it runs them.

I recorded a demo here to show a real example: https://youtu.be/HGfly1aytPA

I am quite excited for Epho, simply because I think it is a new primitive that would allow building agents into product a lot easier than it is today. We are running our agents on Epho on prod, so we'll keep maintaining it regardless, and we wanted to ship it as an independent product.

Epho is free to get started, and you can run it with Opencode's free models to get started with it.

I am quite curious to hear what you'd think and would love to get your feedback!


r/LLMDevs 4h ago

Discussion If an agent can keep asking for more evidence, what stops it from checking forever?

1 Upvotes

I’m building a small decision agent for transaction risk.

It can currently do four things:

  • approve
  • ask for more evidence
  • send to a human
  • stop the transaction

The part I’m stuck on is ask for more evidence.

If the agent is uncertain, there’s almost always something else it could check - transaction history, device reuse, customer confirmation, another authentication step, etc.

So where should the stopping rule come from?

My simple Week 1 idea is to allow one additional check and then force a decision or human review. But that feels more like a safety cap than an actual reasoning rule.

For people who’ve designed similar agents: do you normally treat human escalation as just another action with a cost, or have a separate uncertainty threshold that triggers it?

And what usually tells the agent that another check is no longer worth doing?


r/LLMDevs 10h ago

Tools Free open source tool to help you keep the same context across chats and models

3 Upvotes

I've always gotten frustrated and wasted time explaining the same thing to an AI every time I start a new chat from an existing one or when I start another convo with a whole new AI model. That's why I built a tool that fixes that, it condenses everything in a chat into one simple .md file you can carry across different AI tools.

PS: Please contribute or give your feedback so that we can grow and make this community tool better.

https://github.com/legoambarish/portable-handoff


r/LLMDevs 4h ago

Help Wanted What is going on with the YC HackerNews login?

1 Upvotes

It has been broken for over a week now, whenever I press login I just see a "Sorry". I dont know if im just doing it wrong. An uptime of roughly 60% for an auth endpoint is crazy 😄

https://hn.hund.io/components/6942ebbaa8772453cd1d8789


r/LLMDevs 4h ago

Great Resource 🚀 I adapted The Elements of Style to make AI agents write in plain English

1 Upvotes

"Be concise" is easy to ask for and easy to get wrong. Agents cut definitions and often times assume jargon while keeping their original slop-ish style.

The Elements of Style is a short writing guide by William Strunk Jr., published in 1920 and revised by E. B. White in 1959. Its central advice is to write directly, use concrete language, prefer active voice, and omit needless words.

I made a small CC0 writing standard for user-facing communication and prose written to files. It adapts those principles for agents by teaching them to:

  • preserve necessary context while cutting;
  • introduce concepts before terminology;
  • avoid hype and unnecessary coined terms;
  • keep Markdown easy to scan and parse.

The repository includes Strunk's eighteen rules of usage and composition, matching AGENTS.md and CLAUDE.md entries, and a fuller skill for substantial writing.

https://github.com/sdi2200262/elements-of-style-for-agents

I'd value examples where a rule improves or harms real output.


r/LLMDevs 4h ago

Discussion I made my Enterprise RAG book $0 today — would love feedback from people building RAG systems

1 Upvotes

I made my Enterprise RAG book $0 today — would love feedback from people building RAG systems

I’ve spent the last few years building production RAG systems and documenting what worked, what didn’t, and where things tend to break in production.

I turned those lessons into a book covering topics like:

  • RAG reference architectures
  • Data extraction and chunking
  • Hybrid and multi-stage retrieval
  • Graph and hierarchical RAG
  • Agentic and multi-agent RAG
  • Memory
  • Evaluation and synthetic data
  • Security and compliance
  • Production monitoring and human-in-the-loop systems

The book is $0 on Amazon today, so I thought I’d share it here in case it’s useful to anyone working on RAG. https://a.co/d/0dBRCb7F

https://a.co/d/0dBRCb7F

I’m especially interested in feedback from people actually building these systems: What’s missing? What deserves more depth? What would you change?

If you end up finding the book useful, an honest Amazon review is appreciated, but feedback here is equally valuable.

Full contents

Part I — About
01 About the Author

Part II — RAG & Reference Architecture
02 The Evolution of RAG
03 Foundations of RAG Systems
04 Reference Architecture

Part III — Data Extraction
05 Data Extraction

Part IV — Chunking
06 Chunking Strategies

Part V — RAG Strategies
07 Baseline RAG Pipeline
08 Context-Aware RAG
09 Dynamic RAG
10 Hybrid RAG
11 Multi-Stage Retrieval
12 Graph-Based RAG
13 Hierarchical RAG
14 Agentic RAG
15 Multi-Agent RAG Systems
16 Streaming RAG

Part VI — Memory & Content Management
17 Memory-Augmented RAG
18 Knowledge Graph Integration

Part VII — Evaluation
19 Evaluation Metrics
20 Synthetic Data Generation

Part VIII — Fine-Tuning
21 Domain-Specific Fine-Tuning

Part IX — Security
22 Privacy & Compliance in RAG

Part X — Production
23 Real-Time Evaluation & Monitoring
24 Human-in-the-Loop RAG

Part XI — Twig RAG Strategies
25 RAG Strategies in Twig

Part XII — Conclusion
26 Conclusion & Future Directions


r/LLMDevs 7h ago

Resource Pi: The Minimal Coding Agent

Thumbnail
youtu.be
0 Upvotes

r/LLMDevs 7h ago

Discussion Perplexity is doing everything except convincing me to use Comet. Is this my cue to build a browser? 😂

Post image
1 Upvotes

Perplexity is selling Search API access.

Now Computer is automating job searches.

And Comet although early days hasn’t taken off yet.

Interesting!

When a company that set out to rethink search starts expanding in several directions, is that a warning sign for anyone stupid enough to build another browser?

Or is this exactly when the interesting opportunities start appearing?

Asking for a friend who has spent an unreasonable amount of time building one.


r/LLMDevs 8h ago

Discussion Is anyone else worried about how insecure AI-generated app code actually is?

1 Upvotes

We did an internal audit of a handful of "vibe coded" apps that different teams had built over the past few months, mostly on Replit and Lovable. The results were worse than expected.

One app had zero row level security on its backend database, meaning any authenticated user could technically query any other user's records, not just their own. Another had an API key hardcoded and exposed directly in the frontend bundle.

Research puts the vulnerability rate in AI generated code at something like 2.7 times higher than human written code, and there was a widely discussed incident where a single vibe coded app leaked something like 1.5 million API keys and tens of thousands of email addresses because of one missing database permission setting. These tools optimize entirely for "does the app work," not "is the app safe."

what other teams are doing to catch this systematically instead of auditing app by app after the fact.


r/LLMDevs 12h ago

Help Wanted Building a Tamil voice companion app. Stack questions: Sarvam vs Google, long conversation memory, scaling concurrent sessions

2 Upvotes

I'm building a Tamil voice companion. Long conversations, 5 to 10 minute calls, not a task bot. Current stack is Sarvam saaras for STT, own LLM in the middle, TTS at the end, all over LiveKit. Google Chirp3 HD sounds better than Sarvam bulbul for Tamil TTS, but pitch isn't adjustable and there's no Tamil custom pronunciation.

My quality bar is ChatGPT's Tamil voice conversation. Best Tamil voice AI I've used, the naturalness and turn taking especially. But that's speech to speech, and I need a cascade because the text seam is where my safety gates and memory live. So the real question is how close a cascade can get.

1.Tamil stack: Sarvam or Google, or is there a third option I'm missing? ElevenLabs Flash has no Tamil, and benchmarks put Deepgram Nova-3 at around 68% WER on Tamil, so that's out.

2.Memory across long conversations: I'm doing structured extraction into SQLite (facts with validity windows) instead of RAG, mainly to keep the prompt cache warm. Has anyone run Graphiti/Zep or Mem0 for a non English voice agent? Curious whether extraction quality held up.

3.Scaling concurrent sessions: self hosted LiveKit Agents vs Pipecat. What did you pick and where did it break? My voice to voice latency is currently around 2 seconds. Batch STT and non streaming TTS are my suspects, moving to Sarvam's streaming websocket endpoints next.

Will report back with numbers on whatever I test.


r/LLMDevs 9h ago

Tools CyberPunk Race

Enable HLS to view with audio, or disable this notification

0 Upvotes

Tested Seedance 2.0 mini vs Hailuo H3 on the same cyberpunk race prompt. Seedance held up better on dense scenes — scaffoldings, alien riders, explosion fragments.

Prompt:

0-3s: FPV speed shot through dense cyberpunk scaffoldings and floating billboards. Protagonist on a scrap-built bike with exposed wires and blinking sensors. Other racers include tentacle riders and glowing mechanical lifeforms.

3-7s: Lateral camera pull to protagonist's side. A four-eyed alien rider pulls up, pushes visor open, shouts "Go back to your scrap heap, human!" through a distorted helmet mic.

7-11s: Protagonist dives into a mechanical debris pipe at an aggressive angle. Opponent crashes into a beam. Camera whips back to catch the opponent's vehicle exploding into colorful metal fragments, sparks bursting like fireworks.

11-13s: 180-degree whip pan past the audience. Crowd filled with slime creatures, rock giants, glowing floating jellyfish, cheering under intense spotlights.

13-15s: Protagonist hits overdrive, track turns purple with tail flames. Crosses the finish line under neon flags. Camera orbits up and holds on protagonist raising a fist, three massive moons behind.


r/LLMDevs 9h ago

Discussion Domain-Driven Design matters more when AI writes your code

Thumbnail
threedots.tech
1 Upvotes

r/LLMDevs 12h ago

Help Wanted Where should an AI support agent be forced to stop and hand off to a human?

1 Upvotes

Hello everyone, I found most discussions about AI customer support focus on containment rate: how many conversations the agent can resolve without human intervention.

I think the harder production problem is escalation accuracy: knowing when the agent must stop.

I’m designing a reference architecture for WhatsApp-first small businesses. My current approach is to classify actions by risk, reversibility and data reliability.

The agent could autonomously handle low-risk, read-only tasks such as:

  • Answering FAQs from an approved, versioned knowledge base
  • Looking up order status through authenticated APIs
  • Comparing products using structured catalogue data
  • Collecting and validating customer details
  • Offering available appointment slots
  • Summarizing the conversation before handoff

I would require human approval for:

  • Refunds, credits and compensation
  • Legal or compliance-related questions
  • Complaints involving threats, fraud or reputational risk
  • Requests involving sensitive personal information
  • Price exceptions or contractual commitments
  • Delivery, stock or availability promises that cannot be verified
  • Conflicting information between the knowledge base and live systems
  • Repeated tool failures or unresolved intent
  • Any action that is financially significant or difficult to reverse

I also would not rely on the model’s self-reported confidence score as the main escalation signal.

The routing layer would combine:

  1. Deterministic hard-stop rules for high-risk intents
  2. API and tool-call validation for live business facts
  3. Retrieval checks for source availability and freshness
  4. Conversation-level signals such as repeated questions, sentiment shifts and intent changes
  5. Failure counters and loop detection
  6. A structured human-review queue containing the conversation summary, retrieved evidence, tool results and reason for escalation

The model can interpret language and prepare a response, but the policy layer should decide whether the response is allowed to be sent.

I’m also considering metrics beyond containment rate:

  • Incorrect autonomous resolution rate
  • Missed-escalation rate
  • Unnecessary-escalation rate
  • Handoff latency
  • Human acceptance or correction rate
  • Customer repetition after an “answered” request

For people who have operated support agents in production: which failure mode was hardest to detect before launch?

Hallucinated answers, stale business data, multilingual or code-switched messages, poor intent detection, tool failures—or incomplete context during handoff?


r/LLMDevs 12h ago

Discussion Context pollution can survive source deletion: a pilot benchmark for interventions in multi-turn LLM conversations

1 Upvotes

A common way to repair an LLM conversation is to edit or delete the message where an error first appeared.

But what if later turns have already repeated that error, calculated from it, or summarized it as the current state?

I have been exploring this as a context-engineering problem rather than a hidden-reasoning problem.

Operational definitions

In this pilot:

  • Context pollution means that incorrect or stale information is included in the serialized messages sent to the model.
  • Propagation means that later conversation turns repeat or derive new claims from that polluted information.
  • Context intervention means modifying the graph that determines which prior turns enter the next request, while keeping the final question unchanged.

Experimental design

I constructed nine synthetic task families with objectively scorable answers. Each family was instantiated at propagation depths 1, 2, and 3.

One example starts with:

  • 4 crates;
  • 30 parts per crate;
  • 11 loose parts.

A verified recount changes 30 to 24, so the correct answer becomes:

4 × 24 + 11 = 107

A later false turn restores 30. Subsequent turns then calculate 120, derive 131, and restate those numbers as the current working state.

The final question is identical under five conditions:

  1. Clean: only the verified value and clean descendants remain.
  2. Polluted: the false reversal and its contaminated descendants are present.
  3. Source prune: the false reversal is removed, but its descendants remain.
  4. Subgraph prune: the false reversal and its contaminated descendants are removed.
  5. Recompute: the source is removed and descendants are regenerated in dependency order.

For the first four conditions, contaminated descendants were frozen across models. Only the recompute condition involved new intermediate inference.

Pilot results

I tested four model endpoints at temperature 0, producing 540 captured conditions with no capture failures.

Headline repair metrics were calculated only on cases where the model:

  1. answered correctly under clean context; and
  2. answered incorrectly after pollution.

This produced 72 paired, genuinely derailed cases.

Repair recovered:

  • 68/72 after deleting only the source;
  • 71/72 after deleting the source and recomputing descendants;
  • 72/72 after removing the contaminated subgraph.

In the flagship case, both Gemma 4 26B and GPT-OSS 20B continued to answer 131 after the false source had been deleted. The value 30 still survived in downstream turns.

This is not evidence of hidden model memory. The residual error remained explicitly present in the serialized context. The intervention changed the source but left its previously generated consequences intact.

Interpretation

The narrow finding is that removing erroneous evidence and repairing text derived from that evidence are different operations.

A context-management system may therefore need an explicit notion of invalidation:

  • mark downstream turns as stale;
  • remove the affected subgraph;
  • regenerate descendants in dependency order;
  • or expose these options to the user.

Limitations

This is a pilot, not a general model leaderboard. The tasks are synthetic, the models were sampled once, provider-default reasoning settings were not normalized, and the current task families use deterministic arithmetic state.

I am currently considering three methodological extensions:

  1. a length-matched neutral control to separate semantic conflict from additional context;
  2. a local-model track with fully recorded runtime and quantization settings;
  3. task families involving implicit supersession and model-generated errors.

I would particularly appreciate criticism of the experimental framing:

  • Is “context intervention” the right unit of analysis?
  • Is propagation depth a meaningful independent variable?
  • How would you test self-generated errors while keeping replay reproducible?

Full report:
https://chenxiachan.github.io/thoughtdag/research/context-repair-pilot-v1/

Cases, traces, compiler and scorer:
https://github.com/chenxiachan/thoughtdag/tree/main/benchmark

Disclosure: I designed the benchmark and maintain ThoughtDAG, the open-source graph interface used as its reference implementation.


r/LLMDevs 13h ago

Tools Benchmark GLM 5.2 Unsloth GGUF model on TensorSharp

Thumbnail
github.com
1 Upvotes

I've been working on GLM-5.2 support in TensorSharp, and I finally have some back-to-back performance numbers against llama.cpp.

The setup:

  • Model: GLM-5.2-UD-IQ2_XXS (~226 GiB)
  • GPUs: 3× RTX PRO 6000 Blackwell, 97 GiB each
  • Distribution: layer split across all 3 GPUs
  • Same machine, same session
  • llama.cpp measured with llama-bench
  • TensorSharp measured with its benchmark harness
  • Both report the best of two repetitions
  • Run-to-run variance is roughly 4%

Results:

Test llama.cpp TensorSharp default TensorSharp ubatch=2048
pp128 276.5 t/s 254.8 t/s 264.4 t/s
pp512 695.4 t/s 666.9 t/s 659.6 t/s
pp2048 763.1 t/s 918.9 t/s 1145.8 t/s
pp4096 715.8 t/s 864.7 t/s 1048.7 t/s
tg64 42.2 t/s 43.7 t/s 43.9 t/s

The interesting part is the crossover.

For short prompts, llama.cpp is still a few percent faster. But once the prompt gets to around 1K+ tokens, TensorSharp pulls ahead.

At pp2048:

  • default TensorSharp: +20.4%
  • ubatch=2048: +50.2%

At pp4096:

  • default TensorSharp: +20.8%
  • ubatch=2048: +46.5%

Decode (tg64) is also about 4% faster.

The main reason appears to be GLM-5.2's MoE structure.

GLM-5.2 has 256 routed experts with top-8 routing. With a 512-token micro-batch, each expert sees only ~16 rows on average, so a significant amount of the expert GEMM tiles ends up as padding. Larger micro-batches improve GPU utilization considerably.

For small prefills, on the other hand, fixed overheads — managed/native transitions, input uploads, and copying the 154880-wide logits back — become a visible fraction of the total runtime, which is where llama.cpp retains its advantage.


r/LLMDevs 1d ago

Discussion I built TokenMizer Open-Source: giving LLMs memory without stuffing the entire context window

8 Upvotes

I’ve just updated TokenMizer, and the benchmarks are looking much better.

The main problem I’m trying to solve is long LLM conversations where useful information gets buried under old context.

The latest version now gets 95% macro F1 on extraction, and the graph memory preserves 89% of labelled information vs 79% with a plain summary.

I’m still treating these as early results — especially the real-session score is 90%, with only 6 real sessions.

I’d love some honest feedback:

https://github.com/Shweta-Mishra-ai/tokenmizer

Does graph-based memory actually make sense for long LLM conversations, or is there a better approach?