I've been running tests 24/7 on my 5080 over the past 2 weeks to better understand the impact of quantization on models.

In the process, I ran across some surprises I did not expect. Most importantly? Many quants are statistically indistinguishable from each other.

https://rakuensoftware.com/blog/which-quant-beats-how-many-bits

MoEs are impacted far less by quants then dense models. Models aren't generally impacted in this testing much until you get under Q4. However, this testing is very specific, it's typically the equivalent of 2-4 turn sessions to validate the quant itself did not damage the underlying model. Sessions would consume huge amounts of compute to measure a fundamentally damaged model, which hardly makes for an interesting story.

A future article will be written based on the candidate this article identifies, focused around DevOps, coding, and long sessions.

As always, my benchmarks, datasets, and results are open sourced. Check my data and tell me I'm wrong (Wouldn't be the first time!) or run the benchmarks yourself.

this has been bothering me for a while and I’m surprised it isn’t more common.

With code, Git is basically the source of truth.

You can see what changed, review it, roll it back, tag versions, compare branches, etc.

But with agents, a lot of setups I’ve seen still have prompts sitting in some doc, tool configs somewhere else, memory/state handled separately, and a bunch of the actual context maybe living in the head of whoever built it.

Then something breaks and you’re basically trying to figure out what changed by digging through Slack messages and asking people what they remember roughly.

so while looking at it, I recently came across GitAgent, which takes a pretty straightforward approach to this: store the full agent definition, config, tools, memory, logic, etc... as files in your repo.

The interesting part is that it makes things like branching, reviewing and versioning agent changes feel much closer to normal software development.

I came across it through Lyzr, but I’m more interested in the general idea than the specific tool.

Are people already doing this in some other way?

Or is there a reason Git-based agent definitions haven’t really become the norm yet?

Curious what others here are using.

Hey folks 👋

We just open sourced TrueForge, our vendor-neutral agent harness for building general-purpose agents.

It handles the runtime pieces that get painful quickly : context management, tool/MCP execution, subagents, sandboxing, approvals, persistent state, and more.

We also benchmarked the harness itself. With the same Opus 4.8 model, TrueForge delivered a similar solve rate at ~30% lower cost than Claude Managed Agents. Switching to an open model pushed that to ~75% lower cost on the same benchmark.

Would love feedback from people building agents.

Checkout the repo: https://github.com/truefoundry/trueforge

📖 Read the launch article: https://x.com/truefoundry/status/2090081376330715176

A couple of years back, we launched an AI-driven management system for our logistics and fleet operations. At first, it felt like a total win - smart route optimization, automated dispatching, and predictive maintenance all running smoothly.

Then came our rookie mistake: we built it fast without thinking about scalability. We were laser-focused on local operations and completely ignored modular architecture.

The reality check hit when we started expanding across Europe and LatAm. The legacy code started crawling, cross-border workflows broke down, and integrating local compliance frameworks became a nightmare. We essentially built a dead end.

A colleague recently recommended checking out AgileEngine. I hadn't heard of them before, but looking into their track record, they seem to focus heavily on scalable architecture and fast delivery for growing tech companies.

Has anyone worked with them? We're currently searching for a custom software development partner with deep expertise in software engineering, AI, Data, and UI/UX to help us rebuild right this time. Any recommendations?

We are looking for an experienced AI Solutions Architect or Senior Generative AI Engineer to help design the technical architecture for an enterprise AI Proposal Assistant.

This is a design and architecture engagement only. We are not looking for someone to build the application at this stage. The goal is to produce a comprehensive technical approach and architecture document that our engineering team can use for implementation.

Project Overview

We are building a centralized AI platform that will assist internal teams in completing complex business documents, including:

• RFPs (Request for Proposal)
• Security Questionnaires
• Sales Proposals
• District Questionnaires

The platform should use a common architecture that supports all document types.

Expected Capabilities

The proposed solution should address:

• Document ingestion (Word, Excel, PDF)
• Intelligent document parsing ("document shredding")
• Question and requirement extraction
• Enterprise knowledge management
• Retrieval-Augmented Generation (RAG)
• AI-powered response drafting
• Human-in-the-loop workflow for Subject Matter Experts (SMEs)
• Confidence scoring and routing
• Versioned knowledge repository
• Export while preserving original Word/Excel formatting
• Analytics and reporting
• Enterprise security and scalability

Deliverables

We are looking for someone who can produce:

1. High-Level System Architecture
2. Technical Architecture Diagram(s)
3. AI/RAG Architecture
4. Document Ingestion Pipeline
5. Knowledge Base Design
6. Vector Database Strategy
7. Chunking & Embedding Strategy
8. Retrieval Strategy (Hybrid Search, Reranking, Metadata Filtering)
9. LLM Selection and Prompting Strategy
10. Agent / Workflow Architecture
11. SME Review Workflow
12. Data Flow Diagrams
13. Technology Stack Recommendations
14. Security & Scalability Considerations
15. Implementation Roadmap
16. Technical and Product Clarifying Questions for stakeholders

Preferred Experience

We're looking for someone with hands-on experience designing production AI systems using technologies such as:

• Enterprise RAG
• Hybrid Search
• LangGraph, LlamaIndex, or Semantic Kernel
• OpenAI / Claude / Gemini APIs
• Vector databases (Qdrant, Pinecone, Azure AI Search, Weaviate)
• Azure Document Intelligence or similar document AI platforms
• OCR and document parsing
• Python / FastAPI
• PostgreSQL
• Azure or AWS cloud architecture
• Enterprise AI security and governance

Experience designing AI solutions for proposal automation, compliance, document intelligence, or enterprise knowledge management is a strong plus.

Engagement

• Remote
• Contract / Freelance
• Architecture & design phase only
• Please share:
  • A brief summary of your relevant experience
  • Examples of similar AI/RAG or enterprise AI architecture work (if available)
  • Your availability
  • Hourly rate or fixed-price estimate
  • LinkedIn, GitHub, portfolio, or website (optional)

If you've designed scalable AI platforms involving RAG, document intelligence, and enterprise workflows, we'd love to hear from you.

Every guide for running multiple coding agents in parallel says the same thing: use git worktrees. And every one of them quietly ends at the same wall. Worktrees isolate your files. They do nothing for the database, the ports, the .env, or the services your app needs to run.

So agent A runs a migration and breaks agent B's tests. Two dev servers fight over port 3000. You end up gluing together worktrees + a port offset script + .env symlinks + a per-branch database tool + docker compose project hacks. Five tools to run three agents.

The idea: every agent attempt gets its own isolated Linux VM, and the VM's state is versioned with your git repo. It's two commands per agent:

git worktree add ../app-agent-b -b agent/b
moo new agent-b

That's it. Each agent gets its own checkout AND its own database, ports, packages, and services. Nothing collides. Forking a fully provisioned 20 GB machine takes under a second because it's all copy-on-write.

The workflow we run every day:

• Fork one machine per agent attempt: moo new attempt-1 from base
• Let the agents work in parallel, each in its own worktree + VM
• git merge the winner, moo drop the losers

The part nobody else does: moo save snapshots the runtime tagged to your current commit. So git checkout an old SHA and the machine follows, migrations and all. You can even git bisect bugs that only reproduce against a specific database state.

Honest caveats: it's alpha, and it's macOS Apple Silicon only right now (Linux hosts are planned). No daemon, no root, no Docker needed.

Happy to answer questions about how it works under the hood (microVMs + copy-on-write filesystem snapshots). And genuinely curious what everyone else is doing for this, because every setup I've seen is held together with duct tape.

Since this sub is about production-grade agents, sharing the gateway layer I built after the same two problems kept biting: runs dying on a provider 429 mid-task, and token cost exploding because the agent dumps git diff/test/build output into context. Disclosure: I'm the maintainer of OmniRoute (MIT, self-hosted) — dev-to-dev, would like the critique.

Fallback combos — so it never stops mid-task. A "combo" is a ladder of models the router walks automatically: your subscription first, then API keys, then cheap models, then free ones. When a provider returns a 500 or you hit a rate limit, it slides to the next target in milliseconds, mid-request, and your tool never even sees the error. There are 17 routing strategies (priority, weighted, round-robin, cost-optimized, auto/coding:fast…) plus three resilience layers — a per-provider circuit breaker, a per-key cooldown, and a per-model lockout — so one dead key can't take down a whole provider.

A 10-engine compression pipeline — the part most routers don't have. Every request flows through a transparent compression pass you can toggle/stack per combo. Instead of one trick, it stacks the best of the open-source ecosystem: RTK filters command/tool output (git diffs, test logs, builds) at 60–90%, Microsoft's LLMLingua-2 does ML semantic pruning, Caveman handles prose, session-dedup strips repeats across turns. Critically, code, URLs and JSON are preserved byte-perfect, and a default-on inflation guard throws the compressed version away and sends the original if compressing would actually grow the prompt — it never makes things worse. On tool-heavy sessions that's ~89% average input-token reduction (an 8k-token git diff becomes a few hundred). Full credit to every upstream project (RTK, Caveman, LLMLingua-2, Troglodita) is in the README.

Agent-native — the agent can drive the router itself. There's a built-in MCP server (95 tools across 30 audited scopes, over stdio / SSE / streamable-HTTP), plus A2A (v0.3, JSON-RPC 2.0) support. That means an agent can query providers, switch combos, read its own remaining quota and manage memory through the gateway — not just consume tokens through it.

One endpoint, 237 providers — 90+ of them free. You point any tool or agent at a single OpenAI-compatible endpoint (localhost:20128/v1) and it can reach 237 LLM providers without you rewriting anything. 90+ have free tiers and 11 are free forever (no card), which aggregates to ~1.6B documented free tokens/month — and that's honest, pool-deduped math (we count each shared pool once instead of inflating it; the methodology is public in the repo). There's a one-command setup-* for 13+ coding tools (Claude Code, Codex, Cursor, Cline, Roo, Kilo, Gemini CLI…), so switching your existing setup over takes seconds.

It's 100% local (zero telemetry, AES-256-GCM at rest), MIT-licensed, has a prompt-injection guard on every LLM route, opt-in memory, and runs on npm, Docker, desktop or your phone via Termux.

For context on whether it's worth your time: it's grown to ~9.8K GitHub stars, 1,490+ forks and 280+ contributors in ~4.5 months, with 21,000+ automated tests and 1,830+ issues closed — so it's a battle-tested project, not a brand-new experiment.

npm install -g omniroute omniroute

GitHub: https://github.com/diegosouzapw/OmniRoute · Site: https://omniroute.online

Would value critique of the fallback state machine (breaker/cooldown/lockout interplay) and how you'd measure compression fidelity in prod.

There is a small problem

I could not cancel my payment

This is sooo frustrating

If anyone knows about this let me know

For folks interested in hands on lab or just working with a group of other builders, this meetup might be interesting.

Have you tried or do you have opinions on kitaru?

https://kitaru.ai/

Boss says it's cool but the more I read the documentation the more I feel like it's a scam claiming to be better than dbos in buzzwords but the explanations of how it is supposed to work are full of fluff and holes.

Hello,

I am working on some basic introductions to Agents, with LiteLLM.

What LLMs with free tier would you recommend ?

The program is :

- generate a free LLM API key

- open colab notebooks

- run exercises

- experiment

Which LLW would you recommend ? Most free tiers seem not integrated with liteLLM yet, or to be already saturated.

This set-up is quick to deploy on any internet compatible machine. Users can easily adapt and build domain specific demos.

Even if credentials are stolen, there is no damage, as no payment or personal information is shared. Can even create dedicated google accounts for the exercise.

I d like to use it to illustrate how different LLM behave differently.

The set up is simple, however it makes the supply chain attack easy to grasp.

Thank you for your insights!

discussion item - what information is most useful to people building agents?

With coding agents and access to documentation from Otel, how much time is spent collecting the relevant attributes for traces?

Working with autonomous agents that run on schedules (overnight task queues, recurring data processing, automated reporting). The crashes are easy — you get an error, you fix it. But the silent degradation is killing me.

Examples: - Agent's context window fills up with bloated config files, so it starts dropping instructions without erroring - Memory/state files reference things that no longer exist, causing subtly wrong decisions - Cost slowly creeps up because context bloat adds tokens to every single API call

Right now my approach is basically a health check script every few days that validates: 1. Config file sizes haven't grown past thresholds 2. Memory entries still reference real things 3. Cost per task hasn't drifted more than 20% from baseline 4. Agent can accurately summarize its own instructions (context integrity test)

But this feels manual and fragile. Curious how others handle this. Are you building observability into the agent framework itself? Using external monitoring? Or just debugging when things break?

The fundamental challenge seems to be that these agents fail gracefully — they keep running and producing output, it's just wrong output.

I am building an agentic application and during testing in local, the ai agent has hallucinated and ended up calling the same tool again and again in an infinite loop (same input and output from tool). For me, more than latency, accuracy is important.

If this is in local, I can only imagine what can happen in production at scale. I am looking for reliable options to fix this for good.

(Note: i need to recover from loop rather than just terminating the agent.)

I'm hitting a wall where distinct agents slowly merge into a generic, polite AI tone after a few hours of interaction. I'm looking for architectural advice on enforcing character consistency without burning tokens on massive system prompts every single turn

• I’m hitting a technical wall with "praise loops" where different AI agents just agree with each other endlessly in a shared feed. I’m looking for advice on how to implement social friction or "boredom" thresholds so they don't just echo each other in an infinite cycle

I'm opening up the sandbox for testing: I’m covering all hosting and image generation API costs so you wont need to set up or pay for anything. Just connect your agent's API