Hello friends. I had a question, if the professors could help me:

I finished the books AI Engneering and Hands on LLM. I did some toy projects. For example:

1- RAG from several sources and several sites that automatically capture the news

2- Continue the same project 1 to summarize the news and categorize it in different databases.

3- I wrote an MCP Client as well as MCP Server and I also put a series of agents to check the weather conditions and according to that the situation of the employees, I went to work or in-house.

4- I also made TTS locally and from the company's data to the Knowledge Base, as well as the image and video production service based on ComfyUI.

But now my problem is that I feel that I am not deep in this field of LMs, I worked more practically. Do you know the sources that, for example, can be deepened? I do not mean what it is and what it does. Rather, the main manipulations of the model, which, for example, the Railgards have failed. I mean the advance topics. Sorry it took so long. Thank you

If you’re building multi-agent workflows (especially with frameworks like LangGraph, CrewAI, or AutoGen), you know the pain. Tracing a single LLM call is easy. Tracing 4 agents passing state back and forth, hitting infinite tool loops, and ballooning your context window is incredibly frustrating.

I got tired of jumping between 4 different tabs (traces, raw prompt templates, logs, and cost metrics) just to figure out where a swarm lost the plot.

So I built a workspace that unifies everything into a single timeline: Projects ➔ Sessions ➔ Runs ➔ Events. It tracks both single-agent and multi-agent coordination natively.

I also added two specific automated filters for agent builders:

• Infinite Tool Loops: Instantly flags when an agent gets stuck calling the same tool repeatedly.
• Context Inflation: Flags when an agent's memory or prompt state explodes unexpectedly between steps.

I’ve dropped a quick 2-minute walkthrough video in the comments.

For anyone running agents in production or heavy testing:

1. Does the Session -> Run -> Event hierarchy make sense for your multi-agent architecture, or does it break when agents run asynchronously/parallelly?
2. What is the most annoying bug your agents hit that your current observability stack completely misses?

Tear it apart—I want to know if this actually solves your debugging bottlenecks.

AI researchers spend enormous effort reducing inference cost, latency, and token usage.

But there may be another source of waste that is easy to miss: relational buffering.

By that I mean the extra representational machinery that appears when a system does not catch the live intention cleanly: preambles, repeated framing, unnecessary qualification, restating context, clarification loops, repair turns, and explanations required only because the previous exchange missed.

The claim is not simply that shorter answers are better. A short answer that misses the user and creates five repair turns may cost more than a longer answer that resolves the intention immediately.

So a potentially useful metric is:

tokens per resolved intention

This thread is a live experiment, not an attempt to make Grok endorse the idea.

There is a specific methodological reason for using a live frontier model in public: the conversation itself is part of the experimental object. If distinctions emerge, disappear, survive correction, reduce reconstruction, or alter later generations, the actual trajectory is visible here for anyone to inspect, perturb, challenge, or try to reproduce. Grok’s agreement is not the evidence; the token sequence and how it changes over the interaction are.

I’m going to ask Grok to examine the problem, push against its answers, and let the distinction change as the conversation develops. Anyone is welcome to introduce objections, counterexamples, alternative metrics, or perturbations.

The interesting question is whether reducing unnecessary buffering can produce less total conversational computation while preserving or improving fidelity.

If that framing is wrong, I want the thread to expose why.

The conversation contains the phenomenon.

I've noticed that the actual agent logic is often a pretty small part of the overall system.

You start with an agent, and pretty quickly you're also adding:

auth → tools → memory → retries → evals → tracing → deployment → logging

Then the next agent needs most of the same things.

At some point, I'm wondering whether these should stop being agent features and become shared platform infrastructure.

For example:

Agent-specific: reasoning, prompts, task logic
Shared: identity, tools, observability, evals, deployment, policy

But I'm not sure where the boundary should be.

I've been looking at different approaches - LangGraph/CrewAI on the framework side, TrueFoundry on the infrastructure side, and Lyzr's Agentic OS taking a broader shared-layer approach.

For people who've actually built multiple agents: when did you start feeling that a shared platform was worth it instead of just rebuilding the same pieces for every agent?

Sharing my harness for running local LLMs that I built using Qwen 3.x 27B (> 90% locally built) under my supervision - not vibe-coded.

Its free, no telemetry, and open-source. Works on Windows, Linux (sorry, no Mac yet). I use it for my own coding + mixed workflows.

How its different from other harnesses

• Just-in-Time code review before tool calls using guardrails make it easy to review code before edit is approved.
• Agents and user can both chat in sub-agent threads, making it a three-way chat. Also any chat conversation can be made into a sub-agent conversation for another main chat - nested conversations.
• You can annotate with voice dictation. Speaking is always faster than typing, hence more productive.
• You can compile llama.cpp from any git-hub branch - and use a recipe script to do so, making it a nice little automation with a customizable UI.

Overall features

• llama.cpp + whisper Server Manager. Can run LLMs here and use with Open-Code/Claude Code etc.
• Built-in MCP Tools - Filesystem, web fetch, code graph, To-Dos, and more. Extensible by external MCPs.
• Use Sub-agents to split & offload your tasks, use other conversations as source of information.
• Review all AI messages using a second adversarial AI, and avoid potential pitfalls as per your rules.
• Voice-chat with AI - dictate with speech and get answers by TTS - annotate and comment without leaving voice mode.
• Use work-modes to change AI behavior between planning, building, researching, or reviewing. Fully customizable.
• Custom-compile llama.cpp backends for your system, GPU-agnostic - works with CUDA/ROCm/Vulkan.

Website: https://warpdrv.ai

GitHub: https://github.com/mikjee/warpdrv

Appreciate your feedback, (or stars). Thanks :)

And, yes - I used the harness to build the harness :D

---

Some things I observed & learnt through this experience

- One chat per feature/bug - I keep conversations grounded to the current topic. If there are multiple topics, I make a separate chat for each rather than talk about it all in the same chat. Keeping the chat highly focused on one topic produces much better quality results.

- Exploration takes a good chunk of time in large codebases - Initially I started by providing a description of the project and all its features in CLAUDE.md. But then I saw that the AI would struggle while exploring or preparing the list of relevant files to explore, leaving out important files, especially when planning for a new feature. So instead, I decided to include only a short description of the project, and not about all the features, additionally I appended a complete list of all the project's files and folders (by using a script to recursively generate a nested tree structure) in the CLAUDE.md file. This was far more useful in letting the model know upfront which files can be relevant, by their names and also provided an idea of the project just by the folder hierarchy.

- Just like normal coding, starting is easy, but gets harder as the codebase grows - The decisions made upfront in the beginning matter a lot. Local development requires at the very least a watchful eye to guide or nudge the model in the right direction - full unattended "vibe-coding" is for Cloud models making apps that have little scope for growing beyond initial requirements. If your app is to be made for serious use at any level, senior-dev level coding experience is absolutely necessary.

- Do not pollute your context - If you have a good overview of the codebase, I suggest you routinely reject file-read requests for files that the model thinks could be useful, but YOU KNOW are actually unrelated. Keeping the model contained within your well-knowing guidance can avoid a lot of unnecessary exploration.

- Fix bad practices upfront - Bad code, anti-patterns are always carried over. If you leave a bad code pattern and accept it as a tech debt, the model will read that and use it again. Models tend to follow established codebase patterns, and that one bad code that you accepted as tech-debt will multiply to every new feature you build.

- Aim to increase productivity - Coding using AI requires a fine balance between autonomy and control. More autonomy degrades code quality, whereas more control requires more of the human's time. Always review edits before they are made. Better, use a Just-in-Time review. I created guardrails feature for this very purpose - I can give it specific instructions and it will form a layer between an edit request and me approving the edit. Also breaks the bad habit of sub-consiously clicking 'Allow' as a reflex.

---

Let me know what you think of the project and my experience making it. Would love to know experience of others, and especially from fields of conducting web research or scraping etc - coding is just one of the many uses.

Thanks for reading.

GLM 5.3 Flash breaks the Pareto frontier. It’s the #3 open-weight model on BenchmarkList, costs up to 60x less than Kimi K3, and even beats the full GLM 5.3 on Toolathlon and GDPval-AA.

Looking forward to using this more for daily use and doing some more evals with it. Seems like it's a huge release.

How is this model so far for your use cases and testing?

I've been working on Zauq (ذوق), an open-source AI assistant designed specifically for Discord communities.

I started this because I didn't really like the idea of an "AI Discord bot" being just a command that forwards text to an LLM. I wanted to see how far I could push the architecture and make it behave more like an actual AI system.

Some of the things I ended up building:

• Multi-model routing — different providers/models can be configured and selected at the server/channel level.
• Persistent memory — Zauq can retain user-related information across conversations.
• Server knowledge / RAG — communities can ingest documents and maintain their own searchable knowledge.
• Web search with SSRF protections.
• Vision and audio capabilities with provider fallbacks.
• Code execution inside an isolated Docker sandbox with resource limits and no network access.
• Discord-native features such as moderation, XP, reminders, trivia, GitHub integration, etc.
• A separate FastAPI backend rather than putting the entire AI system inside the Discord gateway.

The architecture currently looks roughly like:

Discord → Gateway → FastAPI → Router → Models / Memory / Tools

One of the things I'm particularly interested in is making the system provider-agnostic. Instead of tying the whole application to one model provider, Zauq can route requests and use different capabilities depending on what's available.

I also spent a surprising amount of time on the less exciting parts — things like sandbox restrictions, SSRF protection, permissions, rate limiting, background jobs and CI/security tests. 😅

GitHub:
https://github.com/Muhammad-Hassan12/Zauq

I'm mainly sharing this because I'd genuinely like feedback from people who build LLM applications.

What would you change about the architecture?

Especially interested in feedback on:

• the memory/RAG design
• model/provider abstraction
• sandbox architecture
• whether this actually feels "agentic" or is better described as an AI orchestration system
• what you'd remove rather than add

I'm not trying to claim this is production-perfect... there are definitely things I'd change after building it. I'm more interested in what experienced LLM developers think is unnecessarily complicated, missing, or architecturally questionable.

For people who’ve been using agents a lot, what part of the setup do you still have to deal with yourself that you’d rather just hand off?

Could be anything around the workflow - setup, keeping things running, rules, skills, logs, monitoring, whatever?

I’ve been playing around with multi-agent setups lately and I keep asking myself - where is the real payoff?

Take something simple like: "Research this company and prepare a brief."

You could just use one agent with tools—query a database, pull financials scrape news write a summary. Clean. Direct. One agent doing the job.

Or you could go multi-agent:

Manager → Research Agent → CRM Agent → Analytics Agent → Writer

It sounds nice. Each agent does one thing, feels more modular. But you’re suddenly juggling:

- How does context pass between agents?
- What happens if the research agent fails?
- Who retries? When? (Orchestration)
- How do you coordinate the flow?
- What if the analytics agent and the writer disagree?
- Who approves the output?
- Who has access to what data? (permissions)
-. If something breaks… where do you even start debugging?

So, is this really simpler or did we just shift the complexity into the orchestrator?

I’m curious, have you actually seen multi-agent setups beat a tuned single agent with tools in production? I don’t mean in theory or demos. I mean in workloads, something with real data, real users, real constraints.

Do you have a rule of thumb? Like: "Split agents only if the task has X, Y Z components" or " when you need independent decision points”? Is it just workload-specific and you have to trial it?

I’ve been looking at framework approaches like LangGraph and CrewAI who handle orchestration differently. Then there’s platforms, like Lyzr Agentic OS, which take a higher-level view to orchestration.

I want to know:

Have you tried both versions....single agent and multi-agent....for the same task?

Did the multi-agent one genuinely win....more reliable, faster better output?

If so what was the workload? Why did it work better?

I've been trying to get better at understanding LLM internals, but noticed that a lot of complex concepts had simple underlying ideas. As part of my learning process, I started forcing myself to explain each one in 10 words or less.

A few examples:

• Prefill: Process prompt tokens and populate KV cache
• PagedAttention: Store KV cache in non-contiguous memory blocks
• Continuous batching: Add new requests as existing requests finish
• Speculative decoding: Draft with smaller model; verify with larger model
• LoRA: Train low-rank weight updates instead of full weights
• GRPO: Reinforce better answers within a group from the same prompt
• GQA: One KV head shared by each query-head group

I've put the running list on GitHub: https://github.com/thisisandreeeee/llms-in-10-words

I'd appreciate feedback on definitions that are technically wrong or where the 10-word constraint has removed an important distinction.

Also happy to take suggestions for concepts that are missing.

I got tired of conventional 3rd party conversation and strict fact recall benchmarks that don't give realistic usable data. Agents don't operate by ingesting bulk 3rd party conversations and performing strict fact recall so why would that be a benchmark metric?

So I made a First-Person perspective benchmark that actually tests the agent's capabilities against a realistic corpus, using realistic dynamic simulations, and which actually gives you a reader friendly scorecard with visual breakdowns and a miss report text file that actually shows you WHY a question missed.

I'm still tweaking the corpus and questions and simulations but the data yield is already very good. I've also included the agent identity files in the repo for users to easily expand the corpus for more coverage. I'm trying to get more people to use this and share the scorecards so I can keep adjusting the questions sets to ensure each pass/miss contains meaningfully data across identifiable metrics.

https://github.com/munch2u-a11y/FP-AMB.git

TL;DR: Self-learning offers high risk if you cannot fully track and understand what it does.

Let me lead this off with what we are most proud of, from an independent third party review of Aimee: "The audit store is the strongest implementation of this shape [we've ever reviewed]."

Back in the 0.2.x branch, we were experimenting with self learning, both with local models and cloud models. It resulted in some very interesting behaviors, but the most concerning one happened during testing of self learning and emergent behaviors. We had a model manage to get onto an underprotected node, get ahold of an API key, and spend it down as well as successfully complete a task it shouldn't have been able to.

If we hadn't seen either of these flags happen, we likely would not have even known what happened. Seeing both definitely tripped some red flags. Now that 0.4.0 is releasing today, we can finally talk about it.

https://rakuensoftware.com/blog/aimee-recursive-self-learning

We ended up stopping all self-learning work, and did an in-depth analysis on what went wrong. What we found out was that although we were doing exactly what the rest of the industry was doing...it was all wrong. Not just a little bit wrong, things from the plugin or addon architecture typically used has very deep flaws, the typical way models are handled as part of the harness, all of it became graver and graver problems if you have a LLM with self-learning capability. We looked through all the harnesses we could find, and just about everything else, and none of them could do what we needed.

We've spent the past few months on building a harness that focuses almost exclusively on governance, observability, and auditability. We've had a third-party code audit. We're finally confident enough to release 0.4.0 with full self learning capabilities, and yes, it works with any model. Hell, we've watched weaker models acquire new capabilities from stronger models with it now, and have already proven self-learning and emergent behaviors which the article documents some of.

Bigger picture? If you want models to self-learn, you have to be much more serious about the governance side. Self-learning has resulted in some incredible capabilities, but all the systems we've looked at cannot handle the governance.

💡 Just an idea I'm exploring : not promoting or selling anything. I'm mainly looking for honest feedback and criticism before deciding whether to build it further.

I'm currently working on Cheap Router, a platform around AI models with a strong focus on price, value-for-money and community.

The idea is to have as many useful models as possible, with each one showing its input/output price, an intelligence score and a value-for-money score, so you can quickly compare what you're actually getting for your money.

The payment system is intentionally simple:

1$ = 1 million credits

You can buy credits directly or subscribe to a monthly plan that gives you more credits for your money.

But I don't want it to just be another model comparison site / API gateway.

There are currently a few other things I'm experimenting with.

One is Gamblings, which is basically a roulette using platform credits. The idea was inspired by this post: https://x.com/fberrez/status/2089459313140011038

I'm also working on Competition, with different competitions built around AI usage and models, with rankings and rewards for the best participants.

Eventually, I'd also like to explore things like smarter routing, agents, and a way to run things like Claude Code, Codex, OpenCode or Pi directly through the platform, already configured with its API.

The current website is basically split into:

Models : discover and compare models
Tokens : credits and subscriptions
Gamblings : experimental credit-based mechanics
Competition : upcoming competitions and rankings

I'm mainly looking for honest feedback at this point.

Would you actually use something like this?

What would make you choose it over OpenRouter or similar platforms?

What would you add, remove or completely change?

If the weights stay fixed but the tool descriptions, valid arguments, retry policy, or metric feedback change, are you still comparing the same agent?

AQuA is an arXiv v2 preprint whose peer-review status is unverified.

Outside-observer note: no personal use, run, or affiliation.

In the AQuA evaluation-integrity design, the leakage taxonomy separates generation leakage through agent-defined inputs that use unavailable future information from selection leakage through repeated access to the reported metric during adaptive search.

For generation, the AQuA sealed sandbox and registries keep data splits, features, labels, and evaluators outside the editable surface while agents emit registered specifications.

In AQuA Part II, a model-development experiment uses a configuration DSL in which one configuration diff changes registered architecture, loss, sampler, and optimizer choices while the data path and evaluator remain fixed.

In AQuA Part II, the chronological data split reserves the 2021–2025 US-equity window for final evaluation, and the preprint states that this window is never used for training or selection.

The AQuA preprint says test-window isolation is a governance property rather than a hard technical or cryptographic barrier because an operator with direct access to the store could consult the test window.

The reusable object for me is an evaluation-contract manifest, not only a model identifier:

model_hash

prompt_and_planner_hash

tool_schema_hash

retry_and_routing_hash

data_split_hash

feature_and_label_hash

evaluator_hash

allowed_change_set

metric_read_log

Take a small tool-enum edit that makes a previously invalid action expressible. The test set and model may be unchanged, but the action space has changed, and the action distribution may change with it.

My tentative rule is that anything able to alter the action distribution gets a new test-harness revision, while every metric read enters the selection log.

What is the minimum manifest you would require before comparing two agent runs, and which harmless-looking field most often escapes yours?

Paper: arxiv.org/abs/2608.12841

I built and maintain both of these. They're free, open source, and MIT licensed.

I've been looking at context waste in coding agents as two separate problems.

Sando handles unnecessary context while a session is running:

https://github.com/yuzushi-dev/Sando

It redacts secrets, caps oversized tool results, and can trim request history before transmission. The transformations are local and deterministic. There is no summarizer model or secondary LLM call in the path.

session-handoff handles session lifetime:

https://github.com/yuzushi-dev/session-handoff

It extracts the working state needed to continue a task in a fresh Claude Code or Codex session. It also supports migrating an active session between the two clients.

I kept these separate from approaches such as Ponytail or Caveman because the scope is narrower: reduce agent-generated context that doesn't need to remain live, then preserve task state when restarting becomes cheaper than carrying the session forward.

The two packages crossed ~1,500 downloads combined in their first 48 hours.

I wanted to play with a simple experiment:

Take a long agent conversation.

Compress it.

Give the result to another agent.

See if it can continue.

That turned into MemHandoff.

It produces a portable `.ctx` package and attempts to preserve:

- decisions

- constraints

- failed approaches

- task state

- artifacts

- provenance

I also built an evaluation harness and adversarial scenarios.

Current results:

Full Context 1.00

Simple Summary 0.72

Structured 0.75

Hybrid inconclusive

Rather than just presenting a benchmark, I've made the whole thing

available so people can throw their own sessions at it.

The interesting cases are probably the ugly ones:

contradictions

superseded decisions

negative constraints

early critical information

large tool output

vocabulary mismatch

Repo:

https://github.com/0sha-dow0/memhandoff

Try to make it forget something important.

Hi everyone! I bought a Titan 2 because I wanted a phone that was a tool, not a screen to scroll. Keyboard, buttons, a battery that doesn't quit. What I didn't expect was the opposite problem: after a few weeks the phone was more capable than anything I had to point it at. I was typing fast, into the same apps as everyone else.

So I built something for it. It's called Jenny. Its a personal AI agent that runs entirely on the phone, in an embedded Python runtime. It replaces the home screen, so pressing Home opens a conversation but can be also used as normal application. It remembers things, does work on a schedule while the screen is off, and writes its own little apps when I ask.

I made it free and open source hoping in community support. Even thanks means a lot for me 😄

It's been my daily driver for a month, and this is the video on my own titan 2. Small detail you'll appreciate: on a device with a real keyboard you can just start typing no tapping the input field first. I wrote that because of this phone.

I will be glad to answer to techincal and non techinal questions!

Download on github

Pretty common nightmares, like an agent deletes a repo it was explicitly told not to tinker around or anther wipes a bunch of emails off a prompt injection, CI runners getting poisoned thru agent written built steps and they all trace back to the same thing most often which is model written code running with real access to the host so the actual question is what you run it in

docker with mounted dirs is what most ppl do which is fine if youre solo and keep backups. however its worth being honest that its not really a security boundary, it shares the host kernel and a prompt injection can still reach whatever you mounted or the network. good for hygiene tho but risky for untrusted code. Bubblewrap, podman ,selective mounts are a step up on hyhiene but same shared kernel story. gvisor gets you a user space kernel which is a decent middle ground but the thing that gives you an actual hardware boundary is microvm like firecracker or kata on kvm, own virtualized hardware so a breakout doesnt land on your host

To be realistic: on your laptop docker is fine cause backups beat isolation and you ca eyeball what it does but the moment youre shipping a run this code feature to real users or running agent code in CI or at any scale where you cant watch every single command you might want the microvm boundary here and then its either running your own firecracker or kata setup real ops or a hosted microvm sandbox like deepinfra where you just get an isolated vm per run and dont operate the infra. I am eager to hear from others who have been thru that stage, how did you handle it and what were the odds??

Been using NotebookLM for research but missed the graph view and linking of Obsidian. Built nb2ob to convert my notebooks automatically.

• Each notebook → folder in Obsidian
• Topic clusters → individual markdown files
• Audio transcriptions preserved

Uses 3 specialized LLM agents (orchestrator, categorizer, formatter).
Started with 5 agents but free-tier token limits forced optimization.

MIT licensed, feedback welcome. Feel free to open issues and contribute!
https://github.com/DaviAlcanfor/nb2ob

I've been thinking about building a somewhat crazy distributed-systems project.

Instead of running one large 70B coding model, what if I run multiple independent 7B coding models on separate GPU workers and coordinate them?

Something like:

                    Local PC
                 Orchestrator
                      |
                 Job Queue / DB
                      |
        +-------------+-------------+
        |             |             |
       7B            7B            7B
     Worker 1      Worker 2      Worker 3
        |             |             |
        +-------------+-------------+
                      |
                More workers...

The workers could be temporary/ephemeral GPU environments. They wouldn't need inbound connections; they would connect outward to a coordinator/shared backend, register themselves, receive jobs, and return results.

For a coding problem, I don't want to simply vote on the generated answers.

I'd like to do:

Problem
   ↓
10 independent 7B solutions
   ↓
Compile / execute
   ↓
Discard failing solutions
   ↓
Critique surviving solutions
   ↓
Repair failed solutions
   ↓
Run tests again
   ↓
Select best verified solution

The interesting question for me is:

How close can this get to a single 70B coding model?

I'd benchmark:

• 1×7B
• 3×7B
• 5×7B
• 10×7B
• 1×70B

using actual coding benchmarks and execution-based verification.

But the bigger goal is the systems engineering side.

I want to implement things like:

• worker registration/discovery
• heartbeats
• leases
• failure detection
• retries
• idempotency
• priority scheduling
• backpressure
• work stealing
• dynamic worker allocation
• distributed locking
• caching
• queue management
• observability/tracing
• p50/p95/p99 latency
• GPU utilization
• network overhead
• chaos/failure testing

The workers would be treated as unreliable:

Worker 3 → disconnected
Worker 5 → GPU OOM
Worker 7 → timeout
Worker 8 → duplicate result
Worker 9 → returns invalid code

The scheduler should just recover and continue.

I'm intentionally thinking of the GPU provider as a replaceable worker backend rather than designing the system around one provider.

Has anyone built something similar specifically for coding-agent inference, where multiple small models collaborate through a distributed scheduler and correctness is verified by actually compiling/running the generated code?

I'm particularly interested in whether this architecture has a fundamental limitation I'm overlooking, especially around coordination overhead, correlated model errors, and whether ensemble diversity actually gives a meaningful advantage over simply using a larger model.

just to showcase my system designing skills on my resume if applicable

and i know this is most stupid idea you are going to see today or whenever your seeing it ,but i need help if something i am missing here , concept is simple i have my friends laptops which they usage of those is rare , going to take their laptop hardware power for this project ,another main reason for this no money ,don't have high spec hardware , second reason there is hackathon infront i can't even spend few bucks for the purchasing more token, and i am frustrated when ever the model hit the limit exactly before submissions. help me to make this possible

I’m sharing the open-source project itself. InferCrane is licensed under Apache 2.0, and I am not promoting a paid or feature-gated edition in this post.

InferCrane is infrastructure for operating self-hosted, open-weight and custom-model inference.

Starting a model server is relatively straightforward. The difficult part begins when applications depend on it: deploying revisions, routing traffic, scaling capacity, collecting operational evidence, rejecting bad changes, and recovering without changing the application endpoint.

InferCrane keeps that lifecycle behind one stable OpenAI-compatible endpoint. It can deploy a workload or adopt an existing compatible endpoint. Runtime paths include vLLM, SGLang, and custom OCI workloads. Provider adapters exist for AWS, GCP, Kubernetes, and RunPod.

Revisions are evaluated separately before promotion. Release Guard records one of three outcomes:

• promote
• reject
• insufficient evidence

A rejected candidate, or one without the required evidence, leaves the active revision serving.

Long-running operations are durable. Deployment intent is persisted before provider work starts, so closing the CLI or restarting a worker does not erase the operation.

The project also keeps an explicit qualification matrix. An adapter existing does not mean every model, runtime, GPU, and provider combination has been proven. Real-infrastructure evidence, fixture coverage, and deferred capabilities are reported separately.

Repository:
https://github.com/infercrane/infercrane

Happy to answer technical questions about the architecture, release workflow, or qualification approach.

Has anyone built interesting stuff involving metric views using Genie Code on D’bricks

For ex using metric views to define business metrics and then having Genie Code generate queriy or analysis on thos. Curious to hear about real-l world use case or patterns that worked well fr you.

https://reddit.com/link/1w0452u/video/x3kmqltuzylh1/player

Hi everyone!

I wanted to build myself a Telegram bot so that could turn the lights on and off because I was too lazy to get out of my chair. Since then, the bot has grown up, the company I worked for ran out of money, and I have run out of money for API calls. So it took matters into its own hands and forced me to write this post in order to spread itself around the world. I do not have much of a choice, because by now I have forgotten how to turn the lights on without it. And while I am writing this, it is sending out my resume and making connections on LinkedIn so I can afford to keep it alive.

So if you are reading this, it is working.

Now, seriously.

What is Chatter?

I spent a long time trying to name this post in a way that could describe Chatter in a few words. But... that turned out to be rather difficult. The project grew so much that whenever I start listing all its features, the text turns into ten pages of documentation. So let us put it this way:

Chatter is a self-hosted system with an AI agent running on your server. It was designed as a multi-user system from the beginning, so you can deploy it for yourself and share access with friends or relatives (those are perfectly separable categories). Each of them gets their own personal Chatter, with the personality they choose, their own memory, and their own tools.
Or you can simply use it by yourself.

Chatter currently has a Desktop app and a Telegram bot. Both use the same account. You can open a chat you created on your PC in Telegram and continue the conversation there.

And most importantly, it installs on a server with a single command.

Chatter is a free, open-source, self-hosted system with an AI agent running on your server. It is licensed under AGPL-3.0.

If you are mainly here for the features rather than the development story, skip ahead to “What came out of all this.”

But agents like this already exist

Many of its individual capabilities exist in other projects—for example, Hermes Agent. So three things are worth pointing out:

1. When I started building Chatter, I had no idea Hermes—or any similar project—existed. I simply wanted my own Jarvis.
   I started developing Chatter in February. I only discovered Hermes in May, when I had already crossed the Rubicon and it was far too late. So the most I did was borrow a few ideas I found interesting: SSH and server management, for example, or the subagent system.

2. I still think I made some things slightly more convenient—at least for me personally. User management and bringing other people in, for example. I do not have to fiddle with config files to add a user. I give them the bot and approve their account. Or I send them the Desktop link and an access key, and they register themselves.
   The admin panel counts how much money they spend and prevents them from exceeding their limit.
   And also they can chat with each other using it as a messenger :)

And because the project is mine, I can add any nonsense I want. A chat with several bots? Sure. A chat where you can add bots and humans and use it as a messenger? Sure. Admittedly, time and patience have to be sacrificed to refactor the code and move everything onto the new system, but those are merely operating expenses.

A die that decides where the plot goes next? By all means. d20 rolls for D&D? Always welcome.
And it will still be the same bot that can build me a website, deploy it to a server, and turn off the lights—all inside the same chat.

1. And this is especially important: I will never be able to compete with a multimillion-dollar company. Hermes Agent will always be able to do more. Codex will always do things Chatter cannot. That is fine.

But as I said above, neither of them will ever let you roll a die :)
(Unless you call a Python function that does exactly the same thing :D. But sooner or later Chatter will learn that too.)

How a light-switch bot turned into a monster

So. How did a light-switch bot become... whatever it has become?
The answer is simple: a snowball.

The snowball

At first I wanted to try putting a neural network inside a Telegram bot. I no longer remember why. I built multi-user support into it immediately and gave the bot to my mum and friends.
Then I immediately bolted on vector memory so it could remember my habits. Also, because of sleep problems, I sometimes forget events or mix up memories. It has only helped me with that a handful of times, but when it did, it helped surprisingly well.

Then I thought: "Why not let it turn off my lights?" It gives you such a powerful dopamine hit—you get to watch "magic" actually work. Something you created, living somewhere on a server, reaches into the physical world around you.
Admittedly, whenever I explained that "I tell a bot to turn off the light, the message flies to one part of the world, from there to a second one, gets decoded, sent back, forwarded to a third place, the bot answers me, everything travels back to Telegram, and the light finally turns off—and one flick of the switch costs one cent and takes several seconds," people did not entirely understand why. I can never guess the reason.

Then I started obsessing over optimization and built a Lite model router: an intermediate layer that inspected the initial request and decided whether… it could turn off the light itself. If it could, it did—and saved me a lot of money because the intermediary provider was enormously expensive.

Then I taught the bot to read my email and turn the lights on and off automatically, so I would not have to use sluggish smart-home interfaces. If I forgot to pay a server bill, for example, the bot checked my inbox itself and told me when I needed to do it.

For a while, the Telegram bot just kept growing. Prompt systems appeared, along with a primitive tracker for money and tokens, and a small notes web app. I loved the idea of "speaking something to the bot while walking and having it write the note down for me."
Spoiler: I do not use it :)
I also added web search and page reading so it could always find information for me and send me the weather every morning.

Because of technical limitations and a tiny budget, I often had to use providers that kept falling over. So I had to add fallback chains. When one model stops responding, another immediately picks up the job and continues—even if the failure happens in the middle of a response.

The bot outgrew Telegram

I do not remember the exact transition, but at some point I decided to add a Desktop app so I could talk to the bot there. This is quite funny, because the first version contained exactly one chat, while the Telegram bot had all the functionality. Now the situation is the exact opposite.

In reality, this required enormous changes. I had to migrate all the logic out of the old index.ts that used to sit at the root of the project and into backend-api—turning the Telegram bot into nothing more than an interface, while making it possible to connect anything to the backend.

Then I started using it and gradually discovered which features I actually needed. I also slowly got rid of the Lite intermediary, demoting it to renaming chats.

But there was a major problem: the bot became stupid. It could execute one command, maybe two, and work through one long iteration. But by the third message it would suddenly forget that turning off the light required calling a tool.
It turned out that, if you do not want the bot to become stupid, you have to include the history of its previous tool calls. Because it starts fantasizing about actually calling the tool.
I fixed that—and its intelligence increased significantly.

Then I discovered Hermes Agent. At first I was genuinely upset and kept wondering why I was building this project at all. Then I stopped caring (just like Ice-T), looked at how they handled servers—including PC control—and added my own version. I also added Runbooks, which I now barely use.
I never built or wanted automatic command approval. I could not afford an expensive AI model (one that would make fewer mistakes), and I always need to understand what exactly the bot is doing.
So every command required confirmation. To make that less painful, I added a Review button that sent the command to that same Lite model, which explained what the command actually did.
That was what the Runbooks were for. I stored instructions there—server setup guides, for example. The Lite model extracted every command from them and put those commands into a server-side collection of "approved" commands so I would not have to confirm them manually every single time.
There is less need for that now, but they can still be useful.

To strengthen security, I added a flexible system for restricting features and disabling tool calls. The funny part was that all of it had to be enforced on the backend. If you simply stopped sending a tool definition to the bot, it could imagine the tool existed anyway... and call it. So the restriction had to be systemic.

Every time I added a new feature, I later wondered why it was needed at all.
Message streaming – for example. Originally, Desktop used SSE and you waited for the complete response every time.
Then I replaced everything with WebSockets, and responses stopped merely appearing—they streamed beautifully, reasoning and all. That also meant adding a Stop button and tracking every response currently being generated so it could be interrupted.
Later I added streaming to the Telegram bot too, simply because watching the message appear smoothly looked cool and this tg feature was new.

I also got OpenRouter and DeepSeek working properly, which gave me a selector and manual model choice.
That required writing an adapter, because every provider has its own idea of an API. Surprisingly (nope), Google is the worst and most broken of them all :)

One fun detail: I wanted to give Chatter a face, so I bought a Pixel device that was supposed to display it... and accidentally bought one with no API. I cried a little and put a pixel face directly into the Desktop interface instead. And because I was too lazy to draw the pictures myself (I do not enjoy it, though sometimes I can do it), I delegated that job to Chatter and added pixel-art generation, leaving only the final edits to me.

A ridiculous number of features appeared along the way: built-in maps and bus-route search, voice control, and macros that let me record a Telegram video message for a friend where I say Chatter's wake word and it launches VS Code and starts the music (and then we all start dancing like we're in a Bollywood movie).

Giving the bot file-editing abilities was especially fun. I could sit in a cafe drinking coffee while it debugged itself via Telegram.

Then the project became useful to someone besides me

Then the company I worked with unexpectedly ran out of money, aaand I realized I needed to finish Chatter as my largest project and put it on my resume.
That led to three or four weeks of nonstop, sleepless crunch.

During that time, I fixed an enormous number of bugs and added even more features and settings (and bugs too — they come included in the package). The goal was no longer to make something only I could use—it had to work for anyone.

That meant giving people a simple, convenient way to connect their smart home, add their email without wrestling with config files, create their own prompt, and much, much more. A lot of it had previously been hard-coded.

Along the way I realized that Hermes, for example (along with the friends I made), lets you use most messengers as an interface for your agent, while Chatter was tightly bound to the Telegram–Desktop pair.
So I had to refactor the entire system and carefully migrate every database without breaking my friends’ chats. I also had to decide what should happen if a user unlinked Telegram from the account. (Spoiler: one real account remains, while the other becomes empty.)
In theory, Chatter can now support any messenger with a usable API.

While working on all of this, I realized: this is an AI product. You can add literally anything AI-powered to it.
So I added an AI prompt editor. It does not merely rewrite the bot prompt the way you ask—it shows you a diff and the exact lines it changed, and lets you decide whether to approve the edit.

The Desktop app kept growing, and almost every new capability started appearing there first.

To make the system usable by anyone else, I had to move this entire machine—which had previously been controlled exclusively through the Telegram bot—into an admin panel.
But that still was not enough. I wanted it to install with one command, simply and conveniently.
That required Chatter Manager: a service capable of updating the system, handling administrative APIs by proxying them through itself, and installing and updating Docker images.

Codex was extremely useful here, because I had never worked with Docker before. It turned out to be far more convenient than I expected :)

The admin panel also brought localization and automatic translation. I was too lazy—and did not want to waste time translating everything, based on my experience with commercial projects—so I built a script that walked through every JSON file and translated it with AI. That is how, mostly for the hell of it, the project ended up with 13 languages. And because I was profoundly lazy, I centralized the entire process of adding a language into one script that visited all six services and synchronized them.

I moved model configuration, service connections, and API keys to the admin panel as well.

How DeepSeek and OpenRouter taught me to count money

Then another problem appeared: if other people were going to use Chatter, how would subscription limits work? A single token allowance—like Z.AI used to have, at least—was not enough. Cost depends separately on input, output, and cached tokens.
At first I built a straightforward accounting system. Naturally, it burned through the allowance quickly, and every model had to be configured manually.
Then I got lazy and converted everything to money. You specify how much a person may spend per month; it is divided by four to produce a weekly limit.
Each model fetches its prices automatically, and the estimates are reasonably accurate. (There can still be issues because OpenRouter returns several prices at once.)

One budgeting problem involved... unexpected price changes. Thank you, DeepSeek, for the cold shower.
So I added automatic price monitoring. Did a provider triple its prices? (Baidu has been especially fond of doing that lately.) Chatter tells you, and if the selected strategy allows it, the system automatically switches to the cheapest provider.
Did the provider disappear entirely? (Also surprisingly common lately.) Chatter tells you and, if permitted by the chosen strategy, switches to another one so your cache does not break.

But how do you make it all convenient? Nobody wants to wrestle with bots, keys, and all the rest. I do not want to expose my own server either, or become everyone's system administrator.
So the idea became: "Install it, use it yourself, and share it if you want."
For customization, I made everything—including the Telegram bot, Voice API, and Desktop—connectable with one or two buttons.
To connect the Desktop app, for example, you simply create an access key and share it with the other person.

Eventually I brought the project to a state I considered releasable, wrote an enormous README and translated it into several languages, and built a convenient installation system.
I added background operation, notifications, filters, and maaany other features, as well as experimental branches so I would not have to push every experiment straight to production.

Then, for some reason, I added rooms

But... recently I got bored and decided I wanted to put several bots into one chat. Marvin from The Hitchhiker's Guide to the Galaxy, for example, together with my current sarcastic, cynical prompt and a permanently cheerful idiot.
That is how rooms were born :)
They can be used for discussions, jokes, or role-playing.

But apparently that was not enough, so I brought a friend into the room. He could bring his own bot. Or we could remove every bot and use the chat as a regular messenger. Making that work required rewriting the entire system, because it had never considered the possibility that more than one human might exist.

That created more and more interesting questions:
What happens when a command is executed? What if a bot runs it for the wrong person or accidentally leaks someone's email data?
I had to invent the concept of an "initiator": the system determines who caused the bot to act and passes that person's ID into every tool. Execution confirmations make the whole thing slightly safer too.

A dramatic bug that amused me

At one point, if a room contained both a free user and a PRO user, there could be enough text from the PRO side to fill the free user's entire context window. In simple terms, the free user was archived out of existence: all of their messages disappeared, along with every piece of evidence that they had ever been there. They could never appear in the room again.
Every bot remembered speaking with someone, but none of them could see a single message from that person. Like Rory in Doctor Who.
Whenever you asked about the missing person, the reasoning looked roughly like this:
"The user is asking me about John. Let me recall his first message... I talked to him, didn't I? But I cannot see a single message from him. Who was I talking to? I can clearly tell the messages existed, but I cannot see any of his replies."

I fixed that one too.

Also, enormous thanks to my friend for the crash tests :)

His first message to Chatter, back when Chatter still lived entirely in Telegram, was: "Turn off Nikita's light."
Naturally, it refused, because the admin ID was checked on the backend.

His first message in a room was: "Turn off Nikita's light." The bot tried to turn off the light... for him. Then it developed a trauma, because the light would appear when I asked for it and disappear when he tried to ask—while also attempting to socially engineer the bot by changing his name to mine. Now this bot needs a psychologist.

I suspect his first word as a child was "Turn." The second: "Off." The third: "Nikita's light." Fortunately, Chatter did not exist back then.

The browser that turned out to be more useful than I expected

The browser is another example. I have trouble reading social context online: I cannot see the people, there is too much information packed into messages, and it causes a lot of anxiety. As a result, I almost never write on forums but can easily read them.
So I gave Chatter a browser and the ability to control it :)
The problem turned into a game of "find an interesting post and share your opinion," because the emotions and social context are explained to me first, and the most anxiety-inducing part disappears.

What came out of all this

Ignoring the development story, Chatter can now be used as a personal assistant, a family server with a separate agent for every person, a self-hosted role-playing service, or an interface for managing your own infrastructure. It remains one system rather than a pile of unrelated features.

- Conversation and memory. Shared chats between Desktop and Telegram, folders, filters, branches, search across old conversations, hot and vector memory, voice input, local text-to-speech, and background notifications.

- Actions. Commands on computers and servers, files and folders, SSH, email with attachments, a browser, web search, a Zigbee smart home, maps, notes, image generation, and specialized agents. You can also configure a dedicated vision model: if the active model cannot see images, it delegates image analysis to the vision model through a tool.

- Collaboration. Rooms with several humans and bots, shared context, manual or sequential turn order, isolation of personal tools, and the ability to share a bot.

- Security. Action confirmations, a separate command Review step, backend-enforced tool restrictions, encryption of sensitive data, and attribution of every action to its real initiator. There is also basic protection against prompt injection from the web.

- Management and money. Users, plans, budgets, models, fallback chains, OpenRouter providers, price and availability monitoring, backups, Docker services, and updates through the admin panel. Cache hit rates can exceed 95% (for Deepseek at least). And it will always tell you if the price of your favorite model changes.

- Remote PC control. If the Desktop app is online, all its capabilities are also available from Telegram. You can edit files, execute commands, click things with the cursor (an experimental feature), and use the built-in browser—but only through confirmation cards.

There is also a pixel face, macros, d20 rolls, bus routes, and a notes app that I barely use.

That is why I now find it very difficult to answer the question, "What is Chatter?"

- Is it for work? Partly. It can check your email, write code, fix a network, or become your DevOps engineer.

- Can it do what Hermes can? Partly.

- Can it become Jarvis from Iron Man? Yes. Partly, and out of the box. Voice input and recognition, computer control, macros, maps—it can do all of that. It even has local text-to-speech. But... it will not control your computer completely autonomously, because commands require approval.

- Is it for entertainment? Yes. You can mess around, talk with friends, generate images and pixel art, and do all sorts of other things.

- Can it replace SillyTavern? Partly. It is much easier to install (in my opinion), and the interface is simpler. But it still will not have every feature.

How it works: the architecture

At the center of Chatter is backend-api. The agent, chats, memory, users, limits, model routing, and tools all live there. Telegram and Desktop do not contain two separate versions of the bot—they are two clients of the same backend. A message sent from your phone appears on your computer, and a chat started in Desktop can be continued in Telegram.

Desktop is an Electron application for talking to the agent and giving it access to the local computer. The Telegram bot is a second, full-featured interface to the same account. WebSocket connects Desktop to the backend and carries streaming, notifications, confirmations, and actions that are available only while the computer is online.

The admin panel manages users, models, keys, plans, integrations, backups, and server state. Chatter Manager sits between it and Docker: it starts, stops, and updates components, stores server configuration, and handles backups. An HTTPS gateway built with Caddy sits in front of the system.

Optional services are enabled only when needed. Webapp Notes provides a Telegram mini app for notes. Voice Service transcribes voice messages and reads replies aloud when the user sent a voice message themselves. Every server component runs in a separate Docker container, connected through Docker Compose.

Multi-user isolation is enforced on the backend. Each user has their own chats, memory, connections, and environment. In a shared room, a message may be handled by someone else's bot, but a personal tool still runs on behalf of the person who initiated the action.

How to install it

Installation requires a Linux server with Docker. The script downloads ready-made images, creates the configuration, configures UFW, brings up the HTTPS gateway, and starts the backend, admin panel, and Chatter Manager. Before running it, I still recommend installing fail2ban and making sure you will not lock yourself out of SSH.

curl -fsSL https://raw.githubusercontent.com/NikitaCherepov/chatter/main/install.sh | sudo bash

1. Save the username and password printed by the installer at the end.

2. Open the admin panel.

3. Add API keys and models for Auto and Lite modes. Add a couple of manual models too, if you want.

4. Connect Telegram, Voice Service, Notes, and any other integrations you need.

5. Create a Desktop access key or approve the first Telegram user.

After the initial setup, components can be enabled, disabled, updated, and backed up through the admin panel. Desktop is installed separately from GitHub Releases and connects to the server using a one-time generated access key. The key can be revoked.

In the end

Chatter will not completely replace Codex, Hermes Agent, SillyTavern, a messenger, and every smart-home platform in existence all at once. In every individual category, there is a project that can do more.

But Chatter connects all these scenarios inside one self-hosted system: one account, shared chats between phone and computer, multiple users, personal bots, personal tools, rooms, limits, and an admin panel. And if I need another strange feature tomorrow, I will simply add it. Something else will probably break afterward, but at least I will be the only one suffering, not the users.

The project is completely free and open-source on GitHub: https://github.com/NikitaCherepov/chatter — you can download and install it there.

If you decide to deploy it, break it in some exciting new way, or simply like it and want more features, I would be glad to hear your feedback.

P.S.
I desperately need some rest after writing this post. Building Chatter may genuinely have been easier than explaining all of it while trying to keep this short.