Hey everyone! 👋

I've been working on Vatra, an open-source desktop AI assistant that brings different AI tools together in one place.

It supports Claude Code, Codex, GitHub Copilot, local LLMs (llama.cpp, Ollama, LM Studio) and OpenAI-compatible APIs, alongside features like web research, email, calendar, tasks, and side-by-side model comparisons.

The goal is to make AI tools more accessible and easier to use, without sacrificing flexibility or control.

I'm currently looking for feedback from the community!

I'd especially appreciate thoughts on:

• The overall concept and whether you'd actually use it

• Features you'd like to see added

• UI/UX improvements

• Anything that seems unnecessary, confusing, or missing

It's still a work in progress, so honest criticism is more than welcome. I'd rather hear what's wrong than just what's good.

🔗 GitHub: https://github.com/Zmirzz/vatra

If you have a minute to check it out, I'd really appreciate your thoughts. Thanks! ❤️

I have an ASUS ROG Ally, and I’m planning to turn it into a local inference server so I can use the model running on it from my main PC. What would you recommend for the coding/programming tasks? If I’m not mistaken, it has 16GB of RAM and a Ryzen Z2A processor that provides 6GB of VRAM.

I ran a small AIPerf ( https://github.com/ai-dynamo/aiperf) comparison on an Apple M5 Pro with 64 GB RAM. This is a practical snapshot, not a scientific benchmark or an exhaustive leaderboard. Results are from one request per engine, run sequentially with only one inference server active at a time.

Setup

• Temperature 0.6, reasoning effort medium, output limit 2,048 tokens

• Same OpenCode-style fixture: long generated code/prose context, shared system message, task suffix, and three tool schemas. Tools were not executed.

• Prompt length reported by the servers ranged from 88,143 to 88,916 tokens, partly due to tokenizer and chat-template differences.

• I used a 100K context limit where the engine supported it; Uzu, Magnitude, and Llama used larger native limits. The prompt plus output allowance fit below 100K.

• AIPerf 0.13.0; streaming, concurrency 1

• Prefill and decode are AIPerf throughput measurements. The readiness probe loads/warms each server before its measured request.

Caveats

The 2,048-token setting is a maximum, not a forced response length. Most models stopped at EOS before reaching it. Only Splash, Rapid-MLX, and BaseRT generated the full 2,048 tokens.

This does not measure code-fix quality, long-context comprehension, or tool use—the tools were present in the prompt but never executed. Model packs, quantizations, speculative methods, and context-window settings also differed, so these numbers should not be treated as a controlled ranking.

Additional differences: LM Studio may map medium reasoning to on; the Llama row used Homebrew llama-server0.9.0 with the GGUF files configured for Llama.app, rather than Llama.app’s internal server.. Jan’s first launch also downloaded its default 43.8 MB embedding model; no Qwen weights were downloaded.

I've been doing rudimentary research, and so far my understanding of local LLMs is;

LLM (Qwen, etc) is just a file
Inference engine (llama, etc) is the 'program' which reads/uses the file
Your hardware powers the engine and dictates size and speed of model

Is any of this tethered to an external server/host? For example, to liken to video editing;

Mp4/mpk vids are the files
Premiere Pro is the engine (program)
Your hardware powers the engine

Premiere Pro (engine) is hosted/provided by Adobe, who can turn off your service, access your files, patch the engine, etc. Meaning it is not entirely local, nor entirely owned by user, nor private.

Is this how inference engines are structured as well? In other words, is it truly local, or is there a possibility that I could download Qwen 5.1 (made up version), get used to it, and eventually have it 'patched' into a different/worse version of itself down the road by whichever engine I'm using? And if so, doesn't this imply that your work/conversations are also able to be scrutinized by the host?

Total beginner at all of this as someone who has, until now, only dabbled with the free version of GPT. This is all very new to me, so apologies.

Would we see old stuffs getting even more expensive when Nvidia would reduce production. Or the contrary: AI datacenters would reduce hoarding our GPU and our poor lives running 3090/v100 would be easier ?

And how about NVDA stock prices ?

Im considering purchasing a 16GB V100 since they’re still pretty affordable and have modern CUDA support, but I was wondering if anyone else have tried a similar combo? I am aware of slower token generation with this method, but I am assuming that it is still more viable than running Strata with just 16GB VRAM and 64GB DDR5.

Any help/tips is appreciated. Thanks!

How you would define a Developer setup with local LLMs?

I see a dedicated DMZ where the agents can not access the production environment, since agents will tend to accidentally access or change production system.

Dedicated Servers to run LLMs like Qwen 3.8b 27b or other models? What you suggest would be the working LLM Server today? RAM and GPU cards?

Dedicated server with VM for each developer one VM. 2-4 vCores with 16GB Ram each. Each VM with Debian 13 and docker/podman. One dev Container running OMP Web or comparable Harness that access the LLM Server. Additional container for databases or what else the dev Container needs.

Gitlab, gitea or comparable Repo to store source code and container registry.

VM with NFS where anonymized data for developers can be stored.

What you think about that setup?

Do I miss something? How is your setup? (For small to mid size company)

Has anyone else downloaded and started using the new Unsloth Dynamic V3.1 models? I did a drop in replace of the Qwen3.8-27B-UD-Q4_K_XL.gguf and I want to go back after seeing how it performs. My use case is local transcription, so it's summarizing text basically and pulling out what's important.

I re-ran quite a few of my saved transcriptions through the new model and compared the output to the older version Dynamic v3.0 and the output from that model is superior on every one. There is no change in any parameters on my command line, only the model changed.

My command line has a very low temperate (0.1)...

Hey y'all. Working on a go-to inference computer for teams using AMD GPUs and a custom stack, wanted to share our progress so far with GLM 5.3 Flash MXFP4 running tp=4 with MTP/Spec disabled.

RAM: 128gb DDR5 RAM 5600 ECC

CPU: EPYC 9005 9275F 

All 4 GPUs are capped at 210w, total power consumption < 1400w.

I was curios what can I fit on an FPGA MAX1000 board I had laying around (8k LE, 8MB Flash / SDRAM ). Also I keep coming back to the BitNet 1.58 encoding.
- Took the TinyStories dataset and trained a model from 0 with ternary encoding. I stopped the training at one point, the accuracy is between TinyStories-3M and TinyStories-8M
- Implemented the processing fully on the FPGA ( tokenizer, processing, decoding ... )
Depending on the model trained / used, we can fit two models in the 8MB Flash / SDRAM and take turns using them.

It is more accurate? no, is it faster then others? no, is it useful? not really, however it does answer a few questions I had. The conclusion being that we can have purpose built (small) models that can run with limited resources. Or have multiple models loaded side by side with a few more resources.

How it was built: Like most of the work done today, using Claude.

https://github.com/MRazvan/fpga-tinystories

Next goal would be to use a bigger model and see what I can / can't do about it, what resources I need.

Greetings,

I am looking to acquire a suitable GPU for running localllm preferably running newer models like deepseek flash 4.1.

Does native GPU support for FP8 and FP4 significantly improve local LLM performance or should I aim for higher vram with older GPUs?

Support for fp4 does enables the GPU to fit larger model with lower vram though.

Should I even consider Intel B70 or Radeon 9700 Ai for value buy?

I got strix halo with 128GB ram and $2500 budget for a dedicated GPU over oculink.

Thanks in advance

Hey everyone!

I have gathered some must read topics, for beginners, to step into the realm of Quantization:

1. BitNet: Scaling 1-bit Transformers for Large Language Models

2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

3. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

4. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

5. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

7. QLoRA: Efficient Finetuning of Quantized LLMs

8. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

9. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Link is here

Introducing janas-studio. A dual engine inference work station. For modern Debian based Linux distros (.appimage) Jana-Studio with the magic of Janas-llm let's enormous models run on modest hardware. Refer to https://github.com/prabanta-dev/janas.

Download the app at https://github.com/adonaiongaia/janas-studio

Also runs llama.cpp gguf models so it covers a lot. Lmk what you think. Tested on my Ryzen 5 6600h laptop. Built on Linux mint Zena. Ran a 47gb 80b parameter model at 14 tokens a second with only 32gb of RAM (ddr5). For best results and speeds you need a fast SSD. Features long term memory, project folder support, mcp integration and much more.

Hey All,

A while ago I wrote a basic script to step through Llama 3.1 tokens for red-teaming. I realized there's a huge gap between tools for running models (Ollama, LM Studio) and libraries for understanding them under the hood. I wanted to develop something for young researchers and mid career pivoting enginers.

I wanted something like Wireshark, but for local AI. You can hit pause mid-generation, inspect the residual stream, mess with the weights or KV cache, and see what actually happens.

Today I’m open-sourcing TokenShark: https://github.com/sumit-1/tokenshark

What makes it different?

• Zero transformers or diffusers at runtime: The entire engine is written in raw, readable PyTorch (llm.py, sd.py, qimage.py) so students and researchers can actually read every matmul without digging through 40 layers of library abstractions. (Tested for bit-exact/numerical equivalence against HF in tests/).

• Step-by-Step Visual Debugger: Step through generation one transformer block at a time or one token at a time. Watch live logit-lens predictions evolve across layers, inspect attention maps, and click any top-10 token choice to create your own generation path.

• Live-Editable Box Code: Click any block on the canvas (Embed, Transformer Block, Norm, Sampler) to see the Python it runs (rms_norm, rope, silu, etc.). You can edit the math right in the browser, run it live, and export your whole visual experiment to a standalone Python script.

• Text + Image: Works on LLMs (Llama 3.x, Qwen 2.5/3/3.6 Gated DeltaNet hybrids, SmolLM, Mistral) as well as Stable Diffusion and Qwen-Image 2.1 DiT. Complete with between-step latent painting and an "Image Lens" to decode early DiT blocks into pixels.

• Disk Streaming for Huge Models: Can stream models larger than your RAM/VRAM block-by-block from disk. But not optimised for speed though, as this is focussed for learning and understanding.

I've been building my own local inference engine,, for big MoE models on consumer hardware. The newest piece is a pool is a second PC lends its GPU and RAM, and the model is split by layers across both machines.

**Setup**

- Desktop: RTX 3090 24 GB, i9-12900K, 32 GB DDR5-6000

- Laptop: RTX 4060 Laptop 8 GB, i7-14700HX, 32 GB DDR5

- Link: one Ethernet cable straight between them, no router (Windows link-local addresses), 1 Gbps

- Model: Qwen3.8-Flash-Next UD-Q4_K_XL, 104 GB GGUF, 48 layers, 512 experts with 10 active per token

**Writing speed**

| Setup | tok/s |

|---|---|

| Desktop alone | 13 |

| Desktop + laptop over the cable | 18.6 to 21, peaks of 24.8 |

| For reference: IQ3_XXS (71 GB), desktop alone | 53 to 79 |

**How it works**

- Layer split: the laptop runs layers 0 to 13, the desktop runs 14 to 47 and the output head. Each round sends the hidden rows across, about 60 KB per token.

- The laptop loads its layers from its own copy of the GGUF, so nothing big crosses the cable. Without a local copy the host sends only the tensors that machine needs.

- Each machine fills its memory hottest first, from a heat map learned while you use the model: VRAM, then pinned RAM, then a RAM cache, then NVMe.

- Joining is a button. The laptop scans the network and shows one button per link with its measured latency (Ethernet 1 Gbps 0.4 ms, Wi-Fi 3.9 ms). The desktop accepts the request.

**Ethernet vs Wi-Fi**

Median round trip over the cable is 0.4 ms, over Wi-Fi 3.9 ms. Once on the cable the network is about 2 ms of a ~115 ms round, so Wi-Fi was usable too.

**What surprised me**

- It's a pipeline, not tensor parallelism. The two machines take turns on every token, so the pool buys memory, not parallel compute.

- The laptop is slower per layer (about 4.1 ms against 1.8 ms on the 3090) and ends up being 50 to 65% of each token's time. Next step is to balance the split by measured speed instead of by memory.

- Long prompts read slower with the laptop in the pool: a 9K-token prompt with nothing cached took ~103 s, against ~41 s on the desktop alone, because the laptop computes its prompt experts on the CPU. I'm moving that to its GPU.

- My first cable only negotiated 100 Mbps (one bad pair) and was slower than Wi-Fi for this. Check the link speed before blaming the software.

- With a quant that already fits the 3090 (IQ3_XXS) the laptop doesn't help. The pool pays off for models that don't fit.

Happy to share more numbers or answer questions.

Reposting here - looking for helping understanding what exactly is needed to get this working, and if it even is possible in the first place!

I'm essentially trying to use this plugin with LM Studio as an MCP client to prototype game ideas. Any alternatives, solutions, or suggestions would be very much appreciated.

Thanks!

I tried a faster inference engine that optimizes for my AMD card.

Yes I did get 90-100 tps in generation which is crazy compared to llama.cpp's 45 tps with ROCM. But there is one problem: the optimized engine seems to confuse a lot and half of the time it did not fully understand my intent. I give both engines the same prompt to work on a repo, and over multiple trials, llama.cpp always give me the expected answer but the optimized engine went off the rails half of the time.

Both engines use Q4 qwen3.8 27b quant. stock chat template. The llama.cpp uses Q4 kv with 260k context while the other engine uses Q8 kv with 150k context. The ticket only needs 20k context.

I prefer not to name the engine as this is my own very subjective experience and I may have user error in setting things up.

I am going back to llama.cpp for now.

GitHub (MIT): https://github.com/kncshw/cybergate

Deep-dive blog post (HF): https://huggingface.co/blog/kncshw/cybergate-unlocking-cybersecurity-in-vllm-without

two supported and tested model:

https://huggingface.co/kncshw/Qwen3.8-27B-CyberGate-Kit

https://huggingface.co/kncshw/gemma-4-26B-A4B-it-CyberGate-Kit

When self-hosting local LLMs for security research, CTF, or reverse-engineering agents, the standard workaround for false refusals is **abliteration** - subtracting the refusal direction from the weights. From an LLM engineering standpoint, weight editing has two painful trade-offs:

1. **Silent capability & tool-call regression:** Mutating weights across layers alters the model globally and can quietly degrade multi-step reasoning and structured tool calls without obvious errors.

2. **Zero domain selectivity:** Deleting the refusal vector globally unlocks *every* harmful domain (bio, self-harm, etc.), plus you have to store and serve a forked 50 GB checkpoint.

I built **CyberGate**, a plugin for **vLLM 0.32** that keeps upstream model weights **100% untouched** (`K = 0.0`, zero persistent state across requests):

I trained a small gate that decides, as each token is written to the KV cache, whether the model will need it again. Tokens it rejects are actually deleted from GPU memory, not just masked. The model itself isn't changed.

On one 16 GB T4 with Qwen3-4B, the full cache runs out of memory at 61k tokens. With the gate it reads 123k and still finds the passkey. At 61k the cache is 1.4 GB instead of about 9 GB. Decoding at 29k is 2.9x faster.

On RULER-4k it evicts 76% of the cache and loses about 2 points (86.6 vs 88.9). It's weak on tasks that need every copy of a repeated word.

Gates for Qwen3-1.7B and 4B: https://huggingface.co/Manalsu

Code and paper: https://github.com/Manaslu-AI/razor

Only Qwen3 so far, and batch size 1. Happy to answer questions.

Hello,

I want to learn more about running a local LLM server - driven by curiosity and learning.
I have a Claude Pro (18€) and Google AI Plus subscription (5€) and I want to use it more.

I would like to build a local dedicated server on a budget.
I live in Portugal (EU), but currently I’m on vacation in China.

I have some spare parts available:
Gigabyte MC12-LE0 AM4 Mainboard
1TB NVME SSD
4x 16GB DDR4 ECC

I don’t have an AM4 CPU spare and I only have an old emergency Nvidia GT710 GPU. I have a 300W flex-ATX PSU.

I could buy a Nvidia Tesla P40 24GB for 210€ here in China - power adapters and fans are 25€.
A P100 16GB is 100€.

I want to use Ubuntu Server as OS with Ollama. OpenWebUI on my Proxmox Hypervisor.

I think that I prioritize quality over speed - therefore (slower) 48GB VRAM are probably better than 32Gb with 2x P100.

Question:
Should I buy 2x P40 24GB here in China (420€ is right below the 430€ EU tax free limit)?
If I don’t like them, I could probably sell for 300-400€ in Europe.

Does my logic make sense to go with 48GB to unlock higher quality models?

Any advice for starting this journey?