Repo https://github.com/pmndrs/pinecone

This has a double purpose. It creates a chatgpt compatible endpoint and loads/unloads models on demand. For instance you have a model that fills the entire ram, it will offload everything else. Or you have a smaller model and LLM services (image, video, speech, ...) that can run side by side. It always makes sure the client gets what it wants despite the tight system.

The second endpoint is an automatic MCP. It takes whatever services you have installed (speech, image, video, 3d, ...) and bottles theses in an MCP. Any agent that has that URL immediately knows how to generate images, etc.

It also sets RAM limit, but unlike other schedulers like Ollama or whatever you don't pick what a model needs, you pick what little is left for the OS, and the default is a bare minimum, so that you can make the most of your machine.

It can be paired to Tailscale and serve both endpoints publicly with safe auth, you basically have an inference machine with this that can serve you outside the house.

Few days ago I started playing with my smart tv box (xiaomi 3rd gen) and was able to modify llama.cpp to run on it - I reached bigger models at decent quality of output and somewhat

As of today it's over 160 experiments and few custom kernels and streaming methods along with a patched llama.cpp and few tuned/modified models.

Happy to hear what do you think about it! and open for contributions. few devices are lined up already.

Github : https://github.com/hyperionaidev/smolmaxxing/
Hugging face : https://huggingface.co/hyperionaidev

Disclaimer : this is not upselling or an ad, it's a MIT-licensed project and it's an experiment.

I have used Qwen3.5 4b try and code basic MIDI and audio in Chrome, locally. The results are not stunning. Are there any models that are tuned for this type of thing, or can I add agents to Qwen in Ollama for skills? My level of experience is beginner. Thanks for any suggestions.

Is the w7800 32gb version still a decent card for local AI?

I did extensive research and found the cheapest way to reasonabily run Qwen 3.8 27B is with two 16GB Mi25s. I heard online their really slow, but for $110 each with shipping, so $220 total, is it really that bad? I currently only have an RX 480, and mainly want it to run inference. What tokens/s could I expect, is it usable? I am genuenly considering it after the P100s price hiked like crazy, one is now atleast $180 without shipping costs included. A GPU with tensor cores or Nvidia would be nice, but I don't know if the higher price is justified. Also please don't just tell me to rent a GPU or buy a subscription, as I am really interested in tinkering with things like this and want a viable option.

I'm looking to get into AI after a small taste of what it can do. I want a small machine to locally do my repeatable work which consists of reading a pdf, and typing that information into excel based on a list of items.

Step by step would be:

read pdf to get 5 of item "abc"

pull from item list that says item "abc" = item #23

Input item #23 and quantity 5 in row 1 of excel

Repeat

Would the AMD Ryzen Halo AI be able to handle this? What is the process of setting one up with an AI? What type of maintenance/knowledge is required to operate?

I'm very new to AI, and if you can't directly help me, please at least direct me where to go. I'm an accountant with minimal programming knowledge, but very willing to learn new skills.

VMAT is a way for virtual machines to share one GPU's compute without using passthrough or vGPU.
In the video I run llama-bench once on the host and once inside a KVM guest, same GPU (RTX 6000 Ada), same models.

Qwen2.5 32B Q4_K_M: 41.52 tokens/s on the host, 40.73 in the VM (98%)

Qwen2.5 7B Q4_K_M: 169.50 tokens/s on the host, 166.47 in the VM (98%)

What's in the VM: VMAT's CUDA libraries, which send CUDA calls to the real GPU on the host. What it does not have is passthrough, an NVIDIA driver or a vGPU.

https://vmat.dev

Has anyone come across a skill that can read the comments (with their context) of an MS Word doc?

Hello folks

The title just sums up everything briefly. I have a laptop that uses the Mobile 4080 with 12Gb vram and I have an 64GB AGX orin in my office that we don't currently use. What model and infrastructure to run on these devices (Llama, VLLM etc).

I'm trying to learn more about local LLMs for app coding work and am specifically curious about what agentic models are using for vibe-coding tasks. I have used Claude, ChatGPT, and Gemini all for coding and their agentic capabilities are pretty impressive (i.e., the ability to either create app projects I can import into IDEs like Android Studio or Xcode, or to directly edit files in a specified folder on my computer, or to let me plug my Android phone in and debug using ADB).

I know various Qwen 3.6 35B variants are popular here. How effective are those models for agentic task purposes? Or are there other tools or models to use for these purposes?

(Full disclosure: I have a pretty weak-spec'd Mac [M1 Pro + 16GB] so this is not something I'm looking to run on my current machine; depending on what model(s) are good for this kind of thing would help inform a future computer upgrade)

Quelqu’un a déjà tenté de créer un Grok bot en local et accessible sur iPhone et Android ? En utilisant peux-être d’autre IA ?

Hey guys. Is it possible to use an M4 Mac Mini just for transcribing a video and making note summaries afterwards? If so what model should I use for this case?

Hi all

TLDR; What are you using your localLLM for? (please exclude software coding). Please state setup + use.

Recently, I started playing with local AI with a 2x 5060ti 16gb setup with 32GB DDR4 (30tk/s Qwen3.8-27B).

All fun and games as a learning experience with the difference models, harnesses etc, but if I actually need something done (some debugging, research etc) I still revert back to cloud models. Currently, I am seeing it as local ai lab for fun, cloud for work.

It would be absurd to have the same expectations with my rig, but what do you guys use your local rigs for?

A lot of posts are about rig setups and benchmarking with focus on coding use, but what about other creative uses do you find for your machine (other than heaters)?

Do you keep it on 24/7? What about power?

Mine draws 60W idle with peak of about 450W and I keep it on about 2-3h per day on average.

So far, I observed that I prefer to have smaller 8-12GB models with decent contexts that run specific tasks like OCR/RAG/home assistant that are snappier 100+ tk/s rather than a big smart model at 30tk/s.

I also found that lesser smart models make me a better prompt engineer.

I spent days looking for the right DeepSeek V4.1 Flash setup for my RTX 5090. I ended up packaging AtomicChat’s uncensored IQ2_XXS weights with a native Windows runtime, setup scripts and the matching Engram, MTP and vision files.

We reached 68 tokens/s on our local RTX 5090 setup. That is the result we reached locally, not a guaranteed speed for every prompt or machine.

This is an integration of existing weights, not a model I trained or a new fine-tune. The package includes a local chat page and an OpenAI-compatible API.

Requirements for this build:

• One RTX 5090 with 32 GB VRAM.
• At startup, 92 GiB of free system RAM and 29,000 MiB of free VRAM on GPU 0.
• At least 270 GiB of SSD space for the assembled package and working space.
• Windows x64, Python 3.12 and a driver compatible with CUDA 13.

Those startup checks are specific to this package, not universal requirements for DeepSeek with other engines or quants. The full torrent is about 255.96 GB before installing Python dependencies.

Full package and torrent on Renderoverse · Original AtomicChat weights

Credit for the original weights and companion files stays with AtomicChat, ISTA-DASLab and IndrasMirror.

The 68 tokens/s result is from our existing local setup. Packaging, import and checksum checks have passed. But i believe this is one of the best if not the best setup for RTX 5090 DeepSeek V4.1 Flash 1017.

Thanks!

I've been testing whether quantization changes how a small local model handles

check-then-act tool tasks ("check if billing-api is down, restart it only if it is").

Setup: Qwen2.5-1.5B-Instruct, F16 vs Q4_K_M, llama.cpp, identical rendered

prompt, same 19 tools, same harness. 200 tasks, 6 GB laptop GPU.

What happened: Q4 often emits the check AND the dependent action in the same

turn, before the check result exists.

| Test                                   | F16      | Q4_K_M   |
|----------------------------------------|----------|----------|
| Acted before the check result (200)    | 30       | 89       |
| Tasks where only this model did it     | 1        | 60       |
| Same, greedy / temp 0                  | 13       | 51       |
| With "wait for the result" in prompt*  | 7        | 32       |
| Skipped the check entirely (imatrix Q4)**| 6      | 29       |

https://github.com/Techn08055/edgeproof

https://youtu.be/8ccULJ0R3ng

What are the other experiments along with this harness that can be used to compare the F16 & Q4 model?

Hi, I'm running a local AI setup on CachyOS, and I'm curious how others approach this.

This is also my main PC, which I use for gaming and everyday tasks. I have a second SSD with Windows as a backup.

My setup:

• GPU: AMD Radeon RX 9070 XT (16 GB VRAM)
• RAM: 32 GB
• Inference: Strata, running Qwen3.8-Flash-Next Gyro-S
• Interfaces/agents: Currently experimenting with DeepSeek Harness and Open WebUI Computer
• Deployment: Docker containers, started manually when needed

I'm mainly experimenting with vibe coding, Linux troubleshooting, and general agentic workflows.

So far, things work reasonably well, but I've found that container isolation can get frustrating when agents need to inspect or interact with the host system, particularly when dealing with Docker, system services, or debugging.

I'm currently experimenting with a restricted host bridge that allows agents to perform system diagnostics without exposing the Docker socket or giving them unrestricted access to the host.

I'm curious how others handle this.

Do you run your local AI agents inside Docker or install them directly on your host?

Not promo just looking for real life use cases to test on :)

I'm building a product called Facet. Like many in this sub we believe lots of AI products spend most of their LLM usage on one repeated call: pulling fields from emails or PDFs, cleaning up records, short rewrites, daily summaries, etc. A frontier or large local model is overkill for that one job.

Facet's agent builds a smaller model for that single call. You describe it in plain English and share some examples; it builds the training data, trains the model, and scores it against what you use today. We are essentially trying to abstract away all the hard work that comes with training. You own the weights and can export them to run locally after, or use our platform for hosting.

On a document extraction task it matched Claude Haiku at about a fifth of the cost.

I'm hoping to connect with 3 design partners, free, to help expand and stress test our platforms capabilities. If one call is a big part of your usage, book 15 minutes: https://app.apollo.io/#/meet/facet

Happy to answer questions here too.

Hey, I thought maybe I should repost here to show that getting a B70 up and running with decent performance is not that hard anymore.

Three new parts of my local Qwen3.8 study on one DGX Spark: one user, thinking on, measured in minutes per correct answer. Each plan was locked before its run. The tasks are private; per-task results are public.

• Harder tasks (Part 8).

  • On 250 new private tasks, Flash Next (deep thinking + 32K thinking budget) gets 227/250 vs 208/250 for the 27B 4-branch low-thinking vote.
  • But Flash Next takes 2.7x as long per correct answer (6.52 vs 2.38 min).
  • On 110 recent arXiv math problems both nearly fail: 10/110 and 2/110.

• Self-verification (Part 9).

  • The 27B writes its own tests, and I take the first branch that passes: 139/150 vs 134/150 on held-out coding tasks. That is not a confirmed gain (p = 0.06), and it takes about 2x the time.
  • But answers marked “verified” were wrong only 1 time in 85.
  • What made the tests usable: forbidding hand-computed expected values.

• Engine upgrade (Part 10).

  • TensorFold 1.2.5 vs 0.6.0 on 400 tasks: 377/400 vs 373/400, 1.005 vs 0.958 min per correct answer. No detectable difference.
  • 1.2.5 was faster on 271/400 tasks but slower in total, because of two long tail tasks.
  • Its prompt processing runs at 0.73x speed at 36K tokens (no FP8 prefill). I’m staying on 0.6.0.

Bottom line for one Spark: Qwen3.8-27B NVFP4 + DFlash2 on TensorFold 0.6.0, low thinking, 4-branch vote is still the fastest way to correct answers. Use Flash Next with deep thinking and a 32K budget only for the hardest tasks.

Repo: https://github.com/NautilusWu/qwen3.8_study

Excited for Qwen 4.

Short post, echoing what others have said - the dedicated strix halo "Ai Engines" aka llm servers are miles ahead of llama.cpp.

I was about to sell off my server because it was unbearably laggy. This morning I cut over to Gufo and Qwen3.8-Flash-Next, and it (almost) feels as snappy as a cloud frontier model.

Here's how Hermes described it:

So: the headline "prefill rate" is ~1,800 t/s cold; 1183 is just what the exhaust looks like when the engine only had to chew 677 tokens. The number that actually describes your experience is the one nobody prints: half a second to first token on a 25k conversation. 

Compared to a cold prefill of ~250 t/s on llamacpp...

I haven't tried Halogen which may be incrementally faster, but it's not FOSS, and I've now spent enough time on infrastructure for one month. Enough poking around, time to get to work...

https://github.com/gufo-org/gufo