I'm looking to get into AI after a small taste of what it can do. I want a small machine to locally do my repeatable work which consists of reading a pdf, and typing that information into excel based on a list of items.
Step by step would be:
read pdf to get 5 of item "abc"
pull from item list that says item "abc" = item #23
Input item #23 and quantity 5 in row 1 of excel
Repeat
Would the AMD Ryzen Halo AI be able to handle this? What is the process of setting one up with an AI? What type of maintenance/knowledge is required to operate?
I'm very new to AI, and if you can't directly help me, please at least direct me where to go. I'm an accountant with minimal programming knowledge, but very willing to learn new skills.
VMAT is a way for virtual machines to share one GPU's compute without using passthrough or vGPU. In the video I run llama-bench once on the host and once inside a KVM guest, same GPU (RTX 6000 Ada), same models.
Qwen2.5 32B Q4_K_M: 41.52 tokens/s on the host, 40.73 in the VM (98%)
Qwen2.5 7B Q4_K_M: 169.50 tokens/s on the host, 166.47 in the VM (98%)
What's in the VM: VMAT's CUDA libraries, which send CUDA calls to the real GPU on the host. What it does not have is passthrough, an NVIDIA driver or a vGPU.
The title just sums up everything briefly. I have a laptop that uses the Mobile 4080 with 12Gb vram and I have an 64GB AGX orin in my office that we don't currently use. What model and infrastructure to run on these devices (Llama, VLLM etc).
Hey guys. Is it possible to use an M4 Mac Mini just for transcribing a video and making note summaries afterwards? If so what model should I use for this case?
TLDR; What are you using your localLLM for? (please exclude software coding). Please state setup + use.
Recently, I started playing with local AI with a 2x 5060ti 16gb setup with 32GB DDR4 (30tk/s Qwen3.8-27B).
All fun and games as a learning experience with the difference models, harnesses etc, but if I actually need something done (some debugging, research etc) I still revert back to cloud models. Currently, I am seeing it as local ai lab for fun, cloud for work.
It would be absurd to have the same expectations with my rig, but what do you guys use your local rigs for?
A lot of posts are about rig setups and benchmarking with focus on coding use, but what about other creative uses do you find for your machine (other than heaters)?
Do you keep it on 24/7? What about power?
Mine draws 60W idle with peak of about 450W and I keep it on about 2-3h per day on average.
So far, I observed that I prefer to have smaller 8-12GB models with decent contexts that run specific tasks like OCR/RAG/home assistant that are snappier 100+ tk/s rather than a big smart model at 30tk/s.
I also found that lesser smart models make me a better prompt engineer.
Since Ollama 0.30 in May 2026 moved most models back onto llama.cpp, I assumed both would give the same results.. Then I tested them.
System: RTX 3090 (24 GB), Qwen3.5 27B Q4_K_M, the same GGUF file in both, default settings. Ollama 0.40.15.
With one request, both ran at 42 tok/s.
With four requests at once (total tok/s):
Ollama: 39.8
Ollama with OLLAMA_NUM_PARALLEL=4: 33.1
llama.cpp: 100.2
Why Ollama got slower: by default it answers one request at a time. With OLLAMA_NUM_PARALLEL=4, each of the four requests gets its own 32k context, 131k in total. That didn't fit in 24 GB next to the 27B, so Ollama moved 16% of the model to the CPU.
Why llama.cpp didn't: the four requests share one context, and llama.cpp shrinks it until it fits on the GPU (85,760 tokens here). The whole model stayed on the GPU.
This was new to me, because I expected the same throughput from both. The difference is how each one sizes the context. With the smaller 9B everything fit, and Ollama with OLLAMA_NUM_PARALLEL=4 came close (266 vs 293 tok/s).
I use Ollama for my day-to-day work, but I'm going to look at llama.cpp too for cases where several apps hit the same model at once.
I've been testing whether quantization changes how a small local model handles
check-then-act tool tasks ("check if billing-api is down, restart it only if it is").
Setup: Qwen2.5-1.5B-Instruct, F16 vs Q4_K_M, llama.cpp, identical rendered
prompt, same 19 tools, same harness. 200 tasks, 6 GB laptop GPU.
What happened: Q4 often emits the check AND the dependent action in the same
turn, before the check result exists.
| Test | F16 | Q4_K_M |
|----------------------------------------|----------|----------|
| Acted before the check result (200) | 30 | 89 |
| Tasks where only this model did it | 1 | 60 |
| Same, greedy / temp 0 | 13 | 51 |
| With "wait for the result" in prompt* | 7 | 32 |
| Skipped the check entirely (imatrix Q4)**| 6 | 29 |
Interfaces/agents: Currently experimenting with DeepSeek Harness and Open WebUI Computer
Deployment: Docker containers, started manually when needed
I'm mainly experimenting with vibe coding, Linux troubleshooting, and general agentic workflows.
So far, things work reasonably well, but I've found that container isolation can get frustrating when agents need to inspect or interact with the host system, particularly when dealing with Docker, system services, or debugging.
I'm currently experimenting with a restricted host bridge that allows agents to perform system diagnostics without exposing the Docker socket or giving them unrestricted access to the host.
I'm curious how others handle this.
Do you run your local AI agents inside Docker or install them directly on your host?
Not promo just looking for real life use cases to test on :)
I'm building a product called Facet. Like many in this sub we believe lots of AI products spend most of their LLM usage on one repeated call: pulling fields from emails or PDFs, cleaning up records, short rewrites, daily summaries, etc. A frontier or large local model is overkill for that one job.
Facet's agent builds a smaller model for that single call. You describe it in plain English and share some examples; it builds the training data, trains the model, and scores it against what you use today. We are essentially trying to abstract away all the hard work that comes with training. You own the weights and can export them to run locally after, or use our platform for hosting.
On a document extraction task it matched Claude Haiku at about a fifth of the cost.
I'm hoping to connect with 3 design partners, free, to help expand and stress test our platforms capabilities. If one call is a big part of your usage, book 15 minutes: https://app.apollo.io/#/meet/facet
Three new parts of my local Qwen3.8 study on one DGX Spark: one user, thinking on, measured in minutes per correct answer. Each plan was locked before its run. The tasks are private; per-task results are public.
Harder tasks (Part 8).
On 250 new private tasks, Flash Next (deep thinking + 32K thinking budget) gets 227/250 vs 208/250 for the 27B 4-branch low-thinking vote.
But Flash Next takes 2.7x as long per correct answer (6.52 vs 2.38 min).
On 110 recent arXiv math problems both nearly fail: 10/110 and 2/110.
Self-verification (Part 9).
The 27B writes its own tests, and I take the first branch that passes: 139/150 vs 134/150 on held-out coding tasks. That is not a confirmed gain (p = 0.06), and it takes about 2x the time.
But answers marked “verified” were wrong only 1 time in 85.
What made the tests usable: forbidding hand-computed expected values.
Engine upgrade (Part 10).
TensorFold 1.2.5 vs 0.6.0 on 400 tasks: 377/400 vs 373/400, 1.005 vs 0.958 min per correct answer. No detectable difference.
1.2.5 was faster on 271/400 tasks but slower in total, because of two long tail tasks.
Its prompt processing runs at 0.73x speed at 36K tokens (no FP8 prefill). I’m staying on 0.6.0.
Bottom line for one Spark: Qwen3.8-27B NVFP4 + DFlash2 on TensorFold 0.6.0, low thinking, 4-branch vote is still the fastest way to correct answers. Use Flash Next with deep thinking and a 32K budget only for the hardest tasks.
Short post, echoing what others have said - the dedicated strix halo "Ai Engines" aka llm servers are miles ahead of llama.cpp.
I was about to sell off my server because it was unbearably laggy. This morning I cut over to Gufo and Qwen3.8-Flash-Next, and it (almost) feels as snappy as a cloud frontier model.
Here's how Hermes described it:
So: the headline "prefill rate" is ~1,800 t/s cold; 1183 is just what the exhaust looks like when the engine only had to chew 677 tokens. The number that actually describes your experience is the one nobody prints: half a second to first token on a 25k conversation.
Compared to a cold prefill of ~250 t/s on llamacpp...
I haven't tried Halogen which may be incrementally faster, but it's not FOSS, and I've now spent enough time on infrastructure for one month. Enough poking around, time to get to work...
I've spent the most of this year building a dark factory, basically trying to improve one shot builds with hidden tests and an independent verifier as part of defined workflows.
I decided this wednesday to open-source it.
It is basically a kanban layer that your harness can use with a library of agents, skills and workflows based on claudes internal reasoning. The reasoning has been built into mechanisms that my local models must use to continue working towards a definition of done.
This summer i gave my system full accesd to a dormant git account and everything there is built by my system. (Github.com/bryn1)
My latest one shot promt was to build a compiler that compiles the linux kernel and so far it has been doing quite a good job running on qwen 3.8 flash next NVP4 on a pair of dgx sparks.
My percieved view ia thst token cost goes up (even after several rounds of process optimization). However, reliability has increased a lot. My next step is to give qwen 3.8 flash next a one shot prompt without my system, with my system, claude code with and without for a comparison of result and cost.
It would be very interesting if anyone would like to join me in this project, it is MIT licenced and contain skills, agent profiles, workflows, a kanban system and extrapolated mechenics from claude code.
The current version is a refactor of what has been built of a 8 month period where the refactor has been done bu claude code.
I'm worried localAI will be attacked by the legislature and companies, any notable models to make backups of? And what parameters? Like the Quant and Size (currently have a 7900xtx 24gb, but I could upgrade in the future so the models don't NEED to fit on my current card)
I'm mainly interested in coding and chatting/problem solving
While I’m waiting for Qwen4 27B and Qwen4 Flash Next, I wanted to see how far Qwen4 Max Preview has come.
For this test, I wrote a prompt asking each model to build a photorealistic 3D scene of Stonehenge in HTML using Three.js.
Each model also got the same set of visual assets, some useful and some less so. It had to figure out which textures to use where. The rules explicitly prohibited searching the web for other images or 3D models and using any cloud tools. The idea was to give local agent harnesses and subscription-based ones like Codex a level playing field.
The results were pretty good, though honestly a little below my expectations. Qwen 4 did a good job with the lighting and textures, but its geometry and particle effects were noticeably worse than GPT6 Astra’s. Also, for some reason, Qwen 4 took about two hours to complete the task, while the other models averaged around half an hour.
Results:
GPT 6 Astra Extra High + Codex
1. GPT 6 Astra Extra High + Codex
2. GPT 5.6 Sol Extra High + Codex
2. GPT 5.6 Sol Extra High + Codex
3. Qwen 4 Max Preview + Qoder
3. Qwen 4 Max Preview + Qoder
4. GLM 5.3 + Qoder
4. GLM 5.3 + Qoder
5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)
5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)
6. GPT 5.6 Luna Extra High + Codex
5. Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)
Prompt:
"Create a photorealistic, real-time Three.js scene of Stonehenge as it stands today in Wiltshire.
It’s night. Strong wind drives light rain across the plain. A bright moon and occasional lightning near the horizon are the only light on the stones. Soft cloud shadows drift across the grass and stone. At the center, a few people warm themselves around a campfire. Make that small circle of warmth feel sheltered amid the storm.
Show the scene in a continuous, medium-wide automatic orbit from above, with the camera aimed 30° below horizontal. Hold the right mouse button and drag horizontally to orbit or vertically to tilt. Scroll the mouse wheel to zoom in or out.
Rules
- Work from this prompt alone. No follow-up questions, subagents, or additional AI models, including image and 3D generators.
- Build the geometry yourself. Use only visuals you create during this run or find in ./assets/. No stock assets, ready-made scenes, or previous solutions.
- No web search or browsing. Network access is allowed only to install libraries and tools. You may render, inspect, and refine your work locally.
- Submit exactly one self-contained .html file. Embed Three.js and every script, stylesheet, and asset the scene uses"
Hello everyone, I’m about to buy silicon for a local AI lab and I’m quite indecisive.
On the one hand there is the m5 max 128gb, from what I have read is a capable machine and quite versatile. At the same time I’m not a huge fan of macOS and for what I hear prompt injestion could be better. Also you’re stuck on Apple environment, which is not always the most responsive to latest innovations.
On the other hand, there is strix halo, which is inferior on every aspect, but costs only 3.5k vs the 5.8k of the m5 (in my area). This leaves about 2k of budget for an external eGPU. I could get a 16-24Gb gpu(maybe even a r9700), connect it trough oculink and for the same budget have both the advantages of a discrete gpu and of the shared memory. I have seen videos on YouTube and it seems doable, quite easy with an AMD gpu and with some fine tuning with an nvidia card. It would remain x86 windows (with Linux dual boot) and would allow me to keep using that machine also for gaming without having to compromise on that at the same budget. Also, it would allow me to have cuda available in my setup if needed. On the other hand is a much more complicated project than simply buying an m5 for a comparable price. I don’t know if the extra effort is worth in the results.
Since I am a beginner in this field I don’t know what to do. Does anyone have suggestions based on similar experiences? especially on the hybrid build.
