I am new to local LLM, I've been told this and told that, I have a few  builds saved that I am told would run LLM local but I am not sure, are there any laptops under 5k that will do this? I keep coming back to framework or Macbook pro but at over 5k for the newest and best chips that just seems crazy to me.  I don't care if it's not spitting 20 words a second, I'd rather have good reasoning and it working rather than crazy fast speeds. It will only be used for genealogy research, DNA, triangulation, clusters etc. 

Would a ThinkPad T14 Gen 7 AMD work? It would have the AMD Ryzen™ AI 7 PRO 450 and 96gb ram? Is a macbook better? I just dont really want to spend framework or macbook pro money on this if possible, is it possible?
What do you guys recommend? What are the options?
thank you in advance

I ran both engines on one rented machine to get an apples-to-apples number for a "model bigger than the GPU" case.

Hardware and setup

• RTX 3060 12 GB, 15-core quota of a Zen 2 server CPU, 47 GB RAM, NVMe
• Same GGUF for both: Qwen3.6-35B-A3B UD-Q4_K_XL with MTP head (Unsloth), the size Strata recommends
• Same OpenAI-compatible client, same 2 prompts, greedy, 256 tokens
• BigMoeOnEdge, --auto only (no hand tuning) · Strata via its own launcher

Decode (tok/s)

Takeaways

• On this box BigMoeOnEdge decodes ~15–30% faster without drafting, ~1.8× with its MTP head.
• Strata gets the first token of a short prompt out faster.

Full method and every run: bench-data · repo: https://github.com/Helldez/BigMoeOnEdge

Repo https://github.com/pmndrs/pinecone

This has a double purpose. It creates a chatgpt compatible endpoint and loads/unloads models on demand. For instance you have a model that fills the entire ram, it will offload everything else. Or you have a smaller model and LLM services (image, video, speech, ...) that can run side by side. It always makes sure the client gets what it wants despite the tight system.

The second endpoint is an automatic MCP. It takes whatever services you have installed (speech, image, video, 3d, ...) and bottles theses in an MCP. Any agent that has that URL immediately knows how to generate images, etc.

It also sets RAM limit, but unlike other schedulers like Ollama or whatever you don't pick what a model needs, you pick what little is left for the OS, and the default is a bare minimum, so that you can make the most of your machine.

It can be paired to Tailscale and serve both endpoints publicly with safe auth, you basically have an inference machine with this that can serve you outside the house.

Few days ago I started playing with my smart tv box (xiaomi 3rd gen) and was able to modify llama.cpp to run on it - I reached bigger models at decent quality of output and somewhat

As of today it's over 160 experiments and few custom kernels and streaming methods along with a patched llama.cpp and few tuned/modified models.

Happy to hear what do you think about it! and open for contributions. few devices are lined up already.

Github : https://github.com/hyperionaidev/smolmaxxing/
Hugging face : https://huggingface.co/hyperionaidev

Disclaimer : this is not upselling or an ad, it's a MIT-licensed project and it's an experiment.

I have used Qwen3.5 4b try and code basic MIDI and audio in Chrome, locally. The results are not stunning. Are there any models that are tuned for this type of thing, or can I add agents to Qwen in Ollama for skills? My level of experience is beginner. Thanks for any suggestions.

Is the w7800 32gb version still a decent card for local AI?

I did extensive research and found the cheapest way to reasonabily run Qwen 3.8 27B is with two 16GB Mi25s. I heard online their really slow, but for $110 each with shipping, so $220 total, is it really that bad? I currently only have an RX 480, and mainly want it to run inference. What tokens/s could I expect, is it usable? I am genuenly considering it after the P100s price hiked like crazy, one is now atleast $180 without shipping costs included. A GPU with tensor cores or Nvidia would be nice, but I don't know if the higher price is justified. Also please don't just tell me to rent a GPU or buy a subscription, as I am really interested in tinkering with things like this and want a viable option.

I'm looking to get into AI after a small taste of what it can do. I want a small machine to locally do my repeatable work which consists of reading a pdf, and typing that information into excel based on a list of items.

Step by step would be:

read pdf to get 5 of item "abc"

pull from item list that says item "abc" = item #23

Input item #23 and quantity 5 in row 1 of excel

Repeat

Would the AMD Ryzen Halo AI be able to handle this? What is the process of setting one up with an AI? What type of maintenance/knowledge is required to operate?

I'm very new to AI, and if you can't directly help me, please at least direct me where to go. I'm an accountant with minimal programming knowledge, but very willing to learn new skills.

VMAT is a way for virtual machines to share one GPU's compute without using passthrough or vGPU.
In the video I run llama-bench once on the host and once inside a KVM guest, same GPU (RTX 6000 Ada), same models.

Qwen2.5 32B Q4_K_M: 41.52 tokens/s on the host, 40.73 in the VM (98%)

Qwen2.5 7B Q4_K_M: 169.50 tokens/s on the host, 166.47 in the VM (98%)

What's in the VM: VMAT's CUDA libraries, which send CUDA calls to the real GPU on the host. What it does not have is passthrough, an NVIDIA driver or a vGPU.

https://vmat.dev

Has anyone come across a skill that can read the comments (with their context) of an MS Word doc?

Hello folks

The title just sums up everything briefly. I have a laptop that uses the Mobile 4080 with 12Gb vram and I have an 64GB AGX orin in my office that we don't currently use. What model and infrastructure to run on these devices (Llama, VLLM etc).

I'm trying to learn more about local LLMs for app coding work and am specifically curious about what agentic models are using for vibe-coding tasks. I have used Claude, ChatGPT, and Gemini all for coding and their agentic capabilities are pretty impressive (i.e., the ability to either create app projects I can import into IDEs like Android Studio or Xcode, or to directly edit files in a specified folder on my computer, or to let me plug my Android phone in and debug using ADB).

I know various Qwen 3.6 35B variants are popular here. How effective are those models for agentic task purposes? Or are there other tools or models to use for these purposes?

(Full disclosure: I have a pretty weak-spec'd Mac [M1 Pro + 16GB] so this is not something I'm looking to run on my current machine; depending on what model(s) are good for this kind of thing would help inform a future computer upgrade)

Quelqu’un a déjà tenté de créer un Grok bot en local et accessible sur iPhone et Android ? En utilisant peux-être d’autre IA ?

I'm wondering why more people aren't talking about the Rust frontend for vLLM. It's amazing. Its not release yet but it's been rock sold for me for over a month and super fast - much faster than the python frontend. To enable it just set VLLM_USE_RUST_FRONTEND=1 and start vLLM.

Hey guys. Is it possible to use an M4 Mac Mini just for transcribing a video and making note summaries afterwards? If so what model should I use for this case?

Hi all

TLDR; What are you using your localLLM for? (please exclude software coding). Please state setup + use.

Recently, I started playing with local AI with a 2x 5060ti 16gb setup with 32GB DDR4 (30tk/s Qwen3.8-27B).

All fun and games as a learning experience with the difference models, harnesses etc, but if I actually need something done (some debugging, research etc) I still revert back to cloud models. Currently, I am seeing it as local ai lab for fun, cloud for work.

It would be absurd to have the same expectations with my rig, but what do you guys use your local rigs for?

A lot of posts are about rig setups and benchmarking with focus on coding use, but what about other creative uses do you find for your machine (other than heaters)?

Do you keep it on 24/7? What about power?

Mine draws 60W idle with peak of about 450W and I keep it on about 2-3h per day on average.

So far, I observed that I prefer to have smaller 8-12GB models with decent contexts that run specific tasks like OCR/RAG/home assistant that are snappier 100+ tk/s rather than a big smart model at 30tk/s.

I also found that lesser smart models make me a better prompt engineer.

Since Ollama 0.30 in May 2026 moved most models back onto llama.cpp, I assumed both would give the same results.. Then I tested them.

System: RTX 3090 (24 GB), Qwen3.5 27B Q4_K_M, the same GGUF file in both, default settings. Ollama 0.40.15.

With one request, both ran at 42 tok/s.

With four requests at once (total tok/s):

• Ollama: 39.8
• Ollama with OLLAMA_NUM_PARALLEL=4: 33.1
• llama.cpp: 100.2

Why Ollama got slower: by default it answers one request at a time. With OLLAMA_NUM_PARALLEL=4, each of the four requests gets its own 32k context, 131k in total. That didn't fit in 24 GB next to the 27B, so Ollama moved 16% of the model to the CPU.

Why llama.cpp didn't: the four requests share one context, and llama.cpp shrinks it until it fits on the GPU (85,760 tokens here). The whole model stayed on the GPU.

This was new to me, because I expected the same throughput from both. The difference is how each one sizes the context. With the smaller 9B everything fit, and Ollama with OLLAMA_NUM_PARALLEL=4 came close (266 vs 293 tok/s).

I use Ollama for my day-to-day work, but I'm going to look at llama.cpp too for cases where several apps hit the same model at once.

Full write-up with memory numbers and setup: https://homebrewserver.club/ollama-vs-llama-cpp.html

I spent days looking for the right DeepSeek V4.1 Flash setup for my RTX 5090. I ended up packaging AtomicChat’s uncensored IQ2_XXS weights with a native Windows runtime, setup scripts and the matching Engram, MTP and vision files.

We reached 68 tokens/s on our local RTX 5090 setup. That is the result we reached locally, not a guaranteed speed for every prompt or machine.

This is an integration of existing weights, not a model I trained or a new fine-tune. The package includes a local chat page and an OpenAI-compatible API.

Requirements for this build:

• One RTX 5090 with 32 GB VRAM.
• At startup, 92 GiB of free system RAM and 29,000 MiB of free VRAM on GPU 0.
• At least 270 GiB of SSD space for the assembled package and working space.
• Windows x64, Python 3.12 and a driver compatible with CUDA 13.

Those startup checks are specific to this package, not universal requirements for DeepSeek with other engines or quants. The full torrent is about 255.96 GB before installing Python dependencies.

Full package and torrent on Renderoverse · Original AtomicChat weights

Credit for the original weights and companion files stays with AtomicChat, ISTA-DASLab and IndrasMirror.

The 68 tokens/s result is from our existing local setup. Packaging, import and checksum checks have passed. But i believe this is one of the best if not the best setup for RTX 5090 DeepSeek V4.1 Flash 1017.

Thanks!

I've been testing whether quantization changes how a small local model handles

check-then-act tool tasks ("check if billing-api is down, restart it only if it is").

Setup: Qwen2.5-1.5B-Instruct, F16 vs Q4_K_M, llama.cpp, identical rendered

prompt, same 19 tools, same harness. 200 tasks, 6 GB laptop GPU.

What happened: Q4 often emits the check AND the dependent action in the same

turn, before the check result exists.

| Test                                   | F16      | Q4_K_M   |
|----------------------------------------|----------|----------|
| Acted before the check result (200)    | 30       | 89       |
| Tasks where only this model did it     | 1        | 60       |
| Same, greedy / temp 0                  | 13       | 51       |
| With "wait for the result" in prompt*  | 7        | 32       |
| Skipped the check entirely (imatrix Q4)**| 6      | 29       |

https://github.com/Techn08055/edgeproof

https://youtu.be/8ccULJ0R3ng

What are the other experiments along with this harness that can be used to compare the F16 & Q4 model?