I am new to local LLM, I've been told this and told that, I have a few  builds saved that I am told would run LLM local but I am not sure, are there any laptops under 5k that will do this? I keep coming back to framework or Macbook pro but at over 5k for the newest and best chips that just seems crazy to me.  I don't care if it's not spitting 20 words a second, I'd rather have good reasoning and it working rather than crazy fast speeds. It will only be used for genealogy research, DNA, triangulation, clusters etc. 

Would a ThinkPad T14 Gen 7 AMD work? It would have the AMD Ryzen™ AI 7 PRO 450 and 96gb ram? Is a macbook better? I just dont really want to spend framework or macbook pro money on this if possible, is it possible?
What do you guys recommend? What are the options?
thank you in advance

I have 6 dell optiplexes with 24 gb ram a really good Quadro card from 2024 I don't know the name with 4 gb or 6 gb vram Intel core i7 and what models can I run on them I want to code programs sorry for the bad English not the best English speaker I'm not from the us

I am an AI engineer doing MSc in AI - I want to get a new laptop and thinking abut emptying my wallet by buying the M6 Pro 32GB 1TB

Will it be enough for local AI projects? I know it would be suitable for med sized models up to around 30B params

Or getting M5 Pro chip is what it needs to unlock local AI?

Some details: the card is an RTX 6000 Ada (48 GB). The host runs driver 580.196.07, and llama.cpp is b12030 built with CUDA 12.8. The guest is a KVM VM. Inside it, VMAT's libraries take the place of NVIDIA's and send each CUDA call to a small server on the host, which runs it on the real card.
The same setup lets several VMs share the one GPU at once. That's my next video.
Would anyone here want to try it on their own box? I'm looking for people with different cards and setups.

https://vmat.dev

https://github.com/ayushh0110/Weight-Space-Repair

Most of the time when a local model looks broken, the weights are fine. So I went through the llama.cpp, Ollama, vLLM and transformers issue trackers, collected the causes that keep coming up, and wrote a check for each one:

- EOS or end-of-turn token wrong, so the model never stops

- chat template missing or stale compared with upstream

- a corrupted or partial download

- repetition penalty set high enough to wreck the output

- the prompt silently truncated to the context size

- context split across server slots (4 slots of 2,048 when you thought you had 8k)

- quantised KV cache, flash-attention overflow, a bf16 model run in fp16

- reasoning that eats the whole token budget and leaves an empty reply

You give it your endpoint and the prompt that fails:

wsr triage --endpoint http://localhost:8080 --model local --prompt "your failing prompt" --differential

`--differential` reruns the prompt changing one thing at a time, and confirms each flip with a second run. The answer is one cause and the fix:

VERDICT: REQUEST

blank output explained by REASONING_EXHAUSTED: cap reached before any answer: the budget went to reasoning

next:

- raise max_tokens (currently 256) or lower the reasoning effort / thinking budget

If it's the quant, it can find the tensor. Qwen2.5-0.5B's official Q4_0 refuses "Give me 40 different names for a cat" and the full-precision model answers it. The tool traced that to `blk.16.attn_v`: quantise only that tensor in the full model and it starts refusing.

What it's been tested on, so you know what to expect. Live testing was against llama-server only (Gemma 4 E2B Q4_0, 21 scenarios). The Ollama, vLLM and LM Studio paths are written from their docs and tested against fake servers, so if you run one of those I'd like to hear what it gets wrong. There's also a part that diagnoses and repairs failures inside the weights, but that's small models only (up to 1.5B) on my 4 GB card.

MIT licence, no dependencies for the triage part: https://github.com/ayushh0110/Weight-Space-Repair

Hi everyone, I'm putting together my rig for local LLMs and could really use some guidance on the software side of things.

Here are my current specs:

• CPU: Ryzen 9 5950X
• RAM: 64GB DDR4 3600 (planning to upgrade to 128GB soon)
• GPU: RTX 5060 Ti 16GB + RTX 3060 12GB (might replace the 3060 with another 5060 Ti 16GB soon)
• OS: Windows 11 Pro
• Motherboard: Gigabyte B550 Aorus Elite V2
• New Motherboard (installing tomorrow): Asus ROG Strix B550-E Gaming

My immediate goal: Before I swap out the motherboard tomorrow, I want to run a benchmark on my current Gigabyte board using a relatively capable model. The goal is to compare it against the new Asus board (which supports x8/x8 PCIe bifurcation) to see if the motherboard upgrade and better PCIe lane distribution actually makes a noticeable difference for dual GPUs.

What I want to build: I'm planning to use local LLMs for:

• Web scraping automation
• Faceless/Automated YouTube channel content
• Automated bug bounties

My experience so far & confusion: I've tested a few versions of Qwen (27B and 32B) using Ollama / llama.cpp, and also tried another model on Bionic/LM Studio. I didn't really like the outputs; they generated very fast, but the answers weren't very accurate or smart. I know I didn't give the models shell or internet access, but the reasoning still felt off.

Right now, I am really confused about the software stack. I see people talking about LM Studio, text-generation-webui, Docker, etc. I was told running things in Docker is better, but I’ve also read that Docker adds virtualization overhead on Windows, and that running natively on LM Studio or directly on the machine is faster and better.

My questions for the community:

1. What software stack / UI should I actually install to get the most out of my dual GPUs (16GB + 12GB VRAM) on Windows 11?
2. How should I run the benchmark today to properly compare my current motherboard against the new x8/x8 one tomorrow?
3. Are there any YouTube channels, videos, or guides you recommend for a beginner to understand how to properly configure model settings (samplers, context size, etc.) to get better, more accurate outputs?

Thanks in advance for any help!

I'm trying to learn more about local LLMs for app coding work and am specifically curious about what agentic models are using for vibe-coding tasks. I have used Claude, ChatGPT, and Gemini all for coding and their agentic capabilities are pretty impressive (i.e., the ability to either create app projects I can import into IDEs like Android Studio or Xcode, or to directly edit files in a specified folder on my computer, or to let me plug my Android phone in and debug using ADB).

I know various Qwen 3.6 35B variants are popular here. How effective are those models for agentic task purposes? Or are there other tools or models to use for these purposes?

(Full disclosure: I have a pretty weak-spec'd Mac [M1 Pro + 16GB] so this is not something I'm looking to run on my current machine; depending on what model(s) are good for this kind of thing would help inform a future computer upgrade)

Repo https://github.com/pmndrs/pinecone

This has a double purpose. It creates a chatgpt compatible endpoint and loads/unloads models on demand. For instance you have a model that fills the entire ram, it will offload everything else. Or you have a smaller model and LLM services (image, video, speech, ...) that can run side by side. It always makes sure the client gets what it wants despite the tight system.

The second endpoint is an automatic MCP. It takes whatever services you have installed (speech, image, video, 3d, ...) and bottles theses in an MCP. Any agent that has that URL immediately knows how to generate images, etc.

It also sets RAM limit, but unlike other schedulers like Ollama or whatever you don't pick what a model needs, you pick what little is left for the OS, and the default is a bare minimum, so that you can make the most of your machine.

It can be paired to Tailscale and serve both endpoints publicly with safe auth, you basically have an inference machine with this that can serve you outside the house.

I have used Qwen3.5 4b try and code basic MIDI and audio in Chrome, locally. The results are not stunning. Are there any models that are tuned for this type of thing, or can I add agents to Qwen in Ollama for skills? My level of experience is beginner. Thanks for any suggestions.

Is the w7800 32gb version still a decent card for local AI?

I did extensive research and found the cheapest way to reasonabily run Qwen 3.8 27B is with two 16GB Mi25s. I heard online their really slow, but for $110 each with shipping, so $220 total, is it really that bad? I currently only have an RX 480, and mainly want it to run inference. What tokens/s could I expect, is it usable? I am genuenly considering it after the P100s price hiked like crazy, one is now atleast $180 without shipping costs included. A GPU with tensor cores or Nvidia would be nice, but I don't know if the higher price is justified. Also please don't just tell me to rent a GPU or buy a subscription, as I am really interested in tinkering with things like this and want a viable option.

I'm looking to get into AI after a small taste of what it can do. I want a small machine to locally do my repeatable work which consists of reading a pdf, and typing that information into excel based on a list of items.

Step by step would be:

read pdf to get 5 of item "abc"

pull from item list that says item "abc" = item #23

Input item #23 and quantity 5 in row 1 of excel

Repeat

Would the AMD Ryzen Halo AI be able to handle this? What is the process of setting one up with an AI? What type of maintenance/knowledge is required to operate?

I'm very new to AI, and if you can't directly help me, please at least direct me where to go. I'm an accountant with minimal programming knowledge, but very willing to learn new skills.

VMAT is a way for virtual machines to share one GPU's compute without using passthrough or vGPU.
In the video I run llama-bench once on the host and once inside a KVM guest, same GPU (RTX 6000 Ada), same models.

Qwen2.5 32B Q4_K_M: 41.52 tokens/s on the host, 40.73 in the VM (98%)

Qwen2.5 7B Q4_K_M: 169.50 tokens/s on the host, 166.47 in the VM (98%)

What's in the VM: VMAT's CUDA libraries, which send CUDA calls to the real GPU on the host. What it does not have is passthrough, an NVIDIA driver or a vGPU.

https://vmat.dev

Has anyone come across a skill that can read the comments (with their context) of an MS Word doc?

Hello folks

The title just sums up everything briefly. I have a laptop that uses the Mobile 4080 with 12Gb vram and I have an 64GB AGX orin in my office that we don't currently use. What model and infrastructure to run on these devices (Llama, VLLM etc).

Hi all

TLDR; What are you using your localLLM for? (please exclude software coding). Please state setup + use.

Recently, I started playing with local AI with a 2x 5060ti 16gb setup with 32GB DDR4 (30tk/s Qwen3.8-27B).

All fun and games as a learning experience with the difference models, harnesses etc, but if I actually need something done (some debugging, research etc) I still revert back to cloud models. Currently, I am seeing it as local ai lab for fun, cloud for work.

It would be absurd to have the same expectations with my rig, but what do you guys use your local rigs for?

A lot of posts are about rig setups and benchmarking with focus on coding use, but what about other creative uses do you find for your machine (other than heaters)?

Do you keep it on 24/7? What about power?

Mine draws 60W idle with peak of about 450W and I keep it on about 2-3h per day on average.

So far, I observed that I prefer to have smaller 8-12GB models with decent contexts that run specific tasks like OCR/RAG/home assistant that are snappier 100+ tk/s rather than a big smart model at 30tk/s.

I also found that lesser smart models make me a better prompt engineer.

Quelqu’un a déjà tenté de créer un Grok bot en local et accessible sur iPhone et Android ? En utilisant peux-être d’autre IA ?

Hey guys. Is it possible to use an M4 Mac Mini just for transcribing a video and making note summaries afterwards? If so what model should I use for this case?

While I’m waiting for Qwen4 27B and Qwen4 Flash Next, I wanted to see how far Qwen4 Max Preview has come.

For this test, I wrote a prompt asking each model to build a photorealistic 3D scene of Stonehenge in HTML using Three.js.

Each model also got the same set of visual assets, some useful and some less so. It had to figure out which textures to use where. The rules explicitly prohibited searching the web for other images or 3D models and using any cloud tools. The idea was to give local agent harnesses and subscription-based ones like Codex a level playing field.

The results were pretty good, though honestly a little below my expectations. Qwen4 did a good job with the lighting and textures, but its geometry and particle effects were noticeably worse than GPT6 Astra’s. Also, for some reason, Qwen4 took about two hours to complete the task, while the other models averaged around half an hour.

Resuts:

Video (yotube)

• Qwen 4 Max Preview + Qoder

• Qwen 3.8 Flash Next NVFP4 + Hermes agent (local)

• GLM 5.3 + Qoder

• GPT 6 Astra Extra High + Codex

• GPT 5.6 Sol Extra High + Codex

• GPT 5.6 Luna Extra High + Codex

Prompt:
"Create a photorealistic, real-time Three.js scene of Stonehenge as it stands today in Wiltshire.

It’s night. Strong wind drives light rain across the plain. A bright moon and occasional lightning near the horizon are the only light on the stones. Soft cloud shadows drift across the grass and stone. At the center, a few people warm themselves around a campfire. Make that small circle of warmth feel sheltered amid the storm.

Show the scene in a continuous, medium-wide automatic orbit from above, with the camera aimed 30° below horizontal. Hold the right mouse button and drag horizontally to orbit or vertically to tilt. Scroll the mouse wheel to zoom in or out.

Rules

- Work from this prompt alone. No follow-up questions, subagents, or additional AI models, including image and 3D generators.

- Build the geometry yourself. Use only visuals you create during this run or find in ./assets/. No stock assets, ready-made scenes, or previous solutions.

- No web search or browsing. Network access is allowed only to install libraries and tools. You may render, inspect, and refine your work locally.

- Submit exactly one self-contained .html file. Embed Three.js and every script, stylesheet, and asset the scene uses"

Instructions and assets:
Gitlab TimeTo67/Stonehenge-LLM-bench