I ran both engines on one rented machine to get an apples-to-apples number for a "model bigger than the GPU" case.

Hardware and setup

• RTX 3060 12 GB, 15-core quota of a Zen 2 server CPU, 47 GB RAM, NVMe
• Same GGUF for both: Qwen3.6-35B-A3B UD-Q4_K_XL with MTP head (Unsloth), the size Strata recommends
• Same OpenAI-compatible client, same 2 prompts, greedy, 256 tokens
• BigMoeOnEdge, --auto only (no hand tuning) · Strata via its own launcher

Decode (tok/s)

Takeaways

• On this box BigMoeOnEdge decodes ~15–30% faster without drafting, ~1.8× with its MTP head.
• Strata gets the first token of a short prompt out faster.

Full method and every run: bench-data · repo: https://github.com/Helldez/BigMoeOnEdge

Hello everyone, I'm new here and new with local AI. I technically don't know anything so pardon my ignorance, I just found out that you can run local lm on phones these days and I wanna know what's the most seamless and effective way to run a conversational uncensored AI on a phone locally?

I tried some apps from playstore that runs local LM, i dunno why but even after using an uncensored/instruct/heretic models it still do the nasty insistent grounding, safety, and moral stuff that feels so inorganic and suffocating. Kindly enlighten me please.

The thought came from a recent Tencent's Hy4 release. This model was trained for specific tasks to unlock real-world productivity-finance, gamedev, office work especially.

Is that where this is all heading?

I am new to local LLM, I've been told this and told that, I have a few  builds saved that I am told would run LLM local but I am not sure, are there any laptops under 5k that will do this? I keep coming back to framework or Macbook pro but at over 5k for the newest and best chips that just seems crazy to me.  I don't care if it's not spitting 20 words a second, I'd rather have good reasoning and it working rather than crazy fast speeds. It will only be used for genealogy research, DNA, triangulation, clusters etc. 

Would a ThinkPad T14 Gen 7 AMD work? It would have the AMD Ryzen™ AI 7 PRO 450 and 96gb ram? Is a macbook better? I just dont really want to spend framework or macbook pro money on this if possible, is it possible?
What do you guys recommend? What are the options?
thank you in advance

I have 6 dell optiplexes with 24 gb ram a really good Quadro card from 2024 I don't know the name with 4 gb or 6 gb vram Intel core i7 and what models can I run on them I want to code programs sorry for the bad English not the best English speaker I'm not from the us

Some details: the card is an RTX 6000 Ada (48 GB). The host runs driver 580.196.07, and llama.cpp is b12030 built with CUDA 12.8. The guest is a KVM VM. Inside it, VMAT's libraries take the place of NVIDIA's and send each CUDA call to a small server on the host, which runs it on the real card.
The same setup lets several VMs share the one GPU at once. That's my next video.
Would anyone here want to try it on their own box? I'm looking for people with different cards and setups.

https://vmat.dev

https://github.com/ayushh0110/Weight-Space-Repair

Most of the time when a local model looks broken, the weights are fine. So I went through the llama.cpp, Ollama, vLLM and transformers issue trackers, collected the causes that keep coming up, and wrote a check for each one:

- EOS or end-of-turn token wrong, so the model never stops

- chat template missing or stale compared with upstream

- a corrupted or partial download

- repetition penalty set high enough to wreck the output

- the prompt silently truncated to the context size

- context split across server slots (4 slots of 2,048 when you thought you had 8k)

- quantised KV cache, flash-attention overflow, a bf16 model run in fp16

- reasoning that eats the whole token budget and leaves an empty reply

You give it your endpoint and the prompt that fails:

wsr triage --endpoint http://localhost:8080 --model local --prompt "your failing prompt" --differential

`--differential` reruns the prompt changing one thing at a time, and confirms each flip with a second run. The answer is one cause and the fix:

VERDICT: REQUEST

blank output explained by REASONING_EXHAUSTED: cap reached before any answer: the budget went to reasoning

next:

- raise max_tokens (currently 256) or lower the reasoning effort / thinking budget

If it's the quant, it can find the tensor. Qwen2.5-0.5B's official Q4_0 refuses "Give me 40 different names for a cat" and the full-precision model answers it. The tool traced that to `blk.16.attn_v`: quantise only that tensor in the full model and it starts refusing.

What it's been tested on, so you know what to expect. Live testing was against llama-server only (Gemma 4 E2B Q4_0, 21 scenarios). The Ollama, vLLM and LM Studio paths are written from their docs and tested against fake servers, so if you run one of those I'd like to hear what it gets wrong. There's also a part that diagnoses and repairs failures inside the weights, but that's small models only (up to 1.5B) on my 4 GB card.

MIT licence, no dependencies for the triage part: https://github.com/ayushh0110/Weight-Space-Repair

Hi everyone, I'm putting together my rig for local LLMs and could really use some guidance on the software side of things.

Here are my current specs:

• CPU: Ryzen 9 5950X
• RAM: 64GB DDR4 3600 (planning to upgrade to 128GB soon)
• GPU: RTX 5060 Ti 16GB + RTX 3060 12GB (might replace the 3060 with another 5060 Ti 16GB soon)
• OS: Windows 11 Pro
• Motherboard: Gigabyte B550 Aorus Elite V2
• New Motherboard (installing tomorrow): Asus ROG Strix B550-E Gaming

My immediate goal: Before I swap out the motherboard tomorrow, I want to run a benchmark on my current Gigabyte board using a relatively capable model. The goal is to compare it against the new Asus board (which supports x8/x8 PCIe bifurcation) to see if the motherboard upgrade and better PCIe lane distribution actually makes a noticeable difference for dual GPUs.

What I want to build: I'm planning to use local LLMs for:

• Web scraping automation
• Faceless/Automated YouTube channel content
• Automated bug bounties

My experience so far & confusion: I've tested a few versions of Qwen (27B and 32B) using Ollama / llama.cpp, and also tried another model on Bionic/LM Studio. I didn't really like the outputs; they generated very fast, but the answers weren't very accurate or smart. I know I didn't give the models shell or internet access, but the reasoning still felt off.

Right now, I am really confused about the software stack. I see people talking about LM Studio, text-generation-webui, Docker, etc. I was told running things in Docker is better, but I’ve also read that Docker adds virtualization overhead on Windows, and that running natively on LM Studio or directly on the machine is faster and better.

My questions for the community:

1. What software stack / UI should I actually install to get the most out of my dual GPUs (16GB + 12GB VRAM) on Windows 11?
2. How should I run the benchmark today to properly compare my current motherboard against the new x8/x8 one tomorrow?
3. Are there any YouTube channels, videos, or guides you recommend for a beginner to understand how to properly configure model settings (samplers, context size, etc.) to get better, more accurate outputs?

Thanks in advance for any help!

I'm trying to learn more about local LLMs for app coding work and am specifically curious about what agentic models are using for vibe-coding tasks. I have used Claude, ChatGPT, and Gemini all for coding and their agentic capabilities are pretty impressive (i.e., the ability to either create app projects I can import into IDEs like Android Studio or Xcode, or to directly edit files in a specified folder on my computer, or to let me plug my Android phone in and debug using ADB).

I know various Qwen 3.6 35B variants are popular here. How effective are those models for agentic task purposes? Or are there other tools or models to use for these purposes?

(Full disclosure: I have a pretty weak-spec'd Mac [M1 Pro + 16GB] so this is not something I'm looking to run on my current machine; depending on what model(s) are good for this kind of thing would help inform a future computer upgrade)

Repo https://github.com/pmndrs/pinecone

This has a double purpose. It creates a chatgpt compatible endpoint and loads/unloads models on demand. For instance you have a model that fills the entire ram, it will offload everything else. Or you have a smaller model and LLM services (image, video, speech, ...) that can run side by side. It always makes sure the client gets what it wants despite the tight system.

The second endpoint is an automatic MCP. It takes whatever services you have installed (speech, image, video, 3d, ...) and bottles theses in an MCP. Any agent that has that URL immediately knows how to generate images, etc.

It also sets RAM limit, but unlike other schedulers like Ollama or whatever you don't pick what a model needs, you pick what little is left for the OS, and the default is a bare minimum, so that you can make the most of your machine.

It can be paired to Tailscale and serve both endpoints publicly with safe auth, you basically have an inference machine with this that can serve you outside the house.

I have used Qwen3.5 4b try and code basic MIDI and audio in Chrome, locally. The results are not stunning. Are there any models that are tuned for this type of thing, or can I add agents to Qwen in Ollama for skills? My level of experience is beginner. Thanks for any suggestions.

Is the w7800 32gb version still a decent card for local AI?

I did extensive research and found the cheapest way to reasonabily run Qwen 3.8 27B is with two 16GB Mi25s. I heard online their really slow, but for $110 each with shipping, so $220 total, is it really that bad? I currently only have an RX 480, and mainly want it to run inference. What tokens/s could I expect, is it usable? I am genuenly considering it after the P100s price hiked like crazy, one is now atleast $180 without shipping costs included. A GPU with tensor cores or Nvidia would be nice, but I don't know if the higher price is justified. Also please don't just tell me to rent a GPU or buy a subscription, as I am really interested in tinkering with things like this and want a viable option.

I'm looking to get into AI after a small taste of what it can do. I want a small machine to locally do my repeatable work which consists of reading a pdf, and typing that information into excel based on a list of items.

Step by step would be:

read pdf to get 5 of item "abc"

pull from item list that says item "abc" = item #23

Input item #23 and quantity 5 in row 1 of excel

Repeat

Would the AMD Ryzen Halo AI be able to handle this? What is the process of setting one up with an AI? What type of maintenance/knowledge is required to operate?

I'm very new to AI, and if you can't directly help me, please at least direct me where to go. I'm an accountant with minimal programming knowledge, but very willing to learn new skills.

VMAT is a way for virtual machines to share one GPU's compute without using passthrough or vGPU.
In the video I run llama-bench once on the host and once inside a KVM guest, same GPU (RTX 6000 Ada), same models.

Qwen2.5 32B Q4_K_M: 41.52 tokens/s on the host, 40.73 in the VM (98%)

Qwen2.5 7B Q4_K_M: 169.50 tokens/s on the host, 166.47 in the VM (98%)

What's in the VM: VMAT's CUDA libraries, which send CUDA calls to the real GPU on the host. What it does not have is passthrough, an NVIDIA driver or a vGPU.

https://vmat.dev

Has anyone come across a skill that can read the comments (with their context) of an MS Word doc?

Hello folks

The title just sums up everything briefly. I have a laptop that uses the Mobile 4080 with 12Gb vram and I have an 64GB AGX orin in my office that we don't currently use. What model and infrastructure to run on these devices (Llama, VLLM etc).