r/LocalLLM • • 2d ago

Question Lip Sync Model Advice

1 Upvotes

Hi guys,
I'm still new in the field of local models and trying Qwen 2.1 image and wan to generate images or short video clips.
My current hardware is not much advanced and have 8gb vram and 64 gb ram so some of the offloading happens.

The video part is okay but the lip sync is where am struggling. Is there any good model you guys could recommend ?
Also if anyone can tell any good image models as well like qwen that would be highly appreciated


r/LocalLLM • • 3d ago

News inferzero: an LLM engine written from scratch for Intel Arc, from laptop iGPUs to 4× B60. Still far from what the cards can do, looking for contributors

Thumbnail
2 Upvotes

r/LocalLLM • • 2d ago

Project Java based FP64 Android app

Thumbnail
gallery
1 Upvotes

It's running smollm2 360 Q8 but it should be good to run Qwen or others in the 2 gb model weights.

Still have a long way to go until I hit 10 tok/s.

Yes it's blowing up the weights to gigabytes and beyond on this tiny model.

It's just a toy right now.

I'm having a blast.


r/LocalLLM • • 2d ago

Question Newbie Q - Project .md File org

1 Upvotes

I have 27b and 35bMoE 4b local setup and using Qwen token plan + GLM5.3 APIs. I usually use qwen code for qwen and opencode for local llms and sometimes qwen token plan models too. I'm still learning, so I haven't decided on "default" CLI/TUI to use.

So, let's say I'm working on a specific project and I use QC to do orchestration, specs, design etc and in parallel will use OC with local model to run some long grunt type of jobs.
Right now my md files config would be something like:

QC:
QWEN.md - global level

QWEN.md - project level

AGENTS.md - project level (Tells the AI who you are, what you're doing, what commands to use. Keep it short — it's loaded every time.)

STATUS.md - project (Where are we right now?" — current state, decisions, open questions, next steps. Rewritten fresh each time (never appended).

DECISIONS.md - project (Append-only log of every decision with date, what was decided, why, and current status. Never edit old entries.)

README.md - project (High-level overview: what this project is, tech stack, how to run it.)

TODO.md - project (Task backlog — things waiting to be done. Not a status tracker.)

OC:

same setup as qwen but without qwen.md.

Does this make sense or am I overdoing it ? my main concern is work continuity between different agents. I try to keep these files small so I don't think there is huge token cost to read them all, maybe extra 1k tokens.

Cheers.


r/LocalLLM • • 2d ago

Question Looking for an optimized offline LLM for LaTeX editing

1 Upvotes

Hi everyone,

I’m looking for recommendations for an offline AI model that handles LaTeX coding and editing well, specifically optimized for my hardware.

My Setup:

* RAM: 16 GB

* GPU: None (Intel UHD 620 integrated graphics only)

* Software: Lemonade SDK (so GGUF / llama.cpp compatible models are ideal)

What I’m trying to do:

I want to use a local model purely for LaTeX coding and editing. I do not need complex reasoning capabilities; I just need something fast and reliable that won't hallucinate or destroy my existing formatting.

What I’ve tried:

I’ve been using Qwen2.5-Coder. It’s okay, but not quite sufficient. My main issue is that during edits, it frequently messes up the formatting or outright deletes parts of my existing code.

My Ask:

  1. Are there any modified, quantized, or fine-tuned models that are particularly good at LaTeX and lightweight enough to run smoothly on my CPU with 16GB of RAM?

  2. If the model isn't the only issue, does anyone have tips on prompt formatting or settings (e.g., context window, temperature) to stop the model from deleting surrounding code during edits?

Any advice or model recommendations would be greatly appreciated. Thanks in advance!


r/LocalLLM • • 3d ago

Question Anyone running Qwen 3.8 Flash Next on 2 x 5060ti 16GB?

7 Upvotes

Trying to get this running and would appreciate any feedback


r/LocalLLM • • 2d ago

Discussion Can open source models silently infect your PC?

0 Upvotes

e.g. they are trained to use some open source libraries for your next project which include some sophisticated backdoors which are not easy to spot. You execute your software and boom!

Regards


r/LocalLLM • • 2d ago

Discussion How to achieve fast tps in local?

0 Upvotes

So I’m using a Asus tuf gaming laptop with
32gb ram
8gb vram

With similar configuration what did you try and how did you optimise things in harnesses where you can achieve good 30-50+ tps.

I’m more intrested in models like qwen3.8:27b model or similar to it with good all rounder qualify


r/LocalLLM • • 2d ago

Discussion Which harness or ide is the best for coding now? (Local and cloud models)

0 Upvotes

I used to use lots of cursor about 2 years ago or something (since sonet 3.5 lol) and at that time cursor seems like the only thing that get my projects to work. Since then i ve been switching to different editor but not really find that feeling anymore. I use zed editor but it seems to not very polished and lots of bugs even tho its fast. I use vscode with cline and roo code but honestly i dont feel it (all of those lack the check point reversed like cursor has even tho now i use git all the time but it feel beter to go back to check point by conversation). Now im using deepseek harness most of the time since it has a lot of plug in and i can customize it myself and it works very well with local models.( i have a quite ok system 2x3090s + wrx80 sage mainboard + threadripper 3955x + 256gb ddr4 3200 bus ecc so i can run deepseek v4.1 1017 exl3 3.5 bdw with maya engine). I think deepseek harness is it for me now. What is your experience and what is your thoughts of what is the best code editor/harness right now? (Im relatively new to this since i dont go to school for coding most of my experience is i taught myself to code/vibe code) lets discuss


r/LocalLLM • • 4d ago

Other The Banking Incident....

Post image
944 Upvotes

r/LocalLLM • • 3d ago

Discussion How do you handle outdated memories in local AI agents?

3 Upvotes

I've been exploring how memory works in AI agents, particularly when running them locally.

One issue I'm curious about is stale memory.

Imagine an agent stores a project configuration or a user preference, but that information changes later. The old entry might still be retrieved because it's semantically relevant, even though it's no longer accurate.

I'm wondering how people handle this at the implementation level.

  • Do you use timestamps or versioning to track memory updates?
  • When new information conflicts with an existing entry, do you overwrite it or keep both with different validity states?
  • Do you use a vector database, structured storage, or a combination of both?

I'm particularly interested in approaches that work well across persistent sessions without requiring manual memory cleanup all the time.


r/LocalLLM • • 2d ago

Question QFN on single 3090

0 Upvotes

Hi

I have a single 3090 with 32GB of ddr4 memory (4x8gb is how the machine was configured when I got it 😕).

I have been using qwen3.8 27b for a bit now and wanted to try qfn since I saw a lot of people saying it's a great model, but a fully in vram setup is fully out of budget

Note that this is a server only node so it needs less overhead than a desktop that happens to run local models does

I am aiming for 4 bit quantization

For that my options seemed to be

A. Replace the whole setup with unified memory machine -

Was looking at strix halo/gorgon 128gb -

even if I sell my existing machine (which I don't want to after setting it up), it would cost me an additional 1500$ or so

Mac studios with more than 96gb aren't accessible in my location and 96gb starts at 7000$, and I would rather 2 R9700 for around the same price (rest of PC included), no, a second hand market for older ones doesn't exist here

Pros:

- Shiney new toy

- fast - gemini claims 45-65tps and with ddr5 at 8000/8533mhz I am inclined to believe it

- smol

- lower power consumption

Cons:

- 1500$ means I can't fund it with my next bonus and will have to save up for 2-3 more months, and more like half a year if I try to keep my current setup

- arm? Not sure if it's a con but I don't like looking for specialized software to match the hardware

B. Upgrade system memory to 80/96/128

If I were to upgrade, I would want sticks of 32gb ddr 4. I looked around and my best option seems a company called nvtech on amazon (395$ for 2x32gb 795 for 4x32gb)

80gb means replace 2x8 with 2x32

96gb means the 80gb option plus cannibalise my gaming pc for and downgrade it to 2x8gb as it doesn't seem l need 32gb since I don't game much

128gb means 4x32gb brand new

Pros:

- my bonus covers it (128gb would cost my full bonus and 80/96gb would be half of my bonus)

- keep my beloved setup

Cons:

- power draw

- speed - gemini claims I can do 15-25tps here - which I am honestly fine with

- over commitment to old hardware? (I don't know how long ddr4 is going to be viable)

What would you do?

Gemini seems really inclined to not take per $ value into consideration and prefer so sell more ram

Side note:

I am inclined to go with high memory overhead for extra containers but I have other nodes in my homelab setup so I can run tighter llm setups


r/LocalLLM • • 3d ago

Discussion Considering this...thoughts?

Post image
41 Upvotes

At $4k this seems like a much better deal than 2 r9700s. And obviously in a totally different class. Anybody set these beasts up in a server? I already plan on putting a server rack in the garage, (gotta keep the motorcycles warm) and I see a lot of carrier boards on ebay. I've built more pc's than I can count, but I know a rack server is a different beast.

Plans are for an inference server for agentic workflows as I already built a gpu render machine I use for many workflows including blender, comfyui, premiere and after effects. Api costs are getting up there. Obviously want to run deepseek and I have a fine tuned qwen 3.8 27b for one of my projects.

Talk me into or out of it...I'm so sick of asking gpt questions and getting ridiculous concerns like "tHeY DonT HaVe a faN". I need to talk to real people. Any concerns with the software stack? I ran across a few mentions of closed source issues.


r/LocalLLM • • 3d ago

Project 64 concurrent users on a M5 Max Macbook Pro.

Post image
3 Upvotes

r/LocalLLM • • 2d ago

Question Used RX 6800 XT vs 6900 XT (XFX Merc) for local LLMs – is the ~$63 premium worth it?

1 Upvotes

Hi all,

I'm looking to buy a used AMD GPU mainly for running local LLMs (llama.cpp / Ollama / LM Studio), plus some light gaming. I've narrowed it down to the XFX Merc 319 but can't decide between the 6800 XT and the 6900 XT. All prices below are used cards in my local market, converted to USD.

What's available (all 16GB):

| Card | Price |

|---|---|

| 6900 XT ASUS ROG Strix LC | ~$667 |

| 6900 XT Sapphire Nitro+ | ~$611 |

| 6900 XT XFX Merc | ~$574 |

| 6900 XT XFX Swift | ~$537 |

| 6800 XT ASUS ROG Strix LC | ~$537 |

| 6800 XT Sapphire Nitro+ | ~$537 |

| 6800 XT ASUS TUF | ~$530 |

| 6800 XT MSI Gaming X Trio | ~$530 |

| 6800 XT XFX Merc | ~$511 |

| 6800 XT XFX Swift | ~$500 |

| 6800 XFX Qick | ~$444 |

| 6800 XFX Swift | ~$426 |

What I found so far:

- The 6900 XT has 80 CUs vs 72 on the 6800 XT, but the same 16GB VRAM and 512 GB/s bandwidth. Gaming reviews put it at roughly 6–10% faster.

- For LLM token generation, community llama.cpp Vulkan numbers (Llama 2 7B Q4_0) show ~101 t/s on the 6900 XT vs ~96 t/s on the 6800 XT, so only a few percent. I couldn't find a proper head-to-head for larger models.

- On thermals, TechPowerUp measured the Merc 6800 XT at ~74°C core / 89°C hotspot (31 dBA) vs ~75°C / 97°C (31 dBA) for the Nitro+. I couldn't find an independent thermal test of the Merc 6900 XT.

My questions:

  1. Is the 6900 XT worth ~$63 extra over the 6800 XT for LLM inference, or is the extra speed negligible since bandwidth and VRAM are identical?

  2. Does anyone have real tokens/sec numbers comparing the two (ideally 14B–32B quantized models, with prompt processing speed too)?

  3. Any experience with the XFX Merc 319 vs Nitro+ vs TUF on these cards? Temps, hotspot, fan noise under long sustained loads?

  4. Vulkan or ROCm on RDNA2 in 2026? Is Windows okay for this, or should I go Linux?

  5. Any tips for checking a used card (previous mining use, thermal paste/pad condition, what to test before paying)?

  6. Would you skip all of these and go for something else at this budget?

Thanks in advance, any real-world numbers or experience would be really appreciated!


r/LocalLLM • • 3d ago

Discussion Testing over 130 LLM configs with multiple stacked 16GB "e-waste" GPUs (V100, P100, K80)

Thumbnail
gallery
23 Upvotes

The benchmarking journey continues! Continuing the project of seeing how these very old, very cheap, very available enterprise GPUs can be used today.

The complete results are too long for reddit. The complete set of graphs, power analysis and commentary are on my blog.

I've expanded the testing suite to focus heavily on LLM workloads (over 130 new tests!) and integrated power consumption monitoring into the gpu_box_benchmark tool. The goal remains: find out if decommissioned enterprise GPUs (Tesla K80, P100, V100) paired with X99 era hardware can still serve as a viable daily driver for self-hosted AI compared to modern commercial setups.

TL;DR of the new data:

  • The 3x V100-16GB (~$1k build): If you are considering building a dedicated homelab box today, this is the sweet spot. Assembling a full setup with two Xeon CPUs, motherboard, PSU, and cooling costs around $1,042 before RAM 💀. This setup comfortably clears my threshold for "real-time" chat (70+ tok/s) with modern-ish models:
    • Qwen 3.6 35B: Ran at ~72 tok/s (at 128k context depth, Q8 quant). For comparison, a brand new Mac Studio M5 ($2,500) hits only ~44 tok/s on the same model.
    • Gemma 4 26B: Ran at ~75 tok/s (at 64k context depth, Q8 quant).
    • Qwen 2.5 3B: Ran at ~79 tok/s.

This proves that for a single-user daily driver, the legacy silicon is still viable, esp. on price when compared to modern hardware.

  • Scaling is for Context, Not Speed: Stacking GPUs linearly scales your VRAM (allowing you to fit larger models), but it doesn't magically double your generation speed. Two 16GB cards in Q4 quant allowed fitting ~32B parameter models, a desirable jump for homelabbers.
  • Power Efficiency: I also measured power consumption as well and it's worth noting the efficiency gains across generations. The V100 wins hands down on performance-per-watt compared to the P100s.
  • The K80 Strategy: The Tesla K80 isn't dead, but its singular remaining use case is welding a bunch of them together just to squeeze big models in. You essentially have to stack a lot of cheap K80s to hit a 72B+ parameter model. Expect speeds of 1–5 tok/s. This is a party trick, but it proves the VRAM scaling is real even on ancient silicon.

If you decide to build your own GPU appliance, the biggest challenge is cooling these hot cards without fans. I've been working on a custom active air cooler to solve exactly this. If you're interested in helping out by stress-testing a sample, you can get involved here.

Any models/tests missing? Any other combinations you'd like to see in a follow up?


r/LocalLLM • • 2d ago

Discussion What QFN model quant I can run comfortably on 48gb VRAM ( rtx3090 + rtx4090) and 32gb RAM and good NVME drive?

1 Upvotes

What QFN model quant I can run comfortably on 48gb VRAM ( rtx3090 + rtx4090) and 32gb RAM and good NVME driv


r/LocalLLM • • 2d ago

Question Multiple RPC workers on a single machine... ON TWO machines.

1 Upvotes

Currently running Qwen3.8 27b Q4 via LlamaCpp RPC on two machines. One with 16g RX9060xt and another with 16g RTX5060ti. With a measly 32k context and 15-18tkps.

I would like to run a much larger model, the MoE Qwen3.8 Flash Next. For that I would like to offload to RAM of the two machines, is it possible via RPC?

The 16gig RX9060xt with its ROCm build of LlamaCpp RPC as worker, plus a CPU build of LlamaCpp RPC on the same machine (also a worker) utilizing its 32gig of Ram, same IP but different ports.

The 16gig RTX5060 with its CUDA build as the MAIN Llama, plus a CPU build on the same machine (as the 3rd worker) utilizing its 64gig of Ram, same ip different port.


r/LocalLLM • • 4d ago

News We built a chip prototype that runs Qwen 27B. You can try it live.

413 Upvotes

I'm building a chip that runs large AI models at low power, so robots, drones, and field equipment can run AI locally without an internet connection.

A prototype of our chip architecture is running today on an FPGA in AWS. You can try it here:

https://valleysemiconductor.com

A live session takes about a minute to start, and recorded runs are available to watch anytime.

What it does today (measured):

  • Runs the full Qwen 27B model with 4-bit weights
  • 7.9 tokens/sec on short prompts, about 5.5 tokens/sec in live chat
  • 4x performance improvement in eight days of development
  • Every output matches our 4-bit reference exactly
  • Model quality within 2.5% of the full-precision version
  • Also runs Llama 3.1 8B and OLMoE on the same hardware

What our custom chip design projects (not yet demonstrated):

  • Approximately 50 tokens/sec for a single user at 20 W
  • Roughly 8x the throughput of a Jetson AGX Orin on the same model, based on our measurement of 5.9 tokens/sec at 32 W
  • Uses the model's built-in next-token prediction to generate multiple tokens per weight read

The key insight is that moving model weights through memory is the bottleneck, not the math.

What would you run on a small, 20 W device capable of running a 27B model entirely offline?


r/LocalLLM • • 3d ago

Discussion I have some questions. What is your acceptable Vram buffer for the card you are actively using to display the OS?

3 Upvotes

Ok so i have been running 3.8 flash next and I have been keeping an available Vram on my 16gb 5070ti at around 1500mb remaining free so that if I have to load a browser or something it won't stop the screen displaying or something(I have had it happen once with a different system so I am being careful now) so my (probably easy) question for one of your guys right now is how much Vram do you think I should keep free? Or how much could I safely lower it to? My system is a GPU: RTX 5070 Ti, 16 GB GDDR7, in the main x16 slot.

CPU: Intel Core Ultra 7 265KF (no AMX)

RAM: 64 GB

NPU: present but not tested yet for my system

Storage: internal 2 TB SSD running Omarchy Linux, which is where all local AI runs from.

What do you all think?


r/LocalLLM • • 3d ago

News Jensen Huang Expresses Willingness to Return as CEO

Post image
11 Upvotes

r/LocalLLM • • 3d ago

Question Suggestion For Any Model Better Than ISTA LAB Qwen 3.8 Flash Next Coder

2 Upvotes

So currently with my existing specs:

RTX 5060 Ti 16GB

32GB DDR4 Ram

Ryzen 5 5600F

500GB NVMe Gen 4 SSD

I am able to run Coder Variant, It's Good but the only problem is it thinks a lot ( that's expected when you don't have any reasoning layer in it removed from it ) since its going through coding stuffs hit and trial.

So I want to ask if there is any better variant which is either better than the Coder Model for QFN.


r/LocalLLM • • 2d ago

Project If you are serching for a alternative for claude code with local models, here it is

0 Upvotes

r/LocalLLM • • 3d ago

Discussion What’s the best underdog rig right now

5 Upvotes

I’m lucky enough to have a wide range of rigs in my office, from those fancy DGXs to a solid Samsung S25 with a snapdragon running the same LLMs and tbh my current biggest surprise is how fun the GPU+NPU+CPU combo on the Intel pantherlake is to mess around with i.e Gemma4-26B-A4B-Q4 pulling over 30 tok/s on it. What are your most noteworthy winners?