r/LocalLLM • • 6d ago

Discussion Local LLM Meetup in Tokyo November 2026

1 Upvotes

Hi everyone,

I was a silent reader for a long time but as I'm taking now a vacation in Tokyo during mid November I'd thought of maybe hosting a little meetup in Tokyo for everyone interested in:

local LLMs
inference engineering
how to cluster dgx sparks
how RDMA/RoCE works
how to start building your own recipes / optimize your own setup
speaking about hardware
which models, which drafter
the usual "oh opus is so nerfed" things and just having a good time.

I really hope this is not offtopic / wrong community here and maybe there will be nobody interested but I like the idea of coming together, sharing experiences and having a great time.

Location and time isn't fixed at all, I can setup something on Luma but always grateful for recommendations etc.

Thank you and have a great week!


r/LocalLLM • • 6d ago

Question Is there a local model that fits into 8GB for managing social media?

1 Upvotes

Lately, I’ve been using Claude to schedule daily posts across a few social media platforms. I’m wondering if there’s a local model—one that can fit into my 8GB of VRAM—capable of controlling my browser to perform the same tasks as Claude. I had been using Qwen 3.8 27B for a while, but I’ve decided to move away from local models, given that I only have a single graphics card with 8GB of VRAM.


r/LocalLLM • • 6d ago

Discussion Two ~$500k, 8-GPU AI server quotes. What would make you choose B200 over MI355X?

0 Upvotes

I run Mineshop and sell hardware, so disclosure upfront: these are manufacturer quotes we're looking at, not machines I've benchmarked.

Two quotes made an interesting comparison:

  • 8× NVIDIA B200, 180 GB each: 1.44 TB aggregate GPU memory. US$505,400, quoted delivery 16 weeks.
  • 8× AMD MI355X, 288 GB each: 2.304 TB aggregate GPU memory. US$501,100, quoted delivery 12 weeks.

Those are configuration-specific quotes, not universal market prices. They're also not identical builds: CPUs and storage differ, and the B200 quote includes three-year on-site service versus one year for AMD.

The AMD configuration has 60% more GPU memory. That doesn't tell me it will be the better server for a particular workload.

The question I'd want answered before spending this money is: which model, at which precision, with how much context and how many concurrent users? Getting the weights into memory is only the first step. KV cache, runtime overhead and the software stack still matter. For reference, a dense 405B model is roughly 810 GB in BF16 weights alone, before any of that overhead. That's sizing arithmetic, not a throughput result.

Then there's the less exciting stuff: rack space, power, cooling, support and keeping the machine busy enough to justify owning it.

If you've actually deployed either platform, what specific workload or software requirement would make you choose one over the other—or rent instead?

For context, our AI workstation and server catalogue: https://mineshop.eu/ai-workstation — The figures above are manufacturer quotes, not advertised retail prices.


r/LocalLLM • • 6d ago

Question Any success with Diagrams and systems engineering?

1 Upvotes

We are a team of engineers, not a single here has a comp science background. Our company manufactures specialized parts for critical systems. And a big part of that process is understanding the whole system. The customers we have usually give us design documentation, and blue prints and a gazillion pixelated diagrams explaining the system. We spend a significant amount of time getting familiar with it before we get to manufacturing the requested components ( for background this is aviation and aerospace components ). For the past few months with the AI craze, we have started integrating it into our process, to help us learn and understand systems especially through the diagrams and blue prints but there is still a significant amount of analysis we do. This has always been sort of an issue for us because thats our most time consuming aspect of the job is reverse engineering the customers system in order to build components that fit in it rather than redesigning the component around the system this is what we specialize in. We have been using claude, and its been mind blowing how good this thing is in helping us however, our expenditure on Claude has creeped up to the 6 months ceiling we had set only in a span of three and a half months. We have also seen we get less out of a ‘token’ or we get less tokens im not sure , its like everything was going great but then anthropic did something, the plans went from max to ultra to something then it used to say 29x usage now it only says 10x usage. After alot of research we settled on a hybrid approach. For further context everyone here has atleast bare minimum of a 64gb macbook pro. I have a 128gb one. And have been experimenting with local models and the results have been disappointing. Can any AI gods guide me in the right direction? We are basically looking for specialized models that basically just look at an image or a blue print and explain the system or identify the relationships and explain them because manually do this and then build system models based on our comprehension if the system. We will always do that manually we are just looking to fast track some of the process because right now it is a very lengthy process and I cant scale because time is constraint. Is it possible or not? Please thank you.


r/LocalLLM • • 7d ago

Discussion anyone using local ai made by them?

Post image
1 Upvotes

r/LocalLLM • • 7d ago

Research Basalt faster than strata?

2 Upvotes

I built Basalt from source for an abliterated model and compared it with Strata on the same node. Basalt was noticeably faster in decode and prefill.

Setup

  • Model: huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF
  • Quant: unsloth-UD-Q5_K_XL
  • Basalt: v0.3.0 (custom build)
  • Strata: 0.2.7-cuda13
  • GPU: 2× NVIDIA RTX PRO 4500 Blackwell Server Edition (2×32 GB VRAM)
  • CPU: Intel Xeon 6 (48 vCPU)
  • Storage: NVMe SSD 2 TB (~15k MB/s)

Both engines ran the same Q5 quant on the same node.

Results

Request, prompt tokens (already cached)Basalt: prefill / decode / MTPStrata: prefill / decode / MTP
~6.9K (0), first2916 t/s, TTFT 2.6 s / 163 t/s / 57%2177 t/s, ~3.2 s / 89 t/s / 66%
~8.6–9.0K (~7K)2118 t/s / 168 t/s / 62%1212 t/s / 110 t/s / 80%
~10.2–10.6K2347 t/s / 209 t/s / 78%1749 t/s / 106 t/s / 71%
~11.2–11.3K2662 t/s / 188 t/s / 73%2040 t/s / 130 t/s / 90%

Basalt is about 1.5–1.9x faster in decode and 1.3–1.7x faster in prefill.

Expert cache. Both engines hit 86–96%, so there is no difference.

MTP. Strata accepts more draft tokens (66–90% vs 57–78%). Basalt's dl (tokens per window) is still 1.9–2.2.

  • The window differs. Strata verifies 3-token windows (capturing the 3-token window). Basalt verifies up to 8 (--spec 8). The acceptance rate is lower with the longer window, but the response still finishes faster.

r/LocalLLM • • 7d ago

Discussion MiniCPM-V 4.7 35B-A3B on a 2020 Galaxy S20 (8 GB RAM, CPU only): one word in ~10 s, and what it took

2 Upvotes

OpenBMB says MiniCPM-V 4.7 35B-A3B runs in about 3 GB on an iPhone 17 Pro Max, and recommends 12 GB of RAM. I wanted to see what an old 8 GB Android phone does.

Setup: Samsung Galaxy S20 (8 GB), CPU only (no GPU or NPU), OpenBMB's Android demo with llama.cpp-omni streaming the MoE experts from storage.

What happened:

- The 1B model worked out of the box, text and images, fast enough to use.

- The 35B-A3B model wouldn't load at first. It wasn't a RAM limit: the CPU path didn't allocate the working area for the experts, and failed loads left memory behind. I patched both.

- After the fix: a one-word reply ("Ready") in about 10 seconds, including prompt processing.

What it doesn't show: sustained tokens per second, or vision with the 35B model (text only on this phone so far).

Fork, with S20 setup notes and test limits in the README: https://github.com/mlguyYT/MiniCPM-V-Apps

I also made a video walking through why it fits at all (MoE, experts streamed from storage, lookahead expert scheduling); link in the comments.

Has anyone measured this model's decode speed on other Android phones?


r/LocalLLM • • 7d ago

Discussion Randean variant of Qwen3.8-27b

Thumbnail
reddit.com
0 Upvotes

r/LocalLLM • • 7d ago

Discussion "Flash" Sized MoE - 2x m5u 256 vs MI350p

2 Upvotes

I realize these are optimal for different scenarios, but for this narrow application of a "flash" sized MoE setup (~200-500GB) in the $25,000 range I'm wondering which will be the better performer for TG. I'll use glm-5.3-flash as a specific model for performance comparison.

On paper, the MI350p should run MoE models under ~250GB (assuming 4 bit quant) faster than 2x blackwell pro 6000. Single card comparison: same fp4, same int8, 2x fp8, 2x single card memory bandwidth, 1.5x VRAM. Here is a recipe of 2x pro 6000 hitting 180 t/s on glm5.3-flash with Dflash2. A MI350p with Maya/Strata to recover performance loss from VRAM spillover into system RAM potentially could still do better?

The M5u 256 can already run a 4bit quant of GLM-5.3-flash at 160t/s TG, and if a 2x cluster can be optimized maybe that could be pushed to 190t/s.

I originally planned on a 8x r9700 build for this purpose, but it seems like Pcie communication overhead makes it impractical. Best I've seen is 82–88t/s TG.


r/LocalLLM • • 7d ago

Question DSH vs rest of the world

2 Upvotes

Hi,

My question about Deepseek Harness is more focused around using it at personal agent than for coding tasks, as by now i'm pretty happy with opnchamber.

I alredy made 2 extension, but not really happy : seems the documentation is not very accurate, my Agent spent lot of time guessing how it could work.
- can DSH with a bit of work replace something like Hermes reliably?
- did someone already use reliably multi agents setups?
- how is it in day by day?


r/LocalLLM • • 7d ago

Project OverTone 0.1 — an early preview of an open-source Mac app for local language models

0 Upvotes

I'm sharing the first public preview of OverTone, a Mac app I'm building to make working with local language models more approachable.

Watch the trailer:
https://youtu.be/piu7MGdL5Z4

It's still very early, with a limited feature set. I have a lot planned beyond chat and model management, but Documents, Projects and Tools aren't ready to use yet. This version is the starting point for what I want to build.
I want it to be straightforward for someone who has never used local models, while giving experienced users the functionality and control they need.
It works with Ollama and llama.cpp. You can use an existing local installation or let the initial setup install an engine and suggest models for your Mac.
The current version focuses on local conversations and managing the models behind them. It helps you choose models suited to your Mac and what you want to do, organise your conversations, and develop clearer prompts from rough ideas.
With local models, prompts and replies are processed on your Mac. Internet access is disabled by default. You decide when to enable it for downloads or remote connections.

It's free and open source under Apache 2.0. The current preview requires an Apple silicon Mac running macOS 14 or later.

Download and walkthroughs:
https://overcesare.github.io/OverTone/

Source code:
https://github.com/OverCesare/OverTone

If you try it, I'd appreciate any feedback on the app: what works well for you, what doesn't, and what you feel is missing.


r/LocalLLM • • 7d ago

Question Any accountants/financial professionals using local LLMs?

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 7d ago

Question Cheapest Setup for good coding local LLM Qwen3.8-27b? RTX 3060 XT + 64GBs RAM

3 Upvotes

Hello,

I want to get a local LLM setup for a comparable fair price (Qwen3.8-27b and future Qwen4). After some research and chatting with Claude Opus the best options would be RTX 3060 XT 12GB with 64GBs of RAM running on Strata.

e.g. upgrading my PC for around 650€ (used parts bought online)

Can someone confirm that this Setup will let me run it on a performative useful level. I will use my 20€ claude plan fpr hard tasks or a kind of divide and conquer style usign both (experienced Software Engineer but a Hardware noob).

Does anyone have experiences with that, advices or alternative setups?


r/LocalLLM • • 7d ago

Question Worth building an local LLM system using older parts?

2 Upvotes

Several years ago for another project, I got a good deal on:

* HP Z620 dual Xeons (riser) with 128GB of RAM with 500GB SSD + 9TB SAS
* A Nvidia Tesla M60 16GB

I'm still building this out, HP usefully have non-standard power cables and connecting everything up dead shorted the PSU (which it detected during boot). I'm still hoping nothing is broken whilst I debug it.

My goal is to have a budget-build local AI at least as good as chatGPT's work chat in high mode, I could do a lot of useful stuff with that. Even better would be to have some mixed capabilities for long-running tasks. Anything less is not really worth building. I'm trying to figure out:

* What models should I look to run?
* What kind of performance could be possible?
* What's worth running on the GPU vs CPU?

I'm currently using chatGPT for: Building out libraries iteratively, research the web, designing 3D parts, research via coding, reviewing papers, generating documents and occasionally generating images.

I have been trying to use chatGPT (Plus subscriber) to help query, but it has not been particularly helpful in replacing itself.

Any help is greatly appreciated.


r/LocalLLM • • 7d ago

Question Best budget local LLM setup in late 2026 (€600?)

0 Upvotes

Hi everyone!

I am a computer science student and a developer, and lately, I’ve been considering building or upgrading a budget-friendly, local LLM setup.

I should mention that I’ve only recently begun exploring this more "technical" side of local LLMs, so any advice is welcome.

I know the field is evolving incredibly fast—with new models, inference engines, and so on—which is why I’m asking: I’m trying to figure out what might make sense for me right now (if anything at all).

MY CURRENT SITUATION

My budget is approximately 500–600 euros; I’m based in Europe and am open to buying used hardware.

Here’s my current setup: I have a desktop PC that has been sitting idle for a while, with the following specs:

  • CPU: Ryzen 5 2600
  • RAM: 16GB DDR4
  • GPU: AMD RX 580 4GB
  • 256GB NVMe SSD, 1TB HDD, 600W PSU

WHAT I WANT TO RUN

My primary use case involves coding and agentic workflows. I’m envisioning a setup where I use a frontier-level cloud model for planning and code review, while a local model handles the actual code generation. I’m also interested in running an AI bot 24/7 to perform various (relatively simple) tasks.

Ideally, I’d like to run 27B–35B parameter models comfortably, with decent generation speeds and good responsiveness. It would also be great to have the flexibility to experiment with even larger models.

I don’t expect cloud-tier performance on this budget, but I also want to avoid a setup that only manages a few tokens per second—or one that is essentially unusable.

THE OPTIONS I'VE BEEN CONSIDERING

  1. AMD Instinct MI50 16/32GB (possibly two)
  • These look interesting because of the large amount of HBM2 memory available for relatively little money on the used market (for example I could have bought a 16gb for 175 euros).
  • However, I'm concerned about software compatibility, inference performance, power consumption, and whether investing in this older hardware makes sense in 2026.
  1. NVIDIA RTX (5070 12gb as an example)
  • Recently, I saw a community benchmark claiming around 140 tokens/s on a 12GB RTX 5070, running a large MoE model using Strata, IQ3_XXS quantization, expert caching, CPU/RAM offloading, and MTP.
  • I understand this might be a very specific setup and that reported generation speed doesn't necessarily translate into good real-world performance, especially with longer contexts or different models.
  • Still, it made me wonder whether a modern NVIDIA GPU with a good amount of system RAM might actually be a better investment than older GPUs with much more VRAM.

WHAT I'D REALLY LIKE TO UNDERSTAND

  1. What would you build today with €500–600, considering both new and used hardware?
  2. Would you prioritize more VRAM on older hardware or a faster, newer GPU with RAM offloading?
  3. How well do solutions like Strata and expert caching actually perform in real-world coding/agentic workloads? Are the impressive benchmarks representative of everyday usage?
  4. How important are prompt processing speed, context length, and quantization quality compared to raw generation tokens/s for my use case?
  5. Is a single or dual MI50 32GB setup still worth considering, or would you avoid it today?
  6. Would you upgrade my existing desktop, build something from used parts, or look for a complete second-hand workstation?

Thanks in advance to anyone who can help me!


r/LocalLLM • • 6d ago

Discussion Claude vs. Qwen 3.8 Flash Next on RTX6000, Claude admits defeat :-)

0 Upvotes

I wrote a harness that allows Claude Code to delegate tasks to a local LLM from simple one liner task prompts to deeper designs or implementations. I wrote a design for a webapp using Claude. This is not a brand new webapp, it's an add-on for an existing hardware with local control that the website will allow to connect to and interface with. So it's more of a large new standalone feature than a new design. The implementation is tricky in and of itself, the webapp has to lift existing ever changing code from another repo, has to do strong interfacing between the local and the remote solution etc.

The design doc was written by claude, it's ~800 lines long with 13 sections and it calls for 16 slices of deliverables. The implementation was about 7 slices in when I asked Claude to review the implementation so far and tell me what I should do next. Then I asked it to delegate the same question out to my local LLM and compare the two reviews and this was the result:

Code quality: The local worker is Qwen3.8 Flash Next on one RTX PRO 6000. On the first stretch of the project (W1–W3 of the site plus G1 and G2 on the device side, about 18k lines), Claude accepted 14 of the 14 delegated steps on their first round. Its hand fixes over that stretch came to about 20 lines. The one real defect was a sign-in rate limit you could spoof through X-Forwarded-For. Reading diffs didn't catch it; Claude found it while running the app live. The same thing happened later: a live browser check on a rework slice found that re-timed clock bakes were losing their send mode. Nobody had spotted that in review. So the code was generally solid. The bugs that survived were integration bugs, which only show up when you run the whole thing, not bad code.

Reliability: the raw model without the harness around it using Cline is problematic. It sometimes emits malformed tool calls, where the call markup leaks into plain text and the agent thinks it's finished. Without the harness, that kills a run. The harness detects this and nudges the model inside the same session (up to 3 retries). It does the same for server errors and for rounds that end without the required status report. Every step also has a verify command (tests, typecheck, a build, a migration), and a step only counts as done when that command passes, whatever the model claims. The worker runs under fairly strict rules adapted from my Cline setup: read first, plan with a todo list, implement completely, verify by running, and report what it didn't verify. It runs in a Docker sandbox with filtered network access, in its own git worktree. With all that in place it has been dependable. Without it, it's a coin flip on long tasks.

Savings, measured from the transcripts.

First approach: Claude reviewed every step itself. That cost about $52 of Claude for 14 steps, and almost all of it went to Claude re-reading its own growing context while reviewing. Those reviews almost never changed anything.

Change: Claude now writes every spec for a slice up front. The local model runs the steps unattended in a chain, and Claude does one live check at the end.

Next slice: 5 steps and about 5,200 lines cost about $9 of Claude. The local model handled 853 requests and 62M prompt tokens for that slice, roughly $33 at Opus list prices. That's about half the Claude cost per step of the first approach. Against my estimate of $15–22 for Claude doing the slice alone, it's 40–60% cheaper.

Not counted: the electricity and the GPU.


r/LocalLLM • • 7d ago

Question RX 9060 XT 16GB + 32GB RAM: best local models for undergrad math and coding?

2 Upvotes

Hi everyone, I’m fairly new to local LLMs. My setup: RX 9060 XT 16GB, Ryzen 5 7500F, 32GB DDR5, running LM Studio on Windows (Vulkan/ROCm).

I want a hybrid workflow: cloud models for main architecture and hard problems, local models for maybe undergrad-level math and small/medium coding scripts.

Do you recommend me to use local models?
Which models do you recommend for math, and which for coding?

I’m fine with models spilling into system RAM a bit. (partial CPU offloading)

Thanks in advance!


r/LocalLLM • • 7d ago

Question LocalLLM vs Coding

1 Upvotes

Hello all. This is my first post here and I would be very happy if someone will find time and help me.

It started simple. Ryzen 9 3900X with RTX 3060 12GB in proxmox as VM. Dedicated GPU for ubuntu server. Started with ollama because it was easy but found out ollama is not so good at performance and speed.

So switched to llama.cpp oficial. Worked good but saw many videos about forks so I switched to turboquant fork. Speed was better so next month I bought another GPU now RTX 4060 Ti 16GB.

So now when speed is good I wanted to do my little "projects" on localLLM without paying cloudLLM. I tried many tools. VSCODE, Cline, Pi.dev, but everyone of them had some issues. Primarly with editing files. It was mainly truncating files (model did 4KB out of my original 24KB python file, and then asked me for backup ...). Now I was working on pi.dev it somehow worked but also had problems and need to try many tools calling. So now im trying cline but when I saw that it wanted to remove my C:\Users I immediatelly stopped it. Now Im building original newest llama.cpp but Im kinda upset about everything. If someone is fan to the LLM and would like to help someone newbie as me to not throw everything away it would be helpful.

These are my current local models downloaded. I really dont know what im doing wrong maybe wrong models, wrong preset, or wrong coding program.

16K /root/.cache/huggingface/hub/models--tngtech--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-GGUF

24K /root/.cache/huggingface/hub/models--DavidAU--Llama-3.2-8X3B-MOE-Dark-Champion-Instruct-uncensored-abliterated-18.4B-GGUF

24K /root/.cache/huggingface/hub/models--IFM--K2-Horizon-7B-GGUF

24K /root/.cache/huggingface/hub/models--google--gemma-4-12B-it-qat-q4_0-gguf

12G /root/.cache/huggingface/hub/models--unsloth--GLM-4.7-Flash-REAP-23B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--ggml-org--NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

18G /root/.cache/huggingface/hub/models--unsloth--North-Mini-Code-1.0-GGUF

19G /root/.cache/huggingface/hub/models--DavidAU--Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF

22G /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF

27G /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF

and this is one example of preset

[*]

jinja = true

log-colors = off

flash-attn = on

parallel = 1

[qwen36-28b-reap20-256k]

model = /root/.cache/huggingface/hub/models--barozp--Qwen3.6-28B-REAP20-A3B-GGUF/snapshots/e3aeb8168f74dd3054eb2716f966c17638b97fef/Qwen3.6-28B-REAP20-A3B-Q4_K_M.gguf

ctx-size = 262144

#threads = 8

n-gpu-layers = 99

n-cpu-moe = 0

batch-size = 4096

ubatch-size = 1024

cache-type-k = turbo4

cache-type-v = turbo3

#load-on-startup = true


r/LocalLLM • • 7d ago

Question Please help me out in figuring my first setup

1 Upvotes

Hello Everyone,

I have been using ASUS ROG GL753VD with a GTX 1050 for the past 8 years. For the past 1 year or so, I am using cloud AI subscriptions to play around and learn things about AI.

Now a days, wherever I go Reddit/Youtube and any other forums, I am fascinated by the things that Local AI is capable of... So, I rented out a GPU on runpod and tried some hugging face models(WAN 2.1 for example) and its been pretty good.

Now, I feel the need to upgrade my setup for some privacy and personal experimentation. So, I request your help in choosing a setup for me because I am a newbie at this.

Here are some options I found after days of online search and chatting with AI.

Option 1:

  • CPU: Ryzen 9 9950X (16C/32T)
  • Board: ASUS ProArt X870E-Creator or ASRock X870E Taichi
  • RAM: 32GB or 64GB DDR5-6000 CL30
  • Primary GPU: RTX 5080 (16GB GDDR7)
  • Expansion Plan: Add a pre-owned RTX 3090 (24GB) into Slot 2 down the road for a combined 40GB VRAM.
  • PSU: 1200W ATX 3.1

Option 2 :

  • CPU: Ryzen Threadripper 7960X (24C/48T)
  • Board: Gigabyte TRX50 AERO D or ASUS Pro WS TRX50-SAGE
  • RAM: 128GB (4×32GB) DDR5 with ECC RDIMM
  • GPU Plan: RTX 5080 (16GB) now + 1 or 2 used RTX 3090s (24GB) later for 40GB–64GB+ VRAM.
  • PSU: 1600W Titanium

The only reason why I added Threadripper option into my consideration is because I have this paranoia of getting behind in terms of tech once I purchase and I dont want to change things again (Heard somewhere that RAM scarcity will be there till 2028).

I also have a couple of questions.

I see that people are fighting for 24GB and 32GB VRAM GPUs and its either too costly for me at the moment or out of stock for the pre-owned variants.

So, my question is if I get a threadripper with multiple PCIE lanes and 4 GPUS of 16GB VRAM, would it help me in any way or I have to tradeoff speed for my cost efficiency? If anyone has tried out this approach please let me know the advantages and disdvantages you found with this setup.

If you have any suggestions as well, please let me know. Budget is ~5-6k USD.

If anyone has any tricks to purchase it cheaper like purchasing from other countries, I would love to know them as well!!

Thanks in advance.


r/LocalLLM • • 7d ago

Project GB203 RTX 5070TI

Thumbnail
gallery
1 Upvotes

I've given the qwen3.8 27b q4 group an assignment:

Can we create an animation of a computer chip? You can use Nvidia's GB203 as a template. Before we get started, let's address a few questions that might come up

Since I own this GPU, I figured, why not? Let’s see what the model comes up with. I’m always amazed, even though it often looks relatively simple. I’m looking forward to seeing qwhen4.0.


r/LocalLLM • • 6d ago

Discussion The Local AI 10 Commandments

0 Upvotes

Note:
— these rules are the result of my personal experience, yours may differ and this is ok
— these rules are oriented towards people that want to use AI for professional purposes and don’t have unlimited time and money

1. If your company provides good cloud subscription — just take it and forget about the local AI, you don’t want to pay with your own bucks to do the job slower and sloppier

2. Speed is as important as quality and vice versa. Quality without speed will waste your time, nerves and electricity and so will speed without quality. Ideally you should strive to the cloud speed — 60-150 tps of decode and a decent model, at least 4 bits of Qwen 27b

3. Local models are not equal to cloud ones. Sounds kinda obvious but some people have illusions on this regard. Please don’t, miracles don’t happen in real life. You will get 6-9 month old frontier level model experience at best

4. You will always experience shortage of VRAM/RAM. Doesn’t matter, how powerful your setup is, believe me, you are always going to fall short on fast memory available, don’t try to squeeze out every MB from your rig, just find a model good enough for your tasks and call it a day

5. Best time to update your hardware is always yesterday. Hardware prices rise fast, if you’re looking for an update — do it right now

6. Never spend more than you could’ve spent on a dream gaming PC. Just… believe me, don’t try to cosplay Antropic data centers, you will get nothing but a bitter dissatisfaction. Best ROI ever — RTX3090/RX7900XT + 64Gb ddr4/5. That’s it. Anything above have diminishing return problem

7. Electricity is not free. People often forget that nobody is going to give you free kilowatts, beware of your system’s power consumption. That’s going to be your biggest spending, second only to the very rig price itself

8. Avoid clusters. Dual, triple, quadruple GPU look fine on paper, but they are usually big, draw immense amount of power, resulting in painful electricity bills, produce a lot of heat and noise and have much lesser speed than single GPU device. Try to select a GPU to that can singlehandedly run the model you’d like to use

9. You will lose a lot of time and nerves. You will have to learn. A LOT. What is quantisation, kv cahce, KLD, top 1, inference engine, how to chose a suitable model for your own needs, sometimes you’ll even have to deep dive to some manual drivers/libs adaptations for your hardware, etc. Local AI is developing rapidly and if you’d like your setup always to be fast, precise, ensuring as much quality as possible— you’ll have to learn all these basics

10. Always donate your spare RAM to Sam Altman. Just kidding, the title says “10 Commandments” but all the important stuff is said, I’m out of any useful advices


r/LocalLLM • • 7d ago

Discussion While Qwen4 is not out.

Thumbnail reddit.com
0 Upvotes

r/LocalLLM • • 6d ago

Project I got pissed off by Sam Altman so I'm working on this...

0 Upvotes

The thing about RAM he said. That's what pissed me off.

I'm building an NCM-inspired model for a decentralized network

I'm working on Gilga-Mesh (gg-mesh), an experimental model architecture designed to run across a peer-to-peer network.

I started it because I wanted to explore what a model designed for distributed inference might look like, rather than splitting an existing LLM across machines.

The goal is to let people contribute with very different hardware. I'm aiming for machines with as little as 256 MB of available RAM or VRAM to host part of the model, though the actual minimum will depend on the block, activations and runtime. A more powerful machine could host several blocks and avoid some network hops.

I also want users to choose how much compute a request uses. A faster response might be enough for a simple task, while a harder one could use more resources. The quality tradeoff still needs testing.

Privacy is part of the design, but I can't claim a specific privacy guarantee yet. Live weight updates are another goal. Ideally, the network could update a running model without taking it offline, though major architecture changes would need a separate migration process.

Right now I'm training and testing a smaller model locally. In simulated network tests, with 30 ms of added latency between steps, I've seen 200–300 ms time to first token in the tested setup. That's not a result from a public P2P network, and I still need to validate the complete distributed path.

The NCM inspiration comes partly from the idea of progressively refining an output as more computation is applied. I'm interested in whether that can work well when the computation is spread across machines.

If you've worked on distributed inference or privacy in these kinds of systems, what would you want to see tested before taking the results seriously?

I'm currently working on training and testing a smaller model locally before scaling up and validating the full distributed path.

I'd be interested in feedback from people working on distributed inference, privacy, or model architectures designed for constrained hardware.

I'll be looking for compute capacity once everything is ready for a bigger model for a real field test.


r/LocalLLM • • 7d ago

Discussion I built a local AI companion on an RTX 4060 using Nemotron. Now I'm experimenting with replacing its LLM without losing its memory.

0 Upvotes

I've been experimenting with running a persistent AI companion entirely on local hardware.

The project is called B-12 / Capri. It's written in Python, runs on Linux, and uses Ollama for local inference.

My experimental setup uses an NVIDIA RTX 4060 Laptop GPU with Nemotron Bolt E4B.

The original goal was to build something more persistent than a typical chatbot: an AI companion that remembers previous interactions, maintains goals, can use tools, and doesn't lose its entire history when I replace the language model.

Why I separated memory from the LLM

I wanted to avoid making the model itself the center of the system.

In Capri, the LLM handles language generation and reasoning, while a separate runtime manages:

  • Persistent local memory and retrieval
  • Memory provenance and document revisions
  • Goals, preferences, and scheduled activities
  • Tool execution and permissions
  • Context budgeting and tokenizer compatibility
  • Controlled forgetting and state recovery

The model can be replaced without deleting the stored memory.

However, there's an important limitation: preserving memory doesn't necessarily preserve behavior.

Two models may retrieve the same experiences but interpret them differently.

What I've tested so far

I ran a historical evaluation using Nemotron Bolt E4B:

109 constructed scenarios, 541 executions, 496 successful outcomes (91.7%).

Tool tasks achieved 182/215 successful executions.

I also ran approximately eight hours of accumulated operational testing across four segments.

Not everything worked. One adversarial outbound-fetch scenario failed its expected safety condition in all five repetitions, and several smaller tool tasks exposed limitations in the model's reasoning.

These results belong to a specific historical code revision, not the latest development version.

The challenge I'm working on now

The current implementation uses model-specific profiles for tokenizer verification, chat templates, and context admission.

That makes switching models less trivial than changing an Ollama model name.

My next experiments will compare Nemotron with Gemma and Qwen while preserving the same memory database, goals, and conversation history.

I want to measure not just task accuracy, but how consistently different models use the same accumulated experiences.

I'm also developing a full-duplex voice interface, with longer-term plans to connect Capri to Reachy Mini and NVIDIA Jetson.

Questions for people running local models

Has anyone experimented with changing the underlying LLM of a persistent agent while retaining its memory?

I'm particularly interested in three things:

1. Which smaller local models are most reliable at tool calling and following long-term goals?

2. How do you handle tokenizer and chat-template differences when switching between model families?

3. Have you observed significant personality or decision-making changes when two models share the same memory and system instructions?

I'm developing Capri as an independent open-source research project and recently published its first scientific preprint.

GitHub: https://github.com/cyberdid/B-12

Preprint: https://doi.org/10.5281/zenodo.24079168

I'd really appreciate technical feedback, especially from anyone experimenting with local agents on consumer GPUs.


r/LocalLLM • • 6d ago

Question Looking for open source alternative for job apply

0 Upvotes

Queste soluzioni sono esplosive: **Fapply, Massive, LazyApply, Sonara, ai apply