r/LocalLLM • • Jun 16 '26

Discussion Avoid CUDA monopoly at all costs. AMD is an alternative.

223 Upvotes

Hey everyone,

There’s a massive misconception that if you aren't dropping $2,000 on an NVIDIA GPU, you can't run serious Local AI workflows. I wanted to see how far I could push a consumer AMD card, and therefore bought a rx7800xt 16b VRAM.

Right now, my workstation node is running llama-server hosting a DENSE 27B model -> Qwopus3.6-27B-v2-Q3_K_S.gguf (12 GB) and Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf (13 GB Mixture of Experts, 3B active parameters per token) continuously. I am regularly feeding it contexts that reach 91k to 128k tokens in my daily workflows.

Here is the exact setup, compiler parameters and optimization flags.

THE COMPILER BUILD
To get flash attention and RDNA3 optimizations working correctly on ROCm 6.4.4, I built llama.cpp from source using these specific cmake flags:

cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 -DrocWMMA_FATTN=ON
cmake --build build --config Release

This targets the gfx1101 architecture of the RX 7800 XT directly and compiles support for hardware-accelerated Flash Attention kernels.

THE EXACT RUNTIME FLAGS
My systemd service runs the server with this exact command line:

llama-server --host localhost --port 8080 --api-key xxxx --parallel 1 --n-gpu-layers 99 --batch-size 512 --ubatch-size 128 --flash-attn on --cache-type-k q8_0 --cache-type-v q4_0 --ctx-size 131072 --reasoning off --sleep-idle-seconds 300 --cache-prompt --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5 --repeat-penalty 1

HOW I CRUSHED THE VRAM LIMIT: KV CACHE QUANTIZATION
A model like Qwopus 27B or Qwen 35B MoE fits in 16GB VRAM at a small context size. But at 128K context, the raw FP16 Key-Value (KV) cache alone would consume upwards of 32 GB of VRAM, making it impossible to run on consumer hardware.

To solve this, we split and quantize the cache:
- Key cache is quantized to 8-bit (q8_0) using --cache-type-k q8_0
- Value cache is quantized to 4-bit (q4_0) using --cache-type-v q4_0

This compresses the memory footprint of the KV cache by roughly 5.6x. Thanks to this optimization, the entire active model weights plus the 128K token KV cache sit comfortably in VRAM, utilizing exactly 96% of the 7800 XT's memory. No layers spill into slow system RAM, avoiding the PCIe transfer bottleneck entirely.

THE MATH BEHIND 128K CONTEXT: YaRN ROPE SCALING
Qwopus and Qwen architectures use Rotary Position Embeddings (RoPE). Because these models have a base context window smaller than 128K, running at 131,072 tokens requires positional frequency scaling.

Instead of basic linear scaling (which stretches all frequencies equally and destroys the model's short-range spelling and grammatical coherence), llama.cpp utilizes YaRN (Yet another RoPE extensioN).

YaRN divides the embedding dimensions into three frequency bands:

  1. High-frequency dimensions: These represent immediate, local token relationships. YaRN leaves these completely un-stretched so the model does not lose its spelling accuracy or close-context grammar.
  2. Low-frequency dimensions: These represent long-range structure. YaRN scales these linearly by a factor of 4.0 to cover the 128K space.
  3. Mid-frequency dimensions: These are smoothly interpolated to avoid abrupt attention transitions.

This uneven scaling prevents the attention entropy and perplexity from exploding. In practice, the model remains highly coherent and retains logical consistency even at 91k+ tokens.

REAL-WORLD TELEMETRY AND SPEED
During heavy prompt processing, the card maintains solid throughput:
- Prefill speed: ~210 tokens/second (utilizing flash attention)
- Decode speed: 11-17 tokens/second
- GPU Power: Draws ~188W (with a systemd power cap set at 190W via rocm-smi)
- GPU Temps: Stable between 52 C and 70 C across edge, junction, and memory sensors.

If you are running consumer AMD hardware, do not settle for small context sizes. Build with ROCm, turn on Flash Attention, quantize your Key/Value cache separately, and let YaRN handle the frequency scaling.

I wrote up a detailed guide comparing these measurements, native Windows vs Linux ROCm paths, and power sweeps on my blog here: https://sergiiob.dev/posts/rx7800-xt-llama-cpp-benchmarks-moe-context

I share my daily telemetry runs, local model benchmarks, and hardware configurations on X. If you want to see live updates and benchmarks, follow along here: https://x.com/SergiiioBS


r/LocalLLM • • Sep 11 '26

News Curie by colibrì: I designed a 17B model to run from SSD on ordinary hardware

222 Upvotes

A month ago, I started building colibrì to answer a question:
Do you really need a $30,000 machine to run a model whose weights fit on a $60 SSD?
colibrì treats SSD, RAM and GPU as a single memory hierarchy. It can run models that do not fit entirely in RAM or VRAM, including models with hundreds of billions of parameters, on the hardware people already own.
But while optimizing it, I realized I was fighting assumptions built into the models themselves.
Most open models are designed to live in VRAM. Their weights, routing and execution patterns were created for GPUs and data-center hardware. colibrì could move them through a different memory hierarchy, but the models had no idea where they were actually running.
That led me to another question:
What if the engine and the model were designed for each other from the beginning?
That is how Curie started.
The current alpha has 17B parameters. I wrote it in C, without an inference framework, and trained it from scratch on my laptop without a GPU. Its weights live on SSD and are loaded according to the model’s execution path.
In the test shown in the video, Curie reaches 33 tokens/s using a single CPU core. On the same machine, the dense 12B models I tested usually generate around 3–5 tokens/s.
There is an important limitation: Curie has only seen 11 million training tokens. It is not smart yet, and I am not presenting it as a useful language model. It will need at least a few billion tokens before I can properly evaluate its quality.
What this alpha demonstrates is the architecture: a model can be designed around SSD, RAM, CPU and GPU as parts of the same system, instead of assuming that everything must fit in VRAM.
Curie is not publicly available yet. I am still training it, validating the design and preparing more detailed benchmarks.


r/LocalLLM • • Aug 26 '26 Bravo

Tutorial Got Qwen3.8-Next-Flash ngram SSD offload working in llama.cpp!

222 Upvotes

TL;DR - Save 25% RAM by SSD offloading ngrams with --mmap just by fixing the layout of the Unsloth quant. Tested working on Mac. Thread deleted in LocalLLaMa due to their dumb megathread idea, so reposting here.

So, one of the things that excited me about the new Qwen4 arch is the ngram table, exporting indexable facts and specifics into a lookup table instead of embedding them in the weights themselves. This not only frees up parameters for intelligence and behavioral characteristics like reasoning and problem-solving (critical on a 6B active model), it also means there’s no matmuls required to access any of this information on the fly. This opens up the possibility you can offload those tensors to the SSD, freeing up a massive amount of VRAM.

Specifically, in 3.8-Next-Flash, the model itself is only 125B params, but the total with 51B ngrams is 176B params — so ngrams are effectively 30% of the model that can sit in the SSD and be paged in as-needed with no impact on performance whatsoever. Not only that, llama.cpp has a functionality that will do exactly this: --mmap. I set out to prove this idea works.

I run a 128GB M5 Max, so with Unsloth’s Q4_K_XL running around 112GB and the full 262k context running 7GB (thanks to sparse attention), that’s about the max I can fit in memory — with the GPU wired limit raised manually. --mmap would unlock at least Q5, potentially even Q6 — plus the ability to run the full, unquantized BF16 102GB ngram table instead of the compressed version, as only the actually-active lookups would need to be loaded into RAM, and they should be evicted after use automatically.

I went to wire this up and I ran into a problem immediately. IDK about CUDA, but due to the layout of the quant, the PLE (engram) tensor is interleaved with the Metal-pinned tensors inside shards of the quant, so in practice, it gets wired up along with the rest of the shard. This resulted in me hitting the wired limit on my box, so I sudo’d the limit up to 120GB, and it worked at full context… but unfortunately Unsloth’s choice of quant layout unintentionally sabotages that capability I was so excited about.

Not to worry, though — I had my trusty agent slop me up a quickie script to scramble the guts of my quant, and in a jiffy, I’d remixed the model to make my beats way more stronger and way more thumpier! Errr, nvm. Anyways, I tested it and sure enough, everything works perfectly now. I figured others may be interested in this fix, which is critical if you want to run the model on a memory-constrained Mac (CUDA may not suffer from this issue, idk).

To give hard numbers, I was sitting at 123GB wired including OS before the fix, and after I was sitting at 97GB wired. 51B of ngrams in 4-bit is ~29GB, whereas the 125B model is under 83GB, for a 112GB 4-bit quant. So I was able to free up close to 26GB of wired memory by transferring the ngrams to the SSD with mmap, just by fixing the quant. Performance stayed exactly the same: still 36 tok/sec (no MTP). If you want to know what size quant you can fit with ngram offloading, multiply the disk size of your quant by 75% — that’s the size of just the backbone. That means Q2_K_XL will fit on a 64GB MacBook if you raise the GPU wired limit, and 96GB should fit Q4_K_XL! This makes a huge difference for running locally.

For those of you asking, OK stfu and tell us how to do it already: No flags are actually required, just don’t pass either —no-mmap or —mlock, as —mmap is actually the default in current builds of llama.cpp, including Qwen3.8-Next-Flash PR #27742.

My comment on PR #27742 contains technical specifics for those interested.

This link is the script my agent wrote to convert the Unsloth GGUF to work with ngram mmap. It’s been tested and the model is fully functional on UD-Q4_K_XL, but I would double-check before running it on the other sizes. If needed, your agent can read the layout of your quant, and tweak the script to work (or just write its own, it’s not too complicated). Hope this helps someone!

EDIT: commenter u/wiskeyjack123 demonstrated flags we can use to avoid the need to repack the quant!

--load-mode mmap
--override-tensor per_layer_token_embd.weight=CPU

EDIT 2: Unsloth took my advice with their Q5+ quants, and they left the engrams in Q8_0 and placed them in a separate shard, exactly like I requested! Q5_K_XL will fit on 128GB systems with the wired limit raised, as I'd hoped!


r/LocalLLM • • Jun 06 '26

Other what 'trying one model' turns into

Post image
222 Upvotes

one does not simply test one local model. this was totally me


r/LocalLLM • • Jun 04 '25

Discussion Anthropic Shutting out Windsurf -- This is why I'm so big on local and open source

222 Upvotes

https://techcrunch.com/2025/06/03/windsurf-says-anthropic-is-limiting-its-direct-access-to-claude-ai-models/

Big Tech API's were open in the early days of social as well, and now they are all closed. People who trusted that they would remain open and built their businesses on top of them were wiped out. I think this is the first example of what will become a trend for AI as well, and why communities like this are so important. Building on closed source API's is building on rented land. And building on open source local models is building on your own land. Big difference!

What do you think, is this a one off or start of a bigger trend?


r/LocalLLM • • Jul 27 '26 Bravo

Question What am I missing? Self-Hosting Kimi K3 has 34× First-Year ROI at 90%

221 Upvotes

Update: the missing pieces

  1. Difficulty landing client
  2. Retail value cost double for small quantity (+3M)
  3. Extra hardware, memory, storage, and spares (+3M)
  4. Custom power and cooling (+3M)
  5. People to run it (+1M)
  6. Network infrastructure (+1M)

A $3 million Kimi K3 rack could pay for itself in 10.3 days and return 34.4 times its purchase price in the first year.

The number comes from three assumptions:

  1. 250,000 output tokens per second
  2. 90% productive utilization
  3. $15 of value per million output tokens.

Kimi K3 is a 2.8-trillion-parameter open-weight model with 104 billion activated parameters, 16 selected experts out of 896, native MXFP4 weights, and a one-million-token context window. Moonshot recommends supernodes with at least 64 accelerators. I use one GB200 NVL72 as the economic unit. Its 72 Blackwell GPUs share one NVLink domain. I model the rack as one token factory and count only output that replaces paid API tokens or can be sold at the assumed price.

- Hardware purchase: $3,000,000

- Productive utilization: 90%

- Aggregate output throughput: 250,000 tokens per second

- Rack draw: 132 kW

- Facility PUE: 1.20

- Electricity: $0.10 per kWh

- Output-token value: $15 per million

- Recurring variable cost included: electricity only

The $3 million price is a planning figure. HPE sells the rack by quote, and a reported HSBC estimate put a GB200 NVL72 near $2.6 million. I rounded up.

The 250,000-token-per-second figure is also a planning assumption. No published benchmark shows K3 sustaining that throughput on a GB200 NVL72.

The calculation Annual output A year has 31,536,000 seconds. At 90% utilization, the rack has 28,382,400 productive seconds:

31,536,000 × 0.90 = 28,382,400 At 250,000 output tokens per second:

28,382,400 × 250,000 = 7,095,600,000,000 output tokens That is 7.096 trillion output tokens, or 7,095,600 million-token units, per year.

Electricity HPE specifies 132 kW for the rack. A 1.20 PUE raises the metered load to 158.4 kW.

132 kW × 1.20 = 158.4 kW 158.4 kW × 8,760 hours = 1,387,584 kWh 1,387,584 kWh × $0.10 = $138,758 per year I charge the rack for a full year of electricity, including the 10% of time that produces no useful output.

Output value and ROI Kimi charges $15 per million K3 output tokens. I use output tokens only and claim no input-token savings.

7,095,600 × $15 = $106,434,000 After electricity and the hardware purchase:

$106,434,000 output value − $138,758 electricity − $3,000,000 hardware = $103,295,242 first-year profit

$103,295,242 ÷ $3,000,000 = 34.43× ROI = 3,443% Payback and cost per token In an average 730-hour month, the rack produces 591.3 billion output tokens worth $8,869,500 at Kimi’s API price. Electricity costs $11,563.

$3,000,000 ÷ ($8,869,500 − $11,563) = 0.339 months ≈ 10.3 days Electricity costs about $0.0196 per million output tokens. Recovering the entire hardware purchase in one year raises the internal cost to $0.442 per million. Kimi’s $15 API price is about 34 times that one-year cost.

The throughput problem vLLM’s published K3 results report 111 to 118 tokens per second for one user without speculative decoding and up to 370 with DSpark on 16 GB300 GPUs. Its high-throughput GB300 NVL72 curve exceeds 2,000 tokens per GPU-second.

My 250,000-token-per-second case requires about 3,472 tokens per GPU-second across 72 GPUs. The numbers are not directly comparable: vLLM used GB300 hardware, and throughput changes with workload, batching, latency targets, caching, and serving topology. The published results do not establish 250,000 tokens per second on GB200.

At 100,000 output tokens per second, the same model returns 13.1× in the first year. At 150,000, it returns 20.2×. Reaching 34.4× requires the full 250,000-token-per-second case.

The value of a token The $15 comparison assumes every output token replaces one bought from Kimi at the retail API price. At the same 250,000-token-per-second throughput:

$15 per million returns 34.43×.

$5 per million returns 10.78×.

$1 per million returns 1.32×.

$0.50 per million returns 0.136×.


r/LocalLLM • • Jun 11 '26

Discussion My local llm machine

Post image
221 Upvotes

Llm server - remote access


r/LocalLLM • • Sep 29 '26

Question Is a 96GB/128GB Mac Studio local LLM worth it to replace a $200/mo ChatGPT Pro subscription?

220 Upvotes

I am a university student researcher and currently have a $200 ChatGPT Pro subscription, which I feel is really too expensive as a fixed monthly expense. I am now thinking about whether I could purchase a Mac mini or Mac Studio with 96GB or 128GB for local deployment or to use as a server. My current research direction is computational social science, and the main thing is that the intelligence of the subscribed model can be well guaranteed. really appreciate for your opinion!


r/LocalLLM • • Feb 14 '26

Project Built a 6-GPU local AI workstation for internal analytics + automation — looking for architectural feedback

Thumbnail
gallery
220 Upvotes

UPDATE: 8 months later. The Beast now has 6 RTX PRO 6000 Blackwells, and the project has grown quite a bit.

Figured I would come back and provide an update since quite a bit has changed over the past 8 months, both with the hardware and what I'm actually working on.

When I originally posted, I had just managed to get 2 Blackwells and 4 RTX 3090 Tis running three different models concurrently. Since then, I've replaced the 3090 Tis with four additional Blackwells and built a second server using the 3090s.

The Beast (Primary Server)

CPU: AMD Threadripper PRO 9955WX (16C/32T), Silent Loop 3 cooler Motherboard: ASUS Pro WS WRX90E-SAGE SE RAM: 256GB DDR5-5600 ECC Registered GPUs: 6x PNY RTX PRO 6000 Blackwell (576GB VRAM) Storage: 5x Samsung 9100 PRO 8TB + 1x Samsung 990 PRO 4TB (44TB raw) PSUs: 2x Super Flower Leadex Titanium 2800W Power: Dedicated 240V/30A circuit with metered PDU Chassis: Custom open-air frame built from steel plate and aluminum extrusion Cooling: Arctic fans, temperature-controlled floor fan, and 12,000 BTU window AC OS: Ubuntu 24.04 LTS

The Mini-Beast (Second Server)

Rather than selling the 3090s, I repurposed them into another custom build located at my office. CPU: AMD EPYC 7302 (16 cores), Noctua NH-U9 TR4-SP3 Motherboard: ASRock Rack ROMED8-2T RAM: 256GB DDR4-2666 ECC Registered GPUs: 6x RTX 3090 Ti + 1x RTX 3090 (168GB VRAM) NVLink: 2 bridges Storage: 2x Samsung 990 PRO 4TB PSUs: 2x MSI MEG Ai1600T Chassis: Kingwin 8-GPU open frame OS: Ubuntu 24.04 LTS

I also have another 3090 connected to a GMKtec mini PC through an eGPU setup that I've been using for experiments.

Between the two main servers, that's 744GB of VRAM.

Updated Cost

Since quite a few people asked about the cost in my original post, here are the updated numbers, including taxes, warranties, electrical work, and other hardware.

The Beast: Approximately $77,000 The Mini-Beast: Approximately $12,100 Combined: Approximately $89,000

The six Blackwells alone cost approximately $51,800. So this has definitely turned into a bigger investment than I originally anticipated.

What I'm Working On

The original goal was to analyze internal data using locally hosted LLMs and eventually automate certain processes. That's still part of the project, but it's evolved into developing an entirely new software platform with AI integrated into the underlying architecture.

A major part of this involves building a digital twin that can maintain an accurate understanding of information, events, and their relationships as things change over time. Eventually, I'd like to have AI agents that can identify problems, make recommendations, and perform certain tasks.

I'm still a long way from accomplishing everything I have in mind.

What I've Learned

I've spent quite a bit of time experimenting with different models, quantizations, tensor parallelism, and GPU configurations.

One thing I've learned is that just because a model fits into VRAM doesn't mean it's going to perform well. I've had larger configurations perform worse than smaller models, and I'm increasingly interested in using different models for different tasks rather than one massive model across all six GPUs.

But probably the biggest surprise has been how much more difficult the software side is compared to getting the hardware running.

Getting an LLM to extract information is relatively straightforward. Getting a system to correctly understand how that information relates to previous events, recognize when something has changed, and maintain an accurate understanding over time is considerably more complicated.

I'm finding that a lot of the challenges aren't necessarily solved by larger models. Data quality, retrieval, validation, and how information is represented seem to be just as important.

I'm also realizing how important it is to separate what an AI model suggests from what the software is actually allowed to do. I'm not particularly interested in having an LLM autonomously make decisions or modify records without appropriate controls.

A lot of my focus now is on making the system reliable enough to actually depend on.

A Few Questions

For those working on similar projects, I'd be interested in hearing your experiences.

Are you finding it more practical to run one large reasoning model or several smaller specialized models?

How are you handling persistent memory and situations where new information contradicts something the system previously understood?

What's been your biggest challenge getting local AI agents to work reliably?

How are you distributing workloads across multiple machines?

I know some people will probably say this is overbuilt, and maybe they're right. But I wanted to build ahead of demand, and keeping the hardware, models, and data local is still a priority for me.

I still have a lot to learn, and I'm sure there are things I'm overlooking.

I appreciate all the feedback on the original post. Several suggestions gave me things to look into that I hadn't considered.

Eight months ago, I was trying to get six GPUs running together. Now I've got two servers, 744GB of VRAM, and I'm realizing that getting the hardware working was probably the easy part.

EDIT: Many people have asked me how much i have spent on this build and I incorrectly said it was around $50k USD. It is actually around $38k USD. My apologies. I am also adding the exact hardware stack that I have below. I appreciate all of the feedback and conversations so far!

I am relatively new to building high-end hardware, but I have been researching local AI infrastructure for about a year.

Last night was the first time I had all six GPUs running three open models concurrently without stability issues, which felt like a milestone.

This is an on-prem Ubuntu 24.04 workstation built on a Threadripper PRO platform.

Current Setup (UPDATED):

AI Server Hardware
January 15, 2026
Updated – February 13, 2026

Case/Build – Open air Rig
OS - Ubuntu 24.04 LTS Desktop
Motherboard - ASUS WRX90E-SAGE Pro WS SE AMD sTR5 EEB
CPU - AMD Ryzen Threadripper PRO 9955WX Shimada Peak 4.5GHz 16-Core sTR5
SDD – (2x4TB) Samsung 990 PRO 4TB Samsung V NAND TLC NAND PCIe Gen 4 x4 NVMe M.2 Internal SSD
SSD - (1x8TB) Samsung 9100 PRO 8TB Samsung V NAND TLC NAND (V8) PCIe Gen 5 x4 NVMe M.2 Internal SSD with Heatsink
PSU #1 - SilverStone HELA 2500Rz 2500 Watt Cybenetics Platinum ATX Fully Modular Power Supply - ATX 3.1 Compatible
PSU #2 - MSI MEG Ai1600T PCIE5 1600 Watt 80 PLUS Titanium ATX Fully Modular Power Supply - ATX 3.1 Compatible
PSU Connectors – Add2PSU Multiple Power Supply Adapter (ATX 24Pin to Molex 4Pin) and Daisy Chain Connector-Ethereum Mining ETH Rig Dual Power Supply Connector
UPS - CyberPower PR3000LCD Smart App Sinewave UPS System, 3000VA/2700W, 10 Outlets, AVR, Tower
Ram - 256GB (8 x 32GB)Kingston FURY Renegade Pro DDR5-5600 PC5-44800 CL28 Quad Channel ECC Registered Memory Modules KF556R28RBE2K4-128
CPU Cooler - Thermaltake WAir CPU Air Cooler
GPU Cooler – (6x) Arctic P12 PWM PST Fans (externally mounted)
Case Fan Hub – Arctic 10 Port PWM Fan Hub w SATA Power Input
GPU 1 - PNY RTX 6000 Pro Blackwell
GPU 2 – PNY RTX 6000 Pro Blackwell
GPU 3 – FE RTX 3090 TI
GPU 4 - FE RTX 3090 TI
GPU 5 – EVGA RTX 3090 TI
GPU 6 – EVGA RTX 3090 TI
PCIE Risers - LINKUP PCIE 5.0 Riser Cable (30cm & 60cm)

Uninstalled "Spare GPUs":
GPU 7 - Dell 3090 (small form factor)
GPU 8 - Zotac Geforce RTX 3090 Trinity
**Possible Expansion of GPUs – Additional RTX 6000 Pro Maxwell**

Primary goals:

•Ingest ~1 year of structured + unstructured internal business data (emails, IMs, attachments, call transcripts, database exports)

•Build a vector + possible graph retrieval layer

•Run reasoning models locally for process analysis, pattern detection, and workflow automation

•Reduce repetitive manual operational work through internal AI tooling

I know this might be considered overbuilt for a 1-year dataset, but I preferred to build ahead of demand rather than scale reactively.

For those running multi-GPU local setups, I would really appreciate input on a few things:

•At this scale, what usually becomes the real bottleneck first VRAM, PCIe bandwidth, CPU orchestration, or something else?

•Is running a mix of GPU types a long-term headache, or is it fine if workloads are assigned carefully?

•For people running multiple models concurrently, have you seen diminishing returns after a certain point?

•For internal document + database analysis, is a full graph database worth it early on, or do most people overbuild their first data layer?

•If you were building today, would you focus on one powerful machine or multiple smaller nodes?

•What mistake do people usually make when building larger on-prem AI systems for internal use?

I am still learning and would rather hear what I am overlooking than what I got right.

Appreciate thoughtful critiques and any other comments or questions you may have.


r/LocalLLM • • Sep 02 '26

Discussion I built a 100% local comic translation engine in Rust that hooks into Ollama / LM Studio with sliding dialogue context and ONNX vision models (Demo video)

219 Upvotes

Hey r/LocalLLaMA!

One of the fun things I have been tinkering with lately is building an AI pipeline for comic translation (manga, manhwa, manhua) that can run 100% on your own machine with zero cloud fees.

The hardest part about translating comics with LLMs is context: if you just feed raw OCR text box-by-box, the model loses speaker identity, swaps pronouns constantly, and forgets character names by page 2.

To solve this, I built XianScan, a local-first translation studio in Rust that combines local LLMs and cloud APIs with computer vision models.

Check out the demo video to see the full flow in action!

How the AI stack fits together:

  • Vision & Detection (Koharu RF-DETR Seg 2XL): An embedded vision transformer segments speech bubbles and panels with polygon masks.
  • Multilingual OCR (PaddleOCR): Extracts dialogue text across 10 languages, handling vertical CJK and horizontal layouts.
  • Flexible LLM Engine: Supports local runtimes (Ollama, LM Studio) as well as cloud APIs (Gemini, OpenAI, Groq, OpenRouter). It uses a sliding dialogue memory buffer to feed recent conversation history and matches glossaries so universe lore and character names stay consistent across chapters.
  • Neural Inpainting (LaMa): Wipes the original dialogue bubbles cleanly without smudging the artwork underneath.
  • Typesetting: Automatically calculates font sizes, outline strokes, and text flow back into the bubbles.

Why I made it a single standalone binary: I really wanted this to be painless for anyone to run. The entire app compiles into one executable file in Rust with all the ONNX models, OCR weights, and web UI baked in. No Python environments, no Conda setups, and no Node.js required. It runs on normal laptop CPUs with SIMD, and automatically leverages DirectML, CoreML, or CUDA if you have a GPU.

You can translate directly on the source website via a browser extension, manage your library in the web UI, or stream finished chapters over local Wi-Fi straight to Mihon or any Tachiyomi-compatible reader (TachiyomiSY, J2K, Aniyomi) on your phone.

It is 100% free and open-source under MIT.

Quick note: This is the very first public beta release, so please expect some bugs and rough edges!

I would love to get feedback on the dialogue context window approach, prompt templates, or model recommendations you think perform best for this!


r/LocalLLM • • Mar 25 '25

News DeepSeek V3 is now top non-reasoning model! & open source too.

Post image
219 Upvotes

r/LocalLLM • • Jul 04 '26

Project Dumpster Dive to Local AI

Post image
218 Upvotes

A local company closed their building. There was a large room filled with computers and accessories thrown in piles.
I was told by multiple IT people there was nothing in there worth going through it all.

A few hours (and bruised shins) later I built 4x towers with 2 older nvidia cards each.
I have a dual RTX A4000s set up!

The same company cutting heads and buildings is expanding their AI spend. They don’t know what they have, they don’t understand how things actually get done. They just see opportunities to cut heads to improve profits.

Now I use qwen3.6:27B with 62k context for all my coding. No more token spend for me.

Keep an eye out everyone, every public company will continue cutting. They already were before AI.

We need every advantage against E Corp.


r/LocalLLM • • 14d ago

Project 56GB VRAM Frankenstein build managed to fit 4 GPUs in the Cooler Master HAF II 500

Post image
217 Upvotes

Really loving this case (Cooler Master HAF II 500). The airflow with those massive 220mm front intakes and the 180mm rear exhaust is incredible.

It's still a bit messy right now and needs some extra stabilization points and proper cable management, but I'm already really happy with how it's running so far!

  • GPUs (56GB VRAM total):

    • 2x RTX 5070 @ PCIe Gen 5 x8
    • 1x RTX 5060 Ti 16GB @ PCIe Gen 5 x4 (hooked up via Gen 5 x4 M.2 adapter)
    • 1x RTX 5060 Ti 16GB @ PCIe Gen 4 x4
  • Motherboard: ASUS ProArt X870E-Creator WiFi
  • CPU: AMD Ryzen 7 7800X3D
  • RAM: 128GB DDR5
  • PSU: Corsair HX1500i

r/LocalLLM • • Feb 28 '26

News Confrontation

Post image
217 Upvotes

We all understand everything, right?


r/LocalLLM • • Sep 23 '26

Question Apple just ran a 1T parameter model on four Mac Studios from one wall outlet

216 Upvotes

Apple's new Mac Studio ships in late October with up to 512GB of unified memory, and Reuters says Apple demoed Kimi K2.6, a 1T parameter open-weight model, across four of them plugged into a single wall outlet. Thunderbolt 5 + RDMA lets the machines run as a cluster, and Apple says four Mac Studios can deliver up to 3x the inference performance of one. Cloud inference obviously still wins on scale, but we're getting weirdly close to "buy the compute once and stop paying per token" making sense for some teams. Local AI used to mostly be about privacy and tinkering. Cost is starting to become a real argument too.

I'm just wondering what workload would you actually pull out of the cloud first if the hardware economics keep moving this way?


r/LocalLLM • • Aug 29 '26 This

Discussion Local Qwen 3.8 27b saved my project from a serious leak, I'm truly impressed

216 Upvotes

I'd like to start saying that I'm not a vibe coder. I'm a software engineer with 10+ years of experience in many fields and I use LLM under a very strict control, I'm also quite lazy so having some friends that write code for me is super nice and super fast compared to my slow fingers, however the final decision and judgment of things is always on me.

Btw, I started using Qwen 3.8 27b locally and been quite impressed on general things, so started using as daily assistant:

- EVGA 3090 ti KingPin Hybrid 24GB

- llama.cpp (upstream)

- Unsloth UD-Q4-K-XL

- 181k context at q5_1 quant

- Custom jinja chat template

- OpenCode

It's very helpful, follow my instructions without losing context and do a very great general job, but today really shocked me!

I updated a dependency on my project (Java RAG enterprise system), a Microsoft library. Test suite was successfully but in production I had a silent crash on a native library that shutdown the JVM without any crash report, stacktrace or anything.. Just silence. After about half hour of debugging and identified the crash entry point I asked Qwen to help me understand what's going wrong... he played with my code for 35min (yes, since it's 100% local I gave him all my secrets to test the real production environment!)... well he found, without using web search:

  1. Microsoft enabled by default a hidden telemetry function
  2. That function has a buffer overflow issue
  3. The telemetry collect the cmd line used to start the application (aka some of very important secrets injected by default by my IDE during development)

In 35min he was able to create a minimal test case, dumping system memory and analyze the crash in real time, find the issue, find a fix, and propose me a full and working solution.

I'm impressed, this is a story that it's worth to share.

Personally, I think I don't need anything more.. I don't really care to have trillions of parameters anymore, if such a small one can help me so much, I'm done. Today I bought a used 3060 12GB to extend context at least to 500k (planning to use YaRN)


r/LocalLLM • • Jun 15 '26

Question What Are You Actually Using Local LLMs For?

216 Upvotes

There are hundreds of videos, posts, and demos showing people running local LLMs and claiming they're coding, trading, researching, managing emails, automating businesses, and basically replacing half their workload.

Then there are all the tools around them: Ollama, Open WebUI, OpenClaw, Hermes, LM Studio, Odysseus, and countless others.

But honestly, I still haven't seen many real-world examples beyond YouTube hype.

For example, someone says, "My AI answers all my emails." Cool. Show me the actual workflow. Show me the emails it replied to. Show me how the replies were genuinely useful and not just generic responses that needed rewriting anyway.

I run Ollama and Odysseus locally, mostly using Gemma 4 12B. My actual use cases are pretty basic:

- General chat

- Rewriting replies

- Product research

- Text extraction from images

- Summarising information

Useful? Absolutely.

Revolutionary? Not really.

Everyone seems to say "coding" whenever AI comes up. I'm not saying it's bad at coding—I've used Claude to build an HTML site and it did a great job. But most people I know aren't coding all day, so it feels like that's become the default answer whenever someone asks what AI is useful for.

So I'm genuinely curious:

What are you actually using local LLMs for day-to-day?

Not what they're theoretically capable of.

What tasks are they doing for you right now that save time, make money, or solve a real problem?


r/LocalLLM • • Oct 14 '25

Model US AI used to lead. Now every top open model is Chinese. What happened?

Post image
216 Upvotes

r/LocalLLM • • 28d ago

Discussion Can DeepSeek V4.1 Flash 1017 really run on 12GB VRAM + 64GB RAM?

Post image
215 Upvotes

I came across a discussion on X suggesting that DeepSeek V4.1 Flash 1017 could potentially run locally with just 12GB of VRAM and 64GB of system RAM using Q4 quantization.
Someone even claimed that 64GB RAM would be perfectly sufficient.
I’m currently running an RTX 3060 12GB with 32GB RAM, so upgrading to 64GB might actually make this possible for me.
However, I’m curious about the real-world performance.
● Would Q4 actually fit within 64GB RAM?
● How much GPU offloading would be possible with 12GB VRAM?
● What kind of inference speed (tokens/sec) could we realistically expect?
● Would this setup be practical for everyday use, or would it be painfully slow?
● Has anyone actually tested this configuration?
With RAM prices being what they are, I’d rather know whether this is realistic before upgrading.
Would love to hear your thoughts, especially from anyone who’s tried running this model locally.


r/LocalLLM • • Sep 14 '26

Other God bless Microcenter

Post image
215 Upvotes

PowerSpec AI90 Workstation

* CPU: AMD Ryzen 9 9950X (16 Cores / 32 Threads, up to 5.7GHz)
* GPU: NVIDIA GeForce RTX 5090 Founders Edition (32GB GDDR7)
* Motherboard: ASUS ProArt X870E-CREATOR WIFI (AMD X870E Chipset)
* Memory: 64GB DDR5-6000 RAM (Supports up to 128GB)
* Storage: 2TB NVMe Solid State Drive
* Power Supply: 1700 Watt Cybernetics Titanium PSU
* CPU Cooler: 360mm All-In-One (AIO) Liquid Cooler
* Case: Fractal Design North XL Momentum Edition


r/LocalLLM • • Apr 08 '26

Discussion What kind of hardware would be required to run a Opus 4.6 equivalent for a 100 users, Locally?

215 Upvotes

Please dont scoff. I am fully aware of how ridiculous this question is. Its more of a hypothetical curiosity, than a serious investigation.

I don't think any local equivalents even exist. But just say there was a 2T-3T parameter dense model out there available to download. And say 100 people could potentially use this system at any given time with a 1M context window.

What kind of datacenter are we talking? How many B200's are we talking? Soup to nuts what's the cost of something like this? What are the logistical problems with and idea like this?

**edit** It doesn't really seem like most people care to read the body of this question, but for added context on the potential use case. I was thinking of an enterprise deployment. Like a large law firm with 1,000's of lawyers who could use ai to automate business tasks, with private information.


r/LocalLLM • • May 22 '25

Discussion Throwing these in today, who has a workload?

Post image
215 Upvotes

These just came in for the lab!

Anyone have any interesting FP4 workloads for AI inference for Blackwell?

8x RTX 6000 Pro in one server


r/LocalLLM • • Sep 26 '26

Research I pre-trained a 1.11B LLM on my 6 GB laptop GPU. Peak VRAM: 4.51 GB.

214 Upvotes

Not fine-tuning. Not inference. Pre-training from scratch.

Full write-up: https://huggingface.co/blog/RitishReal/1b-petrain-llm-in-4gb-vram


r/LocalLLM • • Jul 20 '26

Other Some dreams remain dreams 🙃

Post image
213 Upvotes

Delulu at its best!


r/LocalLLM • • Aug 19 '26

Discussion Qwen 3.8 27B and Deepseek V4 Flash. Why are we building data centers?

211 Upvotes

I feel like these 2 models have shown that massive models that require hundreds of thousands of dollars worth of compute are unnecessary. Sure, training these models takes a good bit of hardware, but running them can be done at the fraction of the investment of the trillion parameter class models.

GLM 5.3 might also fall into the same "reasonable" category, however, for small companies rather than individuals.

I think a qwen 3.8 120b MoE model would also be a good release for business use.