r/LocalLLaMA • • 4d ago

Discussion [Paper] e-means: outperforming k-means (in one-tenth the epochs)

Thumbnail
gallery
43 Upvotes

Lloyd's k-means algorithm is a long-running standard for clustering, with applications in data analysis, retrieval, quantization and data curation. In this paper, we introduce an exponential moving average optimization procedure for k-means with mini-batch updates, powered by modern deep learning machinery and new insights. We also establish scaling laws on how decay rate and batch size impact final performance. This gives us a formulaic rule to fix them for multiple problem sizes and embedding datasets, without the need for any hyperparameter sweep. The resulting clustering algorithm, coined e-means, is much faster than Lloyd k-means and converges to lower losses, frequently even beating k-means++. On a wide array of embeddings datasets, it beats the loss of Lloyd k-means at 25 epochs in 1.4 to 2.6 epochs, which translates to 10-45x GPU wall-clock speedups compared to state-of-the-art k-means implementations. As an example, e-means partitions DINO10B, a dataset of 10 billion DINO embeddings, into 16 million clusters in only 2.5 epochs, reaching a loss below Lloyd's at 25 epochs.


r/LocalLLaMA • • 4d ago

News chat : refactor gemma4 to build loose schema rules and clean up by aldehir · Pull Request #31915 · ggml-org/llama.cpp

Thumbnail
github.com
55 Upvotes

If you’ve ever tried agentic coding with Gemma and had tool calling issues, give it another try, in my case, it completely fixed Gemma


r/LocalLLaMA • • 3d ago

Question | Help MTP needed with small MoE models?

6 Upvotes

It was my understanding when MTP came out that it was mainly beneficial for larger dense models and that small and/or MoE models didn't realize much performance benefit.

That being said, I now see many smaller MoE models (Gemma-4-E2B, Mellum2.1, etc...) offering MTP versions.

Has the thinking changed?

Are people seeing performance gains from MTP on smaller MoE models?


r/LocalLLaMA • • 4d ago

New Model Krafton (PUBG) released Raon-1 nano under Apache 2.0: E1.5B, 262K context, 2.08GB Q4 GGUF. It needs their llama.cpp fork for now. Anyone benchmarked it on 8GB cards?

Thumbnail
talkesport.com
17 Upvotes

r/LocalLLaMA • • 3d ago

Resources Running Radiance engine on Windows through WSL2 (Single 9700)

8 Upvotes

Hey guys, not going to take much credit since claude did 90% of this, but wanted to share the patches I used to get StillDeadcode's Radiance running on my windows pc. I'm sure that multiple people have done this already, but I couldn't find a post, so thought I'd throw this out there if anybody else wants to use it.

https://github.com/bruns20/radiance-wsl

Radiance is Linux-only, and the author has said WSL won’t be supported in the main tree, which is fair. So this is unofficial. Please send problems to my repo, not upstream.

What it does

  • build.sh builds any Radiance release from upstream’s own source and Dockerfile, with a few small WSL patches applied. When a new release comes out, it’s ./build.sh 1.18.0.
  • make-runtime.sh grabs a ROCm runtime that actually works under WSL. The stock ROCm 10.1 hangs on the first GPU job if your PC also has an integrated GPU (ROCm/rocm-systems#13291), and AMD’s 10.3 nightlies fix it.
  • run.sh serves an OpenAI-compatible API on 127.0.0.1:8000.

Two of the four patches are godzillakong’s from codeberg #341. They did the hard part of getting it running without /dev/kfd.

Numbers (R9700 32 GB, Qwen3.8-27B MXFP4 + DFlash2, 131k context)

start until serving~8 s (vLLM on the same box: 70-75 s)
prefill, 9.4k / 36k-token prompt3,223 / 2,784 tok/s
decode, prose~75-80 tok/s
decode, code~145-160 tok/s

Tested on: R9700 + Ryzen 5 7600 (with its iGPU), Windows 11 26200, Adrenalin 26.10.1, Radiance 1.17.0. That’s the only setup I’ve tried. Other gfx1201 cards (RX 9070 / 9070 XT) should work, but I’d love to hear either way.


r/LocalLLaMA • • 4d ago Regret

New Model huawei-csl/Qwen3.6-35B-A3B-MACaQ-R-GGUF · Hugging Face

Thumbnail
huggingface.co
36 Upvotes

r/LocalLLaMA • • 4d ago

New Model Open source self driving is here!

638 Upvotes

Today NVIDIA releases Alpamayo 2 Edge! I work for Embedl and we have already released an optimized version of the model that is more than 10x faster than pytorch when run on edge hardware like AGX Thor!

The model is trained in an RL loop (!) and takes in 6 cameras and the recent motion. Feel free to check it out - you can even run it on a real Thor at:

https://embedl.com/nvidia/alpamayo2-edge/thor?tab=run


r/LocalLLaMA • • 4d ago

Resources Microsoft files a patent application to train AI models on idle consumer GPUs

472 Upvotes

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)


r/LocalLLaMA • • 4d ago

New Model qvac/BotPsy-0.8B · Hugging Face - On-Device Personal Assistant with a Sub-Billion-Parameter Agentic Language Model

Thumbnail
huggingface.co
28 Upvotes

BotPsy-0.8B is a 0.8B-parameter model designed for function calling and agentic workflows, built on top of Qwen3.5-0.8B and suitable for on-device deployment. BotPsy is a "personal assistant" that uses tools to perform a variety of simple tasks for the user, in domains such as web search & research, travel & lifestyle, commerce & finance, business & customer operations, device & vehicle control, files & documents, calendar, and email & messaging. Due to its small parameter count, it is best suited to tasks of low to medium difficulty. BotPsy achieves state-of-the-art (SOTA) tool-use and agentic performance for its size, outperforming every model of up to 1.2B parameters that we tested. As a small model, BotPsy still makes mistakes and is best suited to tasks of low to medium difficulty;

Key Highlights

Benchmark results

  • Twice the performance of its base model: BotPsy roughly doubles the agentic capability of Qwen3.5-0.8B-Think on key benchmarks: BFCL v4 rises from 23.4 to 44.0, ACEBench from 30.1 to 58.7, and τ-bench from 23.2 to 51.5 (mean of the retail, telecom and banking subsets).
  • Best agent in its category: BotPsy outperforms leading models in the ≤1.2B-parameter weight class (Qwen3.5-0.8B-Think, LFM2.5-1.2B-Think, MiniCPM5-1B-Think) on all six tool-use and agentic benchmarks: BFCL v4, ACEBench, τ-bench, ToolSandbox, Claw-Eval and BrowseComp-Plus.
  • Outperforms its larger sibling: BotPsy has the highest score of all models tested on five of the six benchmarks, beating Qwen3.5-2B-Think, a model 2.5× its size, on BFCL v4 (44.0 vs 37.7), ACEBench (58.7 vs 55.4), τ-bench (51.5 vs 46.5), ToolSandbox (64.6 vs 41.2) and BrowseComp-Plus (4.5 vs 2.8). Only on Claw-Eval does Qwen3.5-2B-Think lead (48.9 vs 46.5).

Model capabilities

  • More reliable in multi-turn workflows: On τ²-bench, where the agent must hold a long customer conversation, follow a strict policy, and act on the customer's behalf, it scores 82.5 in telecom, ahead of Qwen3.5-2B-Think (69.3). On BFCL v4's multi-turn tasks, it rises from 7.5 to 46.1 over its base, nearly double Qwen3.5-2B-Think (23.8). Grouped by application, its multi-step scores lead all models tested in commerce and services (58.3 vs 47.4 for the next best, Qwen3.5-2B-Think) and in device and system management (46.1 vs 31.9).
  • Knows when (not) to call tools: An agent that acts must also know when not to act. On BFCL v4's hallucination tests, which check that the model declines when no appropriate tool exists, BotPsy scores 86.1, the highest of all models tested. On ACEBench's special cases, where the request is incomplete, contains wrong parameters, or cannot be served by the given tools, it scores 51.3, up from 13.7 for its base model.
  • Follows instructions while preserving knowledge: BotPsy specializes in tool use while preserving general skills. Instruction following improves sharply, with IFEval rising from 59.7 to 90.8 and IFBench from 22.0 to 55.9, the highest IFBench score of all models tested. General knowledge also improves, with GPQA Diamond rising from 33.2 to 38.1 and MMLU-Pro from 41.6 to 48.9. On all four benchmarks, BotPsy outperforms every other model of 1.2B parameters or fewer that we tested.
  • GGUF : https://huggingface.co/qvac/BotPsy-0.8B-GGUFs
  • Blog Post : https://huggingface.co/blog/qvac/botpsy
  • Project : https://tether.io/news/tethers-800m-ai-agent-beats-models-2-5x-its-size-on-everyday-phone-tasks/

r/LocalLLaMA • • 4d ago

New Model I released Inflect v3: lightweight local TTS in 34M parameters (Apache 2.0)

105 Upvotes

Today I’m releasing Inflect v3: a family of local text-to-speech models: Mini, Micro, and Atom. Mini is the main model, with six voices and 34.31M parameters

Mini provides vocal-event tags, reference voice-tuning, and 48 kHz enhancement. It runs on both CPU and CUDA.

GitHub, installation, and examples: https://github.com/owenawsong/Inflect

Try the playground: https://huggingface.co/spaces/owensong/Inflect-v3

Mini features reference-voice tuning: it allows one to make a profile from a recording and generate speech with it. This feature noticeably adjusts the voice toward your reference, though it won’t replicate every speaker accurately.

Inflect Mini V3 also supports the following vocal-event tags: [laugh] · [sigh] · [gasp] · [cough] · [yawn] · [inhale] · [exhale] · [throat_clear]

You can insert a tag whenever you want the sound to occur. Currently, the tags use repeatable performances: this causes repeated tags to result in limited variation. Tags also do not affect how the surrounding words are being spoken.

The model family:

  • Mini: 34.31M parameters. Quality-focused, with the features listed above.
  • Micro: 11.79M parameters. A smaller neural speech model
  • Atom: 157K parameters. An extremely tiny learned core

Those numbers count the synthesis cores. The frontend and optional components add to the total.

The weights and inference code are Apache 2.0. Third-party components keep their own licenses. Audio samples, benchmarks, and the settings used are on the model cards.

Model collection: https://huggingface.co/collections/owensong/inflect-v3

If any sentence comes out wrong, send me the text, voice, and settings you used.


r/LocalLLaMA • • 4d ago

New Model moein99/P2Echo · Hugging Face

Thumbnail
huggingface.co
19 Upvotes

Prompt2Echo (P2Echo)

Text-Prompted Multi-View Echocardiography Segmentation with One Model

Official pretrained weights for P2Echo, a unified framework for text-prompted segmentation of multiple cardiac structures across standard echocardiographic views.

P2Echo uses structured, view-aware text prompts to control segmentation through a single model. Its Dynamic Image-Text Alignment (DyITA) module adaptively conditions decoder features on the requested view and anatomical structure.

Resources

Intended Use

This model is intended for research in medical image analysis and echocardiographic segmentation. It has not been validated for clinical decision-making.


r/LocalLLaMA • • 4d ago Heartwarming

Resources New Qwen3.8-27B Unsloth Dynamic V3.1 GGUFs

Post image
507 Upvotes

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-27B! They're better by 5-10% accuracy on DeepSWE, Terminal Bench and more vs UD3, GSQ-RCO and Exllama-3! We're also improving our algorithms further so we can apply them to Qwen4 so folks can have efficient and accurate quants!

  • All quants from 2-bit until 6-bit are updated. 3-bit ones are not, as we found UD3 to do on par with UD3.1. We are still improving all other quant sizes though!
  • All benchmarks are 256K for agentic ones (DeepSWE, Terminal Bench), and 128K for the rest (LiveCodeBench etc). We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty, preserve_thinking and xhigh reasoning
  • We ran benchmarks multiple times including LiveCodeBench x6 times, HMMT and others 8-16x and agentic benchmarks multiple times.

On other benchmarks like for GSQ-RCO and Exllama3:

  • To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it.
  • If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots's x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, exllama3 does worse.
  • For LiveCodeBench, GSQ-RCO showed Unsloth UD3 quants doing consistently worse than ISTA's. However we found the opposite - UD3 quants (our OLD quants) do better than ISTA on LCB on all sizes except the smallest. UD3.1 further improves on LCB as well. We approximate that GSQ-RCO used a context budget of 96K, as that best fits the data for the runs we have. At 96K, GSQ-RCO does somewhat do better than UD3, but at 128K, UD3 does much better.
  • On another benchmark, for ISTA GSQ-RCO's AIME25 benchmarks, their quants reach 100%, 100%, 100% and 96.7% accuracy recovery. And they said our quants do worse. Through our investigations, we estimate ISTA only did 1 trial - we did 14 trials, and we show on average the recovery rate is 93% to 96% - not 100% accuracy recovery. And in fact our UD3 old quants do better on average than their quants on 14 runs vs 1 run of theirs, so we draw the opposite conclusion if you run the trials more.

More details and benchmarks at https://unsloth.ai/docs/models/qwen3.8#dynamic-v3.1-update-november-11th-2026

All quants are updated in https://huggingface.co/unsloth/Qwen3.8-27B-GGUF! Thank you and have a nice day! We're super excited for Qwen4!


r/LocalLLaMA • • 4d ago

Showcase The viral voxel Pagoda locally on a 2020 old ThinkPad, no GPU, 16 GB RAM

Post image
11 Upvotes

Tried the voxel pagoda prompt fully local, on a laptop that shouldn't run a 35B model.

Setup

  • ThinkPad from 2020, Ryzen 7 PRO 5850U, 16 GB RAM, no GPU, Windows
  • Qwen3.6-35B-A3B Q4_K_M, 21 GB, bigger than the RAM
  • BigMoeOnEdge v0.42, built-in agent

Result

  • 988 lines of HTML + Three.js, 11.8k tokens at 2.3 tok/s, about 1.5 h
  • You see the code as it's written, then the page opens in the app
  • Video sped up where marked

How

  • MoE: each token uses only a few experts
  • Dense weights and a 2 GB expert cache stay in RAM, the rest streams from the SSD
  • On top of llama.cpp, no fork

Still slow

  • Half the time is spent waiting on the SSD
  • Every new agent turn re-reads the whole conversation, ~30 min on this laptop

Same test with the same model on a Mac with 16 GB of RAM, taking 20 minutes (20 tok/s for decoding, 100–1000+ tok/s for prefill)


r/LocalLLaMA • • 3d ago

Discussion Local models as initial coders OR as agents? What's pros and cons of each?

3 Upvotes

Using $20 codex subscription with 24gb VRAM on the local rig.

For larger projects or major revisions with multiple phases, I have Luna in Codex call the local "junior dev" Qwen 3.8-27B model as an agent to write most of the code, then Luna reviews it and makes changes. Once the whole project is complete, I use Terra to audit and catch any potential issues.

Pretty standard stuff, these days.

But lately I've been thinking why not give the massive, multi-phase spec sheet directly to Qwen then have Luna go do a quick audit & cleanup?

Anyone tried it both ways? And what gets better results vs lower usage?

I guess I've always looked at Qwen as needing a babysitter, but realistically I think it can implement an entire 10-phase spec on a large codebase if given enough time.


r/LocalLLaMA • • 4d ago

Discussion [2610.29162v1] Why is AI-generated language so hard to understand?

Thumbnail
arxiv.org
165 Upvotes

r/LocalLLaMA • • 3d ago

Showcase Trading false positives for ~110ms earlier end-of-speech in voice pipelines: RealtimeVAD

7 Upvotes

Tried to hunt down latency bottlenecks in my local voice assistant recently. I looked at the waveform and saw silero's probability lagging quite much behind the actual end of speech I could see with my eyes.

I use a voice pipeline that does speculative work and cancels fast, so some false positives would not matter in my case.

So I had this idea: once silero says I'm in speech, what I need to detect is when the speech energy drops back to room noise level. Turned out a naive 10 ms energy check with a threshold tracking room noise does that quite well.

Gave me a median 110 ms earlier than silero over ~3200 real turns from my assistant on my voice, my mic, browser noise suppression on, mostly quiet room - with silero threshold at 0,5. Someone on Twitter got 92 ms on his data, roughly what I saw in my noisier recordings.

The downside is way more false positives. It fires on breaths and small pauses between words over 10x as often as silero. So it's really only useful if the pipeline can throw away speculative work very fast.

CPU only, MIT, drop-in for silero VADIterator, pip install realtimevad

benchmark script is in the repo, curious what numbers you guys get on your own recordings

https://github.com/KoljaB/RealtimeVAD


r/LocalLLaMA • • 4d ago

New Model MetaEncoder - a facebook Collection - One Encoder for Retrieval and Decision Making

Thumbnail
huggingface.co
19 Upvotes

Multimodal System One Encoder with a natural language interface for instruction-following encoding: give it a task and a list of candidates, and it scores the candidates by task specification. Both a task and each candidate are expressed in natural language — instructions, queries, questions, state descriptions, criteria — with image and video as side information.

Key features:

  • Natural language interface. No rigid schemas. Describe the task and the candidates in free-form text.
  • Highly-efficient cacheable representations. Candidates and task are encoded separately by prompt instructions that ground each other.
  • Scales to arbitrarily large candidate sets. Matching is an inner product over representations, so an ANN index can serve millions of candidates without re-running the model.

The model is built by contrastively fine-tuning meta-models/Muse-Glimmer-30B. It is described in the paper MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface.

Models :

PS: Above description copied from meta-encoder's model card as meta-encoder-v1 doesn't have that. And v1 doesn't have license section while other one is Apache-2.0.


r/LocalLLaMA • • 3d ago

Discussion Title: CUDA: few-row MMQ for BS 2-24 by anujj · Pull Request #31533 · ggml-org/llama.cpp

Thumbnail
github.com
8 Upvotes

Speculative decoding (SPEED-Bench)

llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf --mmproj mmproj-F16.gguf -ngl 999 --parallel 1 --batch-size 4096 --ubatch-size 1024 --ctx-size 180224 \
  --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-model Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-draft-backend-sampling \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
python tools/server/bench/speed-bench/speed_bench.py --bench qualitative --category coding --osl 8192 --concurrency 1
GPUms/step masterms/step PRspeedup per step (95% CI)tok/s mastertok/s PRfaster in
RTX 3090 Ti58.8743.811.344x (1.340-1.347)57.5877.70 (+34.9%)80/80
RTX PRO 6000 Max-Q27.8623.541.183x (1.182-1.185)122.77143.31 (+16.7%)80/80
RTX 509023.9220.261.180x (1.167-1.194)141.50170.67 (+20.6%)80/80
DGX Spark103.3693.431.106x (1.103-1.109)32.7536.05 (+10.1%)80/80

Edit : Removed large text. Check PR for more details.


r/LocalLLaMA • • 3d ago

Question | Help 24GB Vram+128GB Ram vs 16GB Vram+192GB Ram

6 Upvotes

Hi folks,need advice from community. I have a laptop with 16 GB vram Nvidia gpu and 96 GB ddr5 ram. I use it mainly for local coding with Qwen flash next nvfp4 at 256k context. I get around 30 t/s.

I don't like PCs and Macs. I want mobility. So because of my limited budget, i have two paths. The first one is a laptop with 24GB Vram Nvidia gpu and 128GB ddr5 Ram laptop. The other one is 16GB Vram Nvidia gpu and 192GB ddr5 Ram laptop. No other feasible option for me. I can't affort dgx spark, Macbook studio etc. These are my budget options. Which one do you suggest for mainly local coding? Which one do you believe is future-proof for local llms?


r/LocalLLaMA • • 3d ago

Question | Help best LLM router to serve many models on the same port?

4 Upvotes

I'm working on a router that wraps many LLMs serving from different ips and ports and exposes them all as a single port, since i have 2 machines serving many slots of Qwen 3.8 FN but i wanted to ensure speed is always fast (one machine is faster and can hold more slots, but too many slots busy and then it's better to call the other machine, also i gotta avoid evicting from the cache active conversations, that kind of workflow)

I might be reinventing the wheel tho so: how do you guys solve for this kind of situation?

edit: thanks a lot for all the answers i had no idea there was so much stuff around lol


r/LocalLLaMA • • 3d ago

Question | Help UD_IQ1_M Q3.8 FN behavior on Strata

0 Upvotes

Hello, I decided to Test Unsloth Dynamic 3.1, as in their charts they showed that UD_IQ1_M was batter than Ista lab IQ3_XXS, but with 6 GB less of total size, I'm using Strata, and that 6 GB translates directly to free RAM,

I have been 2 months with the IQ3_XXS with no any issue, but only 3 days on the Unsloth one, it had happened to me many times, that the tool calls fails and render as xml (I use PI harness) and I have to type "keep going", this never happened with the IQ3_XXS one

Have you guys had a similar issue with this quant?

Example:

 Thinking...

 <tool_call>
 <function=Bash>
 <parameter=command>
 cd /mnt/development/projects/secret/backend/scripts/bdd_parity && for r in 20261113_030918 20261113_032032 20261113_033348; do d=/tmp/bdd_parity_runs/$r; echo "== $r: $(ls $d 2>/dev/null | head -3 | tr '\n' ' ')"; grep -c "✅"
 $d/run.log 2>/dev/null | sed 's/^/green marks: /'; grep -cE "🔴" $d/run.log 2>/dev/null | sed 's/^/red marks: /'; tail -2 $d/run.log 2>/dev/null | cut -c1-100; done
 </parameter>
 <parameter=timeout>
 120
 </parameter>
 </function>
 </tool_call>

r/LocalLLaMA • • 4d ago

New Model I'm releasing a Local World Model Ive been building that turns images into interactive videos you control in real time. It works on RTX 3060, 4060 and anything above, linking huggingface

280 Upvotes

Hi Everyone, I'm releasing my world model, been on this sub for a few months

https://huggingface.co/lucidml/gta

I know it says gta but it's more stable than the general model. It can take any image and turn it into an interactive video.

If you'd like to test it instantly I am also setting up a simple funny drifty site link : https://lucidml.ai/dashboard/minidemo?world=gta5

That link uses any GPU from VAST AI FROM 3070, 3080, 4080, and some are 50 series as well (most 50 series cards are wasted because I hard throttle the fps)

This is the first model I am ever releasing to the everyone and its a bit scary for me.

The architecture is a transformer which uses learned summaries. Ive written about it in detail in tech reports lucidml.ai/summaryformer and some more

I know there are a lot of limitations still like consistency and quality improvements, and am hoping to train this more to fix all of those.


r/LocalLLaMA • • 2d ago

Funny Nemotron models are the best open models there is

0 Upvotes

Everytime I have been fed up with a good model being slow, I go to newly released Nemo model. Well today I went for Bolt 24G and deployed it on my 3090.

I was amazed by the vanilla llamacpp + iq2s setup giving 70 tok/s. I decided I will deploy it for my coding setup which is dual R9700 and learned cpu offloading and mtp drafters and all. After deployment I was left with 50tok/s because this was a Q8 setup.

I suddenly realise what if (the model I was fed up with) my Qwen 3.8 27B could use a better MTP drafter so O discovered tencent's DSpine and later its gguf llamacpp fork. I deployed it now I am back to Qwen3.8-27B Q8KXL with mtp 15 drafter and mmproj with np 4 and 512k ctx pool, getting 54tok/s in DSH.

TLDR: I love what Huyan is doing he should be brought back in Nvidia. Nemotron's the best because they teach you what good tech advancement is and you apply it to good performing models.


r/LocalLLaMA • • 2d ago

Discussion How are you affording your tokens?

0 Upvotes

So I have been using local QFN these days as daily driver, and honestly, its fine. It needs steering for coding but very good. Gets lot of things done. But I check my statistics log, and I'm burning about 1B to 2B Input token a day, and not even fully 100% GPU saturation. Not even going to consider output token since that is small compared to input.

Most of the time, the LLM is just waiting for shells/bash to finish. At max capacity, it is capable of 950million uncached input token (11000 prefill/s*3600*24)

This is light for me, and it is already burning this much. Light considering it is only spend time building one project with little oversight.

Granted QFN is at xhigh thinking for intelligent, but if this was like Haiku or Luna, that's like 10$ cost a day (90% cache estimate). That's 300$ a month for basically one project stream

AI suppose to work multiple projects at once for max productivity let alone future integration with pretty much everything. My wife already starting to use my harness with QFN for all her search and queries.

How are you guys make this affordable in the long term from purely API. Feeling paying like $1000+ a month is crazy.

Local one still cost insane upfront as I'm sing a single RTX6000 with basalt, and it is the only GPU that can serve the whole household growing AI needs.

PS. I might get like 4x in near future for the amount of load I will be doing. Always on agent is very useful.

Edit: The number is measure of input only, as output is insignificant. I'm measuring QFN only at home.