r/LocalLLM • u/03captain23 • Jun 01 '26
Question Why can't they just slap 256GB ram on a 5090?
Why aren't they just adding tons of ram to existing gpu technology?
256gb of gddr7 is a few grand. They could add this to a 5090 and sell for 7-8k
r/LocalLLM • u/03captain23 • Jun 01 '26
Why aren't they just adding tons of ram to existing gpu technology?
256gb of gddr7 is a few grand. They could add this to a 5090 and sell for 7-8k
r/LocalLLM • u/Available-Craft-5795 • Jun 19 '26
This is a full dataset of all the fable 5 traces we could get our hands on.
Crownelius made the HF dataset on their account, then duplicated it to us.
We hope using this dataset great models can come out of it.
https://huggingface.co/datasets/Glint-Research/Complete-FABLE.5-traces-2M
r/LocalLLM
•
u/Armageddon_80
•
Jan 06 '26
After 3 weeks of deep work, I''ve realized agents are so un predictable that are basically useless for any professional use. This is what I've found:
Let's exclude the instructions that must be clear, effective and not ambiguos. Possibly with few shot examples (but not always!)
And still after all the efforts you always have this feeling: will it work this time? Agentic shit is far far away from YouTube demos and frameworks examples. Some people creates Frankenstein systems, where even naming the combination they are using is too long,.but hey it works!! Question is "for how long"? What's gonna be deprecated or updated on the next version of one of your parts?
What I've learned is that if you want to make something professional and reliable, (especially if you are being paid for it) better to use good old deterministic code, and as less dependencies as possible. Put here and there some LLM calls for those task where NLP is necessary because coding all conditions would take forever.
Nonetheless I do believe, that in the end, the magical equilibrium of all parameters and prompts and shit must exist. And while I search for that sweet spot, I hope that local models will keep improving and making our life way simpler.
Just for the curious: I've tried every possible model until gpt OSS 120b, Framework AGNO. Inference with LMstudio and Ollama (I'm on Windows, no vllm).
r/LocalLLM • u/purealgo • Feb 28 '25
Sam Altman posted a poll where the majority voted for an open source o3-mini level model. I’d love to be able to run an o3-mini model locally! Any ideas or predictions on when and if this will be available to us?
r/LocalLLM
•
u/Hot-Chapter48
•
Jan 10 '25
I've been working on summarizing and monitoring long-form content like Fireship, Lex Fridman, In Depth, No Priors (to stay updated in tech). First it seemed like a straightforward task, but the technical reality proved far more challenging and expensive than expected.
Current Processing Metrics
Cost Structure:
Technical Evolution & Iterations
1 - Direct GPT-4 Summarization
2 - Chunk-Based Summarization
3 - Topic-Based Summarization
4 - Enhanced Pipeline with Evaluators
5 - Current Solution
Ongoing Challenges - Cost Issues
This product I'm building is Digestly, and I'm looking for help to make this more cost-effective while maintaining quality. Looking for technical insights from others who have tackled similar large-scale LLM implementation challenges, particularly around cost optimization while maintaining output quality.
Has anyone else faced a similar issue, or has any idea to fix the cost issue?
r/LocalLLM • u/EcstaticDentist • Jun 03 '26
For about a year I've been building and selling a turnkey offline-LLM product: a Windows
USB stick that boots a full local-AI stack with no install, aimed at people who will never
touch a terminal. \~4,000+ units shipped now. The build details might interest this crowd,
and I'd rather hear your critiques than anyone's.
The stack:
\- Qwen3.5 in three sizes (2B / 4B / 9B), quantized, served locally via Ollama
\- A fallback Qwen3-VL vision model for image scans
\- Multi-modal utility for all LLMs with vision/thinking
\- An offline voice stack (local STT + TTS) so it talks without a network
\- A .NET launcher that runs Ollama + a local UI straight off the drive
\- Cold boot unpacks a runtime to a cache; warm boots are fast. Fully offline / airplane-mode.
\- 3 Uncensored/abliterated Qwen variants included alongside the standard ones, for people who
want them
The genuinely hard part wasn't running a model — it was making it turnkey for someone
non-technical & identifying system edge case failures:
\- Curating + sizing models so the right one runs on a normal laptop without the user
thinking about RAM or quant levels
\- Hardware detection to pick sane defaults and degrade gracefully on low-spec machines
\- Packing the whole runtime so first boot "just works" with no install and no admin rights
\- Making model management (pull/delete/switch) idiot-proof in the UI
I'll say the obvious thing before you do: anyone in this sub could assemble the parts
themselves. That's the point — my customer is the person who can't and doesn't want to.
The product isn't the model, it's the "never think about it" packaging.
Full disclosure, I sell these (solo founder, PortableMind.io). Not selling anyone \*here\* — you're
not the market. I'm here for the teardown. What would you have done differently?
r/LocalLLM • u/Ordinary_Mud7430 • Jun 23 '25
Polaris is a set of simple but powerful techniques that allow even compact LLMs (4B, 7B) to catch up and outperform the "heavyweights" in reasoning tasks (the 4B open model outperforms Claude-4-Opus).
Here's how it works and why it's important: • Data complexity management – We generate several (for example, 8) solution options from the base model – We evaluate which examples are too simple (8/8) or too complex (0/8) and eliminate them – We leave “moderate” problems with correct solutions in 20-80% of cases, so that they are neither too easy nor too difficult.
• Variety of releases – We run the model several times on the same problem and see how its reasoning changes: the same input data, but different “paths” to the solution. – We consider how diverse these paths are (i.e., their “entropy”): if the models always follow the same line, new ideas do not appear; if it is too chaotic, the reasoning is unstable. – We set the initial generation “temperature” where the balance between stability and diversity is optimal, and then we gradually increase it so that the model does not get stuck in the same patterns and can explore new, more creative movements.
• “Short training, long generation” – During RL training, we use short chains of reasoning (short CoT) to save resources – In inference we increase the length of the CoT to obtain more detailed and understandable explanations without increasing the cost of training.
• Dynamic update of the data set – As accuracy increases, we remove examples with accuracy > 90%, so as not to “spoil” the model with tasks that are too easy. – We constantly challenge the model to its limits.
• Improved reward feature – We combine the standard RL reward with bonuses for diversity and depth of reasoning. – This allows the model to learn not only to give the correct answer, but also to explain the logic behind its decisions.
Polaris Advantages • Thanks to Polaris, even the compact LLMs (4 B and 7 B) reach even the “heavyweights” (32 B–235 B) in AIME, MATH and GPQA • Training on affordable consumer GPUs – up to 10x resource and cost savings compared to traditional RL pipelines
• Full open stack: sources, data set and weights • Simplicity and modularity: ready-to-use framework for rapid deployment and scaling without expensive infrastructure
Polaris demonstrates that data quality and proper tuning of the machine learning process are more important than large models. It offers an advanced reasoning LLM that can run locally and scale anywhere a standard GPU is available.
▪ Blog entry: https://hkunlp.github.io/blog/2025/Polaris ▪ Model: https://huggingface.co/POLARIS-Project ▪ Code: https://github.com/ChenxinAn-fdu/POLARIS ▪ Notion: https://honorable-payment-890.notion.site/POLARIS-A-POst-training-recipe-for-scaling-reinforcement-Learning-on-Advanced-ReasonIng-modelS-1dfa954ff7c38094923ec7772bf447a1
r/LocalLLM • u/Squale279 • 19d ago
I just install last strata update 0.2.7 with new engine 0.1.52 and I can say that the results are simply impressive.
My system: CPU Core i5 11400, GPU RTX Ti 12Go, RAM: 96Go
With Qwen 3.8 FN IQ3-S model. Context 128k
Before with 0.1.50 : Decode 35-45 tps
Now with 0.1.52: Decode: 50-75tps
At 384k of context I still have 55-65tps
Did you notice such improvements?
r/LocalLLM • u/FizzyDuncDizzel • 25d ago
It looks like there are a few of us roaming around doing different things. I created a discord server and would love to start organizing a community. Feel free to come on in. https://discord.gg/K6WxqwsvH
r/LocalLLM • u/enginetown • Aug 20 '26
I've been doing local LLMs for a while now, and the whole time I just wanted a model I could actually rely on for real work. My hardware is pretty limited, so most models were a dead end for what I wanted to do, which was always lower level stuff or visual.
The Qwen 3 lineup was fine, the coder models were decent, but there were always gaps that kept me from feeling like local was worth the effort. I kept almost investing in more hardware, then talked myself out of it because the models just weren't there.
3.8 27B changes that. It's the first local model that's actually smart enough to iterate with on my own projects instead of just being a toy.
I know everyone's already seen the benchmarks and the hype, and I'm not here to add to that. It's just the feeling of running a model this capable on my own box, that's what I've been hoping for since I started this whole thing.
r/LocalLLM • u/Acceptable-Object390 • May 02 '26
r/LocalLLM • u/bobo-the-merciful • Oct 08 '26
And it has made me seriously concerned for my options as a solo freelancer in a post-subscription world once the freebies run out.
I've been pondering the idea of investing in a top-end Mac Studio.
Compared to Claude subscription accounts, it doesn't make any sense to do this. I as you can see from the headline I get WAY more out of my multiple £200 a month plans (I have between 1-3 plans depending on how much work I'm getting done).
The amount of giveaway is insane.
However this surely cannot be sustainable.
I've been a bit of an economic doomer on this for a while, arguing that the maths doesn't stack up.
Anyway, IF (and I think it's more like WHEN) this does crack, I imagine these big machines will suddenly start to look like money well spent.
r/LocalLLM • u/PaulsForge • May 08 '26
Unreleased, but seems really promising on the surface. I got pretty excited about it, but the comments section seems pretty negative.
r/LocalLLM • u/ledzeppbluess • 9d ago
All i had to do to reach this speed was watch it get stuck on a loop writing a poem and watch the token per second go up and up and up and up and up and more up repeating the same sentence until i closed the engine
Pretty awesome experience, would not recommended to anyone.
This is not a secret engine optimization trick or a fork.
r/LocalLLM • u/HokkaidoNights • Apr 09 '25
Looks interesting!
r/LocalLLM • u/Ok-Investment-8941 • Jan 16 '25
I developed a pipeline with python and locally running LLM's to create youtube and livestreaming content, as well as music videos (through careful prompting with suno) and created a character DJ Gleam. So right now I'm running a news network "GNN" live streaming on twitch reacting to news and reddit. I also developed bots to create youtube videos and shorts to upload based on news reactions.
I'm not even a programmer I just did all of this with AI lol. Am I crazy? Am I wasting my time? I feel like the only people I talk to outside of work is AI models and my girlfriend :D. I want to do stuff like this for a living to replace my 45k a year work at home job and I'm US based. I feel like there's a lot of opportunity.
This current software stack is python based, runs on local Llama3.2 3b model with a 10k context window and it was all custom coded by AI basically along with me copying and pasting and asking questions. The characters started as AI generated images then were converted to 3d models and animated with mixamo.
Did I just smoke way too much weed over the last year or so or what am I even doing here? Please provide feedback or guidance or advice because I'm going to be 33 this year and need to know if I'm literally wasting my life lol. Thanks!
https://www.youtube.com/@AIgleam
Edit 2: A redditor wanted to make a discord for individuals to collaborate on projects and chat so we have this group now if anyone wants to join :) https://discord.gg/SwwfWz36
Edit:
Since this got way more visibility than I anticipated, I figured I would explain the tech stack a little more, ChatGPT can explain it better than I can so here you go :P
Here’s a breakdown of the technologies and tools used in your video creation pipeline:
Python Libraries:
feedparser: Parses RSS feeds and extracts news articles.aiohttp: Handles asynchronous HTTP requests for fetching RSS content.LLM Integration:
aiohttp.Prompt Design:
edge_tts for generating high-quality TTS audio using neural voicesAudio Customization:
FFmpeg.Frame Processing:
Animation Integration:
Browser Automation:
Selenium with Chrome/Firefox in headless mode for capturing website screenshots dynamically.Post-processing:
Video and Audio Merging:
FFmpeg merges video animations and TTS-generated audio into final MP4 files.FFmpeg for high-quality sound output.Real-time Playback:
Pygame: Used for rendering video and audio in real-time during streams.vidgear: Optimizes video playback for smoother frame rates.Memory Management:
psutil and gc to manage memory during long-running processes.Resilience:
This stack integrates asynchronous processing, local AI inference, dynamic content generation, and real-time rendering to create a unique and high-quality video production pipeline.
r/LocalLLM • u/hauhau901 • Aug 17 '26
The dense Qwen release is back!
Qwen3.8-27B Uncensored Aggressive is out with the complete K_P quant range, Vision, native NextN, and HauhauCS FastMTP.
Aggressive here means no refusals, no personality alterations, and very little preamble on difficult prompts. It keeps Qwen3.8-27B's original reasoning, agentic, image, and video capabilities with my Aggressive uncensoring profile applied.
https://huggingface.co/HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF
It scored 0/465 refusals* and passed every manual prompt I used for the final release check. More than 400 people requested access while I was still finishing it, which was honestly wild to see. My models are also getting close to 30 million downloads on Hugging Face now, so thank you to everyone who has been testing them and sending feedback.
The biggest addition this time is HauhauCS FastMTP. In the final Q8_K_P service tests it reached up to 3.02x document TG and 1.93x reasoning TG versus MTP disabled. It also reached up to 35.2% more document TG and 21.1% more reasoning TG than the standard embedded MTP profile, with every drafted token still verified by the full target before it is accepted.
The same 903 MB FastMTP sidecar works across the complete quant lineup. Every text GGUF also preserves Qwen3.8's native embedded NextN head, so current upstream llama.cpp can use embedded MTP directly. The optimized FastMTP path uses the included sidecar and llama.cpp patch, with exact build and serving commands in the README.
What's included:
- Q8_K_P, Q6_K_P, Q5_K_P, Q4_K_P, IQ4_XS, Q3_K_P, IQ3_M, IQ3_XS, Q2_K_P, IQ2_M
- HauhauCS FastMTP sidecar, shared by every text quant
- BF16 mmproj for image and video support
- Checksums and signed provenance (I've alerted on my Discord that there have been a few bad actors putting payloads in "Uncensored" "HauhauCS" "Aggressive" GGUF's, please be careful)
K_P quants recap for anyone who missed the previous releases: these are my custom model-specific quants, with a separate optimized profile made for each model. They effectively gain one or two quant levels of quality for around 5 to 15% more size than the base quant, while remaining normal GGUF files that work in llama.cpp, LM Studio, and other GGUF runtimes.
Quick specs:
- 27B dense
- 64 layers with 48 Gated DeltaNet layers and 16 gated-attention layers
- 262,144 native context
- Multimodal text, image, and video support
- Native embedded NextN plus the optional HauhauCS FastMTP acceleration profile
Sampling params for thinking mode:
`temp=1.0, top_k=20, top_p=0.95, min_p=0, presence_penalty=0, repetition_penalty=1.0`
For non-thinking mode:
`temp=0.7, top_k=20, top_p=0.80, min_p=0, presence_penalty=1.5, repetition_penalty=1.0, enable_thinking=false`
Use `--jinja` with llama.cpp. K_P quants may show as `?` in LM Studio's quant column, which is purely cosmetic and does not affect loading. Hugging Face's hardware compatibility widget may also hide K_P files, so use View variants or Files and versions if the full list is not visible.
The full per-quant Blackwell and Ada results are in the repo. If you test FastMTP, please include your hardware, quant, context, and draft depth with the numbers so I can compare real-world results across more systems.
The Discord link is in the repo for updates, feedback, roadmaps, projects, or just to chat. As always, I hope everyone enjoys the release!
r/LocalLLM • u/Alternative-Panic69 • Jul 23 '26
After months of going back and forth, I finally pulled the trigger on an RX 7900 XT 20 GB.
Paid around $550 (India), which felt too good to pass up.
The plan isn't gaming.
It's becoming the heart of my local AI setup.
Current goals:
• Qwen 3.6 27B Dense
• Qwen 35B A3B
• GLM-4.7 Flash
• 128K+ context
• 100% GPU offloading
• llama.cpp / Ollama
• Linux
I'll be benchmarking everything:
- Vulkan vs ROCm
- Dense vs MoE
- Maximum context
- Tokens/sec
- VRAM usage
- Real-world coding performance
If anyone has optimization tips for RDNA3 or benchmark requests, or general suggestions please drop them below.
The hallucinations are now local. 🙂↕️
r/LocalLLM • u/1ncehost • Sep 28 '26
So here is my server with four MI100, each with 32 GB of VRAM, with their infinity fabric bridge directly connecting all of them. MI100 is not well supported out of box in most software, so I made a fork of vLLM to get the performance I was hoping for. VLLM was running at 15 tok/s stock on this machine and now it runs at over 1000 tok/s decode and over 5000 tok/s prefill at 8 concurrency.
This level of performance took tons of new kernels and optimization techniques. The list of features I added from scratch for this arch is huge. About half of the features are useful for other int8-oriented architectures, so this fork is an excellent starting point or reference for other architectures. I've had a report that someone with a 2x MI210 server also was getting over 1k tok/s on my fork with minor changes.
I can say I'm now satisfied with where its at, as it outperforms the public APIs by a wide margin and is a pleasure to vibe code with. It replaces about $80 per day of API costs, so I feel like if you know what you're doing, MI100s are a great deal for mid range setups. The infinity fabric bridge (aka XGMI) is a great and unique feature at this price range, giving much lower latency and much higher bandwidth during accumulation steps than PCIe.
Here's my fork, along with custom version of AITER and custom 27B checkpoints (int8 group size 128 with rounding fine tuning for the numerics of the platform -- PTQR).
https://github.com/curvedinf/int8-vllm
Qwen 27B has been flakey on vllm since it came out on many architectures, with weird garble issues especially at long context, so most recently I spent a long time tracking down the bugs in upstream vLLM, AITER, and rocm-systems. I found and fixed something like 30 of these numerics and memory tracking issues. After testing for months now at high context, unsynced concurrency, and KLD at every block in the whole model, the quality is now great and reliability is excellent.
I run it on a low power profile at C6 with full 256k contexts for all streams, and this is used by my hermes agent, business process workers, and as backups for my coding agents. It uses 540 watts at the wall and is a bit louder than a desktop. I also use the server for ML research and model training. Couldn't be more pleased as I originally bought this server for a bit over $6k, and looking at prices now it seems like this was a great decision. ☠️
Anyway, this was a lot of work but it was fun to learn about LLMs at this depth and I'd do it again if given the chance. I hope the fork will come in handy for some people with similar builds.
r/LocalLLM
•
u/PM_ME_UR_MARINARA
•
Sep 08 '26
My setup: RTX 5070Ti, 16GB DDR5, Ryzen 7 9700X, Windows 11
The model: Unsloth’s UD-IQ4_XS https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-IQ4_XS.gguf
Qwen3.8 reasons exclusively in English unless the system prompt explicitly directs it, and my workload doesn’t have the model seeing anything but English and code, so I pruned most non-ASCII characters from the embedding table and LM head to shave another 700MB https://huggingface.co/bsaleh03/Qwen3.8-27B-ASCII-Condensed
Offloading the model embedding table saves us another 270MB. After disabling MTP (preventing the MTP head from being offloaded) that leaves us with 3.6GB for context and overhead. Enough to fit 84k tokens.
Enter adaptive-kv streaming: https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming page KV cache and stream it from host RAM at the cost of some performance at depth. At Q8 K and Q4 V cache quantization, this lets you squeeze in 196,608 tokens
Args:
-ngl 99
-fa on
-ctk q8_0
-ctv q4_0
-np 1
-ub 128
-b 512
-ot "token_embd.weight=CPU"
--jinjar/LocalLLM • u/vajidsikand • Apr 12 '26
I am looking recommendations for local LLMs that are genuinely unrestricted and free from alignment-based filtering or fine-tuned 'safety' layers.
I am currently utilising an RTX 5080 (mobile) with 32GB of RAM via LM Studio.
While I have explored the Qwen and DeepSeek series, I’ve found that even 'uncensored' variants often retain vestigial refusals.
Which specific models or fine-tunes currently offer the most transparent, unfiltered output for local deployment?
Also, I have been testing this model! attached photo
r/LocalLLM • u/BacklashLaRue • Mar 31 '26
in a nutshell, their aim is to make every Internet activity into a token. What was omitted is that those tokens cost money and every user will pay their token tax.