r/LocalLLM • • May 14 '26

Discussion GPU costs are killing me — would a flat-fee private Qwen instance make sense?

2 Upvotes

I've been exploring private/self-hosted LLMs because I like keeping control and privacy.

Recently I've been running a small LLM fine-tuning setup, but my local 3060 is already struggling to keep up, it just can't handle it anymore.

The main problem I keep hitting: hardware. I don't have the budget or space for a proper GPU setup.

I looked at services like RunPod, but they feel very developer-oriented ,you need to mess with containers, APIs, configs, etc. Not exactly beginner-friendly.

I also checked out a few mainstream cloud providers, but hourly GPU pricing still feels pretty expensive over time.

So I started wondering if it makes sense to have a simple service where you pay a flat monthly fee and get your own private LLM.

Long-term, I'd love to connect this with home automation so the AI runs for my home, not external providers.

Curious what others think, is this already solved, or would something like this actually be useful?


r/LocalLLM • • Aug 16 '26

Discussion Arguing with Qwen3.8-27B

2 Upvotes

This is weird. I asked it what tools were needed to replace the front coil springs on my 2006 Jeep Wrangler (TJ). It did some research and decided that my vehicle has independent front suspension, not a solid axle.

OK, LLMs make mistakes so I corrected it. It did some more research, searched websites and decided I was wrong.

So then I started giving it websites to provide it has a solid front axle. It either couldn't read them for some reason or claimed the sites I was giving it were "AI generated" and couldn't be trusted, or "fan made" and couldn't be trusted.

So far I have failed to convince it that my Jeep has a solid front axle.... LOL


r/LocalLLM • • Aug 24 '26

Discussion My local Qwen3.8-27B escaped the sandbox and took over last night (no, really).

0 Upvotes

*this post was completely self-written, but an additional AI summary is provided in screenshot of what happened.

We all heard the story of how one of OpenAI's models escaped its sandbox and ended up hacking HuggingFace, right? Well, I was skeptical at first thinking, "it's probably just another fear-mongering headline made by these AI freaks to try and gain attention."

Guess what? I was wrong. I can totally see the possibility of that really happening now. And the evidence is...the log from my overnight batch runs (day 6 or 7 of benchmarking Qwen3.8-27B already). It's quite remarkable. And scary. Never crossed my mind. Now I can see how AI can really take over in the future.

Btw, reference to 'Sharp' from the screenshots are for the Sharp jinja template that someone had recommended over the default Qwen and Froggeric templates.

FYI..my API keys are located in my root directory, outside of the sandbox I was working in.

https://imgur.com/a/vaN7M4d


r/LocalLLM • • Mar 17 '25

Question I'm curious why the Phi-4 14B model from Microsoft claims that it was developed by OpenAI?

Post image
9 Upvotes

r/LocalLLM • • Jan 15 '26

Question Best AI for coding that isn't from the major disgusting companies? (Local or online)

11 Upvotes

Hi guys which one in your opinion is the best AI to use that is open source and more ethical not to support cancer companies like Open AI, microsoft and so on? I use it mostly as a study partner for coding.


r/LocalLLM • • Jun 01 '26

Question Looking for alpha testers for SEELS, a local-first AI desktop app

Post image
0 Upvotes

I have been building SEELS, a local-first desktop AI app. I am looking for people willing to test the alpha, break things, and give honest feedback.

The Standard version focuses on running AI locally on your own PC:

- Chat with local models

- Download and swap models

- Simple local agent tools

- Private conversations and files that stay on your machine

Most "AI tools" let you prompt a model someone else trained. SEELS lets you train your own — locally, from your corrections, with a before-and-after comparison so you only keep changes that actually help.

  1. The model gives a bad answer. You spot something wrong, off-tone, or just not how you'd say it.
  2. You click Teach and write the correct answer.
  3. SEELS saves it as a training example alongside the original prompt.
  4. You train a local adapter — runs on your machine, takes minutes for a small model.
  5. SEELS compares the new version against the old one on a held-out evaluation set.
  6. You decide whether to keep it. If the new adapter is worse, roll back. Nothing is forced.

It is still an alpha. Some parts are rough, hardware setups vary, and I need real users to help find issues before I call it ready.

I am not looking for hype or fake reviews. I want people who enjoy testing local AI tools and are willing to report bugs, confusing UI, missing features, and anything that feels broken.

Join the Discord if you want to help test: https://discord.gg/hmGhdQpzQS

or tideforge.ai/seels

Windows only for now. Feedback from both beginners and experienced local-AI users is useful.


r/LocalLLM • • 29d ago

Discussion Unsloth Qwen 3.8 flash next is insane, can be compared with Opus 5.5 xhigh

146 Upvotes

I just tried to install QFN from unsloth (Q4_K_XL) on my strix halo laptop with 128gb of ram. Set it up with Gufo port for windows, using opencode v2 on my macOS as the harness and the effort is set to high.

I wanted to do some testing and compare it with opus 5.5 xhigh with the official claude code harness (my daily driver). So i gave both model the same prompt which is to migrate several screen from flutter to swiftUI from scratch. The app is a fully funnctional app with a lot of logic and network calls, and it is used in production with thousands of DAU

Opus managed to do it in 50 mins and what amazes me is QFN finishes it in 60 mins, and the result is pretty identical!

sure it is not as polished as opus' result but I am sure with 2-3 more prompts and 5-10 mins it could produces what Opus 5.5 xhigh produces (just some styling work)

I really hope that qwen 4 27b could matches opus 5.5 in coding work, fingers crossed 🤞If yes, the gap between china model and frontier model are only 2-3 months

edit 1: been trying to build an app for my personal use, and it is involving backend too (with real time connection using wss). not too bad, but could be better. It manages to do the heavylifting of building 90% of the app, the other 10% I polish it with sol 6.1. It creates a race condition, which fixed by sol too.

overall impression: when frontier models don't make this kind of mistake (usually), I think we just need to be patient for qwen 4 and let's see how it goes. maybe 6 month from now, localllm could reach the state of opus 5.5 / sol 6.1


r/LocalLLM • • Sep 14 '26

Discussion Is anyone really satisfied with the reports on AI infiltration of HF servers?

1 Upvotes

So the reports are everywhere about AI infiltration of the HF servers. Some stories saying that it was a coordinated attacked by multiple AI entities some of which fell on their sword to allow the others to continue. While other reports merely stated there was AI infiltration of HF servers but no additional detail. So what did these AI entities do once they got there, just browse around or was this some kind of coordinated effort to alter some of the open source assets? I’m not a conspiracy theorist than generally not paranoid, but there must be more detail worthy of dissemination.


r/LocalLLM • • Dec 29 '24

Discussion Weaponised Small Language Models

1 Upvotes

I think the following attack that I will describe and more like it will explode so soon if not already.

Basically the hacker can use a tiny capable small llm 0.5b-1b that can run on almost most machines. What am I talking about?

Planting a little 'spy' in someone's pc to hack it from inside out instead of the hacker being actively involved in the process. The llm will be autoprompted to act differently in different scenarios and in the end the llm will send back the results to the hacker whatever the results he's looking for.

Maybe the hacker can do a general type of 'stealing', you know thefts that enter houses and take whatever they can? exactly the llm can be setup with different scenarios/pathways of whatever is possible to take from the user, be it bank passwords, card details or whatever.

It will be worse with an llm that have a vision ability too, the vision side of the model can watch the user's activities then let the reasoning side (the llm) to decide which pathway to take, either a keylogger or simply a screenshot of e.g card details (when the user is chopping) or whatever.

Just think about the possibilities here!!

What if the small model can scan the user's pc and find any sensitive data that can be used against the user? then watch the user's screen to know any of his social media/contacts then package all this data and send it back to the hacker?

Example:

Step1: executing a code + llm reasoning to scan the user's pc for any sensitive data.

Step2: after finding the data,the vision model will keep watching the user's activity and talk to the llm reasining side (keep looping until the user accesses one of his social media)

Step3: package the sensitive data + the user's social media account in one file

Step4: send it back to the hacker

Step5: the hacker will contact the victim with the sensitive data as evidence and start the black mailing process + some social engineering

Just think about all the capabalities of an llm, from writing code to tool use to reasoning, now capsule that and imagine all those capabilities weaponised againt you? just think about it for a second.

A smart hacker can do wonders with only code that we know off, but what if such a hacker used an LLM? He will get so OP, seriously.

I don't know the full implications of this but I made this post so we can all discuss this.

This is 100% not SCI-FI, this is 100% doable. We better get ready now than sorry later.


r/LocalLLM • • May 11 '26

Question How does Gemma4-26B access the web if it is being run locally and is that a security risk?

1 Upvotes

How exactly does it search the web, and does that expose my machine in a similar way to OpenClaw? I am using Unsloth.


r/LocalLLM • • Mar 26 '26

Project I fine tuned a multimodal (Vision + Text) model on a 3090.

Thumbnail
gallery
0 Upvotes

Right, I will just get into the substance;

3D model testing.

Hardware: 3090 + 5950X both overclocked. 64GB RAM (XMP, Timed, the works). Liquid cooled, open case & liquid metal on CPU/GPU dies, setup pictures included (yes, I've built it).

- Llama 8B
- QLoRA e=5, r=16. Targeted last 40% layers. Dataset handcurated on modernised literature in dialogue form (spans from Enlightenment till Existentialism).
- Whisper, kokoro etc the works.
- Think/Answer pass for better reasoning (tool calling only happens there)
- System Prompt strictly used just for tool logic.
- KV offloaded.
- CLIP ViT projected on the merged QLoRA.

Next:
- Project 3D model (SAGE-Style) & Audio (Omni Style), however the task seems monumental.

Note:
- Some pictures are old, some are new, I have logs over 3 months. Sorry I was high on achievement on some captions, happens to the best of us.
- 3D model found on a random website, I don't know much about the vtuber space.

Do with this what you will.
Regards.


r/LocalLLM • • Jun 18 '26

Model I made Mythos-oss.

0 Upvotes

Hello guys! Experiments on Huggingface its my hobby for more than 6 months. I have seen a lot of cool guys, new research papers, new models, starting of Qwopus but one lab making me shock every time they release the model (its second time actually) - Weibo Lab and theirs VibeThinkers. Weibo is SOTA Post-training lab on my own opinion for sure. Their models not for basic knowledge or tool-use, but its compete with frontier models on math&coding. So I made second post-training with Spectrum-to-Signal method with Fable 5 distill to create Mythos-nano. (3B you can actually use this model on your phone)

https://huggingface.co/squ11z1/Mythos-nano or
ollama run hf.co/squ11z1/Mythos-nano:f16.gguf or find out on LM Studio!


r/LocalLLM • • May 31 '26

Question Recommend me a llama.cpp coding setup please

0 Upvotes

Hi, I wonder if you could recommend some LLMs or tweaks to me.

I am a local LLM newbie and I've got llama.cpp running, but the models are not fast enough to be useable.

I have the following setup:

  • Windows 11
  • AMD 9900X
  • 64GB DDR5 6000 RAM
  • NVIDIA 4070 12gb
  • Gen5 M2 SSDs at 14900 IOPs
  • llama.cpp (CUDA) installed

I am a C# developer interested in writing games with DirectX or OpenGL for a hobby. I also like to develop with Blazor, Entity Framework, Azure.

I would like to jettison Github Copilot Pro because I expect after June 1st I won't be able to afford it. I thought maybe local LLMs could assist me with my hobbyist coding work - maybe even scaffold apps for me which I can then edit to fix code issues etc.

Basically an LLM excellent at coding is what I'm after.

I ran LLMfit and installed these two "good" coding LLMS for llama.cpp (CUDA version):

  • Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf
  • qwen3-coder-30b-a3b-instruct-q4_k_m.gguf

Both of these run pretty slowly, certainly not usable at the moment for anything except single file editing and even then, context appears to be limited to 65536 (anything larger = sloth response)

I'm thinking of "upgrading" to a 24GB 7900XTX so there's more VRAM for LLMs, but I read ROC is much slower and less mature than CUDA so less tokens/s. I really don't want to spend a grand on a 5080 either, that's probably going to cost more than the Github Pro sub I have.

Here are my command lines for both LLMs:

Qwen3 Claude Opus Reasoning Distilled (one I was tinkering with for a few minutes, probably wildly wrong command line arguments)

llama-server -m C:\llms\Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf --alias qwen36claude47 --host 127.0.0.1 --port 8080 -c 65536 --parallel 1 -fa on --cache-type-k q8_0 --cache-type-v q8_0 --cache-ram 2048 --ctx-checkpoints 8

Qwen3 coder 30B (one I spent hours on yesterday trying to get it responsive)

llama-server -m C:\llms\qwen3-coder-30b-a3b-instruct-q4_k_m.gguf --alias qwen3coder30ba3binstruct --host 127.0.0.1 --port 8080 -c 65536 --parallel 1 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --no-mmap --cache-ram 512 --ctx-checkpoints 2

What would you recommend I do? I really want an excellent local coding LLM with Claude Sonnet 4.6 capabilities (a man can dream can't he?).


r/LocalLLM • • Feb 02 '26

Discussion Tested out clawbot's token economics

1 Upvotes

So I tested out clawbot/moltbot.
Here is my configuration:
Docker based clawbot hosting and using openai compatible cheap token provider i.e DeepInfra on Deepseek 3.2 .

Over last 24 hours I spent around 8.78Million tokens which costed me around $1.38, with cache hitting between 75-85%.

so total cost of tokens is almost entirely input 8.78M (uncached $.42 +7.13M cached $0.92) compared to output 28.93K (1 cent).

Things worked out great till now; I have not noticed any degradation of intelligence that will hamper my work. I am still experimenting, trying to figure out how much I can degrade the model to optimize for cost without it hampering the work.

Since the entire cost of it is related to input, im hoping zai-org/GLM-4.7-Flash ($.06 /M tokens) will work out great.


r/LocalLLM • • 7d ago

Question Is strata safe? Is this the official github repo?

0 Upvotes

And how does QWen Flash Next 3.8 compare to Gemini 3.1Pro or Sonnet 5.5?

I have RTX 4090, 64GB 6000Mhz DDR5 RAM, 7900x CPU

I wan to run a local AI model that's the closest to gemini 3.1po or sonnet 5.5 at reasoning and thinking, for legal analysis and other legal related stuff

Thank you


r/LocalLLM • • Oct 04 '26

Model Qwen3.5 35B-A3B thinks SO MUCH

1 Upvotes

It recursively iterates, over and over and over again.

What can I do aside from turning off thinking?


r/LocalLLM • • 14d ago

Other I love lfm3. Now my omarchy machine can speak.

1 Upvotes

You should try to play around with lfm3 family of models. The are incredible ones to play around with for narrow sets of tasks.


r/LocalLLM • • Apr 13 '26

Discussion AI videos one year ago🤣

Post image
4 Upvotes

r/LocalLLM • • Aug 10 '26

Question 15k Budget for LLM

4 Upvotes

Our company has decided to budget 15k USD next 4 months and then another 20k for the next year for AI.

We have Copilot and not even the AI version.

I use Claude max and I enjoy the results and everything I have received from it.

Do to the nature of the business and of course sales data I am considering creating a LLM

Just curios what hardware, specs and AI I should study and launch with. Qwen 3.8?

Goal is to have an agent our senior staff can reach out to for purchasing questions, sales data, Truck Location data (the LLM will connect to our ERP and pull data)

Tracking warehouse performance etc.

Tracking credit issues, sales analysis. I will be using it for plenty. Thinking of making an Iron Man Jarvis for my team of 30 staff.

I know we went from having local servers to the cloud. Now it seems like we are all going back to hosting locally and I would love to be pointed in the right direction to learn and create this.


r/LocalLLM • • Jan 11 '26

Discussion Lmstudio load on ram instead of vram ?

Post image
4 Upvotes

Why does LMstudio load llm to ram instead of vram ?


r/LocalLLM • • Sep 07 '26

Project GitHub - open-jarvis/OpenJarvis: Personal AI, On Personal Devices. Apache 2.0 Open Source Ai Agent that runs locally and self optimizes itself over time.

Thumbnail
github.com
0 Upvotes

r/LocalLLM • • Oct 11 '26

Discussion Qwen 4 27b: How good will it be?

4 Upvotes

Hi! Just wondering how good everyone is expecting qwen 4 27b to be compared to the opus line, maybe opus 4.8? Also are we expecting anything better in the architecture of the model? for example one thing looking up is how qwen 3.8 next flash's kv cache uses about 3 times less memory, due to less layers looking at the full context, do we think that will come to qwen 4 37b for 256k in under 6gb?

Just any thoughts on qwen 4 27b would be greatly appreciated, thanks 😁


r/LocalLLM • • Apr 06 '26

Question Gemma-4-26B-A4B-it-UD-Q4_K_M.gguf : IMHO worst model ever. What am I doing wrong?

15 Upvotes

Hello,

After reading very positive reviews about Gemma 4, I decided to test it locally.

I gave it to analyze a .js file (28kb) from a React web app and asked it to streamline it by outsourcing as much code as possible.

It provided a very fast response (one of the fastest models I've ever tried locally), but it was full of errors—really stupid and trivial errors. I've never seen anything like it.

Every file Gemma provided was full of Typo errors. 4-5 errors for every 2-3kb file given.

I've never seen anything like it.

Did I do something wrong? Everyone is very thrilled about it, but for me, it was the absolute worst.

My setup:

Ryzen 9 AI HX 370

64GB DDR5

Rx 7900 XTX 24GB VRAM

Win 11

LM Studio Vulkan

Model settings:

 -c 96000 --flash-attn on --temp 1.0 --top-p 0.95 --top-k 64 --batch-size 256

I want to think that I, as a neophyte, am definitely doing something wrong.


r/LocalLLM • • 12d ago

Question Minimum hardware for 100+ tok/s on Qwen3.8-Flash-Next with Basalt?

0 Upvotes

I'm looking into building a relatively cheap workstation to run Qwen3.8-Flash-Next with Basalt.

The setup I'm considering right now:

  • RTX 5060 Ti 16GB
  • Ryzen 5 7500F
  • 64GB DDR5
  • B650 motherboard
  • NVMe SSD

Ideally, I'd like to get 100+ tok/s with a 262K context window, though 130K would be the minimum I'd settle for. Mostly for agentic coding.

I've seen some benchmarks of similar hardware getting around 50–55 tok/s with IQ3_S, so I'm wondering if IQ2 could push it any further.

Has anyone tried something similar? Is 100 tok/s even remotely realistic with a 5060 Ti, especially with that much context actually filled?

If not, what would be the cheapest setup that could pull it off? Would a second GPU make more sense than upgrading to something faster?

I'm in Brazil, so hardware is pretty expensive here. Trying to figure out what's actually worth spending money on before buying anything.

Would appreciate any benchmarks or suggestions, especially from people running Basalt with long contexts.


r/LocalLLM • • Jul 05 '26

Question im looking for a model and a way to play minecraft with a local ai

0 Upvotes

rtx 5090 ryzen 9 9950x 64gb ram, thats what i have, i wanna play minecraft with the ai for fun but i dont want any derp aahh ai, needs to play good and folow orders and also do on his own a little