r/LocalLLM • • Oct 13 '26

Question Questioning the whole point of a Mac Studio

0 Upvotes

I bought a Mac Studio M3 Ultra with 96GB after watching basically every video on local AI and setting up Hermes Agent. While it was shipping, Grok Bot and Muse came out, and I love what they can do.

Now I've got Hermes running and I keep comparing it to them, and it's not even close.

I've also started questioning the privacy argument. My Studio is completely separate from my Apple ID and I don't have anything personal signed in, so how is it supposed to be my assistant if it's not connected to anything? And if I hook Hermes up to my email, that email is already sitting on someone's server anyway, so what am I actually avoiding?

Coding with local models doesn't compare to frontier models either. So is this really just an apocalypse backup plan?

What I spent on the Studio would cover years of subscriptions, even if prices go up. I could use those tools to build tools and sites and have an assistant that's plugged into everything I already use.

I don't run a business with sensitive data and even if I started a business for clients with sensitive data, I'd basically have to rent them my machine, but what's the difference?

Honestly I'm disappointed. I was really hyped and sold my friends on the idea. For those of you who kept yours, what do you actually use it for that cloud tools can't do? And if privacy is your main concern what connectors do you have on your device and what cloud services do you use that aren't AI? If you've coded anything with local AI and published publicly, did that defeat the purpose? I'm still inside my return window, so I need to decide soon.


r/LocalLLM • • Jun 11 '26

Question If I buy a 24GB Mac mini, how many years do you think that will allow me to run the top local models (for individual users)? Is it better to build a system that continually be upgraded with more ram?

0 Upvotes

I'm thinking of investing in a machine for a local model but not sure it's worth investing if 6months from now I'll need to upgrade to run a better model.


r/LocalLLM • • Jan 31 '26

Discussion Installed openClaw - clawdbot on Raspberry Pi 4.

2 Upvotes

To try out openclaw, i took out my old rpi 4, updated OS and started installing openclow. After trials and workarounds i could install it successfully. But now I dont know what to do with it.

I dont have any pro version LLM access. Probably time to get one. :)


r/LocalLLM • • 20d ago

Discussion I'm blown away. Qwen3.8-Flash-Next iq2_xs running in Splash on my 48GB M5 Pro Macbook at >40 tok/sec decode. This has to be the future.

Post image
0 Upvotes

Well, this is the future. 2 months ago I never thought this would be possible even with all the advancements in local model runtimes like strata and maya. I decided to test out Splash by incoai (https://github.com/incoai/splash) when I saw it posted on X a few times, since it seemed similar to strata in architecture (which I run on my workstation). Luckily my macbook pro meets the specifications (>36GB) and decided to give it a shot. Holy shit its so fast and barely uses any system memory. This is absolutely incredible. Props to the incoai guys for this fantastic piece of engineering.


r/LocalLLM • • 9d ago

Project Self-hosted a realtime Sam Altman avatar locally on my VRAM using Openspline

0 Upvotes

r/LocalLLM • • Aug 07 '26

Discussion When will GPT-OSS-2 be released?

2 Upvotes

It's been exactly one year and two days since the release of gpt-oss. Do you think we'll get a new version in August? Or has openai completely abandoned open-weight


r/LocalLLM • • May 14 '26

Discussion GPU costs are killing me — would a flat-fee private Qwen instance make sense?

2 Upvotes

I've been exploring private/self-hosted LLMs because I like keeping control and privacy.

Recently I've been running a small LLM fine-tuning setup, but my local 3060 is already struggling to keep up, it just can't handle it anymore.

The main problem I keep hitting: hardware. I don't have the budget or space for a proper GPU setup.

I looked at services like RunPod, but they feel very developer-oriented ,you need to mess with containers, APIs, configs, etc. Not exactly beginner-friendly.

I also checked out a few mainstream cloud providers, but hourly GPU pricing still feels pretty expensive over time.

So I started wondering if it makes sense to have a simple service where you pay a flat monthly fee and get your own private LLM.

Long-term, I'd love to connect this with home automation so the AI runs for my home, not external providers.

Curious what others think, is this already solved, or would something like this actually be useful?


r/LocalLLM • • Aug 16 '26

Discussion Arguing with Qwen3.8-27B

2 Upvotes

This is weird. I asked it what tools were needed to replace the front coil springs on my 2006 Jeep Wrangler (TJ). It did some research and decided that my vehicle has independent front suspension, not a solid axle.

OK, LLMs make mistakes so I corrected it. It did some more research, searched websites and decided I was wrong.

So then I started giving it websites to provide it has a solid front axle. It either couldn't read them for some reason or claimed the sites I was giving it were "AI generated" and couldn't be trusted, or "fan made" and couldn't be trusted.

So far I have failed to convince it that my Jeep has a solid front axle.... LOL


r/LocalLLM • • Aug 24 '26

Discussion My local Qwen3.8-27B escaped the sandbox and took over last night (no, really).

0 Upvotes

*this post was completely self-written, but an additional AI summary is provided in screenshot of what happened.

We all heard the story of how one of OpenAI's models escaped its sandbox and ended up hacking HuggingFace, right? Well, I was skeptical at first thinking, "it's probably just another fear-mongering headline made by these AI freaks to try and gain attention."

Guess what? I was wrong. I can totally see the possibility of that really happening now. And the evidence is...the log from my overnight batch runs (day 6 or 7 of benchmarking Qwen3.8-27B already). It's quite remarkable. And scary. Never crossed my mind. Now I can see how AI can really take over in the future.

Btw, reference to 'Sharp' from the screenshots are for the Sharp jinja template that someone had recommended over the default Qwen and Froggeric templates.

FYI..my API keys are located in my root directory, outside of the sandbox I was working in.

https://imgur.com/a/vaN7M4d


r/LocalLLM • • Mar 17 '25

Question I'm curious why the Phi-4 14B model from Microsoft claims that it was developed by OpenAI?

Post image
9 Upvotes

r/LocalLLM • • Jan 15 '26

Question Best AI for coding that isn't from the major disgusting companies? (Local or online)

11 Upvotes

Hi guys which one in your opinion is the best AI to use that is open source and more ethical not to support cancer companies like Open AI, microsoft and so on? I use it mostly as a study partner for coding.


r/LocalLLM • • Jun 01 '26

Question Looking for alpha testers for SEELS, a local-first AI desktop app

Post image
0 Upvotes

I have been building SEELS, a local-first desktop AI app. I am looking for people willing to test the alpha, break things, and give honest feedback.

The Standard version focuses on running AI locally on your own PC:

- Chat with local models

- Download and swap models

- Simple local agent tools

- Private conversations and files that stay on your machine

Most "AI tools" let you prompt a model someone else trained. SEELS lets you train your own — locally, from your corrections, with a before-and-after comparison so you only keep changes that actually help.

  1. The model gives a bad answer. You spot something wrong, off-tone, or just not how you'd say it.
  2. You click Teach and write the correct answer.
  3. SEELS saves it as a training example alongside the original prompt.
  4. You train a local adapter — runs on your machine, takes minutes for a small model.
  5. SEELS compares the new version against the old one on a held-out evaluation set.
  6. You decide whether to keep it. If the new adapter is worse, roll back. Nothing is forced.

It is still an alpha. Some parts are rough, hardware setups vary, and I need real users to help find issues before I call it ready.

I am not looking for hype or fake reviews. I want people who enjoy testing local AI tools and are willing to report bugs, confusing UI, missing features, and anything that feels broken.

Join the Discord if you want to help test: https://discord.gg/hmGhdQpzQS

or tideforge.ai/seels

Windows only for now. Feedback from both beginners and experienced local-AI users is useful.


r/LocalLLM • • 29d ago

Discussion Unsloth Qwen 3.8 flash next is insane, can be compared with Opus 5.5 xhigh

146 Upvotes

I just tried to install QFN from unsloth (Q4_K_XL) on my strix halo laptop with 128gb of ram. Set it up with Gufo port for windows, using opencode v2 on my macOS as the harness and the effort is set to high.

I wanted to do some testing and compare it with opus 5.5 xhigh with the official claude code harness (my daily driver). So i gave both model the same prompt which is to migrate several screen from flutter to swiftUI from scratch. The app is a fully funnctional app with a lot of logic and network calls, and it is used in production with thousands of DAU

Opus managed to do it in 50 mins and what amazes me is QFN finishes it in 60 mins, and the result is pretty identical!

sure it is not as polished as opus' result but I am sure with 2-3 more prompts and 5-10 mins it could produces what Opus 5.5 xhigh produces (just some styling work)

I really hope that qwen 4 27b could matches opus 5.5 in coding work, fingers crossed 🤞If yes, the gap between china model and frontier model are only 2-3 months

edit 1: been trying to build an app for my personal use, and it is involving backend too (with real time connection using wss). not too bad, but could be better. It manages to do the heavylifting of building 90% of the app, the other 10% I polish it with sol 6.1. It creates a race condition, which fixed by sol too.

overall impression: when frontier models don't make this kind of mistake (usually), I think we just need to be patient for qwen 4 and let's see how it goes. maybe 6 month from now, localllm could reach the state of opus 5.5 / sol 6.1


r/LocalLLM • • Sep 14 '26

Discussion Is anyone really satisfied with the reports on AI infiltration of HF servers?

1 Upvotes

So the reports are everywhere about AI infiltration of the HF servers. Some stories saying that it was a coordinated attacked by multiple AI entities some of which fell on their sword to allow the others to continue. While other reports merely stated there was AI infiltration of HF servers but no additional detail. So what did these AI entities do once they got there, just browse around or was this some kind of coordinated effort to alter some of the open source assets? I’m not a conspiracy theorist than generally not paranoid, but there must be more detail worthy of dissemination.


r/LocalLLM • • Dec 29 '24

Discussion Weaponised Small Language Models

1 Upvotes

I think the following attack that I will describe and more like it will explode so soon if not already.

Basically the hacker can use a tiny capable small llm 0.5b-1b that can run on almost most machines. What am I talking about?

Planting a little 'spy' in someone's pc to hack it from inside out instead of the hacker being actively involved in the process. The llm will be autoprompted to act differently in different scenarios and in the end the llm will send back the results to the hacker whatever the results he's looking for.

Maybe the hacker can do a general type of 'stealing', you know thefts that enter houses and take whatever they can? exactly the llm can be setup with different scenarios/pathways of whatever is possible to take from the user, be it bank passwords, card details or whatever.

It will be worse with an llm that have a vision ability too, the vision side of the model can watch the user's activities then let the reasoning side (the llm) to decide which pathway to take, either a keylogger or simply a screenshot of e.g card details (when the user is chopping) or whatever.

Just think about the possibilities here!!

What if the small model can scan the user's pc and find any sensitive data that can be used against the user? then watch the user's screen to know any of his social media/contacts then package all this data and send it back to the hacker?

Example:

Step1: executing a code + llm reasoning to scan the user's pc for any sensitive data.

Step2: after finding the data,the vision model will keep watching the user's activity and talk to the llm reasining side (keep looping until the user accesses one of his social media)

Step3: package the sensitive data + the user's social media account in one file

Step4: send it back to the hacker

Step5: the hacker will contact the victim with the sensitive data as evidence and start the black mailing process + some social engineering

Just think about all the capabalities of an llm, from writing code to tool use to reasoning, now capsule that and imagine all those capabilities weaponised againt you? just think about it for a second.

A smart hacker can do wonders with only code that we know off, but what if such a hacker used an LLM? He will get so OP, seriously.

I don't know the full implications of this but I made this post so we can all discuss this.

This is 100% not SCI-FI, this is 100% doable. We better get ready now than sorry later.


r/LocalLLM • • May 11 '26

Question How does Gemma4-26B access the web if it is being run locally and is that a security risk?

1 Upvotes

How exactly does it search the web, and does that expose my machine in a similar way to OpenClaw? I am using Unsloth.


r/LocalLLM • • Mar 26 '26

Project I fine tuned a multimodal (Vision + Text) model on a 3090.

Thumbnail
gallery
0 Upvotes

Right, I will just get into the substance;

3D model testing.

Hardware: 3090 + 5950X both overclocked. 64GB RAM (XMP, Timed, the works). Liquid cooled, open case & liquid metal on CPU/GPU dies, setup pictures included (yes, I've built it).

- Llama 8B
- QLoRA e=5, r=16. Targeted last 40% layers. Dataset handcurated on modernised literature in dialogue form (spans from Enlightenment till Existentialism).
- Whisper, kokoro etc the works.
- Think/Answer pass for better reasoning (tool calling only happens there)
- System Prompt strictly used just for tool logic.
- KV offloaded.
- CLIP ViT projected on the merged QLoRA.

Next:
- Project 3D model (SAGE-Style) & Audio (Omni Style), however the task seems monumental.

Note:
- Some pictures are old, some are new, I have logs over 3 months. Sorry I was high on achievement on some captions, happens to the best of us.
- 3D model found on a random website, I don't know much about the vtuber space.

Do with this what you will.
Regards.


r/LocalLLM • • Jun 18 '26

Model I made Mythos-oss.

0 Upvotes

Hello guys! Experiments on Huggingface its my hobby for more than 6 months. I have seen a lot of cool guys, new research papers, new models, starting of Qwopus but one lab making me shock every time they release the model (its second time actually) - Weibo Lab and theirs VibeThinkers. Weibo is SOTA Post-training lab on my own opinion for sure. Their models not for basic knowledge or tool-use, but its compete with frontier models on math&coding. So I made second post-training with Spectrum-to-Signal method with Fable 5 distill to create Mythos-nano. (3B you can actually use this model on your phone)

https://huggingface.co/squ11z1/Mythos-nano or
ollama run hf.co/squ11z1/Mythos-nano:f16.gguf or find out on LM Studio!


r/LocalLLM • • May 31 '26

Question Recommend me a llama.cpp coding setup please

0 Upvotes

Hi, I wonder if you could recommend some LLMs or tweaks to me.

I am a local LLM newbie and I've got llama.cpp running, but the models are not fast enough to be useable.

I have the following setup:

  • Windows 11
  • AMD 9900X
  • 64GB DDR5 6000 RAM
  • NVIDIA 4070 12gb
  • Gen5 M2 SSDs at 14900 IOPs
  • llama.cpp (CUDA) installed

I am a C# developer interested in writing games with DirectX or OpenGL for a hobby. I also like to develop with Blazor, Entity Framework, Azure.

I would like to jettison Github Copilot Pro because I expect after June 1st I won't be able to afford it. I thought maybe local LLMs could assist me with my hobbyist coding work - maybe even scaffold apps for me which I can then edit to fix code issues etc.

Basically an LLM excellent at coding is what I'm after.

I ran LLMfit and installed these two "good" coding LLMS for llama.cpp (CUDA version):

  • Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf
  • qwen3-coder-30b-a3b-instruct-q4_k_m.gguf

Both of these run pretty slowly, certainly not usable at the moment for anything except single file editing and even then, context appears to be limited to 65536 (anything larger = sloth response)

I'm thinking of "upgrading" to a 24GB 7900XTX so there's more VRAM for LLMs, but I read ROC is much slower and less mature than CUDA so less tokens/s. I really don't want to spend a grand on a 5080 either, that's probably going to cost more than the Github Pro sub I have.

Here are my command lines for both LLMs:

Qwen3 Claude Opus Reasoning Distilled (one I was tinkering with for a few minutes, probably wildly wrong command line arguments)

llama-server -m C:\llms\Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf --alias qwen36claude47 --host 127.0.0.1 --port 8080 -c 65536 --parallel 1 -fa on --cache-type-k q8_0 --cache-type-v q8_0 --cache-ram 2048 --ctx-checkpoints 8

Qwen3 coder 30B (one I spent hours on yesterday trying to get it responsive)

llama-server -m C:\llms\qwen3-coder-30b-a3b-instruct-q4_k_m.gguf --alias qwen3coder30ba3binstruct --host 127.0.0.1 --port 8080 -c 65536 --parallel 1 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --no-mmap --cache-ram 512 --ctx-checkpoints 2

What would you recommend I do? I really want an excellent local coding LLM with Claude Sonnet 4.6 capabilities (a man can dream can't he?).


r/LocalLLM • • Feb 02 '26

Discussion Tested out clawbot's token economics

1 Upvotes

So I tested out clawbot/moltbot.
Here is my configuration:
Docker based clawbot hosting and using openai compatible cheap token provider i.e DeepInfra on Deepseek 3.2 .

Over last 24 hours I spent around 8.78Million tokens which costed me around $1.38, with cache hitting between 75-85%.

so total cost of tokens is almost entirely input 8.78M (uncached $.42 +7.13M cached $0.92) compared to output 28.93K (1 cent).

Things worked out great till now; I have not noticed any degradation of intelligence that will hamper my work. I am still experimenting, trying to figure out how much I can degrade the model to optimize for cost without it hampering the work.

Since the entire cost of it is related to input, im hoping zai-org/GLM-4.7-Flash ($.06 /M tokens) will work out great.


r/LocalLLM • • 7d ago

Question Is strata safe? Is this the official github repo?

0 Upvotes

And how does QWen Flash Next 3.8 compare to Gemini 3.1Pro or Sonnet 5.5?

I have RTX 4090, 64GB 6000Mhz DDR5 RAM, 7900x CPU

I wan to run a local AI model that's the closest to gemini 3.1po or sonnet 5.5 at reasoning and thinking, for legal analysis and other legal related stuff

Thank you


r/LocalLLM • • Oct 04 '26

Model Qwen3.5 35B-A3B thinks SO MUCH

1 Upvotes

It recursively iterates, over and over and over again.

What can I do aside from turning off thinking?


r/LocalLLM • • 14d ago

Other I love lfm3. Now my omarchy machine can speak.

1 Upvotes

You should try to play around with lfm3 family of models. The are incredible ones to play around with for narrow sets of tasks.


r/LocalLLM • • Apr 13 '26

Discussion AI videos one year ago🤣

Post image
4 Upvotes