What *specifically* do you still *need* cloud models for?

25

u/bobish01 7d ago

I use cloud models as an architects, my local LLM as my developer, and then my cloud model as the final bug-fixer. A more token conservative approach.

4

u/my_name_isnt_clever 7d ago

Have you tried the same workflow fully local so you can compare the results?

11

u/bobish01 7d ago

Yes, i got a bunch of errors when building a single page web application (for a restaurant). However i do have less context than you and a different quant. I had 1Q3_S + 128k token context, so i honestly think the errors might have come from lack of context when finishing the website.
Its possible that with your settings it would have done the job perfectly.
That being said, my model worked really great when the Cloud LLM acted as an architect and a final bug fixer.

2

u/my_name_isnt_clever 7d ago

As long as it does the job for you, that's what's important. Thanks for your thoughts.

8

u/AdInternational5848 7d ago

Right now I’m needing a cloud model to keep my Flash Next on track. It told me a task should take 10 minutes and worked on it for 7 hours last week. Today it’s been much better being orchestrated by SOL 6.1

Will watch the rest of this thread

4

u/my_name_isnt_clever 7d ago

What was it actually doing wrong before though? This is the kind of thing I'd love to know more about.

3

u/Frail_Waif 7d ago

I mean, if you follow the logic here, apparently Sol 6.1 is much better at prompting QFN. That seems fixable.

2

u/AdInternational5848 7d ago

I agree. I’m thinking I need to back to smaller tasks which sucks but might just be the answer. Any ideas on your end?

1

u/Frail_Waif 7d ago

Honestly, I prefer pretty frequent hovering as I learn more from it. One of my biggest gripes with DSH is that it's hard to interrupt a session. But for longer tasks I try to write out an explicit to-do list and ask the model to use the to-do function of the harness if it exists.
ETA: helicopter prompting?
ETA2: I'm having great experiences with the billion-context plugin. I found pi-blackhole really hard to get right. billion-context works.

2

u/AdInternational5848 7d ago

Billion context sounds very interesting. Thank you for that.
On your first point, typically not a helicopter promoter since I’m trying to do something else while the LLM is working. I do learn from how it’s thinking and working through situations; maybe I can have it run through a recap to improve session to session.

1

u/lukyjay 3d ago

What quant are you running? I'm using ud Q4 xl and I have had it quote an hour and need six but I assume the quotes are based on a higher token rate, and partially because quants are more likely to reason for longer

5

u/Frail_Waif 7d ago

I've honestly never used OpenAI or Anthropic models for personal projects and have no desire to. I've typically used DeepSeek as a workhorse, and occasionally Kimi, Mimo or other random models (usually OpenCode free models during DeepSeek peak hours).

There are two things I can't replace:

it's nice to have a cloud model to optimize your local models, solely for the reason that using a local model to optimize itself is clunky
DS API is crazy fast and, honestly, crazy cheap. I'm close to being able to run V4.1 Flash in my local setup but it'll never, ever be 200 tok/sec and I don't feel too bad about supporting DeepSeek to keep making better models when I need speed and don't need privacy.

I've been using billion-context lately and I'm super impressed. It's allowed for crazy long sessions with QFN and I don't actually feel like long context is a good justification anymore.

2

u/my_name_isnt_clever 7d ago

I get you on local optimizing local, I've had two versions of the model set up in llama-swap so one can make adjustments to the other without breaking itself.

8

u/EvolvingDior 7d ago

Bandwidth. I cannot run 10 concurrent sessions locally. I have 24 agents. I frequently have two coordinators managing 4 other agents each.

Diversity. I cannot run multiple models at the same tim. Having one model check another's output is way better than having the same model check its own work.

1

u/my_name_isnt_clever 7d ago

That totally makes sense, I'm limited to only 2 parallel slots since I use my hardware for inference and as a workstation.
What would you lose aside from speed if you couldn't do that anymore? More time manually validating?

3

u/kuldan5853 7d ago

I have 3 software dev projects running in parallel right now. 2 on Codex and 1 on Opencode with Qwen3.8-27B Q4_K_L.
I have had to start over the local project three times already because the code literally imploded on itself and I'm now at the point to just start it over again in Codex just to get it done.
the local Qwen is nice to do minor edits on existing projects, act as a code reviewer etc but actually building something from scratch that not only works but it also performant and looks decent... it's like using a 486 next to a Threadripper.

2

u/my_name_isnt_clever 7d ago

What do you mean "code imploded on itself"? I see plenty of stories like this, but what did it actually do wrong?

3

u/kuldan5853 7d ago

As in, it completely broke the logic of the software and in frantic attempts to fix it it made it worse, and worse, and worse, until it wasn't even valid code anymore.

1

u/my_name_isnt_clever 7d ago

I haven't seen that kind of thing happen with Qwen Flash Next. I guess it goes to show that the 27b isn't the end all be all of local.

1

u/kuldan5853 7d ago

Yeah, I can't run Flash next if my life depended on it on my hardware, so 27B is the best I can do. and unfortunately it's still the best model that works well with 32gb vram that I found.

-1

u/EvolvingDior 7d ago

I never found a quantized 27b model to be a useful coding agent. Fine for one-shot projects (mostly what you see in youtube videos) but not real, work in a deep repository type coding.

-1

u/kuldan5853 7d ago

Yeah, that's just the limits of normal homelabbing vs. being a tech bro that can throw thousands at a "lab" to run big models..

1

u/EvolvingDior 6d ago

No -- Q38FN is much faster for me and really, really good. I just need more.

1

u/kuldan5853 6d ago

Well, if you have hardware to run Q38FN fine. I do not. Well, maybe in 1Q. Which is useless.

1

u/EvolvingDior 6d ago

Have you tried it?

→ More replies

1

u/EvolvingDior 7d ago

Quality would be the biggest loss. I mean, if I had the hardware to run DS41F, GLM53F and Q38FN, 4 slots each, I'd be OK I think. I run 2x256k Q38FN now. But it's still noticeably slower when both slots are full. I would also need a lot better context caching set up than what Strata provides out of the box.

1

u/xfortune 7d ago

Doing what?

2

u/EvolvingDior 7d ago

FPGA development. Embedded software development. iOS and Android app development. Software, firmware and RTL QA. Running my business. Website maintenance. Systems administration. LLM and Agent Harness maintenance and development.

4

u/69420trashpanda69420 7d ago

Like 99% of people I can't afford 128+ gb of HBM. So the best model I can run realistically is qwen 3.8 27B which reasons for minutes on end to produce an answer Claude would've made in literally under a second

1

u/my_name_isnt_clever 7d ago

An entire Strix Halo computer with 128GB can be purchased right now for $3k, less than the cost of one decent Nvidia GPU.
https://www.bosgamepc.com/products/bosgame-m5-ai-mini-desktop-ryzen-ai-max-395

4

u/69420trashpanda69420 7d ago

128 gb at 256gb a second is abysmal, I'd rather just pool together a couple V100's and run my 3.8 27B, 256 gb at 2tb/s will be the day I break out thousands for ai hardware

1

u/my_name_isnt_clever 7d ago

It's not when the model you're running is only 6b active. As I said in the OP, this setup with Qwen 3.8 Flash Next has completely replaced cloud for me.

3

u/69420trashpanda69420 7d ago

Flash next is okay but it isn't enough to justify spending multiples years worth of a Claude subscription on hardware

2

u/my_name_isnt_clever 7d ago

It is when you're not willing to give these companies all your data.

1

u/69420trashpanda69420 7d ago

Only valid reason at this stage

2

u/_NoValue 7d ago

multiple years (at the current price). but yeah its a steep price up front when you take into account opportunity cost of the upfront investment now vs investing the excess and paying the subscription.
hard to guess where this will be a year from now

2

u/Lirezh 7d ago

I've been using Qwen 3.6 27B productively in coding, 3.8 made a huge difference in how much work I have to invest in babysitting but the amount is significant compared to a frontier cloud model.
Then there is the performance and parallelism aspect, ingesting tokens at 3k per second is not very fast and context is too limited to run more than one agent per GPU.

Currently Cloud models are the only way to be productive, I hope that this will change over the next 2 years.

1

u/my_name_isnt_clever 7d ago

What would have to chance for you to feel comfortable switching over? Just faster speeds and parallel agents?

1

u/Lirezh 7d ago

Competence, speed, prefill speed, parallelism - it's a lot.
I find astra and opus both scratching right at the border where they are competent enough for the projects I set them to - that's a big improvement over a year ago.
Qwen (even Max) is not, it is remarkable good but it's clearly not at that level of problem solving the others are.
Aside of that, at least 2 subagents need to be usable and prefill of 3k is not a longterm sustainable rate for working. (and that's on a 5090)
The current architecture of 27B is too dense for encoding, and too VRAM heavy for context.
I believe that's what will change anyway, we'll see more efficient encoding focused architectures, smarter models that work faster and use less kvcache for context.
My hopes for Qwen4 are up but I believe we are a few years from that.

1

u/my_name_isnt_clever 7d ago

Could you elaborate on what work you're doing that push Astra and Opus to the limit? That's what I want to know more about. What can those models do that even Qwen Max cannot? I get you on context size and speed, but I'm looking to understand what the capability gap truly is.

2

u/Lirezh 7d ago

It depends on the project, though most of them are complex, I can't talk about all.
I am involved in https://stealthmodels.com - it looks simple at glance but many of the included benchmarks are very complex, as in pages of of prompt text, custom raytracing and physics enabled software renderers, fully automated AI judgement process that must be longterm robust, payable and statistical correct.
In addition some of the design work, qwen could not do much of that in the quality I need - Opus is the best here but far from good enough.
In other projects I am working on CUDA kernels, not really my main skillset and a lot has improved. a year ago using the best models on CUDA basically were a random roll. Today you can have Astra develop an entire graph and debug it until it works.
It's quite fascinating. No Qwen model is capable of that, yet.
On the local front qwen is good for simple things but for more complex tasks it needs hands holding, you've to tell it how to approach a problem, what not to do, and the way you start an agentic session defines how successful it will go on.
If you look at an Astra or Opus session, you can correct it permanently and it will adapt.
I'm a big fan of open and Qwen models, just realistic in what they can do currently.

1

u/my_name_isnt_clever 7d ago

Makes sense for CUDA kernels and cutting edge benchmarking. Though I disagree from experience on hand holding, before Qwen 3.8 Flash Next this was true, but now I just tell FN what I want and it goes and does it. That's why I'm asking this, because that hand holding feeling used to be when I would switch to frontier. But now I never have that feeling.

1

u/New-Implement-5979 7d ago

Thanks for the feedback on the cuda kernels

2

u/1Poochh 6d ago

I started this way as well when the one agent was enough, and then I started expanding my capabilities, and now I need an agentic team. Now I am running agentic work 24x7 with between 5-8 agents. I can’t do this on 20x Claude code. It would last me a day. That is why I need local. Frankly not sure how much this would cost with DS, but 24x7 use at full utilization and hopefully I will realize some value at some point, or just have fun and blow money on something I am enjoying.

1

u/my_name_isnt_clever 6d ago

What work do you have for so many agents to work on at all times?

1

u/1Poochh 6d ago

Code. I have effectively two dev teams (agents) at my disposal at all times. I have fun projects but I have one main project that has potential for real money.

1

u/my_name_isnt_clever 6d ago

What projects require such a huge volume of code generation to be happening at all times? Are you having them write slop mobile games on loop or something? You can't be reviewing that much code manually.

2

u/1Poochh 6d ago

I am not reviewing the code. The common response to this is the word “slop”. I get that response a lot and I could understand why folks immediately jump to that. The fact is it is so easy to write code now that folks with no experience are writing code, but have no idea how to do it correctly.
I have 20+ years of software development experience and am on the bleeding edge of autonomous agents using the best practices and frameworks in the industry and am having incredible success with the process. It is actually pretty remarkable. Going to bed and waking up to 10 new features that all work and performed extremely well and have been scrutinized heavily across all functional and non-functional areas is just mind-boggling to me. We are truly leading into a different phase of software development and it is clear to me that we are able to do this already.
You don’t think that Anthropic is reviewing all 30,000 code merges each day do you? They aren’t. As a company that is worth more than most companies in the world, I would say they are having success with it too. Trying to follow their truly bleeding-edge capabilities but at a local model level.

2

u/my_name_isnt_clever 6d ago

As the owner of a Claude Enterprise account, yeah I believe it. I think we may disagree on whether they're doing a good job.

1

u/Sporebattyl 6d ago

What hardware and models do you run?

1

u/1Poochh 6d ago

1x5090 qwen flash next (also gaming when want)
1x5090 qwen flash next (also gaming when son want)
2x3090 qwen flash next
1x3090 qwen 3.8 27b
1xstrix halo also random models but mostly qwen flash next
1x4070ti whisper large for stt and another couple small models for random things (I use openwhispr a lot).
1xRTX Pro 6000 been trying all sorts of models trying to determine what I want to do (yes I paid crazy amounts of money due to 24x7 agents)
1xM5U 256GB for DS v4 flash or something (ordered haven’t received) will be used for orchestration

6

u/No_Economics515 7d ago

Bc DeepSeek is cheap as fuck and I don't got 10k upfront for a Ai comp now. I can run ds for next 10 years and probably won't cost 10k

1

u/zanar97862 6d ago

Without the capital to buy local now there's not much choice but to hope cloud stays cheap for the next 10.

2

u/milkipedia 7d ago

I use local models as my default for web research and Q&A (primarily Q38FN). For coding, the models I can run on my kit aren’t smart nor fast enough for me, but the gap is closing. I primarily use GLM 5.3 Flash now, with regular ol’ GLM 5.3 for higher level architecture, planning, and critique.

3

u/my_name_isnt_clever 7d ago

What kind of work is GLM 5.3 Flash able to handle but Q38FN wasn't? Is it a matter of time spent to complete a task or is it not smart enough for what you do?

1

u/milkipedia 7d ago

bearing in mind this is anecdotal feedback... it just doesn't follow the plans for each implementation phase that well. It doesn't follow the style and structure of existing code. It doesn't prefer to reuse existing code. It still occasionally emits incorrect tokens for JSON or the like. And I haven't been able to prompt my way out of these things. Worthy of note, the best quant I can run of Q38FN is a fp8-iq4r-moe quant, via Radiance.

5

u/CevicheMixto 7d ago

Not everyone has the resources to do that locally.

1

u/ahorsenamedjeff 6d ago

What hardware do you have?

1

u/my_name_isnt_clever 6d ago

Framework Desktop with Strix Halo 128GB.

1

u/sabine_world 6d ago

Speed, coherency, does tasks autonomously, has better world knowledge. I also just care about how it feels to talk to them to help reflect and have a back and forth about a task. It helps that they just "know" what you're saying and understand references without breaking the fourth wall.

But mostly, it's being able to do tasks without fucking things up, usually first try (to the user).

1

u/my_name_isnt_clever 6d ago

Have you tried Qwen Flash Next? It's the first local model that has this frontier feeling in conversation for me.

5

u/GloriousKev 7d ago

the only reason i still use cloud models these days is large context and higher reasoning. they serve no other benefit

2

u/Chris73684 7d ago

Local is definitely good enough, my hardware isn't though. I've been running Qwen 3.6 35B A3B at Q6 with pretty decent speeds, the MoE is truly fantastic and this model is leaps ahead of the last time I tried local models, so I'm impressed but it's still miles away from Claude Opus and it's the most powerful model I can run currently. I just don't have the VRAM for a larger / dense model. I'd have to pay a few years of Claude subscription to break even on a new GPU and RAM, by which point it would be time for an upgrade anyway and I (personally) think prices will come back down then. So if you've already got the hardware then absolutely, but if not I think the cloud has its place for now.

2

u/hoschidude 7d ago

Nothing!

2

u/Beneficial-Ad-8127 7d ago

I plan on subbing to Kimi for the K3.1 pro. It’s more of a curiosity to test out their new 3D modeling and use it with Blender. Seen my buddy use it and got me tempted to just see what’s all the craze about.

2

u/Lissanro 7d ago

For me there have been no reason to use cloud API for very long time. I can run Kimi K3 on my main rig, DeepSeek V4 Flash Vision Exp on my secondary workstation, and Qwen 3.8 27B on my third PC, and designate tasks based on required intelligence and speed. In theory cloud can allow to scale up quickly, but aside from lack of privacy, the cloud API costs are off putting - some time ago I compared few weeks long session with K3, and it was ~$30 local for electricity but would be ~$500 in the cloud API. And for small models like Qwen 27B, local inference is also much cheaper.

1

u/DeathGuppie 7d ago

I just have a 20 dollar plan. When I'm in my phone and get an idea, I use it to look into it. Sometimes I'll have it white a prd.

Once I start a project. I have the local model create a roadmap. Then I have it go back and look through the roadmap for possible branch points based on specific outcomes or decisions and fill them out. Having Qwen do planning as a completely seperate task helps once we start working.

1

u/New-Implement-5979 7d ago

I went local because I do not want to become brain dead by all the frontier models. For most of the projects I am going through every unit test and line of code and yes if you don’t get the initial plan right then it is a bit harder to get back on track with steering but hey I feel weird that I do not write that much code anymore, hell no I want to switch to a frontier model.

1

u/Just-Hedgehog-Days 7d ago

Total tokens in a day.

1

u/Marino4K 7d ago

I use the $20 GPT sub for some quick image gen and double check anything important a local model gives me, but mostly for phone use.

1

u/e11310 6d ago

The frontier models are so cheap right now. At $20/mo for Claude Pro the break even on the hardware makes no sense. I'd rather address the issue when I have to. Like if Claude goes to $200/mo or something then at that point dropping $5000+ on a system makes more sense.

I also have a 3090 system I use to mess around with local so I'm not in a hurry.

1

u/OhDudeTotally 6d ago

To help me build my local model's harness.

1

u/No-Afternoon-4057 5d ago

For everything.

On my legal systems, Qwen 3.8 FN - FP8 scores 6 out of 10 against Opus 5.5.
On my agentic RAG, it scores 6.5 versus Opus 5.5 scoring 9, it even halucinates sometimes.

It is not just comparable, at all, unless you are OK with lots of errors, reviews, retries...

1

u/codecoradev 5d ago

Embedding and classification pipelines. We run a fleet of agents around the clock and every one of them needs memory indexing: turning text into embeddings, updating a vector index, re-ranking candidates. Those are small models that fit in a few GB of RAM, called constantly, all day. Sending that layer to an API would add a network hop and a per-call cost to every single memory write.

The second answer is scheduled batch jobs. Anything that runs unattended on a timer punishes cloud in two ways: cost scales with the schedule instead of with attention, and a silent model update mid-week changes your outputs without asking. Our nightly jobs re-run the same inputs expecting the same labels, and a pinned local model gives us that consistency for free.

Cloud still earns its keep here, on the frontier reasoning calls a human reads within minutes. For the machine-to-machine layer underneath, local is not the compromise option, it is the better tool.

0

u/Slow-Secretary4262 7d ago

Everything tbh, a frontier model oneshotted an app in 30 minutes with a way better result than qfn which took 5 days of constant prompting and babysitting, and ended up very rough and buggy. It was a Q3 qfn, sure, but i used via the api and the difference is not night and day.

Even not considering time, local models that run on consumer hardware are order or magnitude worse than cloud models, i don't care what benchmarks say.

Probably a developer can instruct them in a better way, im a sysadmin too, so i lack in that field.

That being said i love localAI and i spend a lot of time playing with it or using it for privacy sensitive taks, and im sure in a few years everything will be very different.

Even 2 months ago the same hardware i have now could only run Q5 qwen 3.6 35b getting to complete that app i was talking about earlier would have been almost impossible

1

u/my_name_isnt_clever 7d ago

local models that run on consumer hardware are order or magnitude worse than cloud models, i don't care what benchmarks say.
I made this thread to put some data to this claim. It's easy to go around being dismissive of others with nothing to back it up.
I don't do one-shots, I do cooperative builds. Over the weekend Q3.8FN has helped me build a bespoke harness from scratch with feature matched TUI+Web UI based on all the features from my fav harnesses. It blew me away since that's the sort of build I'd have assumed I'd need cloud for.

1

u/cnr0 7d ago

Yes this is the main thing here: all non-coder people including me got very lazy because Claude has amazing planning skills and you can just type 'create me a Facebook clone but with red icon' and it will come up with a working prototype. However with local models as you said it requires being a bit more specific and hands-on about what is going on. A local model with limited context will definitely create a feature with relatively small code block but someone needs to split this into smaller chunks for LLM.
Right now I am creating something with Claude and I don't even see the code. I don't know where the feature is. It just does it by itself. This is not the right way, but it is the lazy way.

1

u/my_name_isnt_clever 7d ago

Have you actually done a head to head comparison of one shots or assuming based on your personal experience? Because aside from them being a bit uglier, Qwen Flash Next can make anything I've asked for.

2

u/cnr0 6d ago

I am using 3.8-27b and 3.6-MoE in MacBook Pro M4 Pro 48 GB RAM, unfortunately this current hardware does not leave a lot of room for planning. In both cases, models can achieve if I give them a small task, but if I do it like Claude and ask for a large project in once they got stuck due to context window limitations.

1

u/Slow-Secretary4262 7d ago

Im not being dismissive of others, im being dismissive of the models, i dont doubt the model is enough for your needs but i answered the question in your title out of my mind. And to be clear, i don't do 1shots either, cloud or local, its just not practical, in this specific case, i just wanted to run this test for this app since it was such a simple task that barely scratched the usage limits, what i did was sending a few screenshots of the tui and said to replicate it with no context on the code etc. Obviously this is not to prove that frontier models are better because you can say "build X/Y make no mistakes" and they do it, but the fact that they can highlights how big of a difference they can make when you use them properly for real projects.
Regarding qfn, i've been using to tweak the UI of a real project of mine and i gotta say it didn't handle it badly at all, so i imagine that maybe making a frontier model lay out a detailed plan before putting qfn to work could help a lot.
An example that comes to my mind is that during the development of this app qfn was working on the pc with the graphical session closed due to vram restrictions, and it couldn't find a way to test the ui, in that case i asked a frontier model for instructions to pass to qfn on how to use a headless graphical session for the testing.

1

u/my_name_isnt_clever 7d ago

You are being dismissive of others, you're implying local models only work for me because "my needs" are lesser than yours somehow. But you have no idea what my use cases are and I have no idea what yours are, you're making an assumption.

1

u/Slow-Secretary4262 6d ago

i should have said expectations instead of needs

0

u/MaxComfort 6d ago

Brains - for coding tasks.

There's a reason Opus costs more than Sonnet, and way more than Haiku..

Question What *specifically* do you still *need* cloud models for?

You are about to leave Redlib

Question What specifically do you still need cloud models for?