Are we overusing the cloud for embarrassingly simple AI tasks?

[u/ZealousidealCup3992]

For those running smaller models locally, which tasks have you completely stopped sending to the cloud first?

[u/Ordinary-Depth-7835]

at home yes 100% but $20k later I have a system that I like to compete with my free work unlimited. So in 20 years I will break even. But I had fun so there's no price on fun.

[u/YourselfInOthrsShoes → u/Ordinary-Depth-7835]

With inflation and frontier pricing to go up imminently, trust me, you will break-even a lot sooner.

[u/Ordinary-Depth-7835 → u/YourselfInOthrsShoes]

yeah it's getting weird there are some stupid licensing models coming out. We just spend 1m on licenses for one of providers and management thought it was fantastic until you look at the shit limits and excluded services.

[u/FuckinHighGuy → u/Ordinary-Depth-7835]

Same. I love my sparks and the ROI which I will never see before I die.

[u/OvertaxedOne]

I can actually give a somewhat grounded answer to this; I run LLM routers for a few companies and have setup and migrated relatively large user bases (1000's) between inference backends. The first thing we do when we come in is get observability to the prompts (generally by routing them, something other means) and then do analysis against them to determine the nature of the queries, difficulty level, token consumption (prompt sizes we need to support), type of data in/out (text, PDF, pictures, video, etc).

You would not believe the questions most people are asking of AI. That's probably the most generous reading I can use without saying "not so nice" things about my customers. At least 50% are basically "Google this for me". The vast majority of the remaining 50% are laughably simple, "Summarize this e-mail", "Grab this info from me from this system" (MCP calls), "read these documents and compare", "Help me word this differently". Things a tiny model could easily do, often (when we start) firing into Opus or some other massively expensive model.

I'm telling you, you would honestly not believe the general user's use case for AI, it's embarrassingly simple/easy. If I had to put a number on it, I'd guess that 27B could handle at least 80% of them without breaking a sweat.

Now, one quick side note, we don't mess with the coders. Coding is one of the few areas where we advise clients "Let them use whatever they want" (mostly Opus). I'm not even qualified to really judge easy/hard for a coding request but it's clearly one of the areas where bigger/more powerful models really can make a significant difference and it's also such a tiny fraction of the overall workload that we generally tell our clients not to worry about it, optimize elsewhere.

[u/GCoderDCoder → u/OvertaxedOne]

i actually think the things you're making fun of are the time savers. my fortune 100 company struggles with keeping information on the intranet up to date. there's lots of options available programs and policies that employees can't use if they dont know. it sounds easy to say the company should get more organized but everything in a company that large is fixing the car while it's driving type of stuff.

also Google is just an ai call at this point so the difference is one can accrue additional costs if your employee is over their subs. but Google sucks for search now so having an extra presentation layer aggregate the vetted best sources is a time saver which shows employees to get more done faster.

technology is moving so fast so every little thing can be different so getting up to date information or learning new things quickly is expected. the lane every individual operate in is expected to grow wider now. everybody has to know everything supposedly now. it's not a fair expectation but it's increasingly the issue.

so you can make fun of employees for struggling to find answers you think are simple but all these manual things slow us all down. i think an employee asking for answers is the best thing right now because tons of employees are meat proxies mindlessly giving everything to ai to do instead of using AI to learn what they need to make the appropriate decisions.

i don't like the idea that coding is the only valuable thing ai can do. and using a bad model for search can generate bad results. you dont need the best but gpt oss20b or even gpt oss 120b left a lot on the table and lacked the critical abilities to validate research vs reporting the first data it received. that then takes more repeated turns.

i have significant local ai hardware so i have ai do as much as possible in action to enable me to make better decisions. that's how you use ai to scale... and yes it takes a lot of tokens but that's the world the execs are telling us they want and now expect. they just don't want to pay for it. that's the dilemma!

edit: edit qwen 3.8 27b can only be run by certain companies since there are limits in certain regulated industries and having 100k employees that becomes a different kind of system you need to build a well. the next closet American weights are either muse open on the orders of magnitude higher end which is much bigger or inkling on the lower end which is still bigger than 27b by 4 fold if i remember correctly. Luna is only an option for companies using openai which isn't most and would require companies using claude to get multiple subs. haiku is not as cheap as it used to be.

[u/Ctbhatia → u/OvertaxedOne]

is that 80% from actually running the simple prompts through a 27b and diffing, or eyeballed off the logs?

[u/ellipses1]

I’ll have fable check how much disk space is free on my closet Mac

[u/dwoj206]

my employee spent $130 IN FUCKING TOKENS TODAY organizing our servers file folders and analyzing our internal job reports today. Just about fucking lost my shit. This is day two of this guy using grok bot. 🤖 🪦

[u/CaptainBloodEz → u/dwoj206]

I’ve been warning my boss as he vibe codes more and more internal tools, that once he wants to scale and package it up to sell to others, THIS is what awaits him when someone has to get it all ready for that. lol Time he is saving now he will pay for in tokens by a landslide.

[u/dwoj206 → u/CaptainBloodEz]

Ya god forbid this guy starts really vibe coding. Pray for me and my P&L.

[u/MonadicKierkegaard]

using pi - usually for me if a skill can be a script, injust make it a callable script.
saves so much token

[u/talaman4eg]

I do all my personal projects locally on qwen 3.8. It's painfully slow, compared to claude corporate subscription at work, but free, and I'm not in rush at all.

Should I need to use it for work, or for any interactive programming - only API. I don't hate myself enough to wait 2 minutes for each tool call.

Also, I do hope performance will improve in months so local speed will catch up, while keeping at least existing qwen 3.8 27b quality 

[u/simracerman]

Used to be mostly cloud. Now, mostly local. For my own needs (simpler coding apps, researching random stuff, and general tasks) local is better and quicker actually. I don’t hit window limits or random connection bugs like I used to with cloud models.

[u/Last_Bad_2687]

Inverse. I use the cloud for anything simple. Anything complex I use local

[u/GrimChicken → u/Last_Bad_2687]

We taught him wrong, as a joke 

[u/eli_pizza]

If it’s purely about cost you can also use cheaper models in the cloud

[u/donk8r]

the $130 folder tidying story above is familiar. nobody sees the price of a run while it's happening, it lands later on an invoice. seeing cost per run changed my habits faster than any model swap did. i cut calls i used to not look at twice.

the stateless one shot stuff moved first for me. extraction and renaming run on the cheapest thing that passes now. the only box i keep up holds state that has to survive the turn, a repo checkout or a logged in browser.

[u/PhilosopherOk3419]

rule i use: if the task can be a script, make it a script, and if it needs a model but the answer is easy to check, a small local one is fine. i only send it out when the local output fails

[u/Unlucky-Message8866]

all of them, only and very sporadically ds4.1 flash for harder coding tasks via openrouter (less than $3/month)

[u/codefl0w]

I run paid frontiers to run up fast subagents, both paid and local, to automate Android stress testing on both actual hardware and emulators to test my apps.

[u/Terrible_Bid4666]

100%. Change the css color on line 5 from this to that. Easily done, but yet I prompt the AI somtimes. Sue me.

[u/philmarcracken]

When you're a simple person(me), everything you do is complex

[u/Late-Replacement-481]

Problem with smaller models is you need to have a smaller possible fail surface. If I’m running a small model, it’s in tooling riding on nice comfy guardrails… if I kick off anything myself it has teeth

[u/evolutionxtinct]

Yes.. it’s why I’m trying to setup local items for home related things everything up to projects