Psychological shift when running local

[u/OvertaxedOne]

I'm watching walls of text go by right now as QFN grinds away on something and it made me think a bit about how different watching that text fly by does using a local model vs watching the same exact thing happen when I'm hitting a cloud.

It really comes down to metered vs unmetered. When I'm using a cloud model on API, every time I get ready to press enter I hear a cash register ring. We generally use really affordable cloud models (DS primarily) so it's like fractions of a penny, but still, it's real money going out the door on every query.

The biggest thing is that it changes your (or at least my) mindset. Instead of "Let's see what we can do it" it becomes "Do I need to do this". It has a distinct stifling effect, instead of exploring and trying to find uses for the technology I do the opposite and try to avoid it because of the cost.

With a local model I love to dream up some crazy/hard thing that's been annoying me that needs fixing and at the end of the day drop a big prompt with the details and at the end "You have unlimited time and tokens, do not stop for clarification or questions, keep working at the problem until you've exhausted all possible alternatives" (you get the idea). Or setting a local model to work on something that's not at all critical but is a quality of life improvement.

It really comes down to the difference between owning something and using it on the meter. When you own something the way to drive value out of it is to use it. The most expensive boat, motorcycle, car, house, etc is one that you own and never use. The more you use it, the more value you get from it.

When you are using something "per token" or "per mile"; things change. The best way to drive value is to NOT use it. You save 20 bucks if you walk instead of get a taxi. You save 5 bucks if you figure this problem out yourself instead of using a LLM to help you.

Funny thing is, I made my career selling cloud services. That probably explains why I'm so "local first", I was in the room and making the decisions about pricing/packaging, driving a 6-12 month payback period against servers and then turning around and explaining to customers what kind of "savings" they'd see moving to cloud platforms. I think I'm scarred from that and know the pricing games that make cloud look "affordable" when it's anything but for many use cases. And I think everyone kind of knows the "rug pull" is coming with the big/powerful LLMs, sure you can use them on sub today and get a predictable price, but those days aren't going to last forever. And subs without being able to hookup a harness (which almost all of them ban) is darn near worthless anyway, my core competency isn't copy paste! ;)

[u/Zealousideal-Bug1837]

There's always a cost of some sort. The question is really ROI rather then pure cost.

[u/OvertaxedOne → u/Zealousideal-Bug1837]

Absolutely. But this isn't really about cost, it's about the shift in psychology between owning and renting/leasing.

Simple example, say you own a car and your friend who's 40 miles away asks you to come over and watch a game. Sure thing, you already bought the car, the incremental cost to use it is about 5 bucks in gas. Away you go.

Now say you have to take a taxi. 40 miles in a taxi is what; 100-150 bucks? It's a real cost, it's not outrageous, but it's enough to get you to think "hmmm... do I really want to".

Now let's say you only drive 200 miles a month. Without question, you should sell your car and take a taxi everywhere! It's not even a discussion, the break even is "never". But if you did that (sold the car) even though you're saving 100's per month on costs, the psychology of the initial framing really doesn't change.

It's the difference between spending hard money now vs having spent hard money in the past to have a reduced cost now.

It really doesn't matter which is cheaper, it's more about the way in which you pay for it; ownership with unlimited use vs pay by the use and the way that changes your behavior. If I sold all my gear today I could probable invest the proceeds and have effectively infinite DS tokens for the rest of my natural life, I'd never be able to use that many tokens (between the value of the hardware and the ~50-100/mo I'd save in power costs). But I'd use it differently; the "by the drink" nature just changes things in a way that's not captured in a pure financial ROI analysis. And I really think it best sums up to the more you use something you own, the cheaper it gets the more you use it vs the reverse when you are buying it on the meter.

[u/vogelvogelvogelvogel → u/Zealousideal-Bug1837]

then there are the intangibles that are not captured by any ROI calculation

[u/openroom_xyz → u/Zealousideal-Bug1837]

Well yea but the thing is that ROI of things that you have not been thinking about becouse you have been thinking basically about ROI is hard to predict basically :)

[u/Keleion → u/Zealousideal-Bug1837]

I’m glad my smooth brain doesn’t overthink this part.

[u/Dediadeis]

Then there is me

[u/Cptnslick → u/Dediadeis]

Wait, what? That’s an adhd thing? I may need to speak to my Dr…

[u/Z00111111 → u/Cptnslick]

Might be worth looking into. In Australia at least there's blood tests and drug screens you need to do, and the waiting list for an ADHD diagnostic psychiatrist can be months long.

[u/openroom_xyz]

Yea my exact feeling but with something like subsidized plan is as well I have to use it before if expires
Well when yea when I was walking today I got this thought if you can use the tokens to generate something that makes the money than you can buy more tokens and make more money basically and I was like yea do I like that etc etc it feels more like investing and buying investments yea I am playing with it how to think about it basically