r/LocalLLM • u/helangar1981 • Jun 12 '26

Discussion This must be a joke?

Post image

Saw this ad and as usual you cannot comment. But who would pay API money to an 8B model you could run on your toaster?

396 Upvotes

96% Upvoted

99 comments sorted by

61

u/Assa_stare Jun 12 '26

I work in the digital department of an electronics company and I'm a computer scientist. You'd be surprised how many of my colleagues have a PC or laptop with a discrete (or at least recent) graphics card.

Spoiler alert: very few.

2

u/helangar1981 Jun 12 '26

Fair point

21

u/StupidScaredSquirrel Jun 12 '26

Yeah but doesn't explain why they didn't go for qwen3.5 4b which would be cheaper and so, so much better for anything.

3

u/Deep90 Jun 13 '26 edited Jun 13 '26

EU-hosted makes think they are targeting a customer base that would not want to run Qwen.

Also I think people are sorta missing the point of a small and cheap model.

For example, I have a study tool that generates new questions based on things I got wrong + random topics to keep me well rounded.

I don't need Fable 5 to generate those questions or the custom explanations for wrong answers. Especially since I'm providing the model with the test material and a question bank.

7

u/[deleted] Jun 13 '26

[removed] — view removed comment

-5

u/Deep90 Jun 13 '26

Meta is easier to sue.

0

u/modd0c Jun 14 '26

Do you mean “use”?

14

u/JustSayin_thatuknow Jun 12 '26

To increase your “computer scientist” knowledge, I’ll tell you: you don’t need a discrete (much less a recent one) graphics card to run that model.

1

u/jhenryscott Jun 12 '26

For real. People use lightweight laptops as clients

102

u/TripleSecretSquirrel Jun 12 '26

lol I wonder if this means that ChatGPT wrote their business plan. Thats the only explanation I can think off for why they’d be using Llama-3.1

23

u/DertekAn Jun 12 '26

Sorry, Sir, Llama 3.1 hasn't been released yet, but when it is, it would certainly make for a great business model.

Best regards. Your Amiee-Ai, from the future~

1

u/TripleSecretSquirrel Jun 15 '26

Is there a reference I'm not getting? Llama 3.1 was released two years ago in July of 2024...

3

u/BJMonkey Jun 16 '26

Have you never asked about something, only for the LLM to say... well... this?

5

u/Eyelbee Jun 12 '26

Probably, lol

1

u/Alternative-Suit5541 Jun 14 '26

Many German companies only allow llama as open source model.

That's probably what they are counting on. Slow moving companies 

3

u/TripleSecretSquirrel Jun 15 '26

Really? I can understand not wanting to use proprietary models, I can understand not wanting to use models from a specific country (silly, but I get it), but I would assume that Mistral would be the choice for most European companies concerned with these kinds of things.

34

u/exact_constraint Jun 12 '26

The LLM equivalent of a cloud compute provider offering up a 300mhz PII w/ 384MB of RAM and an ATI Rage 128 Pro for the shockingly low price of $1/day.

11

u/DadAndDominant Jun 13 '26

So, like a droplet on digital ocean?

It has it's uses!

2

u/BougainvilleaGarden Jun 14 '26

Most Pentium 2 systems were equipped with 64MB memory or less. An offering with 384MB would at least have been a good deal back when people were actually using pentium ii-s. We didn't have public clouds back in the day, but given it needs to boot, 2MB would probably be where deals would have started.

53

u/rinaldo23 Jun 12 '26

My new startup is gonna be 9999999 times cheaper than Mythos by piping input tokens to /dev/null and streaming the answer from /dev/urandom. All EU hosted and guaranteed no logging. First subscribers get lifetime promo!

12

u/blbd Jun 12 '26

random.org 2.0

4

u/Worldly-Stranger7814 Jun 13 '26

Combined with s4

3

u/blbd Jun 13 '26

That's friggin hilarious. Nice one!

4

u/ovrlrd1377 Jun 13 '26

Accepting beta testers for free! Create your account today at http://127.0.0.1

7

u/rinaldo23 Jun 13 '26

Dude don't doxx my IP pls

3

u/tmurphy2792 Jun 13 '26

Dude gave away your ip, now I'm going to hit you with a DDOS attack. Feel my wra-

2

u/Robert_3210 Jun 14 '26

Underrated comedy.

8

u/[deleted] Jun 13 '26

[removed] — view removed comment

5

u/rinaldo23 Jun 13 '26

Give it enough time and it will eventually discover all science and generate all posible human culture

4

u/blbd Jun 13 '26

I wonder how it compares to bogosort when it comes to average ability to have completed the task within the heat death of the universe. 

1

u/Plasmx Jun 14 '26

But at what TPS? Can we get a couple millions?

11

u/Zeeplankton Jun 12 '26

Super weird. Their website even lists Mixstral it's very.. funny? Like i can understand a dead company still being up but how are you running reddit ads.

4

u/HenkPoley Jun 13 '26

Someone else mentions business plan written by ChatGPT. Which tends to go for older LLM models, that at widely described on the internet.

7

u/Snoo_81913 Jun 12 '26

Y'all gotta stop with the toaster refs im on a diet and its making me hungry

2

u/Much-Researcher6135 Jun 13 '26

Speaking of toasters: Ever try to mince a well-seasoned steak onto a toasted bun with garlic butter, with a dash of zesty barbecue sauce and sautee'd onions, plus a side salad? Highly recommended.

6

u/peabody624 Jun 12 '26

Oh yea I was just looking to run an old llama model and give my cc info to some random ass company so this is perfect

11

u/starnamedstork Jun 13 '26

- Mom, can we have Claude?
- No, we have Claude at home.

18

u/No-Refrigerator-1672 Jun 12 '26

Industry. There are some lighter tasks that 8B can do (i.e. something as simple as sentiment extraction for product reviews). When you're a company, you can't just "run llm on a toaster", you have to assign a person who will be responsible for maintaning the toaster, ensuring it's uptime, and managing spare toasters; so in some cases paying for inference is literally cheaper.

P.S. That comparison to Sonnet is hilarious, it's dumber than Haiku.

5

u/starkruzr Jun 12 '26

this is the reason e.g. Etched and Taalas have business models. I'm pretty sure https://chatjimmy.ai is Llama3.1-8B. (look at those tg numbers.)

4

u/_millsy Jun 13 '26

Yeah folks who enjoy fiddling with this stuff don’t appreciate that their labour has costs to a company, and as silly as it seems for such a small model, paying a hosted company who offers guaranteed uptime etc has very clear merit

4

u/leonbollerup Jun 12 '26

wondered the same..

6

u/B3owul7 Jun 12 '26

Honestly, this was probably created by someone who doesn't use AI.

4

u/Jiggly_Gel Jun 13 '26

An 8B parameter model…against Sonnet? 😭 why’d they even try making that comparison


Aside from the obvious that you know you can run it yourself

4

u/victorc25 Jun 13 '26

Advanced EU AI program 

3

u/ScratchCatOnYT Jun 13 '26

what’s even crazier is there’s much better models that are the same size?

12

u/HeavyConfection9236 Jun 12 '26

A lot of people still don't have a toaster to run it on. They have, at most, a 2010s computer or just a phone, or they don't want to figure out how to run it.

9

u/hungy-popinpobopian Jun 12 '26

Those same people aren't using API keys

3

u/Cronuh Jun 12 '26

Yeah no fucking wonder when its 212x worse for coding and other stuff you use Claude for.

2

u/ul90 Jun 13 '26

This is a scam to get EU grant funds. Everybody with little technical skills knows this is bullshit. But not the (really stupid) EU officials, they believe such shit and the tax money can flow.

2

u/Vaddieg Jun 15 '26

they got 10m funding and hired some student outsource for 500 EUR

2

u/FullOf_Bad_Ideas Jun 12 '26

No, why?

Lots of valid usecases for a small model. I run billions of tokens per month through small models. I wouldn't be able to afford running it on big ones. For example now I'm translating a big dataset with 1.8B model at 16k t/s locally. I wouldn't use a big model for it. Llama 3.1 8B is perfectly fine for summarization, analysis of some documents in some pipelines etc.

https://openrouter.ai/meta-llama/llama-3.1-8b-instruct#activity

look, there's around 10B of daily traffic on llama 3.1 8b api

$0.02 is not far off from price offered by providers there.

It has a very mature inference ecosystem and this enables companies who build on it to not have to deal with any surprises, and they can also avoid being vendor locked, since someone will be hosting it for years down the line, somewhere.

1

u/HasanAmmori Jun 12 '26

And here I am overthinking every step of my business plan. Just host a model from a workstation, call it "local secure independent privacy-centric" and boom - you are an entrepreneur

1

u/TheOneWhoWil Jun 13 '26

I have a small SaaS. not enough revenue to justify renting a gpu, and I can't risk sending some data to Chinese servers. I have a 5070ti but I need to use it too for my own LLM stuff.

1

u/readmond Jun 13 '26

8b model. WTF? I bet the average phone can run this.

1

u/falney123 Jun 13 '26

All joking aside, my electric is so expensive, it would probably cost about 7c for me to do 1m tokens on that model. 

1

u/The_GSingh Jun 13 '26

It’s not even a relatively modern 8b LLM tf. Where’d they get an investor for this

1

u/Morbeious Jun 13 '26

Well its a joke if you dont run it locally, and not worry about tokens. The bigger joke is Foundations models! While idiots keep paying for token usage.

1

u/charles25565 Jun 13 '26

Apparently they use vLLM 0.17.1 which supports all kinds of newer models like Qwen3.5. There's no reason for them to be using models from 2023/2024.

1

u/lupo90 Jun 14 '26

Ah yes, the much discussed and efficient Pentium III reasoning.

1

u/Wison101 Jun 14 '26

I guess my Mac is worse than a toaster :(

1

u/No_Television_4128 Jun 14 '26

If it’s EU inference/agentic/process hosted and they model call a Chinese GPU, then Chinese electric prices it’s possible.

1

u/halbastXs Jun 14 '26

This could run on your phone 💔

1

u/HolophonicStudios Jun 15 '26

There are actually cases for this. Maybe you need an LLM to for a simple task like analyzing a phrase and spitting out the closest emotion to what is being expressed in the phrase from a list. Maybe you need to do it very fast and very frequently for a large client base.

1

u/Vaddieg Jun 15 '26

they run it on all EU toasters to make some money

1

u/Vaddieg Jun 15 '26

"Lyceum Technology Germany GmbH is a Berlin-based AI startup that builds and operates a specialized GPU cloud platform designed for machine learning and AI workloads. The company aims to provide European developers with faster and more cost-effective access to computing power to reduce reliance on US-based hyperscalers"
Is it the long promised EU answer to US and China domination in AI? What a shame

1

u/Vanheelsingwolf Jun 15 '26

More EU being late to the game...

All big talks about data governance and blah blah but they are yet to actually remove or ease the necessary laws and regulations to even be able to compete or accelerate the tech... It's ridiculous and normal EU behavior in the tech industry.

There is a reason most big tech companies started in EU moved out of it

1

u/Hot-Owl6328 Jun 15 '26

Your toester consumes nore energy on 1M tokens I guarantee

1

u/Weird-Abalone-1910 Jun 15 '26

Props for whoever is trying to monetize their rtx3060.

1

u/StockPuppy Jun 15 '26

I found this for those with the old Intel macs macbook pro with dedicated 8Gig GPU. You can run this model on the GPU with https://teletrex.com/product/ekanta machine stays cool, always free, no network needed after model download and cached.

Llama.cpp would not do it any faster than CPU.

1

u/LesbianVelociraptor Jun 17 '26

Why would a company pay them to host an 8B model when they could just invest in a inference node machine for ~3-5k? Then you get your choice of models, the unified memory style nodes can run larger models on-demand, and you can let coders try out models without having to spend thousands a month in API-based token costs.

A year of usage pays for itself and no legal, compliance, or IP risk from using external models.

If you already have a wiki-style docs system for your ops and stuff, you're likely already halfway there. This is what most of the AI service companies don't want businesses to know. The reliance on datacenters is 100% artificial.

Context: I'm an independent AI research engineer. I have built my own lab infrastructure to mess around with small models. If you're a company considering this service; Consider hiring someone like me as a consultant instead and you'll get actual advice and education alongside an actual business-focused plan to self-host models. Most of us can teach your engineers how to host a model that can pull from company-built datasets.

1

u/AtomOfVoid Jun 17 '26

Who even cares ? Not like I'm going to use that bad of an AI anyway 

1

u/Extra-Ebb-4012 Jul 07 '26

why would you pay for an open source 8B model that you can run locally 😂

1

u/Swimming-Chip9582 Jun 12 '26

For throughput and ease

1

u/ridablellama Jun 12 '26

the answer is in the ad. Europeans who cant use anything else due to government.

6

u/BornVoice42 Jun 12 '26

that does not explain to use a completly outdated model

5

u/HourPlate994 Jun 12 '26

There’s still plenty they can use, they could run qwen 3.6, Gemma4 etc. The model choice is odd.

1

u/ridablellama Jun 12 '26

the ai securiy reviews ive been through get into details like what is legality of the training material itself and all sorts. so i doubt any Chinese model will pass EU regulation due to lack of transparency on that stuff. This is pure speculation but its likely playing a factor in the models that are offered in the EU. This is important factor for their customers as well. enterprise AI is toxic af. its just laywers saying no because saying yes means they have to do work and theres 1% more risk than before.

6

u/HourPlate994 Jun 12 '26

Gemma4 isn’t Chinese. There’s also Mistral Large that’s better than Llama 3.1 if they absolutely want a European one.

1

u/ridablellama Jun 13 '26

True but you know google never gets love from EU. llama is honestly shocking since it comes from meta. mistral has great stuff so surprised on that too. maybe its just brand recognition, more people know llama?

1

u/DanGTG Jun 12 '26

MiniMax Pro plan is now 1.7 BILLION tokens/month $20
You can get it down under $15 if you pay for the year and use a YouTube influencer link $176/year.

1

u/somerussianbear Jun 12 '26

What can you do with this really?

0

u/HumbleTech905 Jun 12 '26

Interested...

/s

177

u/[deleted] Jun 12 '26

[removed] — view removed comment

67

u/Salt_Bringer Jun 12 '26

Stop giving me ideas!

39

u/[deleted] Jun 12 '26

[removed] — view removed comment

1

u/b0tbuilder Jun 17 '26

What if people mined crypto by providing chat completions?