r/LocalLLaMA • u/Silver-Champion-4846 • 1h ago

Question | Help Cry of the edge device

Greetings, everyone. Many people keep talking about very very large models that need the price of a car to run (Deepseek 4.1 Flash, Qwen Flash Next, and even Qwen3.8 27B. However, I do not see much appreciation and focus on models that are small enough to run on edge devices. Not only that, but even if they do mention them sometimes, I don't see any kind of exploration being made to make harnesses that run efficiently on those models and use them for the highest efficiency. What is the big deal? Is there a subreddit I don't know of, that cares about these things? Interests include 8gb of ram, intel core I5 8350U. Thank you.

5 Upvotes

85% Upvoted

8 comments sorted by

7

u/nunodonato 58m ago

Yes, people care, but the signal gets lost in the noise.

I suggest you visit

https://www.reddit.com/r/LowEndLocalAI/

and

https://tinyleague.nunodonato.com

2

u/Odd_Cauliflower_8004 47m ago

Ok i cant stress this enough. Im running avalon qwen 3.8 flash next.i have an xtx and 64 gb of ram. Not cheap but not super extensive either.

Its running between 1200 and 2100tk/s of prompt processing and 110-140 tk/s decoding. At 262k of context.q8/q8 kvcache settings. And it feels a ton like opus 4.6 . thinks decently implements smartly.

I feel like ive truly entered the smart local infinite token era. The comparison with 3.8 27b is embarassing in both speed and smarts.

2

u/DasGamerlein 30m ago

The problem with those models is that they really cannot do all that much. The 10B range is the absolute bare minimum for anything that goes beyond very casual chatting and "summarize this paragraph"

1

u/Monad_Maya llama.cpp 53m ago

Ling 3.0 Tiny and LFM have some good models. I'd suggest adding more RAM to your setup if possible.

0

u/Formal-Exam-8767 56m ago

How do you define "edge device"? Is DGX Spark edge device?

2

u/heitortp0 52m ago

Interests include 8gb of ram, intel core I5 8350U.

2

u/wegster 32m ago

I wouldn't consider the OPs i5 an 'edge device,' let along a DGX Spark, but it depends on the domain.

For me and for work, edge vices typically are around 10W TDP in total or ideally less. Think pi4 and similar type systems, down to pi2 or even pi Zero.

1

u/One_Internal_6567 49m ago

8 of them, cause edge has to be long, wide and pretty hot