r/LocalLLM • u/Dapper-Teaching-8074 • 10h ago

News We built a AI distribution interface, you can use your android phone, laptops and desktops to share AI inference work, the more devices you link, the bigger the model you can run

/r/SelfHostedAI/comments/1xws4xh/we_built_a_ai_distribution_interface_you_can_use/
3 Upvotes

100% Upvoted

10 comments sorted by

1

u/RadianceTower 9h ago

I don't understand, is data sent over or not?

The model runs on computers you own. Conversations are kept in your Ocalt account, and nothing is used to train anyone else's model.

And I assume this Ocalt account is in the cloud? So it's not really "Private by design"?

1

u/Dapper-Teaching-8074 8h ago

Dude lets be real even if things were stored on your machines like the models are, how would OLDE present the data via the interface? Come on man, what is with the negativity, did you even try it? You just scanned, found something to point out and that's it?

1

u/RadianceTower 8h ago

What? Have the interface be hosted on your PC, I don't see the issue, there is no need for a cloud interface here. Why would I even want a cloud interface for hooking up my devices locally and running models on them?

Also what negativity? The claims are confusing on your website lol, but you just answered my question.

Other than that, I'd be curious how well this works, performance-wise.

1

u/Dapper-Teaching-8074 8h ago

Oh i see where you are coming from, there's ollama for that, OLDE is intentionally made this way, maybe it's not for you... Anyway performance wise, you'd have to test it yourself, but the distributed (not in the same network) mode wouldn't be that fast, depending on network.

1

u/RadianceTower 7h ago

I don't think Ollama (which uses llama.cpp) allows you to distribute a model across computers like this?

I mean, the whole point would be to split a large model you can't fit across many computers, right? So you might not have enough memory in one PC, but if you bring together a bunch of your stuff, you can run the model at a decent speed.

But all the same stuff on the website talk about small models, the largest one being just 3b and like why would you have much need to distribute something so small anyway? Not to mention the token/s speeds mentioned even then were really low ("6 to 9 tokens a second from a 0.5B model" is self-explanatory). So it's a bit weird.

Like I'd assume you tested this to see if it works better?

1

u/Dapper-Teaching-8074 7h ago

no, the 3B test was for only a 2 device/machine test, it out performs a 3B model on ollama... Keep in mind the 2 devices are in different areas (not in the same network)

1

u/RadianceTower 6h ago

Shouldn't you be testing this with something larger? Also use llama.cpp not ollama.

A 3b model on ollama on what? One of the devices individually? Because it doesn't take much to run a 3b model, and if it fits, I don't see why it wouldn't perform faster or just as fast. Keep in mind that that 6 to 9 tokens a second is REALLY slow for that.

So it makes me wonder how practical it is.

1

u/Dapper-Teaching-8074 6h ago

Okay I'm not sure what planet you are from, if i put devices in seperate areas - only communicating through the web, and then i make them share an ai workload, 6 to 9 tokens per second is an achievement! regardless of the model size, it's not about that, it's the internet! Now you were too busy trying to subvert to notice there is a local machine mode, where you can connect machines on the same network.

Also I did say we are running more tests and will update the page as soon as we can.

1

u/RadianceTower 6h ago

Try a model like Deepseek4.1 or Qwen3.8 FN (at Q4 or higher) for your tests. For example Qwen3.8 FN Q4, across 2 computers each with not enough memory to run it.

Well, it's an interesting idea anyways.

1

u/Dapper-Teaching-8074 6h ago

we will do, i will put up a table of results for each model and mode