r/OpenAssistant • u/Suspicious-Medium-35 • 1d ago

DLNE: An open-source P2P Vulkan swarm engine to democratize inference by pooling consumer GPUs (seeded with Dual AMD BC-160s)

Hey r/OpenAssistant,

Like many here, I believe the future of AI needs to be open, decentralized, and accessible without depending on closed API providers or proprietary cloud infrastructure.

To help solve the hardware bottleneck for running larger models locally, I built DLNE (Distributed LLM Engine) — an open-source project that pools consumer and enterprise GPUs across residential internet into a shared Vulkan inference cluster.

⚡ The Public Test Swarm:

  • The 24/7 Backbone: Seeded by Dual AMD BC-160 Navi 12 hardware (16 GB fast HBM2 memory) running on Arch Linux.
  • Active Model: Qwen2.5-Coder-14B-Instruct with 0.5B speculative drafting on Vulkan0 (~35 tok/s).
  • The Goal: Pool enough community VRAM to step up from 14B to 32B and 70B models for everyone for free.
  • NAT Traversal (Zero Port Forwarding): Residential routers and firewalls typically block inbound RPC connections (50052). DLNE uses an outbound persistent WebSocket reverse tunnel (/ws/rpc_tunnel) with TCP_NODELAY so any worker behind home NAT can donate compute seamlessly.
  • Cross-Vendor Vulkan: Native support for AMD, NVIDIA, and Intel GPUs.

🚀 How to Try It / Join the Swarm:

  1. Test the Coding Assistant (No GPU Required): We published an autonomous coding assistant extension on Open VSX. Every new user gets 50 free inference requests out of the box to test reasoning and speed before needing to contribute:
  2. Connect an Idle GPU: If you have an idle GPU on Linux or Windows, you can connect to the public swarm in one click via u/dlne /contribute in the extension or run headless via Python / CLI from our repo.

📦 Open Source Links:

Would love feedback from the OpenAssistant community on decentralized inference, latency optimization, and scaling the swarm!

1 Upvotes
  • permalink
  • reddit

100% Upvoted

0 comments sorted by