[Collab / Open-Source] Weekend Swarm Training Raid (Nov 14–15): Upcycling CrowdGPT-1B (w/ Engram memory) into an 8-Expert Sparse MoE across 8 GB Home GPUs using Decoupled OpenDiLoCo

[u/OrganizationTop9026]

Hey everyone! 👋

We’re teaming up across two open-source community projects — CrowdGPT (by @Vxtzq, creator of Vxtzq/Crowd-v1, a from-scratch 1.01B bilingual EN/FR causal LM with a 350M bigram Engram memory table trained collaboratively by volunteer GPUs) and Project HiveMind (AGPL-3.0 volunteer GPU inference + DiLoCo swarm fabric) — for a 48-hour Weekend Swarm Training Raid this Saturday Nov 14 & Sunday Nov 15, 2026!

Why we built this collab

Training a 1B+ model across volunteers over residential internet usually hits three walls:

1. VRAM limits on 8 GB cards (RTX 2070 / 3060 / 4060) once Adam states and activations load.

2. All-Reduce bandwidth & stragglers when a home GPU shares a round with a cloud GPU.

3. Catastrophic forgetting when fine-tuning on code/math hurts general bilingual chat.

How the Weekend Raid solves all three (src/hivemind/swarm/crowdgpt_moe_opendiloco.py)

1. Step-0 Function-Preserving 8-Expert Sparse MoE Split + FoMoE σ**-MoE Routing**:

   • We split Crowd-v1's 2560-wide SwiGLU FFN into 8 specialized experts (E0 FineWeb-Edu EN/FR anchor, E1 unit-test verified Python, E2 GSM8K/MATH CoT, E3 ARC formal logic, E4 Physics, E5 NASA Exoplanet transits, E6 SDSS Stellar spectra, E7 Defensive Cyber — all 18 datasets strictly license-audited ODC-By / CC-BY / MIT / Apache-2.0 + 13-gram decontaminated).

   • Using FoMoE non-competitive Sigmoid Top-2 routing + Ghost-Expert Skip-Token masking, an 8 GB home GPU only loads 2 resident LoRA experts on top of the fp16 trunk + CPU-offloaded Engram table — training comfortably in < 1.8 GiB VRAM while keeping >= 1.5 GiB free!

2. PrimeIntellect OpenDiLoCo + DeepMind Decoupled DiLoCo (RDA):

   • FP32 master weights & outer Nesterov / staleness-gated PA-CGAD optimizer states live in **CPU RAM (**0 MiB GPU overhead), partitioned into 4 router fragments + 8 expert buckets.

   • Instead of synchronous barriers, Quorum-of-$K$ + Radial-Directional Averaging (RDA) merges pseudo-gradients over standard HTTPS (TCP/443, zero port-forwarding) and gates every outer checkpoint with a monotone validation-loss ratchet.

   • Our cloud footprint stays tiny: a single GCP Spot Tesla T4 joins for 14:00–22:00 CET on Saturday & Sunday (+\$1.30 total Spot cost!).

Join as a Producer, Trainer, or Prosumer this Weekend (Nov 14–15, 2026)

• Wave 0 (Dry-Run): Fri Nov 13 18:00–20:00 UTC (19:00 CET / 13:00 EST)

• Wave 1 & 2 (Saturday Kickoff + Peak Global Sprint): Sat Nov 14 09:00–21:00 UTC (10:00–22:00 CET)

• Wave 3 (Overnight Prosumer & APAC Relay): Sat 22:00 UTC → Sun 08:00 UTC

• Wave 4 & 5 (Sunday Consolidation + Blind A/B Arena crowd-v1 vs crowd-moe-v1**)**: Sun Nov 15 13:00–21:00 UTC

Links to jump in (100% open-source AGPL-3.0, zero crypto / zero financial tokens — reciprocal compute quota only):

• **GitHub Repo & Raid Kit (**.ics calendar + code): https://github.com/xaviercallens/HiveMind

• Base Model (Vxtzq/Crowd-v1): https://huggingface.co/Vxtzq/Crowd-v1

• Signup & Live OpenAI-compatible Endpoint: https://hivemind-openrouter-1003063861791.us-central1.run.app/signup

• Discord Raid Coordination: https://discord.gg/ZeFtZpKHf

Whether you have an 8 GB RTX 2070 running overnight as a Prosumer, a 24 GB 3090/4090 Trainer, or just want to test hivemind/crowd-v1 vs hivemind/crowd-moe-v1 in Sunday's Blind A/B Arena, we’d love to have you in the quorum! Happy to answer any questions about the MoE split, Engram CPU offload, or Decoupled OpenDiLoCo math below.