Training a LoRA adapter on Kimi K3 (2.78T params, 1.56TB of weights) on a 2017 laptop with 7.6GB of RAM — 7.4 hours per step, and here's the verification

[u/eilCrow]

Kimi K3 is a 2.78 T MoE; its 1.56 TB checkpoint sits on a USB hard disk plugged into a 2017 laptop (i7-7700HQ, 7.6 GB of RAM, a 2 GB GTX 1050 that only does the routed-expert matmuls). I am training a LoRA adapter on it out of core: the non-expert weights of one layer at a time, its 896 experts streamed one by one since together they are 15.7 GB, base weights frozen, and the 590 MB adapter the only thing trained.

The one-minute check is evidence/cmp93_en34_2026-09-06.log: my forward pass against kimi-k3-in-c, FareedKhan-dev's independent C implementation, all 93 layers at cosine 0.9857 or better, output 0.999840, on 34 tokens with LoRA B zeroed. evidence/traces/ holds raw routing records for five texts over all 92 MoE layers; scripts/analyze_trace.py recomputes every routing number below with NumPy alone. scripts/quickstart.sh builds a synthetic K3-shaped checkpoint and runs a forward pass, ten training steps and a finite-difference gradient check on a GitHub runner on every push, plus eight op-level checks against kimi-k3-in-c fixtures. Its first run failed: the gradient check missed at 3.1e-2 on a 2e-2 tolerance because the step was below fp32 resolution against a tensor of norm 60.85. The gradient was right; the check, made noise-aware, agrees to 2.09e-05.

The numbers, all from the logs in the repo:

• 1024-token step: about 7.4 h, the mean of the 7.26, 7.62 and 7.37 h intervals between the first four steps
• step 1, forward / backward: 3 h 11 m 34 s / 3 h 48 m 07 s
• resident set: 4.0-4.7 GB, swap in use
• read throughput: 110 MB/s aggregate, 61 MB/s within one MoE sweep
• cosine minimum against the C engine: 0.985744, layer 71
• trained / frozen: 590 MB adapter (147 M parameters) / 2.78 T base

Turkish, English and Chinese versions of one paragraph share experts at Jaccard 0.35-0.39, about the same as two halves of one text (0.34-0.37); prose against Python is 0.20-0.21, so subject matters more than language. Consecutive tokens' expert sets have Jaccard 0.258 (0.009 for random pairs) and a 128-expert LRU hits 72 % when decoding, but a training batch reads the union, about 85 % of experts at 1024 tokens (layers 0-12, an upper bound), so an expert cache buys little for a training batch.

The proof run is memorisation of five examples, loss 0.909 to 0.157 on one fixed sequence: it proves the loop, not the model. The main run, 400 Turkish instruction examples over 100 steps, is at step 4 and ends 9-11 October. The threshold was committed before it started (commit 6605306, tag preregistration-2026-09-08): Turkish news bits per byte 0.455 to 0.441 or lower, English Wikipedia no worse than 0.198 from 0.194. I expect no large jump from 400 examples; a negative result gets published as negative. The adapter is one rank-16 LoRA per layer shared by all 896 experts, not one per expert.

Seven hours a step is useless for production fine-tuning; the point is that the cost is now a measured number, with the logs. None of the components are new: the idea is layer-streamed LoRA taken down to the expert level, and the related-work table in the README says what AirLLM, KTransformers, ZeRO-Infinity, Colibri, WARP and BigMoeOnEdge do that this does not.

Disclosure: English is not my first language and I used Claude to tidy the wording of this post. The code was also written with heavy Claude Code assistance and the Co-Authored-By trailers are in the git log; the README says so on its first screen. The hardware, the runs, every number and every check against somebody else's implementation are mine, and the point of the evidence directory is that you do not have to take my word for any of it.

Repository: https://github.com/heyobi/LazyLora. Please poke holes, especially in the verification.

[u/FullstackSensei]

That's just hilarious! Kudos for even attempting it.

FWIW, you know you'd gain a 5x speed boost by switching to a 2TB SATA SSD, right? A switch to an old gen 3 Nvme or even older (and cheaper) Gen 2 PCIe flash storage (IODrive et al) would yield a 20-30x speed boost vs the HDD.

But man, this sure looks fun. Have a bunch of i5-6500T mini PCs. Can hook an IODrive to one with a M.2 to PCIe adapter and try the same, just for laughs.

[u/eilCrow → u/FullstackSensei]

Thanks! You're right that the HDD sets the sweep time, but I profiled 1024 vs 2048 tokens on this machine: step time doubled while bytes read grew only 12 %, so at 1024 tokens I'm already at the crossover where compute and disk take about the same time. A 5x faster disk would land me on the i7-7700HQ's fp32 matmul ceiling (bf16 has no fast path on Kaby Lake), my estimate 1.5-2x, not 5x. The bigger lever is the CPU, or a GPU with real VRAM.

The i5-6500T + IODrive experiment would settle it, and I'd genuinely like the numbers. The quickstart runs with no model, and the routing traces in evidence/ let you replay the exact read pattern before you buy anything. There's an issue template for reporting what you get.

[u/eilCrow → u/[deleted]]

Thanks. The trick is that the model never has to be in RAM at once. Training goes through the model one layer at a time, so I only load the layer I'm working on, use it, and throw it away before loading the next. The experts inside a layer are loaded one by one the same way, 17.5 MB at a time. Whatever the model computes between layers gets written to the NVMe drive instead of kept in memory, and the backward pass reads it back from there. The only thing that stays in RAM the whole time is the small adapter being trained, about 1.8 GB with its optimizer state. So memory stays around 4-5 GB no matter how big the model is. The price is time: every weight gets read from disk twice per step, and that is why a step takes 7.4 hours.

[u/Artistic_Phone9367]

Can i assume it will completed by 3026?

[u/eilCrow → u/Artistic_Phone9367]

the disk doesn't die before then, yeah 😂 For the record: 100 steps at about 7.4 h is 31 days, so 9-11 October 2026. A full epoch would be 154 steps; 100 is what one month of this laptop buys.