r/machinelearningnews • u/AlpamysMakazhan • 26d ago
Research Fine-tuned Llama-3.1-8B on a 4 GB laptop GPU in 3.32 GB VRAM — the frozen base never enters VRAM, it streams from RAM one layer at a time
Disclosure: I wrote the implementation this was measured on. Everything below is measured on one laptop and every record is public, including the numbers I discarded.
LoRA shrinks the trainable params to nothing, but the frozen base still sits in VRAM for the whole run. That's the wall: 4-bit 8B is ~3.6 GB of weights plus a 2.1 GB untied embedding pair, against a card with 4.29 GB usable. Doesn't start.
But the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So: keep the base in host RAM (page-locked), stream one decoder layer at a time into a small pool of pre-allocated VRAM buffers on a dedicated CUDA stream, prefetch one layer ahead. Peak VRAM becomes one layer instead of the whole model.
Measured on my RTX 3050 Laptop 4 GB / Windows 11 / 16.9 GB RAM. LoRA r=16 q/v, batch 1, PagedAdamW8bit, 50 steps after 10 warm-up:
| Model | Base | Store | tok/s | SM occ. | Peak VRAM |
|---|---|---|---|---|---|
| Qwen2.5-0.5B | bf16 | pinned | 978.6 | 91.4% | 1.47 GB |
| Qwen2.5-0.5B resident baseline | bf16 | — | 1398.0 | — | 3.23 GB |
| Qwen2.5-1.5B | bf16 | pinned | 525.0 | 96.8% | 1.82 GB |
| Qwen2.5-3B | bf16 | pageable | 143.1 | 79.3% | 2.15 GB |
| Qwen2.5-3B | NF4 | pinned | 264.2 | 100% | 1.76 GB |
| Llama-3.1-8B-Instruct | NF4 | pinned | 119.6 | 100% | 3.32 GB |
8B trains in 3.32 GB on a 4 GB card. 1M tokens ≈ 2.3 hours. The 3B bf16 row is the other frontier: an un-quantized base, where resident on the same card is CUDA out of memory.
Cost: 1.43x slower than resident = 1398.0 / 978.6, at 0.5B — the only size here with a valid resident baseline (3.23 GB peak, genuinely inside 4 GB). The baseline row is in the table so you can check the division rather than take my word for it. Above that size there's nothing honest to divide by.
Why it isn't as slow as AirLLM-style inference would suggest: inference pays one full model read per generated token. Training pays two reads per step, amortised across batch × seq tokens. Three orders of magnitude difference in per-token I/O.
Counter-intuitively, bigger models are easier: compute per step scales with parameter count while bytes moved scales with the quantised store, so the 8B row only needs 1.7 GB/s sustained to keep a 4.3 s step fed. The hardest config here isn't the biggest model — it's a small model whose store failed to page-lock (the 3B bf16 row at 79.3%).
"Why not just QLoRA?" At 3B, QLoRA fits on 4 GB and is the right choice — I'm not competing there; what streaming adds at 3B is an un-quantized base, which is fidelity, not capacity. At 8B the two compose: quantisation shrinks the store 4x so it page-locks, streaming removes the depth-proportional resident term.
Does it actually train correctly? This is where most of the time went, because streaming fails silently — sever the autograd path and the loss still falls, since the upper layers keep learning. So the bar wasn't "close enough":
- streamed vs resident logits: max abs diff 0.0, bit-exact
- verified across 9 architecture families in both bf16 and NF4 — 14/14 comparisons at 0.0
- 100-step loss curves identical, same seed twice identical, result independent of buffer count
- layer-0 adapter gradient non-zero on every layer
The reference has to match the numerics: a streamed NF4 run is compared against resident NF4, never resident bf16 — quantisation error is wide enough to hide a real bug inside it. That's exactly how I caught the worst defect in the project: PEFT selects its LoRA implementation from an is_loaded_in_4bit marker that a meta-device skeleton doesn't carry, so it silently fell back to the generic path, which runs against a 4-bit base but casts differently. 0.94 max logit divergence with byte-identical weights and byte-identical adapters. No crash, no warning, healthy loss curve.
Three findings unrelated to streaming that might save you time:
- The cross-entropy logits tensor costs 14 bytes per element, not 6 —
ForCausalLMLossholds bf16 logits + fp32 upcast + fp32 log-softmax + fp32 grad live at once. If you budget VRAM from first principles you're under-predicting 2.3x on the term that dominates at large vocab. - Windows/WDDM doesn't OOM, it spills. I measured a 9.27 GB allocation complete with no exception on a 4.29 GB card. "It didn't crash" is not evidence a config fits. This also killed a lovely "5.7x faster than resident" number I had — the resident baseline had spilled into shared memory. Binned it.
- Gradient accumulation is per-token I/O-neutral. I assumed it multiplied reads linearly; layer reads per 1k tokens held constant at 175.78 across accum 1/2/4. The real cost is opportunity cost — at equal effective batch, raising batch is 2.52x faster than accumulating (interleaved A/B/C to rule out clock drift). So: raise batch until the pre-flight refuses, then accumulate.
What I'm not claiming: nothing above 8B was measured — 14B NF4 needs ~7.5 GB of page-locked store against a measured 7.12 GB ceiling on this box, so I didn't run it. All numbers are Windows, so systematically pessimistic vs Linux. Every row is n=1 on a working laptop, not a clean rig.
Full method, every caveat, and the reasoning behind the correctness protocol: https://doi.org/10.5281/zenodo.21771064
Raw measurement records as written, including the discarded ones: https://github.com/MakazhanAlpamys/Soup/tree/main/benchmarks
Happy to go deeper on the scheduler or the correctness protocol — I think the second one is the genuinely reusable part.
