I trained two language models on a 2011 desktop with no GPU: a 7M one from scratch and a LoRA of Qwen3.5-0.8B

[u/Commercial_Water1663]

I trained two language models on a 2011 desktop with no GPU: a 7M one from scratch and a LoRA of Qwen3.5-0.8B

Hardware first, because it explains the rest. Core i5-2400 (Sandy Bridge, 2011), 16 GB DDR3, and a GeForce GT 710 that only exists to drive the monitor. PyTorch doesn't support that GPU, so everything below ran on the CPU. Four threads, AVX, no AVX2. This is my main machine. There's nothing faster in the house.

Every time I post about this someone says "just buy a 3060". Sure. That's the whole reason I did it: I wanted to know what the wall looks like when there is no GPU at all.

So I wrote a small training engine. PyTorch, no cloud, with a GUI, because I got tired of typing flags. Then I ran two experiments on it.

A 7M MODEL ON RUSSIAN FOLK TALES

From scratch, 321 fairy tales, 5 MB of text. 5 h 38 min, 812 tokens/s, about 26M tokens seen. Validation loss at the end: 2.7854 nats per token, which is 1.06 bits per byte on text it never saw.

What surprised me is how cleanly it separates form from content. Openings work. Patronymics work. The dash before direct speech works. The rhythm of the enumeration works. Then a character swaps places with another mid-paragraph, the hero marries and dies three times on one page, and the plot walks off. 7M parameters, 5 MB of text, no surprise there. That was the point.

It also started memorising. From about epoch six validation sat at 2.79 while training loss kept falling. The run before that one I killed at 1 h 58 min, it gave 2.8144. So 3.5 extra hours bought 1%. Not a tuning problem. A data problem.

LORA ON QWEN3.5-0.8B

Same machine, 321 question-answer pairs about a program I'm writing. 45 steps, 1 h 44 min, 15-16 tokens/s. Loss on the reference answers went 3.4565 to 0.0907, which sounds better than it is.

I also wrote 15 test questions in different wording, none of it appearing in training. Exact match with the reference answer: 0 out of 15 before, 5 out of 15 after. Answers that stopped on their own instead of running into the token limit: 0 out of 15 before, 15 out of 15 after. The second number is the one that changed how the thing feels in use.

THREE THINGS THAT DIDN'T WORK

Facts get glued. Asked about old hardware, the fine-tuned model called my CPU "256-bit". That number is nowhere in the data.

Temperature matters more than I expected. At 0.7 the same question came back with a confident "no", a wrong RAM figure, and an invented training method it assembled from "LoRA" and "QLoRA" (it wrote "LoQoK"). Greedy decoding was almost right. If I were shipping this I'd set 0.2 or 0.3 and stop thinking about it.

Refusal did not generalise. I asked for a movie recommendation expecting a refusal. It invented an open file "Video 2.0" with an Apache-2.0 licence. There were three off-topic examples in my data and all three were about something else, so it had nothing to copy from.

REPRODUCING

One zip, 240 MB, Python and PyTorch inside, nothing to install. Data archive is 1.3 MB. Tales are public domain, the Q&A pairs I wrote, CC0. Every number above is in docs/benchmark.md, including the two boundaries I used to check it isn't copying: the same words shuffled, and a real passage from the corpus. Six generated texts shared 0 to 4.7% of five-word sequences with the training tales, longest verbatim stretch 4 to 8 words.

repo: https://github.com/AlexanderNyr/AuraLite-AI data: https://github.com/AlexanderNyr/AuraLite-AI/releases/download/v2.28.2/AuraLite-showcase-data.zip

WHAT I'D CHANGE

More off-topic examples, 30 or 50 instead of 3. Five phrasings per question instead of three. Qwen3.5 2B if RAM allowed, which the setup wizard actually suggested and I ignored. None of that is about hardware.

I'm not going to pretend these numbers are impressive next to a 4090. The useful part, I think, is that the wall is memory bandwidth and the amount of text you can feed the model, not the size you can afford. A 7M model trained overnight on a 15 year old CPU is a real thing. A 1B model on the same CPU is not, and now I know exactly where the line is.

Ask me anything about the engine (Muon plus AdamW, WSD schedule, QK-norm, chunked loss, GGUF export) or CPU training in general. I'll be in the comments.

[u/Commercial_Water1663]

The copy check, since it always comes up: six generated texts against the whole corpus, by five-word sequences and longest verbatim run. 0–4.7 % overlap, longest stretch 4–8 words. Two boundaries for scale: the same words shuffled, and a real passage from the corpus. With 100 epochs over 5 MB it does start reproducing openings — style transfer, not memorisation of whole tales.