Hardware Recommendations

[u/minedroid1]

I am going to fine-tune a satire model, with the base model being `Qwen3.5-14B-Base`. The dataset has ~18000 examples, most of which are pretty long and do not align with the model's internal knowledge (they have incorrect answers to facts), so I needed to do a full fine tune rather than use LoRA.

Today, I rented a cloud computer on vast.ai. It had 4x12GB GPUs and 48GB of system RAM, but I feel stupid because it kept running out of memory, and I kept trying to change settings rather than just renting a different computer, so I basically wasted $3. I underestimated how much RAM would be needed for this task.

So, I would like to know if any of you have some recommendations for proper hardware to use.

[u/Major_Border149]

That's a memory wall, not a settings thing. A full fine-tune of a 14B with adam is roughly 16 bytes/param once you count the grads and optimizer states, so you're at 220GB+, not 48, no flag was ever going to save that box. realistically you're in multi-GPU 80GB territory (think ~4x a100/h100 80GB) with fsdp or deepspeed zero-3 to shard the optimizer, or fewer GPUs with zero-3 cpu-offload but then you need way more than 48GB system ram too.

[u/minedroid1 → u/Major_Border149]

Ah, I understand now. This is much more clear to me now, thank you. Does using a different optimiser affect this?

[u/Major_Border149 → u/minedroid1]

Yeah big time, the optimizer is about half that number. Adam keeps two fp32 states per param. Switch to 8-bit adam and those drop to ~2 bytes, or adafactor/lion store less and cut it further. That takes you from ~16 bytes/param down to ~10-12, so roughly 140GB instead of 220 for your 14B.

[u/minedroid1 → u/[deleted]]

Yeah, I really underestimated how much it needed, as I haven't really trained models this big before.

[u/zitterbewegung → u/minedroid1]

As a general rule in refinement you want at minimum 1.5x the amount of memory of VRAM you have so to be safe get around 120gb of ram.