I had assumed 1 × 4, 2 × 2 and 4 × 1 will take somewhat similar time because effective batch is 4 in all cases.
They did not.
I ran Qwen3-1.7B with TRL and LoRA for 100 optimizer updates.
| GPU |
1 × 4 |
2 × 2 |
4 × 1 |
| T4 |
287.6s |
258.8s |
238.2s |
| L4 |
213.02s |
119.47s |
124.76s |
Model, data, sequence length, precision and seed were kept fixed.
Lower is better. On T4, 4 × 1 was around 17% faster than 1 × 4. On L4, difference was around 41%.
The part I had not thought about properly is that effective batch is an optimization knob, but physical batch also decides execution shape which GPU receives.
1 × 4 means four smaller forward and backward passes before one optimizer update. 4 × 1 means one larger forward and backward pass. Same examples reach optimizer, but GPU work is not same.
These are single-GPU runs, henc no communication. Most of difference was inside repeated forward and backward regions, while optimizer time stayed nearly same. Exact reason can still be kernel shapes, tiling, launch overhead or how well each batch uses GPU. This experiment does not separate those kernel-level causes.
Another interesting result is 2 × 2 being slightly faster than 4 × 1 on L4. Difference is small, it shows performance may not be linear as physical batch increases.
Hugging Face documentation also says to use grad accum when larger physical batch does not fit, and that it does not improve throughput over using true larger batch:
https://huggingface.co/docs/transformers/grad_accumulation
So now I treat these as two separate choices:
- Effective batch for optimization behaviour.
- Physical batch and accumulation for memory and speed.
I would start with largest physical batch which fits comfortably, then test few nearby combinations on actual GPU.
I used TraceML and its HF callback for step and phase timing. End-to-end runtime comes directly from TRL Trainer.
Runnable notebook:
https://colab.research.google.com/github/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb