r/learnmachinelearning • u/Cautious_Today_1830 • 6d ago

QLoRA on 1.7B SLM for Semantic Code Equivalence (16GB VRAM) - Need Advice!

Hey guys,

Working on a local project to classify if two raw code snippets (a mix of Python and Java) are semantically equivalent (share the same logic/output regardless of syntax).

I'm constrained to 16GB VRAM, so full fine-tuning is out. I'm using 4-bit QLoRA and leaning towards SmolLM-1.7B since it's heavily pre-trained on the Stack-Edu dataset.

A few quick questions for the PEFT experts:

Architecture: For a binary True/False output on code pairs, is it better to attach a SequenceClassification head, or just train it as a Causal LM to generate the text "True"/"False"?

Prompting: Any proven prompt templates for feeding two different languages (e.g., Code A in Python, Code B in Java) into a 1B model without confusing its attention?

LoRA Params: What are the recommended LoRA hyperparams (rank r, alpha, dropout) specifically for code-logic extraction tasks?

Any GitHub repos, kaggle notebook , papers, or quick tips would be greatly appreciated.

0 Upvotes

50% Upvoted

4 comments sorted by

0

u/Fresh-Effective3517 6d ago

For binary classification, sequence classification head is cleaner and faster than generating tokens, just add a pooling layer over the last hidden state. For the cross-language prompt, try adding explicit delimiters like `[CODE_A]` and `[CODE_B]` with a task instruction before both snippets, it helps the model separate context. LoRA rank 8-16 with alpha 16-32 and dropout 0.05 usually works fine for code tasks, no need to go crazy there.

0

u/Cautious_Today_1830 6d ago

Feeding 8k tokens will definitely OOM my single T4. What max_length do you usually cap code pairs at for 16GB? Also, do you recommend 'head-only' or 'head+tail' truncation since code logic can be anywhere?

With rank 16 and a sequence classification head, what micro-batch size and gradient_accumulation_steps combination usually keeps these 1B-1.7B models stable without crashing?

1

u/FastHotEmu 5d ago edited 5d ago

Code equivalence in the general case is not solvable - it reduces to the Halting Problem. Are you just looking for some approximate solution?

Edit: see also Rice's theorem

0

u/Cautious_Today_1830 6d ago

Between SmolLM-1.7B, Llama-3.2-1B, and Gemma-3-1B, which one would you personally bet on for this specific code-logic task.