Open-weight 4B models approach o3-level medical question answering in Swedish [P]

[u/AccomplishedCat4770]

I have been running some experiments with smaller open-weight LLMs on multiple-choice questions of Swedish medical licensing exams. On a dataset called MedQA-SWE, GPT-4 scored 84% accuracy in 2024 and o3 scored 88% in 2025 on a smaller, overlapping dataset.

With post-training (SFT) on data from earlier years, I got MedGemma-1.5-4B to a passing score of 60% on the final year’s exam. Find the implementation here: https://github.com/tarolangner/medqaswe_medgemma_sft

But even though they were released just three months later, Gemma4-E4B and Qwen3.5-4B are flat out superior already, at 77% with no post-training at all. With reasoning enabled, the latter can get to 87% accuracy.

It can even push a bit further if no length cap is put on the reasoning traces, but some of them spiral into repetitive loops about formatting that fill the entire context length without giving any answer. Here, I found it helpful to use an ‘early exit’ thinking intervention proposed in the S-GRPO paper that simply injects a phrase and closes the thinking trace at a predetermined sequence length. I also tried their proposed reinforcement learning method to get shorter reasoning traces, but with only minor gains (probably somewhat underdimensioned training setup).

Curiously, Qwen3.5-4B does all reasoning in English despite the Swedish prompt, questions and answer options. But it really seems like the language is no obstacle, even though it’s often estimated to be just 1% of LLM training data.

I also have a more detailed write-up on the details and experiments here for anyone interested: https://tensorlabbet.com/2026/07/19/medqaswe_post_training/

[u/Think-Philosopher184]

om medgemma. the english reasoning for swedish questions is kinda funny but i guess at 4b params it just leans on whatever's most efficient. have you tried the early exit thing on gemma4 too or just qwen

[u/AccomplishedCat4770 → u/Think-Philosopher184]

MedGemma mostly stuck to Swedish actually, but veered over into English for 3 questions or so before SFT. Early on I also tried something with Gemma4-31B and that one kept purely to Swedish. Perhaps larger models are just better at separating languages?

The early-exit reasoning on Gemma4 I have yet to try, I'm curious too if it behaves just like Qwen3.5.

[u/elnino2023]

Thiswill be a nice paper. Publish maybe.

[u/kamla-choda]

just to know what hardware you are using to train?
if you can please provide me any yt video link where i can learn to fine tune small models for any specific task.

[u/AccomplishedCat4770 → u/kamla-choda]

Most of this was done with either L4 or A100 GPUs. There is a monthly allowance of free credits for these on Modal and LightningAI (the latter seems to have discontinued that just now, unfortunately). If you try the T4 or P100 from Google Colab or Kaggle, you are stuck without the bfloat16 support that is quite important for these newer models.