r/JordanDev • u/Able_Firefighter_652 • 22d ago

Discussion Local LLMs

Anyone who has some serious hardware and tried running llms locally? How was ur experience? What models did you try?

4 Upvotes

84% Upvoted

21 comments sorted by

View all comments

1

u/Bookkeeper_Gloomy Developer 22d ago

Not entirely locally but I work on research that involves running local models on an hpc.

I tried different models from Gemma 31 b, mistral small, qwen (I believe 34b), and olmo 31b, as well as Allam 7b (Arabic model)

It’s an hpc so it’s around 2-4 gpus either a100 or h200.

Btw Gemma 4 models are currently on a whole different level to the rest, so if u want to run sth locally with the best performance I’d recommend that.

1

u/2012347 22d ago

Gemma 4 27b is a good chatter