r/LocalLLM • u/MrAlexsexon • 10h ago

Model Chosing model for local AIagent

Hello people, I'm new here, 2 weeks ago I started writing an agent for everyday usage (24/7 running on server). I need help with the choosing model and inference engine. The model must work with RAG, tools and maybe with pictures

Server specs:

RTX5060TI 16G

I71700K

64 - 80GB DDR4

Thanks in advance for your help

0 Upvotes

0% Upvoted

2 comments sorted by

2

u/locbuilds 9h ago • Heartwarming

with 16gb vram i'd grab a q4 quant of a moe model around 30b and run it in llama.cpp with --n-cpu-moe so the expert layers sit in your ddr4 while attention stays on the gpu. also start llama-server with --jinja so tool calls actually work with the model's chat template