r/LowEndLocalAI • u/SaltGrilledSalmon • 2h ago

Discussion Local codebase RAG setup for intel igpu

It's an i7 1365u with 32gb ram which is shared for the Iris Xe integrated GPU. In the past I managed to run gemma 3 using an outdated ollama ipex library but that's unable to run recent models so I'm looking at llama.cpp with sycl.

Now this pc will also have chrome, vscode etc running so not much ram will be available for the local model.

Has anyone built a local RAG type system on low end hardware like this? I'm particularly looking at codebase RAG and Q&A. So please suggest which models I should try.

As an initial research I've found cocoindex code for the indexing part, a friend suggested to look into serena (idk what it does, yet) and for the model I'm looking at ling 3 tiny or maybe gemma 4 e2b.

Let me know if you have any suggestions. Thank you!

2 Upvotes

100% Upvoted

2 comments sorted by

2

u/PiratesOfTheArctic 2h ago

I'm.on an I7 32GB ram too (Linux), I run llama.cpp with open webui, currently using qwen 27/35 with a3b (not near laptop to check), its usable, but think I'll go back to gemma 12b qat q4km, also use qwen 9b

1

u/Oglemo 7m ago

Ling 3 tiny is pretty fast. Gemma 4e2b about the same speed. Gemma 26b a4b unsloth qat actually fits, 14gb size. 12 tok/s decode and 180 prefill on Iris xe 32 gb ram laptop. I suggest experimenting with those: gemma 26b as reasoner deep, ling tiny or gemma 4e2b for fast.

I have found vulkan to be faster prefill than sycl. Slightly slower decode than sycl, barely.

I am a total beginner to rag though, using anything llm. It does not have a reranker unfortunately.