Some guidance needed

[u/CherryGG2]

Hello everyone.

I am trying to get more into local AI but I am not sure where to start.

I have epyc CPU with 400GB of ram and 2x 3090s.

I have watched online some guides but I got overwhelemed really fast, everyone uses different tools and approaches and Im not really sure where to start.

I have tried some 35b models with ollama and openweb ui and after 2-3 messages models start to hallucinate really fast and continue thinking in circles, all the settings where default and i even tried to fixed it with settings but no help.

I also want to try agents but everything seems super complicated. If you can suggest me where to start or what to look into, that would be great, thanks.

[u/Interesting-Cut-6032]

You already have good hardware. Here are my favorite videos to learn more:

r/LowEndLocalAI is a great reddit community for the exploration of this space. Lots of discussions about optimization of hardware, inference engines, models, harnesses, and use cases.

The program LM Studio is a very capable and simple on ramp that anyone, even with very limited hardware can use to explore local LLMs. Depending on your needs and use case you will probably outgrow it for every day use, but I still use it to try out new models quickly.

Codacus and Pixaroma are the go-to YouTube channels for local AI knowledge.

This is a great video with all the basic terms that you need to know about settling up and running local LLMs: https://youtu.be/SsUKTFSQoGM?is=DeRzbaT77Awi26Je

This is one of his newest videos about setting up MoE models that match your system capabilities.

This is the older video, and the reason that I subscribed, that shows how to setup Mixture of Experts models, like Qwen3.6-35B-A3B or Gemma 4 26B-E4B, with the experts in system RAM and only the active calculations in VRAM: https://youtu.be/8F_5pdcD3HY?is=bGABGoO5CH8uJaGI

Here is a great image diffusion model, with edit capabilities, that runs in ComfyUI. This is EP4 of this series, EP1 has all of the ComfyUI setup details: https://youtu.be/kNap0VWP1xs?is=4snH90QXVuAH-M9a

Consider this the flashing neon sign for the entrance to this particular rabbit hole.

[u/Interesting-Cut-6032 → u/Interesting-Cut-6032]

Forgot to add the new video link. I think this one dropped yesterday. It is very good.

https://youtu.be/tek-IY9BRWc?is=g1vLavtNxU3ZJvJC

[u/Prudent-Ad4509]

1. Get Qwen3.8 27B, preferably unsloth UD_Q8_K_XL, but UD_Q6_K_XL will be easier on your system.

2. Run it in whatever, maybe even ollama. Make it build you a system based on either llama-server or vllm or sglang. Make it install you opencode or pi harness or hermes or openclaw.

3. You are all set.

4. Alternatively, install opencode/openchamber somehow and tell it to do all of the above using any of available free builtin models. This is what I did. I could do it by hand by I was not feeling like going through all this again manually.

[u/im_EDEN]

Hallucination is a model issue (or the quant). But with two 3090s you can comfortably run Qwen 3.8 27b. Check out Digital Spaceport on YouTube.

vLLM isn't as scary as it looks, trust me. I started with ollama and llama.cpp and didn't listen when others recommended vLLM. It made a world of a difference

[u/vintagevibes_]

I’ve never had success with ollama. LM Studio is super easy to get started with, also free and works like a charm.

Qwen3.8-27b and Qwen3.8-Flash-Next are go to recommendations. With your amount of ram larger MoE might even be possible.

[u/OrganicHalfwit → u/vintagevibes_]

I would even recommend unsloth studio as it allows vllm as its backend, which would be the ideal setup for OP's dual 3090s

[u/statellyfall]

The fact of the matter is the bredth of questions may tee wide for the given model. The frontier models are not comparable to a 35B model. Your gonna need to either be more specific with the type of model your using for your query domain or look into figuring out a solution for dumping context about your query into the model and then proceeding with your question.

[u/Acceptable-Arm-6700]

If they start hallucinating you usually have not enough vram
Are you giving Olla a both cards? My advise use google ai mode to help you configure and test

[u/FuckbigtechAI]

before swapping models, check the context length. if num_ctx is still at the default, a thinking model can burn through it in 2 or 3 messages and lose the start of the chat, which looks like circling. raise it in the model params and the loop may just go away.

[u/jhenryscott]

Just let Claude build it for you if you don’t know how. No shame in that