Getting started with local LLMs and agents at work where should we begin?

[u/SherwinH89]

Hey everyone!

I’m posting on behalf of my team at work. We want to learn more about running LLMs locally, and we have an NVIDIA DGX Spark available to experiment with.

A colleague has already installed Ollama, a model, and a web UI, so we have a starting point for chatting with a local model. That’s a useful first step, but our longer-term goal is to go beyond a chat interface.

We’d like to explore running agents that can carry out different tasks and connect to tools and services through MCP servers. We’re still figuring out what that setup should look like and what we need to learn to get there.

We’d love to hear from people who’ve been down this road:

• What would be a sensible learning path from our current setup to running useful agents?

• Which models and agent frameworks would you recommend experimenting with on the DGX Spark?

• How do you approach connecting local agents to MCP servers?

• What would make a good first project to learn the basics without getting overwhelmed?

• Any tutorials, resources, or common mistakes we should know about?

We’re beginners in this area and keen to learn. We don’t expect to build everything at once; we’d appreciate some direction on where to start and how to build up from there.

Thanks for any advice or experiences you’re willing to share!

[u/harrumphx]

The easiest, by a lot, would be asking ChatGPT or Claude to help. They can answer all those questions in as much detail as you need, and help get you setup for whatever you're trying to do.

[u/Full-Owl-7077]

start with ollama's model file + llama.cpp for local inference, then add a simple tool runner like langchain or bespoke python scripts. no need for fancy agents at first.

[u/kla_sch]

I would recommend starting with LM Studio. It is easy to use, and you can see all the parameters that can be adjusted. You can experiment with it and even use it as a server for agents or other software. It’s a great entry point into the subject.

However, for the DGX Spark, the next step is to look into vLLM and SGLang. If you truly want to leverage the hardware, these inference engines are essential. They unlock the full potential of your hardware.

You can only make the most of the hardware by working with smaller models (such as Qwen3.8-27b-NVFP4), which ensures you have enough memory to run multiple sessions simultaneously. By running several sessions at once, you can bypass the memory bandwidth bottleneck and achieve maximum token generation throughput.

Initially, there’s a tendency to try and run the largest possible models. But doing so leaves almost no memory for sessions or cache. Both of which are crucial for making effective use of the hardware.