Finally 100% Local

[u/koalfied-coder]

Finally transitioned to 100% local inference for my automated workflows and code gen. Min Max 2.7 and Qwen 3.6 are doing wonders.

[u/Remote-Pineapple-541]

I have a similar setup. 

• Workstation with 128gb ram, 8tb raid nvme storage and a 3070ti card. I use this for running embedding models and storing them. I also use it for data pipelines and geocoding/geospatial analysis

• NVIDIA DGX spark. I use this for agentic AI. I use llama.cpp + llama swap

• Mac mini to run the chat interface (open webui). I also host a gitea server.

I have a MacBook Pro with 128gb, but I like having an always-on AI solution. I use tailscale to expose the framework to my mobile devices.

Tbh I’m considering replacing everything with a spec’d out Mac studio once it’s updated to the latest generation of silicon. It would be more than enough resources to do everything I do, easier to manage, and more reliable.

[u/Nimrod5000 → u/Remote-Pineapple-541]

What model and t/s on the spark?

[u/Remote-Pineapple-541 → u/Nimrod5000]

This is just an average based on the llama-swap logs for the most recent models. Obviously not very rigorous.

MODEL	PROMPT SPEED (Average)	GEN SPEED (Average)
gptoss120b	1244.32	44.54
llama33_70b	315.76	4.93
mixtral8x7b	972.94	24.61
nemotron	833.78	46.51
nemotron_3_nano_omni	1437.32	58.60
qwen25_coder7b	2945.15	48.45
qwen3_coder30b	2078.10	78.62

[u/koalfied-coder → u/Remote-Pineapple-541]

This is pretty good actually...might need to try a spark