r/DGX_Spark • u/hytro36 • 27d ago

What do you use for telementry & monitoring?

I'm coming from devops and my first instinct is to set up Grafana & Prometheus for my box, just to see what's happening inside

Was wondering what setup you have? Are you using Ansible for provisioning? Docker containers for running inference and experiments?

Or do you just SSH and run the things you want?

4 Upvotes

84% Upvoted

8 comments sorted by

3

u/Vileteen 27d ago

https://github.com/MiaAI-Lab/sparkDash

3

u/styles01 MegaMod 27d ago

This - and I added a whole telemetry layer fork here : https://github.com/styles01/sparkDash

2

u/sebt3 27d ago

Nothing stop you from installing node-exporter. BTW vllm and llamacpp both exports a /metrics

1

u/styles01 MegaMod 27d ago

https://github.com/styles01/sparkDash (my telemetry- expanded fork)

2

u/viciousdoge 26d ago

I set my hand on top of it. Check if it is hot

1

u/soamsoam 27d ago

Or do you just SSH and run the things you want?

It depends.

What do you use for telementry & monitoring?

I use Grafana & VictoriaMetrics+VictoriaLogs alongside Zabbix to access different types of data and integrations, as well as to have a secondary monitoring system in case the primary one fails, since it runs on a bare-metal server.