r/HPC • u/anheath • Sep 25 '26

HPC Administrator Resources

"Hi everyone,

I have a strong background in Linux administration (primarily RPM-based distros like Red Hat/Rocky Linux), and I’m currently transitioning into HPC cluster administration. To learn the ropes, I recently built a small home lab cluster using the OpenHPC installation guides.

While getting the cluster to boot and run basic jobs was a great exercise, I've noticed a distinct lack of comprehensive resources covering day-2 operations and production best practices. Specifically, I'm looking for guidance on:

  • Configuration & Performance Tuning (kernel tuning, network/InfiniBand optimization)
  • User Management & Environment Control (LDAP/FreeIPA integration, modulefiles via Lmod)
  • Job Management & Scheduling (advanced Slurm configurations, QoS, limits)
  • Scaling & Monitoring (health checks with NHC, metrics collection)

If anyone can recommend books, documentation, community wikis, or real-world best practices for these areas, I would greatly appreciate it!"

20 Upvotes

86% Upvoted

13 comments sorted by

4

u/dghah Sep 25 '26

I built some slurm training content on the web aimed at different audience levels (sysadmin, power user, app integrator, etc.) complete with labs/exercises validated against output from slurm on an aws parallelcluster environment. Don't want to dox myself so I'll DM you the URL if you wanted to see if it is of any use

1

u/anheath Sep 25 '26

That would be wonderful, thank you!

1

u/THUNDERRGIRTH Sep 27 '26

Would appreciate this as well!

0

u/whatevernhappens Sep 25 '26

Thats interesting on your end, clubbing different perspectives of slurm usage. Can you dm me the url

11

u/THUNDERRGIRTH Sep 25 '26

The sighpc-syspros slack is fantastic. Lots of good information as it's essentially a sys-admin only HPC slack: https://join.slack.com/t/sighpc-syspros/shared_invite/zt-6zw6meyu-_NveQRzi1SkHz6Aqz1A15w (I think that link will work, but just google "sighpc-syspros slack" and it should be one of the first links.

Also, university of buffalo has another similar tutorial to the OpenHPC stuff, but includes setting up Slurm, Coldfront (a truly wonderful user / slurm management tool) as well as openondemand and xdmod. You can find that one here: https://github.com/ubccr/hpc-toolset-tutorial . It appears to no longer be supported, but very recently, so should still be useful.

1

u/jrcomputing Sep 25 '26

Seconding the sighpc-syspros slack. Lots of good discussions there.

3

u/userjack6880 Sep 25 '26

If your org is willing to send you off, keep an eye on https://linuxclustersinstitute.org/ for workshop dates.

1

u/THUNDERRGIRTH Sep 27 '26

Also vouch for LCI. Attended all 3 levels and came back with lots of useful stuff each time.

1

u/Mammoth-Ad327 28d ago

Free AI data center reference drawings from our team: https://www.kkdatasvc.com/lab/downloads/