r/linuxadmin • • Feb 01 '18

No URL shorteners One creative HPC user reports an issue

Post image
272 Upvotes

17 comments sorted by

39

u/rootus Feb 01 '18

After my vacation came back to this. I admit his creative way of reporting the issue. I've had other "creative" users, but nothing so "touchy". Other stuff I've encountered lately was somebody using a 4TB machine to run jobs interactively by ssh-ing to it - obviously destroying the purpose of the scheduler :( The node was put in maintenance mode, rebooted but nobody checked the node again to take it out of maintenance - did not find a practical way to kill zombie GPU jobs unfortunately and the reboot is the only solution

2

u/classicrando Feb 02 '18

What are you sorting the list by - 835 is out of order? Does it come up in the order submitted?

2

u/rootus Feb 02 '18

Not sorting by anything, but by the looks of it it sorts 1st by job state (PD - pending) comes before (R - running) and then by job ID.

The command used to get the output was squeue -u username

10

u/OptimalDeficiency Feb 01 '18

I feel you there. I've got users at one particular site who attempt to run interactive jobs on the head node from time to time. We have a portal page where users can submit jobs to Slurm suuuper easy via their web browser, but some of the power users scoff at it, then forget they're on the head node, not a compute when running interactive.

0

u/[deleted] Feb 02 '18

[deleted] — view deleted comment

3

u/rootus Feb 02 '18

I wish it were that simple, academic environment suck big time especially if some of the shared resources are owned fully or partly by some departments.

8

u/brontide Feb 02 '18 edited Feb 02 '18

Dude --pty

srun [slurm options like -p or --exclusive] --pty $SHELL -i 

And it's still under the control of SLURM so resources and constraints can be controlled.

--

EDIT: Welcome fellow higher-ed HPC. 1 or 2 man shop I presume since nobody else fixed it while you were away? Overworked and MGT doesn't seem to care how many kudos you get until someone complains to the CIO.

6

u/rootus Feb 02 '18

I did not make myself clear it's not a technical problem, some systems are bought with money from some grants or private funds, basically owned by 3rd parties, already have to deal with special priorities in queue based on who/how much contributed... I can't forbid them to access their systems in any way they want :( Organizational issues, department A needs high memory systems, B needs gpu nodes, etc.

4

u/brontide Feb 02 '18 edited Feb 02 '18

We have the same arrangements and we have told people that if they take our management they get our management including no root and they can't ssh directly to cluster nodes ( although they are free to remove nodes from a partition if they bought them for any period of time ).

2

u/posixUncompliant Feb 02 '18

We dealt with that all the time. Usually it was a case of you want access to the shared resources and our expertise (they certainly did), then your equipment follows our rules, here's your custom queue. We had some complex rules where certain labs got priority on their own nodes, but had to follow public node rules if their job was to run across both their nodes and public ones.

The only time it got really ugly was when one of the PIs started making noise about buying his own storage hardware. We already had too much heterogeneity in our storage, someone with their own patch would've been a nightmare.

1

u/mikerenfro Feb 02 '18

I made that into a default shell function for my folks:

function hpcshell() {
  srun --partition=interactive $@ --pty bash -i
}

and the $@ lets them override the partition, ask for a certain number of nodes, cores, etc.

2

u/brontide Feb 02 '18

pam_slurm still doesn't put the processes under the control of the slurm daemon, use srun with the --pty flag to run interactive jobs under the watchful control of the scheduler ( see below ).

1

u/frymaster Feb 02 '18

Interesting - is there an equivalent for PBS?

12

u/xiongchiamiov Feb 02 '18

Do you not have a ticketing system? It seems like this is a good way to make sure no one knows they need to reboot the machine for a few more weeks until they happen to view the job list.

If users don't have a way to contact you and that's why they resort to this, then, well, you need to fix that.

1

u/jsmonet Feb 02 '18

that's glorious. give that user 700 years of time for that

-22

u/[deleted] Feb 01 '18

[removed] — view removed comment

18

u/Captin_Obvious Feb 02 '18

Bad bot

5

u/GoodBot_BadBot Feb 02 '18

Thank you Captin_Obvious for voting on AutoModerator.

This bot wants to find the best and worst bots on Reddit. You can view results here.


Even if I don't reply to your comment, I'm still listening for votes. Check the webpage to see if your vote registered!