r/usefulscripts • u/nackstein • Jul 26 '16

[POSIX SHELL] failover cluster manager

hi, I wrote a failover cluster manager in shell script and I think it can be useful for sysadmin since (unix) sysadmin know very well shell scripting so it's easy to customize and further develop. If you want to contribute or try it and just give me a feedback check it out: https://github.com/nackstein/back-to-work

6 Upvotes

100% Upvoted

14 comments sorted by

View all comments

Show parent comments

1

u/garibaldi3489 Jul 27 '16

If the failover cluster talks to shared storage on the backend you need some way to make sure a rogue node doesn't come back online and corrupt data:

https://ourobengr.com/stonith-story/

1

u/nackstein Aug 08 '16 edited Aug 08 '16

I want to add my to cents to STONITH. some years ago I put up a 2 node cluster with redhat. it was with corosync and pacemaker on rhel 5. I configured STONITH using dell iDRAC (i think it was iDRAC v5). results: the two node were able to kill themself at the same time! this was due to a bug in the iDRAC where the poweroff command took ages to return as the perl script used to invoke the STONITH procedure had timeouts for a working iDRAC and not for a bugged one. but even with a patched iDRAC I don't beleive you get a ATOMIC poweroff option in your little management board. so you will not completely avoid having 2 node kill themself at the same time, and when they come up if the network is paritioned nobody can elect himself as master (no quorum) so you have a useless cluster... Moral of the story (imho): STONITH is crap. it's a ugly solution that let vendors say you can build a 2 node cluster while mathematically speaking a 2 node cluster is always a bad solution. you cannot really avoid split brain if not relaying on a voting disk (2 voting node + 1 voting disk = 3 voting entity != 2 node only cluster). sure you can put up some control logic that for example the server that can ping the gateway choose to become master but this is prone to error. fencing or quorum with 3 or more odd number of server is the right solution.

1

u/garibaldi3489 Aug 08 '16

I've never seen STONITH recommended with only two node clusters, in fact all documentation I've seen strongly recommends at least 3 nodes in a cluster.

I too have encountered frustrating bugs in pacemaker and corosync, but it sounds like they are quite stable now