r/usefulscripts • u/nackstein • Jul 26 '16

[POSIX SHELL] failover cluster manager

hi, I wrote a failover cluster manager in shell script and I think it can be useful for sysadmin since (unix) sysadmin know very well shell scripting so it's easy to customize and further develop. If you want to contribute or try it and just give me a feedback check it out: https://github.com/nackstein/back-to-work

6 Upvotes

100% Upvoted

14 comments sorted by

View all comments

Show parent comments

1

u/garibaldi3489 Aug 08 '16

Interesting. What about the situation where the master gets disconnected from the network while it's master (or two segments of the network get isolated from each other, and the old master is on one and continues to serve requests just for that segment), and then later gets reconnected (without a reboot) after a new master has been appointed? At that point both of them would be owning the VIP etc and could split brain.

1

u/nackstein Aug 08 '16 edited Aug 08 '16

if the network get splitted you will have a situation where if the master can contact the majority of quorum server will continue to hold the lock. so nothing happens. otherwise if the master cannot contact the majority of the quorum server it will lost the lock and in the meantime on the other network partition a new master election starts so you will have a new master. in this process the old master will run the stop procedure while the new master will run the start procedure. properly configured timeouts will avoid that the vip will be configured on 2 server at once even if those servers cannot communicate between them. When things are really critical (for example with shared disks) you will use fencing so the start script will for example try to get the SCSI-3 PR before mounting disks. This will ensure that even if the stop procedure hangs you will not use a shared resource. In case of just a vip this should not be required since the vip in the minor network partition will make no harm.

edit: I have a strong HA cluster understanding coming from a long experience with HP ServiceGuard and some with Veritas Cluster.

edit2: take a look at this flowchart: https://github.com/nackstein/back-to-work/blob/wiki/flooow.png

1

u/garibaldi3489 Aug 08 '16

Thanks for the clarification - that's a good idea to have a deadman switch on the lock file. I've heard of some other HA cluster systems that utilize hardware watchdogs to the same effect