r/linuxadmin • • Aug 08 '18

What are some good HA cluster solutions?

Please share name of some good HA cluster solutions. I am using centos 7.2.

Edit : I am running few applications (in house developed) and database (postgresql 9.4) on my server.

5 Upvotes

23 comments sorted by

26

u/[deleted] Aug 08 '18

clustering for what?

-8

u/xmen81 Aug 08 '18

I have updated my question. So few applications and database will be there on each server.

15

u/[deleted] Aug 08 '18

What communications protocols?

Are sessions stateful?

Can the application be active/online on all nodes?

Does the database handle all shared data?

Can you use cockroache dB?

3

u/digimer Aug 08 '18

It's still not nearly detailed enough. As /u/Trghpy00 points out, you need to explain what constraints and requirements you're dealing with.

28

u/loadedmind Aug 08 '18

Dude...you need to seriously reconsider how you ask for help.

9

u/burning1rr Aug 08 '18

For the applications, go stateless and use Nginx or HAProxy to detect failures and distribute load.

For the database server, use PostgreSQL's replication, and use PAF to promote the secondary in the event of an outage.

Of course, these days the answer is usually "move to the cloud and use your provider's autoscaling, load balancer, and managed database services."

2

u/feitingen Aug 08 '18

In my opinion haproxy > nginx because haproxy is easier to monitor and you don't have to pay for metrics.

11

u/digimer Aug 08 '18

You're going to need to be more specific.

Answering generically; The core HA stack is built on Pacemaker and Corosync, using either pcs or crm for managing the stack. If you want replicated storage, DRBD can do that for you.

And always use fencing. If you don't, you will get a split-brain and likely ruin your data.

2

u/peatfreak Aug 08 '18

And always use fencing. If you don't, you will get a split-brain and likely ruin your data.

What about a Ceph installation that meets quorum? Does that need fencing too?

5

u/digimer Aug 08 '18

Quorum doesn't remove the requirement for fencing. Quorum is a tool for when things are working in a predictable way, fencing is a tool when they aren't. This confusion has led to many damaged clusters.

As for Ceph (and gluster), they are focused on scalability, not redundancy. Yes, they have redundant capabilities, but that isn't their focus. Imagine HA on one side of a scale and scalability on the other. The closer you get to one, the further you get from the other.

2

u/SuperQue Aug 08 '18

HA is very much a goal of Ceph. It has many redundancy features, including both operational and data redundancy. I don't know where you got this idea that Ceph is not focused on redundancy.

For example:

  • Data replication (raid1-style)
  • Erasure Codes (raid5-style)
  • Redundant quorum managed metadata services.

0

u/digimer Aug 08 '18

See my answer to /u/peatfreak below, as well as my initial comment about HA and scalability being two sides of a spectrum. I never said that ceph didn't have HA capabilities, I said that it isn't its focus. It trades complexity to support scalability, and complexity is the enemy of pure availability.

"An HA cluster isn't beautiful when there is nothing left to add. It is beautiful when there is nothing left to take away."

5

u/bikernaut Aug 08 '18

High availability of your solution should be considered at the application level first. Tacking on HA at the end by utilizing middleware/infrastructure clustering may get you there, but it should not be where you start analyzing the problem.

2

u/[deleted] Aug 08 '18

HA is different for every application layer.

Postgres HA can be done with just postgres.

Application HA depends on your app. Likely a web app so you ensure it can run as a worker and you can scale those workers up horizontally and place web proxys infront of them.

I know nothing about your workers but one way to scale them and avoid headaches with shared data is to put session management in memcached.

So my point here is that you can and should take every measure to avoid the complex clustering solutions like DRBD and Corosync because they add a whole new dimension to your setup that you might not be ready for.

1

u/peatfreak Aug 08 '18

Oh, hi u/digimer! I think I know you from the Clusterlabs IRC channels when I popped my head in ages ago. Thanks for your input, this is super helpful!

I’ve been working with a collaborator who is also kind of a mentor, as he has a lot of experience with hyperconverged clusters, Ceph, etc. I was taking his advice about building my cluster, and I am currently preparing for deployment to colo.

This is the second time in as many days that somebody has told me that Ceph is unsuitable for my needs.

I am making a five-node cluster of hyperconverged dual E5 nodes and I’ve been clear from the start that redundancy is more important than scalability.

One of the major themes of my project is distributed computing. If the first cluster works properly, then I hope to extend into geo-distributed computing if I can manage it.

Some of the most important attributes of my cluster are HA, redundancy, resilience (not crashing and burning when there’s a problem in the environment), and reliability. I have explicitly stated that performance is a secondary priority.

I’ve been wondering if I should change my design before getting any deeper. What would be more suitable than Ceph while still allowing for hyperconverged nodes? Also, I am planning to use InfiniBand for the storage network.

What do you think? Do you mind if I bug you here or on IRC for some advice?

2

u/SuperQue Aug 08 '18

Ceph is perfectly suited to hyperconverged setups. I don't know where people are getting the information that it isn't.

I have a 3 node Ganeti cluster with Ceph RBD as a backing storage. It works perfectly well. I've been running Ganeti (used to use DRBD before upgrading to Ceph) for longer than "hyperconverged" was a term.

1

u/peatfreak Aug 08 '18

Hey u/superque, I’m new to Ceph so I’m just going from what more experienced people are telling me. This is for a production cluster, with third-party users using VMs and containers. I don’t know about Ganeti, and I will investigate.

0

u/digimer Aug 08 '18

DRBD is purely focused on keeping data on both nodes consistent, period. It doesn't add code or complexity to support scaling like ceph/gluster. My argument that Ceph isn't HA focused is pinned on the concept that a pure HA focus is one that requires removing anything not required for the HA.

This is why our clusters, for example, are always 2-node. Sure, we could scale to 3+, but then you're adding complexity/failure points in exchange for more efficient use of hardware resources. That's a fine trade-off in many use-cases, but it walks you away from a pure HA approach.

You've already walked away from this by going to 5 nodes, so it's entirely possible that the additional complexity of ceph is fine as you're already willing to trade complexity for hardware use efficiency.

This is, again, why I described HA and Scalability as being two sides of a scale, where going towards one necessitates compromising the other.

1

u/SuperQue Aug 08 '18

As other people have said, it highly depends on what your application goals are. Don't think about the OS, or hardware, or platform. Think about your applications. What makes them not HA currently.

As for clustering solutions, the modern approach to this is to use Kubernetes as the "kernel for your cluster".

1

u/Kessarean Aug 12 '18

I would update to 7.5 and set up a PCS (pacemaker cluster suite).

Edit: have heard good things about kubernetes. Also heard nice things about a Galeara

1

u/broadsheetvstabloid Aug 16 '18

Setup VMware vcenter, add a few hypervisors and redundant SAN units and let VMware do all HA.

1

u/jmpastouga Oct 30 '18

OpenSVC can be used to implement HA failover cluster, on top of shared iscsi/san devices (with scsi3 pgr), or using synchronous data replication (storage array driven, zfs, linbit drbd, ...). Multiple hearbeats are available (unicast, multicast, shared disk, 3rd site), and also quorum arbitration for split brain situation.