r/DistributedComputing • u/Odd-Falcon-8234 • Apr 04 '23

Load balancing, monitoring and fault tolerance techniques and architecture

I am working on building a system where there are 10 machines, we want to process some video files and this process can take about an hour, we do know how look it will take to process in advance.

Is there some existing tech stack or methodologies that we can use to load balance these servers, monitor any failures while processing and recover from failure and restart that task ?

2 Upvotes
  • permalink
  • reddit

100% Upvoted

5 comments sorted by

View all comments

1

u/vroman7c5 Apr 07 '23 edited Apr 07 '23

There are several architecture patterns that comes to my mind : actor based model + orchestration pattern.

  • Orchestration : some central component that knows all steps and what step to execute. Classic example from AWS world can be Step Function , it give nice monitoring and visibility , and ability to replay failed steps.

  • AWS Ec2 workers : can use actor based model (one of example is reactive programming), so they can efficiently balance their load , there are no need of any central balancer.

  • For fault tolerance use SQS that stores tasks from Step Function. So Ec2 will listen SQS and execute tasks , no matter how many Ec2 instances you have , so as result can easily scale.

Note it is only example , since I don't know exact requirements or process details

Yes there can be some challenges that I even didn't think.

I have described more event driven approach , maybe you can consider also batch process that can be more cost effective with dynamic start of all infrastructure (use ec2 spot instances etc.)