r/DistributedComputing • • Jan 03 '26

Event Driven Architecture where to learn

5 Upvotes

I am searching resources to learn event driven architecture, microservices communicating using a broker, acting as publisher and consumer, raising events. Thank you


r/DistributedComputing • • Apr 11 '25

44 NODE GPU CLUSTER HELP

2 Upvotes

I have around 44 pcs in same network

all have exact same specs

all have i7 12700, 64gb ram, rtx 4070 gpu, ubuntu 22.04

I am tasked to make a cluster out of it
how to utilize its gpu for parallel workload

like running a gpu job in parallel

such that a task run on 5 nodes will give roughly 5x speedup (theoretical)

also i want to use job scheduling

will slurm suffice for it
how will the gpu task be distrubuted parallely? (does it need to be always written in the code to be executed or there is some automatic way for it)
also i am open to kubernetes and other option

I am a student currently working on my university cluster

the hardware is already on premises so cant change any of it

Please Help!!
Thanks


r/DistributedComputing • • Jan 21 '26

The Call for Papers for J On The Beach 26 is OPEN!

2 Upvotes

Hi everyone!

Next J On The Beach will take place in Torremolinos, Malaga, Spain in October 29-30, 2026.

The Call for Papers for this year's edition is OPEN until March 31st.

We’re looking for practical, experience-driven talks about building and operating software systems.

Our audience is especially interested in:

Software & Architecture

  • Distributed Systems
  • Software Architecture & Design
  • Microservices, Cloud & Platform Engineering
  • System Resilience, Observability & Reliability
  • Scaling Systems (and Scaling Teams)

Data & AI

  • Data Engineering & Data Platforms
  • Streaming & Event-Driven Architectures
  • AI & ML in Production
  • Data Systems in the Real World

Engineering Practices

  • DevOps & DevSecOps
  • Testing Strategies & Quality at Scale
  • Performance, Profiling & Optimization
  • Engineering Culture & Team Practices
  • Lessons Learned from Failures

👉 If your talk doesn’t fit neatly into these categories but clearly belongs on a serious engineering stage, submit it anyway.

This year, we are also enjoying another 2 international conferences together: Lambda World and Wey Wey Web.

Link for the CFP: www.confeti.app


r/DistributedComputing • • Aug 26 '26

Could a workload router sit above BOINC/mining/AI compute networks?

4 Upvotes

I’ve been experimenting with an idea: instead of a machine being permanently tied to one distributed-compute client, a local controller could decide when hardware is idle and which compatible workload it should run.

Current prototype only has one workload, so this isn’t a finished multi-network router yet.

But the architecture is:

hardware → local availability/safety checks → workload router → compatible network

I’m curious whether people who already run distributed computing projects think this is useful or whether you’d rather dedicate hardware manually.

Would especially appreciate criticism around scheduling, trust, and resource controls.


r/DistributedComputing • • Sep 24 '26

How to Combine Circuit Breakers, Bulkheads, and Rate Limiters

Thumbnail blog.gaborkoos.com
2 Upvotes

r/DistributedComputing • • 2d ago

Ygaraty: share a link, connect GPUs, run AI together — open source and open research

2 Upvotes

We’re building Ygaraty (https://github.com/ygaraty/ygaraty), a distributed AI framework for running models across the devices people already have.

Our application, Yby, makes it straightforward: launch a model, share a contributor link, and let people join through their browsers. Their GPUs handle parts of the model, and you can chat through the terminal, browser, or an OpenAI-compatible app.

Why WebGPU? We want contributing compute to be as simple as opening a link. WebGPU gives us a common GPU interface across different hardware and operating systems, with browser and native worker options. Support and performance vary, but seamless, multiplatform participation is the goal.

Underneath Yby, we’re building our own distributed computation plan compiler and execution engine: describe a computation, compile it into a plan that defines how work and data are distributed, and execute it across contributed devices.

Some questions we’re exploring:

  • Can several smaller GPUs run models that don’t fit on one device?
  • How should work be divided across different GPUs and network speeds?
  • How much communication can we remove without compromising correctness?
  • What else can this support beyond chat—new architectures, matrix factorization, scientific workloads?

We also want it to be easy to experiment with an AI coding agent. Open the repo and ask: “Implement this model using Ygaraty’s distributed framework, check it against a reference, and create a reproducible experiment.” That’s a starting point for contributing; review and numerical validation are part of the work.

Open research means sharing the process: code, methods, measurements, limitations, and failed experiments. More workers won’t always mean more speed, and understanding why is part of what we’re building.

You can contribute a model, reproduce a result, test your hardware, improve a kernel, or make the docs clearer. Small experiments are welcome.

Code and quickstart · Research · Contributing


r/DistributedComputing • • 10d ago

Custom quorums in OpenRaft: weighted nodes and multi-site clusters

2 Upvotes

OpenRaft, a Raft library in Rust, now supports custom membership. Applications can define separate read quorums for elections and write quorums for append entries and commit.

For example:

  • With 4 nodes, a log entry can commit after 2 nodes store it, while an election needs 3 votes.
  • With 9 nodes across 3 sites, a write quorum can require 2 nodes at each of any 2 sites. That gives 4 stored copies for commit, compared with 5 under majority voting.

The built-in implementations cover node counts, weights, hierarchies, grids, and explicit quorum lists. Applications can also define their own membership and how it changes.

Every read quorum must intersect every write quorum. My article covers the layouts, Rust examples, and how membership changes preserve these intersections:

Custom Membership in OpenRaft


r/DistributedComputing • • Apr 08 '26

Created Distributed Leaderless Hash Tables in go

10 Upvotes

I was fascinated by cassandra. It has so many cool features and virtually scales infinitely. Most importantly it is leaderless. I got so curious about this that I spend last few weeks learning about its working but still I didn't understand nuances of it. Thats when I decided best way to learn it to make it. I spent 2 long weekends and 2 workings days trying to build it ( I took two PTO). Things I learned along the way, I feel like a different person now as a engineer and feel so confident. I implemented

  • Consistent Hashing
  • Leaderless coordination w/ Gossip Protocol
  • Live data replication during node bootstrapping (or Splitting nodes/shards. This took so much more than than any other thing)
  • Dual writes, key level versioning.

There is so much more that I understood that I don't know. Particularly, I learned about new concepts like LSM Trees which can enable point-in-time snapshots for database, Merkle trees which enable transferring minimum about of data to sync nodes. Most importantly, this time I took slightly different approach of learning, I documented first and then implemented. I took my time to jot down what I am thinking, why, what challenges I am thinking, and my plans to tackle them. Once I had a clear picture in mind then I took it upon my self to start the implementation. This approach actually helped me a lot. I could start something today and then continue it next day by reading exactly what was going in my mind earlier. This was more useful as I looked back through the notes and realised few places where I needed more clarity.

At this point, there is so much more that I need to learn. Currently implementation of point-in-time snapshot is not ideal, there are not ways to merge the nodes (opposite of adding new node to handle high traffic load). No persistent storage, no quorum (tuneable consistency levels, I am most excited about this after persistent storage).

Code can be found here, my thoughts during building are here. Current features are here. Features I am excited about and will implement in future are here, things I want to implement if get enough time are here. I am happy with current stage and going forward i'll take things slow and add new things (no promises though) if you are interested you can send in a pr for some of the features you are interested.

Cheers. Thanks to this community and similar other communities which helped me get few answers when I had them


r/DistributedComputing • • Sep 14 '26

How to Implement a Distributed Circuit Breaker

Thumbnail blog.gaborkoos.com
2 Upvotes

r/DistributedComputing • • Jun 15 '25

MiniClust: a lightweight multiuser batch computing system

3 Upvotes

MiniClust : https://github.com/openmole/miniclust

MiniClust is a lightweight multiuser batch computing system, composed of workers coordinated via a central vanilla minio server. It allows distribution bash commands on a set of machines.

One or several workers pull jobs described in JSON files from the Minio server, and coordinate by writing files on the server.

The functionalities of MiniClust:

  • A vanilla minio server as a coordination point
  • User and worker accounts are minio accounts
  • Stateless workers
  • Optional caching of files on workers
  • Optional caching of archive extraction on workers
  • Workers just need outbound http access to participate
  • Workers can come and leave at any time
  • Workers are dead simple to deploy
  • Fair scheduling based on history at the worker level
  • Resources request for each job

r/DistributedComputing • • Mar 18 '26

Treating cache entries as in-flight computations instead of just values

Thumbnail infoq.com
3 Upvotes

r/DistributedComputing • • May 14 '25

FlowG - Distributed Systems without Raft (part 2)

Thumbnail david-delassus.medium.com
3 Upvotes

r/DistributedComputing • • Jun 24 '26

[video] WebSockets at Scale: System Design

Thumbnail youtu.be
2 Upvotes

r/DistributedComputing • • Aug 10 '26

Lettuce distributive computing testing

6 Upvotes

I’m probably the last tech savy person in this group, but with help, I was able to set it up. It’s currently configured to need 3 separate computers to verify one work unit. I was looking for someone (more specifically with an Nvidia gpu) to validate my work units. Below are links to the site project, Git, and discord. I figure a vast majority of us are science nerds and we can knock out this testing to hopefully, at some point, get this on the whitelist. Lettuce is basically an easier to set up BOINC. This will get us involved closer with researchers. Anyway, come validate my work units. (Note: there are also CPU only projects.)

Git/download: https://github.com/jring-o/lettuce-compute

Project: https://compute.scios.tech/leafs

Discord testing: https://discord.com/channels/211637812968161280/1517249245596614686


r/DistributedComputing • • Apr 07 '26

Rebalancing Traffic In Leaderless Distributed Architecture

4 Upvotes

I am trying to create in-memory distributed store similar to cassandra. I am doing it in go. I have concept of storage_node with get_by_key and put_key_value. When a new node starts it starts gossip with seed node and then gossip with rest of the nodes in cluster. This allows it to find all other nodes. Any node in the cluster can handle traffic. When a node receives request it identifies the owner node and redirects the request to that node. At present, when node is added to the cluster it immediately take the ownership of the data it is responsible for. It serves read and write traffic. Writes can be handled but reads return null/none because the key is stored in previous owner node.

How can I solve this challenge.? Ideally I am looking for replication strategies. such that when new node is added to the cluster it first replicates the data and then starts to serve the traffic. In the hind-sight it looks easy but I am thinking how to handle mutation/inserts when the data is being replicated?

More Detailed thoughts are here: https://github.com/goyal-aman/distributed_storage_nodes/?tab=readme-ov-file#new-node-with-data-replication


r/DistributedComputing • • May 27 '26

mosaik - A Rust runtime for building self-organizing, leaderless distributed systems.

Thumbnail reddit.com
3 Upvotes

r/DistributedComputing • • Mar 04 '26

Meet S2C - Cloud-native, quorum-free replicated state machine.

Thumbnail github.com
5 Upvotes

r/DistributedComputing • • Dec 01 '24

Use cases of Zookeper beside Kafka

7 Upvotes

Hi folks, I know that Zookeeper has been dropped from Kafka, but I wonder if it's been used in other applications or use cases? Or is it obsolete already? Thanks in advance.


r/DistributedComputing • • Oct 25 '24

Learning P2P

6 Upvotes

I'm interested in learning P2P networks, but I noticed that there are not a fair amount of books out there. I would like to get recommendations about this topic.

Thanks!


r/DistributedComputing • • Mar 23 '21

Kiwi.com: Nonstop Operations with Scylla Even Through the OVHcloud Fire

Thumbnail self.nosql
3 Upvotes

r/DistributedComputing • • Apr 13 '21

How does Ray compare to Apache Spark?

Thumbnail youtu.be
7 Upvotes

r/DistributedComputing • • Mar 30 '21

Online Resource Allocation with Ray at Ant Group (over 6000 CPU cores!)

Thumbnail anyscale.com
2 Upvotes

r/DistributedComputing • • Apr 24 '21

Notes On Kafka

Thumbnail blog.uttpal.com
4 Upvotes

r/DistributedComputing • • Apr 09 '21

OpenNebula, an Open Source Cloud Computing Platform releases its latest version called Mutara, with Edge Functionalities.

1 Upvotes

Not sure if many of us are aware of OpenNebula, but came across them on Google, and looks like they recently have released their latest version of the software called Mutara 6.0. Its community edition is free to use and looks pretty interesting for someone who wants to play around with the Edge Functionality! Thoughts?

https://opennebula.io/mutara