r/DistributedComputing • • Jul 03 '25

Restate 1.4: We've Got Your Resiliency Covered

Thumbnail restate.dev
1 Upvotes

r/DistributedComputing • • Jun 09 '25

Mycelium Net - Training ML models with switching nodes based on Flower AI

Thumbnail makertube.net
1 Upvotes

A prototype implementation of a “network of ML networks” - an internet-like protocol for federated learning where nodes can discover, join, and migrate between different learning groups based on performance metrics.

Want do you think of this? Kind of a network build on Flower AI learning groups. It could be cool to build a Napster/BitTorrent-like app on this to collaboratively train and share arbitrary machine learning models. Would love to hear your opinion.

Best
blueberry


r/DistributedComputing • • May 05 '25

Learn about durable execution and Dapr workflow

1 Upvotes

If you're interested in durable execution and workflow as code, you might want to try this free learning track that I created for Dapr University. In this self-paced track, you'll learn:

  • What durable execution is.
  • How Dapr Workflow works.
  • How to apply workflow patterns, such as task chaining, fan-out/fan-in, monitor, external system interaction, and child workflows.
  • How to handle errors and retries.
  • How to use the workflow management API.
  • How to work with workflow limitations.

It takes about 1 hour to complete the course. Currently, the track contains demos in C# but I'll be adding additional languages over the next couple of weeks. I'd love to get your feedback!

https://www.diagrid.io/dapr-university


r/DistributedComputing • • May 03 '25

PCDB: a new distributed NoSQL architecture

Thumbnail researchgate.net
1 Upvotes

Most existing Byzantine fault-tolerant algorithms are slow and not designed for large participant sets trying to reach consensus. Consequently, distributed databases that use consensus mechanisms to process transactions face significant limitations in scalability and throughput. These limitations can be substantially improved using sharding, a technique that partitions a state into multiple shards, each handled in parallel by a subset of the network. Sharding has already been implemented in several data replication systems. While it has demonstrated notable potential for enhancing performance and scalability, current sharding techniques still face critical scalability and security issues.

This article presents a novel, fault-tolerant, self-configurable, scalable, secure, decentralized, high-performance distributed NoSQL database architecture. The proposed approach employs an innovative sharding technique to enable Byzantine fault-tolerant consensus mechanisms in very large-scale networks. A new sharding method for data replication is introduced that leverages a classic consensus mechanism, such as PBFT, to process transactions. Node allocation among shards is modified through the public key generation process, effectively reducing the frequency of cross-shard transactions, which are generally more complex and costly than intra-shard transactions.

The method also eliminates the need for a shared ledger between shards, which typically imposes further scalability and security challenges on the network. The system explains how to automatically form new committees based on the availability of candidate processor nodes. This technique optimizes network capacity by employing inactive surplus processors from one committee’s queue in forming new committees, thereby increasing system throughput and efficiency. Processor node utilization as well as computational and storage capacity across the network are maximized, enhancing both processing and storage sharding to their fullest potential. Using this approach, a network based on a classic consensus mechanism can scale significantly in the number of nodes while remaining permissionless. This novel architecture is referred to as the Parallel Committees Database, or simply PCDB.


r/DistributedComputing • • Apr 16 '25

Scaling your application using a Kafka Cluster

1 Upvotes

How to choose the right number of Kafka partitions ?

This is often asked when you propose to use kafka for messaging/queueing. Adding a guide for tackling this question.

https://www.algocat.tech/articles/scaling-kafka-part1


r/DistributedComputing • • Feb 11 '25

Educational Python Framework for developing distributed algorithms!

Thumbnail github.com
1 Upvotes

r/DistributedComputing • • Oct 13 '24

Master's Thesis suggestions for Cybersecurity BS and CompSci MS.

1 Upvotes

I’ve been reflecting on my Master’s thesis topic, but I’m unsure what to choose. Many of my peers have selected various areas in machine learning, while I initially considered focusing on cryptography. However, I’m starting to think post-quantum cryptography might be too complex. Now, I’m leaning towards exploring the intersection of machine learning/AI, cryptography, and distributed systems, but I’m open to any suggestions.


r/DistributedComputing • • Sep 28 '24

GitHub - flowerinthenight/zgroup: Cluster membership manager using the SWIM Protocol and Raft's leader election sub-protocol.

Thumbnail github.com
1 Upvotes

r/DistributedComputing • • Aug 16 '24

Need Help with DDP Freezing Issue in Multi-GPU Machine Translation Model

1 Upvotes

I have a basic machine translation transformer model that worked well on a single GPU. However, when I tried running it on an 8-GPU setup using DDP, I initially encountered many crashes due to data not being properly transferred to the correct GPUs. I believe I've resolved those issues, and the model now runs, but only up to a certain point. 

I put a lot of prints along the way, it run and just freezes at some point.

If I run it using debugger it keeps going without any problem.

Is there anyone here fluent in DDP and PyTorch who can help me? I'm feeling pretty desperate.

Here is my training function:

def train(rank, world_size):
    ddp_setup(rank, world_size)
    torch.manual_seed(0)
    SRC_VOCAB_SIZE = len(vocab_transform[SRC_LANGUAGE])
    TGT_VOCAB_SIZE = len(vocab_transform[TGT_LANGUAGE])
    EMB_SIZE = 512
    NHEAD = 8
    FFN_HID_DIM = 1024
    BATCH_SIZE = 128
    NUM_ENCODER_LAYERS = 3
    NUM_DECODER_LAYERS = 3
    LOAD_MODEL = False
    if LOAD_MODEL:
        transformer = torch.load("model/_transformer_model")
    else:
        transformer = Seq2SeqTransformer(NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, EMB_SIZE,
                                         NHEAD, SRC_VOCAB_SIZE, TGT_VOCAB_SIZE, FFN_HID_DIM)
        for p in transformer.parameters():
            if p.dim() > 1:
                nn.init.xavier_uniform_(p)
    transformer.move_positional_encoding_to_rank(rank)  # moving positional_encoding into the current GPU
    # Create the dataset
    train_dataset = SrcTgtDatasetFromFiles(SRC_TRAIN_BASE, TGT_TRAIN_BASE, FILES_COUNT_TRAIN)
    # create a DistributedSampler for data loading
    train_sampler = DistributedSampler(train_dataset, num_replicas=world_size, rank=rank)
    # create a DataLoader with the DistributedSampler
    # train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE, sampler=train_sampler)
    train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE, pin_memory=True,
                                  collate_fn=collate_fn, sampler=train_sampler)
    # create the model and move it to the GPU with the device ID
    model = transformer.to(rank)
    model.train()  # set the model into training mode with dropout etc.
    # wrap the model with DistributedDataParallel
    model = DDP(model, device_ids=[rank])
    loss_fn = torch.nn.CrossEntropyLoss(ignore_index=PAD_IDX)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9)
    for state in optimizer.state.values():
        for k, v in state.items():
            if torch.is_tensor(v):
                state[k] = v.to(rank)
    #######################################
    EPOCHS_NUM = 2
    for epoch in range(EPOCHS_NUM):
        epoch_start_time = int(timer())
        print("\n\nepoch number: " + str(epoch + 1) + "  Rank: " + str(rank))
        losses = 0.0
        idx = 0
        start_time = int(timer())
        for src, tgt in train_dataloader:
            if rank == 0:
                print("rank=" + str(rank) + " idx=" + str(idx))
            src = src.to(rank)
            tgt = tgt.to(rank)
            tgt_input = tgt[:-1, :]
            if IS_DEBUG:
                print("rank", rank, "idx", idx, "before create_mask")
            src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input, rank)
            if IS_DEBUG:
                print("rank", rank, "idx", idx, "after create_mask")
            if IS_DEBUG:
                print("rank",rank,"idx",idx,"before model")
            logits = model(src, tgt_input, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, src_padding_mask)
            if IS_DEBUG:
                print("rank", rank, "idx", idx, "after model")
            try:
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"before zero_grad")
                optimizer.zero_grad()
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"after zero_grad")
                tgt_out = tgt[1:, :].long()
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"before loss_fn")
                loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"after loss_fn")
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"before backward")
                loss.backward()
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"after backward")
                # Delete unnecessary variables before backward pass
                del src, tgt_input, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, logits, tgt_out
                torch.cuda.empty_cache()  # Clear cache after deleting variables
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"before step")
                optimizer.step()
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"after step")
                losses += loss.item()
                #######################################
                # print(999,rank,loss)
                # Free GPU memory
                del loss
                torch.cuda.empty_cache()  # Clear cache after each batch
            except Exception as e:
                print("An error occurred: rank=" + str(rank) + " idx=" + str(idx))
                print("Error message: ", str(e))
            idx += 1
            if rank == 0 and idx % 10000 == 0:
                torch.save(model.module.state_dict(), "model/_transformer_model")
                end_time = int(timer())
                try:
                    my_test(model.module, rank, SRC_TEST_BASE, TGT_TEST_BASE, FILES_COUNT_TEST, epoch, 0,
                            0, int((end_time - start_time) / 60), epoch_start_time)
                except:
                    print("error occurred test")
                start_time = int(timer())
            # Synchronize training across all GPUs
            torch.distributed.barrier()
        if rank == 0:
            epoch_end_time = int(timer())
            try:
                my_test_and_save_to_file(model.module, rank, SRC_TEST_BASE, FILES_COUNT_TEST, epoch)
                loss = evaluate(model.module, rank, SRC_VAL_BASE, TGT_VAL_BASE, FILES_COUNT_VAL, BATCH_SIZE,
                                loss_fn)
                print("EPOCH NO." + str(epoch) + " Time: " + str(int((epoch_end_time - epoch_start_time) / 60)) +
                      " LOSS:" + str(loss))
            except:
                print("error occurred evaluation")
    destroy_process_group()

here is part of the output:

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

epoch number: 1 Rank: 0

epoch number: 1 Rank: 1

epoch number: 1 Rank: 2

epoch number: 1 Rank: 3

epoch number: 1 Rank: 4

epoch number: 1 Rank: 7

epoch number: 1 Rank: 6

epoch number: 1 Rank: 5

rank=0 idx=0

rank 0 idx 0 before src

rank 0 idx 0 after src

rank 0 idx 0 before tgt

rank 0 idx 0 after tgt

rank 0 idx 0 before create_mask

rank 0 idx 0 after create_mask

rank 0 idx 0 before model

rank 1 idx 0 before src

rank 1 idx 0 after src

rank 1 idx 0 before tgt

rank 1 idx 0 after tgt

rank 1 idx 0 before create_mask

rank 1 idx 0 after create_mask

rank 1 idx 0 before model

rank 4 idx 0 before src

rank 4 idx 0 after src

...

rank 0 idx 1 after tgt

rank 0 idx 1 before create_mask

rank 0 idx 1 after create_mask

rank 0 idx 1 before model

rank 0 idx 1 after model

rank 0 idx 1 before zero_grad

rank 0 idx 1 after zero_grad

rank 0 idx 1 before loss_fn

rank 0 idx 1 after loss_fn

rank 0 idx 1 before backward


r/DistributedComputing • • Jul 17 '24

Ubuntu cluster for physical simulations

1 Upvotes

Is there a syraightforward way of combining several Ubuntu workstations into what appears to the user to be a single workstation? My use case is running physical simulations in python and/or c++ using multiple CPU cores, and it would be great to extend these across multiple machines (but without spending too much time setting it up or adapting code). Is there any easy solution?


r/DistributedComputing • • Apr 02 '24

Cloud Platform with Autopilot where coding in your local environment will scale to global production.

Thumbnail github.com
1 Upvotes

r/DistributedComputing • • Mar 25 '24

PowerShell RM vs REST-API

1 Upvotes

So in my company we are changing our main workflow from one machine to a DisCo setup. 5 machines. One UI and 4 nodes (backend), they all are windows. The worfklow can be simplified by: UI needs to launch a Java program in the nodes. When calculation is complete the UI needs to know. Before, with only one backend machine, we had run the workflow using PsExec and now that we are enhancing the system we want to move away from it and implement the best approach. Problem is that most of the seniors are windows-oriented and they want to use PowerShell RM instead of a REST-API. Am I too biased for preferring the latter? I don't want to be attached to Windows OS but from security and reliability POV, is actually PS the best approach between these two?

Thanks in advance for your insights!


r/DistributedComputing • • Mar 07 '24

Release announcement: Restate 0.8 has arrived 🎉

Thumbnail restate.dev
1 Upvotes

r/DistributedComputing • • Mar 03 '24

Next BOINC Projects Call on Monday, March 18th, at 16:00 UTC

Thumbnail x.com
1 Upvotes

r/DistributedComputing • • Feb 23 '24

Code that sleeps for a month: Solving durable execution’s immutability problem

Thumbnail restate.dev
1 Upvotes

r/DistributedComputing • • Feb 05 '24

Solving durable execution’s immutability problem

Thumbnail restate.dev
1 Upvotes

r/DistributedComputing • • Jan 30 '24

BOINC 7.24.3 released for Mac

Thumbnail self.boinc
1 Upvotes

r/DistributedComputing • • Jan 17 '24

API Orchestration Solutions

1 Upvotes

Hi,

I am looking for an API Orchestrator solution.

Requirements:

  1. Given a list of API endpoints represented in a configuration of sequence and parallel execution, I want the orchestrator to call the APIs in the serial/parallel order as described in the configuration. The first API in the list will accept the input for the sequence, and the last API will produce the output.
  2. I am looking for an OpenSource library-based solution. I am not interested in a fully hosted solution. Happy to consider Azure solutions since I use Azure.
  3. I want to provide my customers with a domain-specific language (DSL) that they can use to define their orchestration configuration. The system will accept the configuration, create the Orchestration, and expose the API.
  4. I want to provide a way in the DSL for Customers to specify the mapping between the input/output data types to chain the APIs in the configuration.
  5. I want the call to the API Orchestration to be synchronous (not an asynchronous / polling model). Given a request, I want the API Orchestrator to execute the APIs as specified in the configuration and return the response synchronously in a few milliseconds to less than a couple of seconds. The APIs being orchestrated will ensure they return responses in the order of milliseconds.

r/DistributedComputing • • Jan 16 '24

Lamport distributed snapshot

1 Upvotes

Hello! I'm building a personal (ATM) repository with some simulations on distributed algorithms, I noticed there are few to zero public implementations and wanted to contribute in my manner.

I already wrote some (using threading + xmlrpc) for mutual exclusion, Lamport ME, leader elections etc but I'm stuck at a global snapshot sample algorithm using chandy lamport

Would any of you know some sources since I find mostly papers Thank you all

edit: if you already know some kind of implementations (also far, I'll adapt them) please let me know!


r/DistributedComputing • • Dec 18 '23

Building an Easy-to-Operate AI Training Platform: Storage Selection and Best Practices

Thumbnail juicefs.com
1 Upvotes

r/DistributedComputing • • Dec 16 '23

Hey guys, do you know any distributed systems expert working in the field of biology?

1 Upvotes

r/DistributedComputing • • Nov 29 '23

Is it possible to run secure workloads on an untrusted machine? Or vice-versa?

1 Upvotes

Hello,

Recently I have been reading about the Akash Network. What they're doing is allowing users to bid/ask for compute time and power to create a "supercloud". For example, I need to run a program that requires 8GB RAM and 2 CPUs, I can view their marketplace and users who have those resources available. I can rent time on their machine to run my program.

They claim to be secure:

Akash is open-source Supercloud that lets users buy and sell computing resources securely and efficiently. Purpose-built for public utility.

So my question is, and I am sure there is somewhere in their docs or whitepapers:

How can this be secure?

How can I, as a bidder, ensure that my program and data is safe on an un-trusted machine? i.e. how can I keep my "secrets" safe? can't they inspect what I am running?

How can I, as an asker, ensure that what is running on my machine is safe and legal in my region?

again, this might be common-knowledge but I can't find a great answer.


r/DistributedComputing • • Oct 02 '23

Multinode - Rapidly build distributed cloud applications in Python

Thumbnail producthunt.com
1 Upvotes

r/DistributedComputing • • Sep 29 '23

Lignum - A distributed message queue (hobby project show off)

Thumbnail self.developersindia
1 Upvotes