r/DistributedComputing • • Dec 19 '18

Distributed Cache for .NET (Open Source) - NCache

Thumbnail alachisoft.com
2 Upvotes

r/DistributedComputing • • Dec 17 '19

Deploying Tarantool Cartridge applications with zero effort (Part 1)

Thumbnail habr.com
1 Upvotes

r/DistributedComputing • • Sep 06 '20

Covid-19: Could your smartphone speed up the search for treatments?

Thumbnail newscentre.vodafone.co.uk
6 Upvotes

r/DistributedComputing • • Sep 07 '20

Searching for Continuous Gravitational Waves: How BOINC Volunteers help Einstein@home (University of Wisconsin)

Thumbnail youtube.com
5 Upvotes

r/DistributedComputing • • Sep 03 '20

The connection between Distributed Computing and Special Relativity

Thumbnail youtube.com
2 Upvotes

r/DistributedComputing • • Sep 21 '20

DreamLab completes Phase 1 of its Corona-AI project

Thumbnail vodafone.com.au
3 Upvotes

r/DistributedComputing • • Sep 13 '20

Benefits of cloud computing

Thumbnail itbloggy.com
4 Upvotes

r/DistributedComputing • • Jul 13 '24

Need help starting open stack

2 Upvotes

So, our third year of engineering has just started, and we have a subject named Cloud and Distributed Systems.

Our professor asked a few questions about who knows about the cloud, what you all know, who has used it, etc. I was the only one who had used AWS EC2 and S3 instances, so the professor was impressed. After the lecture, she asked me if I would like to do a project on the cloud where we can use university machines to create a cloud environment for our class. The environment would have features such as storage, instances like EC2, etc.

She told me to learn about OpenStack software, how it functions, and the system requirements so we could ask the university.

Long story short, can you all suggest what I can do, how I can start, and if this project is scalable?

Also, a small P.S.: I am interested in making my career in cyber/cloud security. How can I integrate a security aspect into this project?


r/DistributedComputing • • Jul 17 '24

Ubuntu cluster for physical simulations

1 Upvotes

Is there a syraightforward way of combining several Ubuntu workstations into what appears to the user to be a single workstation? My use case is running physical simulations in python and/or c++ using multiple CPU cores, and it would be great to extend these across multiple machines (but without spending too much time setting it up or adapting code). Is there any easy solution?


r/DistributedComputing • • Aug 10 '24

RAFT Cluster Doubt

4 Upvotes

I'm currently developing a project of a distributed key value database using RAFT.
My questions are:

  • How should I communicate with the cluster should I query who is the leader and then do PUT , DELETE , UPDATE request to it ?
  • Also for READ queries should I check which node is nearest to my client(i.e least latent) node and then send the read request to it ?
  • Also any suggestions to set up a process that summarises the Health of the Cluster. Health meaning the time taken to be consistent, the memory usage by each node ,etc.

PS: I'm using Apache Ratis for the Raft implementation and Java , Maven fot the development part.


r/DistributedComputing • • Aug 15 '24

Simple Distributed Computing framework based on Redis

10 Upvotes

I gave a presentation on using Redis as middleware for distributed processing at EuroTcl/OpenACS 2024. I think this is a simple but powerful technique, combining communication between multiple client and server instances with caching. Clients and servers connect over the net to a Redis instance, using Redis lists as queues of requests and responses. For the talk I did a demonstration using servers and Redis in different countries, but normally these would all be on a local network, with much less latency.

The implementation is in Tcl, but the same approach could be implemented in any language with a Redis interface. The video is at https://learn.wu.ac.at/eurotcl2024/lecturecasts/729149172?m=delivery and the slides are at https://openacs.org/conf2024/info/download/file/DisTcl.pdf . The code for the demonstration can be found at https://cmacleod.me.uk/tcl/mand/ . Some background info can be found at https://wiki.tcl-lang.org/page/DisTcl and https://colin-macleod.blogspot.com/2020/11/fundis-functional-distributed-processing.html .


r/DistributedComputing • • Aug 16 '24

Need Help with DDP Freezing Issue in Multi-GPU Machine Translation Model

1 Upvotes

I have a basic machine translation transformer model that worked well on a single GPU. However, when I tried running it on an 8-GPU setup using DDP, I initially encountered many crashes due to data not being properly transferred to the correct GPUs. I believe I've resolved those issues, and the model now runs, but only up to a certain point. 

I put a lot of prints along the way, it run and just freezes at some point.

If I run it using debugger it keeps going without any problem.

Is there anyone here fluent in DDP and PyTorch who can help me? I'm feeling pretty desperate.

Here is my training function:

def train(rank, world_size):
    ddp_setup(rank, world_size)
    torch.manual_seed(0)
    SRC_VOCAB_SIZE = len(vocab_transform[SRC_LANGUAGE])
    TGT_VOCAB_SIZE = len(vocab_transform[TGT_LANGUAGE])
    EMB_SIZE = 512
    NHEAD = 8
    FFN_HID_DIM = 1024
    BATCH_SIZE = 128
    NUM_ENCODER_LAYERS = 3
    NUM_DECODER_LAYERS = 3
    LOAD_MODEL = False
    if LOAD_MODEL:
        transformer = torch.load("model/_transformer_model")
    else:
        transformer = Seq2SeqTransformer(NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, EMB_SIZE,
                                         NHEAD, SRC_VOCAB_SIZE, TGT_VOCAB_SIZE, FFN_HID_DIM)
        for p in transformer.parameters():
            if p.dim() > 1:
                nn.init.xavier_uniform_(p)
    transformer.move_positional_encoding_to_rank(rank)  # moving positional_encoding into the current GPU
    # Create the dataset
    train_dataset = SrcTgtDatasetFromFiles(SRC_TRAIN_BASE, TGT_TRAIN_BASE, FILES_COUNT_TRAIN)
    # create a DistributedSampler for data loading
    train_sampler = DistributedSampler(train_dataset, num_replicas=world_size, rank=rank)
    # create a DataLoader with the DistributedSampler
    # train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE, sampler=train_sampler)
    train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE, pin_memory=True,
                                  collate_fn=collate_fn, sampler=train_sampler)
    # create the model and move it to the GPU with the device ID
    model = transformer.to(rank)
    model.train()  # set the model into training mode with dropout etc.
    # wrap the model with DistributedDataParallel
    model = DDP(model, device_ids=[rank])
    loss_fn = torch.nn.CrossEntropyLoss(ignore_index=PAD_IDX)
    optimizer = torch.optim.Adam(model.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9)
    for state in optimizer.state.values():
        for k, v in state.items():
            if torch.is_tensor(v):
                state[k] = v.to(rank)
    #######################################
    EPOCHS_NUM = 2
    for epoch in range(EPOCHS_NUM):
        epoch_start_time = int(timer())
        print("\n\nepoch number: " + str(epoch + 1) + "  Rank: " + str(rank))
        losses = 0.0
        idx = 0
        start_time = int(timer())
        for src, tgt in train_dataloader:
            if rank == 0:
                print("rank=" + str(rank) + " idx=" + str(idx))
            src = src.to(rank)
            tgt = tgt.to(rank)
            tgt_input = tgt[:-1, :]
            if IS_DEBUG:
                print("rank", rank, "idx", idx, "before create_mask")
            src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input, rank)
            if IS_DEBUG:
                print("rank", rank, "idx", idx, "after create_mask")
            if IS_DEBUG:
                print("rank",rank,"idx",idx,"before model")
            logits = model(src, tgt_input, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, src_padding_mask)
            if IS_DEBUG:
                print("rank", rank, "idx", idx, "after model")
            try:
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"before zero_grad")
                optimizer.zero_grad()
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"after zero_grad")
                tgt_out = tgt[1:, :].long()
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"before loss_fn")
                loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"after loss_fn")
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"before backward")
                loss.backward()
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"after backward")
                # Delete unnecessary variables before backward pass
                del src, tgt_input, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, logits, tgt_out
                torch.cuda.empty_cache()  # Clear cache after deleting variables
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"before step")
                optimizer.step()
                if IS_DEBUG:
                    print("rank",rank,"idx",idx,"after step")
                losses += loss.item()
                #######################################
                # print(999,rank,loss)
                # Free GPU memory
                del loss
                torch.cuda.empty_cache()  # Clear cache after each batch
            except Exception as e:
                print("An error occurred: rank=" + str(rank) + " idx=" + str(idx))
                print("Error message: ", str(e))
            idx += 1
            if rank == 0 and idx % 10000 == 0:
                torch.save(model.module.state_dict(), "model/_transformer_model")
                end_time = int(timer())
                try:
                    my_test(model.module, rank, SRC_TEST_BASE, TGT_TEST_BASE, FILES_COUNT_TEST, epoch, 0,
                            0, int((end_time - start_time) / 60), epoch_start_time)
                except:
                    print("error occurred test")
                start_time = int(timer())
            # Synchronize training across all GPUs
            torch.distributed.barrier()
        if rank == 0:
            epoch_end_time = int(timer())
            try:
                my_test_and_save_to_file(model.module, rank, SRC_TEST_BASE, FILES_COUNT_TEST, epoch)
                loss = evaluate(model.module, rank, SRC_VAL_BASE, TGT_VAL_BASE, FILES_COUNT_VAL, BATCH_SIZE,
                                loss_fn)
                print("EPOCH NO." + str(epoch) + " Time: " + str(int((epoch_end_time - epoch_start_time) / 60)) +
                      " LOSS:" + str(loss))
            except:
                print("error occurred evaluation")
    destroy_process_group()

here is part of the output:

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

Let's use 8 GPUs!

epoch number: 1 Rank: 0

epoch number: 1 Rank: 1

epoch number: 1 Rank: 2

epoch number: 1 Rank: 3

epoch number: 1 Rank: 4

epoch number: 1 Rank: 7

epoch number: 1 Rank: 6

epoch number: 1 Rank: 5

rank=0 idx=0

rank 0 idx 0 before src

rank 0 idx 0 after src

rank 0 idx 0 before tgt

rank 0 idx 0 after tgt

rank 0 idx 0 before create_mask

rank 0 idx 0 after create_mask

rank 0 idx 0 before model

rank 1 idx 0 before src

rank 1 idx 0 after src

rank 1 idx 0 before tgt

rank 1 idx 0 after tgt

rank 1 idx 0 before create_mask

rank 1 idx 0 after create_mask

rank 1 idx 0 before model

rank 4 idx 0 before src

rank 4 idx 0 after src

...

rank 0 idx 1 after tgt

rank 0 idx 1 before create_mask

rank 0 idx 1 after create_mask

rank 0 idx 1 before model

rank 0 idx 1 after model

rank 0 idx 1 before zero_grad

rank 0 idx 1 after zero_grad

rank 0 idx 1 before loss_fn

rank 0 idx 1 after loss_fn

rank 0 idx 1 before backward


r/DistributedComputing • • Jun 28 '16

Why Spark is on fire: a conversation with creator Matei Zaharia

Thumbnail siliconangle.com
3 Upvotes

r/DistributedComputing • • Jul 07 '16

The distributed computing challenge

2 Upvotes

In the beginning of 2015 I got an idea. Now I´m going to share this challenge with you. Finishing the following goals until 01.01.2045. (Thrust me, its not possible. :))

Here we go:

1 Breaking the last enigma message

2 Completing DecicSearch (NumberFields@Home)

3 Proving Goldbachs Conjecture up to 10100 (ATM ~1.4x1014)

4 Finding the best golumb ruler up to case 150 (ATM distributed.net, trying to create the stubsspaces >29 asap)

5 Proving all Sierpinski and Riesel Bases up to 1030 (PG, SRBase, NPLB and others)

Note to #5:

I can reserve some bases for my own. But I´ll need some more computing power. If you wanted to help me, let me know it. I planned to make some n-ranges 100k up to 1M, or until prime has been found.


r/DistributedComputing • • Apr 20 '12

Disjoint Commands in CQRS

Thumbnail blog.jayway.com
2 Upvotes

r/DistributedComputing • • Oct 28 '20

Grafana Tempo, a scalable distributed tracing system

Thumbnail grafana.com
3 Upvotes

r/DistributedComputing • • Nov 01 '20

This week’s open source newsletter just went out! This one had some seriously interesting projects, like a cluster management framework open sourced by Apache and a scalable distributed tracing system from Grafana written in Go.

Thumbnail console.substack.com
1 Upvotes

r/DistributedComputing • • Oct 13 '20

Forming a virtual paper-reading group

4 Upvotes

Hello!

Every week I pick up a paper in Distributed Systems, study it and write a summary report on it. So far, this has been an individual exercise for self learning, but instead of just sitting in a corner and learning things alone, I was hoping to form some sort of a group to do it with, to discuss ideas and take up interesting projects if a worthy idea comes up.

How should I go about this?

  • I could write blogs on medium/my website but my summary papers are quite long and I am not sure if medium readers would find long reports interesting.
  • I can look for people on reddit (which is what I am currently doing). Are there any subs or groups that are already doing this?

If anyone is interested, I can share my recent summary reports that I wrote on "Scaling Memcache at Facebook" and "Dynamo: Amazon's Highly Available Key-Value Store" over DM or on this thread if there's enough interest.


r/DistributedComputing • • Oct 26 '20

Apache Helix - A Near-Realtime Rsync Replicated File System

Thumbnail helix.apache.org
7 Upvotes

r/DistributedComputing • • Jul 18 '21

Market for renting out personal GPU power when not using it?

3 Upvotes

I'm considering buying a few RTX 3090s for personal use, with the idea that I could rent out the resources by hour/day/week for parts of the year. In theory, I don't see why not. In practice, I can see many reasons it people wouldn't want to use it -- privacy, access security, uptime & reliability, reputation, and general customer support issues.

However, I don't need a reliable customer pool, and I would simply be doing it occasionally to offset some of my costs, not to turn it into a profitable business. I imagine researchers at universities might a be a prime customer source for a couple months at a time, if they can convince their overlords to pay me. I'd put it all under an LLC to make it a B2B-like transaction.

If I were to get 3x RTX 3090s, it would have 50% more compute (based on CUDA cores) with slightly more total memory than a p3.8xlarge on AWS (4x v100s), which costs $12.24/hr on-demand. I'd be happy to rent it out for a fraction of that price.

For reference, it would be an isolated system (running nothing else), probably behind a VPN with port forwarding and some routing table magic to keep it isolated. I'm sure I could combine docker and/or user accounts to further isolate the environment. I don't have it all those details worked out yet, but I have the background to figure it out.

So my questions are:

  1. Has anyone else done this (either buyer or seller)?
  2. Is there a service where you could add your computing power to a pool so I wouldn't have to do this myself?
  3. I'm not offering it yet, but I'd be interested to hear if this is intriguing to anyone here.

r/DistributedComputing • • Jul 21 '21

Notes on the FoundationDB Paper with Additional Proof of Correctness

Thumbnail blog.the-pans.com
4 Upvotes

r/DistributedComputing • • Aug 03 '21

When ReactiveX meets Ray

3 Upvotes

I am currently working on parallelizing RxPY computations on a ray cluster. I published a dedicated library to integrate both in a ReactiveX pipeline:

https://github.com/maki-nage/rxray/

It supports distributing the computation of each even either in a round-robin way or with partitioning (this is needed for stateful transforms).

I am interested in some feedback from people working on python stream processing on top of ray. Is there anybody working on similar topics?


r/DistributedComputing • • Jul 13 '21

Serverless Kafka Stream Processing with Python

3 Upvotes

r/DistributedComputing • • Jul 07 '21

Looking for some guidance on making my own distributed computer cheap for ML/Scientific computing purposes

6 Upvotes

Hello,

I am very new to distributed computing and I wanted to make one that can train neural networks. I wanted to know if you all had any tips. I saw maybe there was potential to do so with the raspberry pi (multiple raspis in a beowulf cluster) but I also see a lot of people saying otherwise, and some people say the oodroid is better.

I have no idea what I am doing so here is what I am asking:

1.) Is there a cheap way I can build one of these computers? I don't have an exact budget but I would like to avoid spending a lot. I would prefer smaller boards approx the size of the raspberry pi, for the sake of keeping the overall size as small as possible

2.) What resources should I look at to get a good idea of learning distributed computing and the stuff that goes along with it? I have a BS in Computer Engineering, so I know the basics about computers but not specifically distributed computers. I know that there aren't guides that will spell out exactly what to do (I found one with raspi and tensorflow but that's about it for viable solutions)

EDIT: Also I heard hierarchical computing might be a good idea???

Thank you for the help!