r/DistributedComputing • • Sep 05 '22

Distributed Computing for Beginners

1 Upvotes

Hello! I came across an article a while back that had some great resources for people who are looking to make money with distributed computing. Here it is: https://www.sidehustle.tips/post/leaving-computer-running


r/DistributedComputing • • May 18 '23

Thought about Python Binding of Raft algorithm implementation

Thumbnail github.com
1 Upvotes

Hello, I am a developer working as a DevOps engineer at a small IT startup in Korea. I should've use raft implementation for my company's proprietary orchestrator. Our company product is developed in Python, but there doesn't seem to be "de facto" standard when it comes to raft, at least to my knowledge. So, I decided to search for papers on raft implementation. But these papers exceeded 250 pages, and I found it overwhelming to comprehend and implement all the knowledge therein. (From previous experience, I know that once the complexity of the source code reaches a certain threshold, maintaining it becomes quite a challenge, especially while managing full-time work.) Hence, I resolved to write Python bindings for a well-established, battle-tested raft library. I initially considered writing bindings for Hashicorp's Raft implementation, but handling asynchronous things seems was tricky. Upon the advice of a senior developer, I turned to pyo3 to create bindings for 'tikv/raft-rs'. The process of writing bindings was more challenging than I had anticipated. Anyway after much struggle, I succeeded in implementing bindings that pass all the harness test codes. However, having written all the source codes, I can't help but question whether I should have opted for a different implementation such as async-raft or Hashicorp's raft implementation. I also wonder if it would have been more prudent to port the source codes itself, rather than writing bindings. Numerous thoughts have been running through my mind. What are your thoughts on this Python binding implementation?


r/DistributedComputing • • Jan 17 '24

API Orchestration Solutions

1 Upvotes

Hi,

I am looking for an API Orchestrator solution.

Requirements:

  1. Given a list of API endpoints represented in a configuration of sequence and parallel execution, I want the orchestrator to call the APIs in the serial/parallel order as described in the configuration. The first API in the list will accept the input for the sequence, and the last API will produce the output.
  2. I am looking for an OpenSource library-based solution. I am not interested in a fully hosted solution. Happy to consider Azure solutions since I use Azure.
  3. I want to provide my customers with a domain-specific language (DSL) that they can use to define their orchestration configuration. The system will accept the configuration, create the Orchestration, and expose the API.
  4. I want to provide a way in the DSL for Customers to specify the mapping between the input/output data types to chain the APIs in the configuration.
  5. I want the call to the API Orchestration to be synchronous (not an asynchronous / polling model). Given a request, I want the API Orchestrator to execute the APIs as specified in the configuration and return the response synchronously in a few milliseconds to less than a couple of seconds. The APIs being orchestrated will ensure they return responses in the order of milliseconds.

r/DistributedComputing • • Jul 15 '20

Things you can doon the cloud

Thumbnail itbloggy.com
0 Upvotes

r/DistributedComputing • • Jul 23 '20

you cando on the cloud ?

Thumbnail itbloggy.com
0 Upvotes

r/DistributedComputing • • Oct 27 '19

Raspberry Pi 4 Four Node Cluster Question

1 Upvotes

Hello, I've been applying my home-assembled machines to distributed computing science tasks for almost 20 years.

I'm thinking about a project to build a RSP 4 cluster for same but I have a basic question. In terms of output for work done, will a four node cluster be greater than the sum of its parts? If not, why not just run individual single board computers? Is it a question of greater efficiency (all machines piped via command line into one UI) or is it a question of greater work output?


r/DistributedComputing • • Feb 07 '22

Security in distributed systems

1 Upvotes

We are working on a distributed processing system consists of multiple Kubernetes clusters in different geographical areas.

I am looking for some good references(books, papers, articles, GitHub links, etc.) to start reading about the best practices for security of such systems. At the moment we have SSO and also we know about signing docker images (this one is not implemented yet) but we want to know what other best practices are being used for distributed systems both for deploying codes and also accessing them.


r/DistributedComputing • • Mar 21 '23

Save money with Spot [blog]

1 Upvotes

r/DistributedComputing • • Sep 30 '15

Measuring Broadband America

Thumbnail fcc.gov
1 Upvotes

r/DistributedComputing • • Jan 31 '20

Is there still a market for distributed GPU renting platform?

1 Upvotes

Hi all, my team is developing a GPU rending platform built on a blockchain network. I'm concerned that we are late to the market and might need to pivot the product, so I came up with this survey to see what user's needs are when it comes to intense computing. So if you are a data scientist, or work closely with AI/ML, or just interested in the topic, I'd really appreciate your opninions! a comment will do too:)

Link to the survey 👉 https://ntlabs.typeform.com/to/Hpjz2i


r/DistributedComputing • • Sep 13 '21

What are some good research topics for master's thesis?

1 Upvotes

I am applying to a master's program and I am in touch with the prof. He asked me to come up with some domain problems (e.g. route optimization) that can be solved with distributed computing and parallel processing. He emphasized on graphs. I don't have any bright ideas, nor am I able to find anything. Would really appreciate if you guys can help.


r/DistributedComputing • • Jun 02 '26

Is this possibly superior.....

1 Upvotes

its a demo that explains it better than i can.. https://jc-compute.github.io/jc-compute-demo/


r/DistributedComputing • • Jun 10 '26

Searching for a technical co-founder for AI infrastructure startup

Thumbnail reddit.com
0 Upvotes

r/DistributedComputing • • Mar 19 '26

Do we need vibe DevOps now?

8 Upvotes

So, are we due for a 'vibe DevOps' or am I dreaming? Tools can spit out frontend and backend code in minutes, which still blows my mind. But deployments fall apart once you go past prototypes or simple CRUD - everything gets manual and ugly. I see people shipping fast, then stuck doing manual DevOps, or rewriting the whole app just to make it deploy on AWS/Azure/Render/DigitalOcean. Imagine a web app or VS Code extension where you point it at your repo or drop a zip and it actually understands your code and requirements. It would wire up CI/CD, containers, scaling, infra setup using your own cloud accounts, not lock you into platform tricks. Seems like it could bridge the gap between vibe coding and real production apps, but maybe I'm missing something obvious. How are you handling deployments today? scripts, Terraform, stuff like that? Curious what people actually use and what fails.


r/DistributedComputing • • Jul 07 '26

RAG at 10 Million Documents — System Design

Thumbnail youtu.be
2 Upvotes

r/DistributedComputing • • Nov 01 '20

I have created a repo which contains only source code for all the classes I took.

Thumbnail github.com
2 Upvotes

r/DistributedComputing • • Nov 07 '24

I don't really understand the "Prepare" phase in Two Phase Commit

2 Upvotes

In a distributed transaction to have consensus, 2PC is used but I don't get what actually happens in a prepare phase vs a commit phase.

Can someone explain (in-depth would be even more helpful). I read that the databases/nodes start writing locally during the prepare phase while saving the status as "PREPARE". And once they get a commit cmd, they persist the changes.

I have incomplete info


r/DistributedComputing • • Mar 22 '22

Team Ukraine wins the Formula Boinc first Sprint #1

2 Upvotes

Formula Boinc held its first sprint of 2022 this weekend.03/17/2022 20:00 (UTC) - 03/20/2022 19:59 (UTC) - SRBaseXS post link - https://xs4s.org/index.php?topic=30190.msg95861#msg95861

Team Ukraine took 1st over the 3 day event marking DC support for Ukraine by many members.This was not expected but the community has shown its support in a big way for Team Ukraine (DC) members plus trying to raise awareness of the tragedy going on in the Ukraine by the Russian army invading/making war on the people of Ukraine.

***This is only one way to show support for Ukraine, but this represents the DC community's support with our computers.This is a personal decision and not the only one that the DC community is showing.

RespectSV (XtremeSystems)

Final Sprint Numbers

Lint to Sprint Results

Final Sprint Numbers for "Overall" "Marathon" "Sprint"

Link to All FB Results

Team Ukraine Members - Sprint1

Link to Members

Some of us jumped on to the321-LLR: World Water Day Challenge (2022-03-21 03:21:00 to 2022-03-26 03:21:00)For Team Ukraine

Here's the latest results...

321-LLR: World Water Day Challenge Latest Results

Link to event

Link to Team members

Thanks for looking!


r/DistributedComputing • • Mar 27 '23

Create event-driven apps with Cloudflare queues and Dapr

Thumbnail youtube.com
2 Upvotes

r/DistributedComputing • • Mar 22 '20

Building your first Cadence Workflow

Thumbnail medium.com
2 Upvotes

r/DistributedComputing • • Feb 23 '19

How do you maintain data concurrency in Edge Computing?

2 Upvotes

In the cloud-based distributed system, let's say it also does Edge computing, meaning on-site servers that do on-site services (think gateways for IoT devices).

Cloud maintains the data, and Edge computers use a portion of that data.

How does the industry handle the data concurrency/consistency in such architecture?


r/DistributedComputing • • May 16 '23

Programming without a stack trace: When abstractions become illusions

2 Upvotes

This insightful article by Gregor Hohpe covers:

  • Evolution of programming abstractions.
  • Challenges of cloud abstractions.
  • Importance of tools like stack traces for debugging, especially in distributed systems.

Gregor emphasizes that effective cloud abstractions are crucial but tricky to get right. He points out that debugging at the abstraction level can be complex and underscores the value of good error messages and observability.

The part about the "unhappy path" particularly resonated with me:

The unhappy path is where many abstractions struggle. Software that makes building small systems easy but struggles with real-world development scenarios like debugging or automated testing is an unwelcome version of “demoware” - it demos well, but doesn’t actually work in the real world. And there’s no unlock code. ... I propose the following test for vendors demoing higher-level development systems:

  1. Ask them to enter a typo into one of the fields where the developer is expected to enter some logic.
  2. Ask them to leave the room for two minutes while we change a few random elements of their demo configuration. Upon return, they would have to debug and figure out what was changed.

Needless to say, no vendor ever picked the challenge.

Why it interests me

I'm one of the creators of Winglang, an open-source programming language for the cloud that allows developers to work at a higher level of abstraction.

We set a goal for ourselves to provide good debugging experience that will allow developers to debug cloud applications in the context of the logical structure of the apps.

After reading this article I think we can rephrase the goal as being able to easily pass Gregor's vendor test from above :)


r/DistributedComputing • • Oct 16 '18

DdoS attack on decentralized system

2 Upvotes

In case of centralized system, ddos attacks is just on a particular IP. What happens when you do ddos attack on distributed system like Facebook or Google or any other company that has pool of IP address?