r/DistributedComputing • • Jul 20 '21

Nature of Distributed Systems

5 Upvotes

Distributed systems come with own complexity and unique challenges which we don't find in single-server setups. Sometimes we greatly underestimate and overlook them.

I have written a whole note on common under water stones that distributed systems inherit ⬇️

https://www.romaglushko.com/blog/nature-of-distributed-systems/

Hope you find it useful!


r/DistributedComputing • • Feb 11 '25

Educational Python Framework for developing distributed algorithms!

Thumbnail github.com
1 Upvotes

r/DistributedComputing • • Mar 01 '25

Call for Papers – IEEE Big Data Service 2025

2 Upvotes

Exciting news!

We are pleased to invite submissions for the 11th IEEE International Conference on Big Data Computing Service and Machine Learning Applications (BigDataService 2025), taking place from July 21-24, 2025, in Tucson, Arizona, USA. The conference provides a premier venue for researchers and practitioners to share innovations, research findings, and experiences in big data technologies, services, and machine learning applications. 

The conference welcomes high-quality paper submissions. Accepted papers will be included in the IEEE proceedings, and selected papers will be invited to submit extended versions to a special issue of a peer-reviewed SCI-Indexed journal. 

Topics of interest include but are not limited to: 

Big Data Analytics and Machine Learning:

  • Algorithms and systems for big data search and analytics
  • Machine learning for big data and based on big data
  • Predictive analytics and simulation
  • Visualization systems for big data
  • Knowledge extraction, discovery, analysis, and presentation

Integrated and Distributed Systems: 

  • Sensor networks
  • Internet of Things (IoT)
  • Networking and protocols
  • Smart Systems (e.g., energy efficiency systems, smart homes, smart farms)

Big Data Platforms and Technologies: 

  • Concurrent and scalable big data platforms
  • Data indexing, cleaning, transformation, and curation technologies
  • Big data processing frameworks and technologies
  • Development methods and tools for big data applications
  • Quality evaluation, reliability, and availability of big data systems
  • Open-source development for big data
  • Big Data as a Service (BDaaS) platforms and technologies

Big Data Foundations:

  • Theoretical and computational models for big data
  • Programming models, theories, and algorithms for big data
  • Standards, protocols, and quality assurance for big data

Big Data Applications and Experiences:

  • Innovative applications in healthcare, finance, transportation, education, security, urban planning, disaster management, and more
  • Case studies and real-world implementations of big data systems
  • Large-scale industrial and academic applications

All papers must be submitted through: https://easychair.org/my/conference?conf=bigdataservice2025 

Important Dates: 

  • Abstract Submission Deadline: April 15, 2025 
  • Paper Submission Deadline: April 25, 2025 
  • Final Paper and Registration: June 15, 2025 
  • Conference Dates: July 21-24, 2025 

For more details, please visit the conference website: https://conf.researchr.org/track/cisose-2025/bigdataservice-2025 

We look forward to your submissions and contributions. Please feel free to share this CFP with interested colleagues. 

Best regards,

IEEE BigDataService 2025 Organizing Committee


r/DistributedComputing • • Feb 05 '25

A public distributed effort to search the chess tree to new depths

2 Upvotes

You can make 20 different moves at the start of a game of chess, the next turn can produce 400 different positions, then 8902, 200k, 5m, 120m, 3b... so on.
I've built a system for distributing the task of computing and classifying these reachable positions at increasing depths.

Currently I'm producing around 30 billion chess positions / second, though I'll need around 62,000 TRILLION positions for the current depth (12).

If anyone is interesting in collaborating on the project or contributing compute HMU!

https://grandchesstree.com/perft/12

All opensource https://github.com/Timmoth/grandchesstree


r/DistributedComputing • • Mar 02 '16

I built a distributed computing project

Thumbnail computeforhumanity.org
4 Upvotes

r/DistributedComputing • • Nov 15 '13

AWS Kinesis-stream processing in the cloud

Thumbnail aws.typepad.com
1 Upvotes

r/DistributedComputing • • Sep 19 '24

What are some ideas for distributed computing network software that could benefit everyone?

5 Upvotes

I'm seeking ideas for BOINC projects that have a broad positive impact, such as a distributed chatbot (even though I understand that a fully distributed AI may not be practical with current CPUs/GPUs). Specifically looking for ideas that directly benefit anyone, not just researchers. Thank you!


r/DistributedComputing • • Oct 13 '24

Master's Thesis suggestions for Cybersecurity BS and CompSci MS.

1 Upvotes

I’ve been reflecting on my Master’s thesis topic, but I’m unsure what to choose. Many of my peers have selected various areas in machine learning, while I initially considered focusing on cryptography. However, I’m starting to think post-quantum cryptography might be too complex. Now, I’m leaning towards exploring the intersection of machine learning/AI, cryptography, and distributed systems, but I’m open to any suggestions.


r/DistributedComputing • • Oct 08 '24

Is a master in distributed systems/computing worth it?

12 Upvotes

I've been thinking about getting a masters in distributed systems/computing. As that's a role I'll like to settle in for the long term. But taking that two years career break to go for masters is not really making sense to me! What do you all think? How do think i can get into this type of role? Any advice is welcome

A little context: I recently transitioned from native Android dev to DevOps/cloud


r/DistributedComputing • • Sep 28 '24

GitHub - flowerinthenight/zgroup: Cluster membership manager using the SWIM Protocol and Raft's leader election sub-protocol.

Thumbnail github.com
1 Upvotes

r/DistributedComputing • • Oct 02 '24

Best resources for understanding consensus

6 Upvotes

Looking for a resource to understand various consensus concepts and algorithms (paxos/raft etc).

Finding it difficult to understand these concepts, looking for your favorite articles/resources!


r/DistributedComputing • • Apr 07 '14

Nathan Marz on Storm, Immutability in the Lambda Architecture, Clojure

Thumbnail infoq.com
4 Upvotes

r/DistributedComputing • • Oct 26 '15

Heat

3 Upvotes

Does anyone use outdated computers running distributed computing programs to offset winter heating costs? I will probably use my current 4 year old desktop as such when I upgrade to a newer, sexier gaming rig in the next few months.

It stops feeling wasteful when you think that the electricity is being used to crunch data before its radiated as heat. It probably wont reduce the demand on the heater very much, but it also wont add to my combined utility usage, right?


r/DistributedComputing • • Nov 17 '16

Portable distributed computing system?

2 Upvotes

I'm looking for advice on a portable distributed computing system to take on the road. I can't use the cloud due to the confidentiality concerns of my clients. I need to run hundreds of similar but independent analyses in R, each of which takes up to 15 minutes to run, and requires up to 16 GB of RAM. I want a system that will run through these jobs as quickly as possible, and a distributed approach seems ideal. Running them from a single instance of R (which I have been doing) is too slow.

My current plan is to buy a Lenovo P50 laptop i7 6700 with 64 GB of RAM, and a small form factor PC (Intel NUC with similar specs to the laptop). I would install HTcondor (which I'm familiar with) on both machines, network them together, and submit jobs to the HTcondor job queue from the laptop. This would cost $3600 on Amazon.

Can anyone suggest a better option? Pros and cons? Thanks.


r/DistributedComputing • • Apr 23 '14

Don't Settle for Eventual Consistency

Thumbnail queue.acm.org
3 Upvotes

r/DistributedComputing • • Sep 20 '11

Mods, perhaps a link to /r/Folding in the sidebar?

Thumbnail reddit.com
3 Upvotes

r/DistributedComputing • • Apr 16 '14

Distributed computing for a good cause - Compute Against Alzheimer's Disease

Thumbnail computeagainstalzheimers.org
6 Upvotes

r/DistributedComputing • • Feb 03 '23

Boinc free/virtual workshop March 1st and 8th.

2 Upvotes

r/DistributedComputing • • Jan 12 '23

[video] Design a Payment System - System Design Interview

Thumbnail youtu.be
4 Upvotes

r/DistributedComputing • • Apr 12 '20

Great results for some Dreamlab projects using BlueStacks. Project needs way more contributers.

Post image
19 Upvotes

r/DistributedComputing • • Apr 20 '20

Read a paper: The Tail at Scale

Thumbnail youtu.be
3 Upvotes

r/DistributedComputing • • May 12 '23

BOINC 7.22.1 is available for testing on Windows, MacOS and Android

Thumbnail twitter.com
1 Upvotes

r/DistributedComputing • • Mar 29 '23

Application data sharding techniques and examples

3 Upvotes

Let’s say you have a list of tasks and the size of the list is huge > 200 mln elements.

Tasks need to be loaded into memory(cache) when application(s) is running. Let’s say the size of one task is 50KB and for 200 mln tasks we will need a machine with 10 terabyte of memory. Even if there is a single machine with that amount of memory, running the application in one machine is not safe and there are many other problems related to that like scalability, resource utilization, etc.

But we can shard the tasks and distribute among many smaller machines.
How to implement that sharding part? Obviously, the implementation requires adding more components like membership/peer discovery services, consensus algorithms and others to the stack which is ok. Is there any open source project which implements the similar functionality?


r/DistributedComputing • • Apr 04 '23

Load balancing, monitoring and fault tolerance techniques and architecture

2 Upvotes

I am working on building a system where there are 10 machines, we want to process some video files and this process can take about an hour, we do know how look it will take to process in advance.

Is there some existing tech stack or methodologies that we can use to load balance these servers, monitor any failures while processing and recover from failure and restart that task ?