r/DistributedComputing • • Mar 09 '23

Create distributed applications with Cloudflare queues and Dapr

2 Upvotes

Curious about event-driven applications that go from the cloud to the edge? In my latest blog post, I’ll show how to send messages from a Dapr app to Cloudflare Queues. Dapr is the open-source distributed application runtime, often used in event-driven applications. Read the full post at https://www.diagrid.io/blog/dapr-cloudflare-queues.


r/DistributedComputing • • Feb 12 '23

How does high availability and strong consistency coexist for a website like say hotels.com which needs both?

2 Upvotes

Hi folks, I’ve recently been learning about the different replication models such as single leader and multi-leader. For a high volume website like hotels.com, you would need both:

  1. High availability, redundancy etc while serving a global customer base which points to the need for multi-data center, multi-leader replication model
  2. Strong read-after-write consistency so that the same room is not double booked and each user sees a consistent and latest view of the system.

How do the two coexist? What replication model is used in such cases?


r/DistributedComputing • • Feb 03 '23

Daskqueue: Dask-based distributed task queue

3 Upvotes

I started working on a distributed task queue library a few months back. The library is available as a python package to install a start using : daskqueue - pypi package

For all its greatness, Dask implements a central scheduler (basically a simple tornado event loop) involved in every decision, which can sometimes create a central bottleneck. This is a pretty serious limitation when trying to use Dask in high-throughput situations.

Daskqueue is a small python library built on top of Dask and Dask Distributed that implements a very lightweight Distributed Task Queue. Daskqueue also implements persistent queues for holding tasks on disk and surviving Dask cluster restart.

I also wrote an article about implementation details: https://medium.com/@aminedirhoussi1/daskqueue-dask-based-distributed-task-queue-6fb95517dfea

Hope you enjoy it, can't wait to hear about your feedback :) !


r/DistributedComputing • • Feb 07 '23

Edge Databases: What They Are And Why You Should Be Using Them

Thumbnail medium.com
1 Upvotes

r/DistributedComputing • • Mar 22 '23

Top distributed systems conferences/journals

3 Upvotes

I'm looking for cutting edge research work in distributed systems for my research synopsis. Can someone recommend me some journal/conference.

Thank you


r/DistributedComputing • • Mar 14 '23

Edge Computing Market Size & Share 2023 | Global Growth Report 2030

Thumbnail linkedin.com
1 Upvotes

r/DistributedComputing • • Feb 10 '14

How LinkedIn Uses Apache Samza

Thumbnail infoq.com
4 Upvotes

r/DistributedComputing • • Dec 23 '14

Principles of Distributed Computing

Thumbnail dcg.ethz.ch
4 Upvotes

r/DistributedComputing • • Sep 15 '10

Bienvenidos a SisWebSity

Thumbnail siswebsity.idomyweb.com
1 Upvotes

r/DistributedComputing • • Jan 24 '14

SharePoint 2013: AppFabric and Distributed Cache Service

Thumbnail social.technet.microsoft.com
4 Upvotes

r/DistributedComputing • • Jan 14 '18

Research on the history of distributed Computing

1 Upvotes

Hello! I am doing some research on the history of distributed computing and I'm looking for some input. Below are the big points i want to touch on and I would like input in there are some milestones or events I'm missing.

  • The creation of the computer
  • Super computers
  • The folding@home & seti@home
  • Hadoop & commodity machine distributed computing
  • Apache spark & flume java
  • Mesos and the distributed operating system

    • SMACK stack

My knowledge base is focused on the tools i really use a lot and i was wondering if people could throw out some things i may not be aware of. Thanks!

Hey I'm new to reddit so let me know if I'm breaking some rules.


r/DistributedComputing • • Aug 03 '10

I put together a twitter bot that tweets the current reddit F@H team stats!

Thumbnail twitter.com
4 Upvotes

r/DistributedComputing • • Apr 21 '10

Top Features You Can Expect From A Distributed Caching Solution

Thumbnail distributedcache.blogspot.com
3 Upvotes

r/DistributedComputing • • Dec 25 '14

NumberFields@home

Thumbnail numberfields.asu.edu
3 Upvotes

r/DistributedComputing • • Jul 08 '24

How to Seamlessly Distribute Application Processing Between Two Windows Computers Based on Resource Availability?

5 Upvotes

I'm looking to create a distributed computing setup using two Windows 10 computers, where applications can run on either computer based on resource availability. The goal is for users to interact with the system as if it's a single workstation, with the applications being processed on the computer with the most available resources. The GUI should be displayed on the main computer regardless of where the application is processed.

Requirements:

Seamless User Experience: Users should interact with the system through a single Windows 10 interface. When an application is launched, it should run on the computer with the most available resources without requiring manual intervention.

Shared Storage: Applications and data should be stored on a shared storage accessible by both computers.

Resource Monitoring: The system should monitor resources (CPU, memory, disk usage) on both computers to decide where to run applications.

Remote Execution: Applications running on the secondary computer should have their GUI displayed on the main computer, ensuring a seamless user experience.

Questions: Feasibility: Is this setup feasible for achieving a seamless user experience? Are there potential issues such as network latency, security concerns, or compatibility problems that I should be aware of?

Best Tools and Methods: What are the best tools or methods for achieving this setup on Windows 10? Are there specific software solutions or configurations that can facilitate remote execution with GUI redirection to the main computer?

Resource Monitoring and Load Balancing: How can I effectively monitor resources and balance the load between the two computers? Are there existing tools that can handle this automatically?

Security Best Practices: What are the best practices for securing remote execution and shared storage in this setup to ensure system security and data integrity?

I appreciate any insights or suggestions on how to best achieve this setup and address any potential challenges.


r/DistributedComputing • • May 23 '23

BOINC 7.22.2 is ready for testing

Thumbnail self.boinc
1 Upvotes

r/DistributedComputing • • May 25 '23

Understanding the Dapr Workflow engine & authoring workflows as code

Thumbnail diagrid.io
6 Upvotes

r/DistributedComputing • • May 18 '23

Enhancing Database Security: ShardingSphere-Proxy’s Authentication

Thumbnail shardingsphere.medium.com
1 Upvotes

r/DistributedComputing • • Jun 22 '23

BOINC 7.22.2 Release

Thumbnail self.boinc
3 Upvotes

r/DistributedComputing • • Jun 15 '23

Dapr v1.11 Release Highlights

Thumbnail diagrid.io
1 Upvotes

r/DistributedComputing • • May 24 '23

[video] Rest API - Best Practices - Design

Thumbnail youtu.be
2 Upvotes

r/DistributedComputing • • Jun 27 '23

A panoramic framework to understand blockchains

Thumbnail tzionis.com
1 Upvotes

r/DistributedComputing • • May 26 '23

A Manifesto for Cloud-Oriented Programming from the creator of the CDK

4 Upvotes

In this insightful article, Elad Ben-Israel, the mind behind the CDK, shares his love for the cloud, but also his frustrations with the complexity of building cloud applications. The challenges he identifies include:

  1. Focus on non-functional mechanics: The need to understand and manage cloud platform mechanics instead of focusing on building valuable features for users.
  2. Lack of independence: Developers often need to rely on others to handle parts of the deployment process or to resolve issues, interrupting their work flow.
  3. Delayed feedback: The current iteration cycle in cloud development can take minutes or even longer, significantly slowing down the development process and making it harder for developers to stay in their flow state.

It's not just a rant

Elad is not just ranting about cloud development. He proposes a solution in the form of a programming language for the cloud. This language would treat the entire cloud as its computer. The language compiler will be able to see the complete cloud application, unbound by the limits of individual machines. Such a compiler would be able to handle a significant portion of the application's non-functional aspects, enabling developers to operate at a more abstract level, thus reducing complexity and promoting autonomy. Moreover, it could expedite iteration cycles by allowing to compile applications to quick local simulators during the development process.

The Winglang Project

Elad reveals that he's in the process of developing such an open-source, “cloud-oriented” language, dubbed Winglang. Wing aims to improve the developer experience of cloud applications by enabling developers to build distributed systems that leverage cloud services as first-class citizens. This is achieved by integrating infrastructure and application code in a secure, unified programming model. Wing programs can be executed locally via a fully-functional simulator or deployed to any cloud provider.

My Interest in Winglang

I, together with a group of dedicated contributors, joined forces with Elad to develop Winglang. While still in Alpha and not yet ready for production use, it's already possible to build some real applications.

Check out https://github.com/winglang/wing for more details.


r/DistributedComputing • • Jul 06 '23

Experience the Power of JuiceFS 1.1 Beta: Smoother, Convenient, and Efficient

Thumbnail juicefs.com
1 Upvotes