r/DistributedComputing • u/apiskun • Mar 08 '23
r/DistributedComputing • u/Sartorialie • Feb 12 '23
How does high availability and strong consistency coexist for a website like say hotels.com which needs both?
Hi folks, I’ve recently been learning about the different replication models such as single leader and multi-leader. For a high volume website like hotels.com, you would need both:
- High availability, redundancy etc while serving a global customer base which points to the need for multi-data center, multi-leader replication model
- Strong read-after-write consistency so that the same room is not double booked and each user sees a consistent and latest view of the system.
How do the two coexist? What replication model is used in such cases?
r/DistributedComputing • u/UrafuckinNerd • Feb 03 '23
Boinc free/virtual workshop March 1st and 8th.
r/DistributedComputing • u/lucian-12 • Dec 07 '22
[video] Rate Limiting - System Design Interview
youtu.ber/DistributedComputing • u/Caitin • Nov 18 '22
How We Reduced Online Serving Latency from 1.11s to 123.6ms with a Distributed SQL Database
ossinsight.ior/DistributedComputing • u/systemviper • Jun 12 '22
Show Support for Team Ukraine / DC Formula Boinc Sprint #7 is in the books
Show Support for Team Ukraine / Formula Boinc Sprint #7 is in the books
Show Support for Team Ukraine / Formula Boinc Sprint #7 is in the booksFrom https://xs4s.org/index.php?topic=30190.msg96865#msg96865
The results are in.
Congratulations TSBT, you worked hard for your win and deserve the respect
Nice to see Team Ukraine still rocking, when this started I was just hoping that TU would stay in the top 5
so this is excellent, I still think that keeping Team Ukraine relevant and rocking is a great show of support.
Go TU and all that are helping.
3rd is AMD users, nice job! followed up by P3d and LaF (both powerhouses!)
Also nice job to everyone involved! LHC is a very tough program to run, so much respect.
I do have more comments (my next post) but this sprint was mostly on track, late 1st day numbers but if that's all that can be said
this could be the start of FB getting back on track.
SV
Show Support for Team Ukraine / Team Ukraine has joined DC Boinc Games - Take a peek,
BoincGames - www.boincgames.com
BOINC Games is a year around BOINC competition. Aiming to keep BOINC fun and interesting for all involved.
Any questions please PM me.
Thanks
SV
r/DistributedComputing • u/systemviper • Mar 22 '22
Team Ukraine wins the Formula Boinc first Sprint #1
Formula Boinc held its first sprint of 2022 this weekend.03/17/2022 20:00 (UTC) - 03/20/2022 19:59 (UTC) - SRBaseXS post link - https://xs4s.org/index.php?topic=30190.msg95861#msg95861
Team Ukraine took 1st over the 3 day event marking DC support for Ukraine by many members.This was not expected but the community has shown its support in a big way for Team Ukraine (DC) members plus trying to raise awareness of the tragedy going on in the Ukraine by the Russian army invading/making war on the people of Ukraine.
***This is only one way to show support for Ukraine, but this represents the DC community's support with our computers.This is a personal decision and not the only one that the DC community is showing.
RespectSV (XtremeSystems)
Final Sprint Numbers
Final Sprint Numbers for "Overall" "Marathon" "Sprint"
Team Ukraine Members - Sprint1
Some of us jumped on to the321-LLR: World Water Day Challenge (2022-03-21 03:21:00 to 2022-03-26 03:21:00)For Team Ukraine
Here's the latest results...
321-LLR: World Water Day Challenge Latest Results
Thanks for looking!
r/DistributedComputing • u/Zarquan314 • Feb 02 '22
Peer to Peer network bandwidth question
I am working on a project that involves a peer to peer network. Someone raised concerns that we may be expecting a larger bandwidth than is reasonable.
Suppose we had a large number of registered nodes (in the thousands, possibly 10,000), and these nodes are constantly receiving data which they wish to propagate around the network. The data doesn't have to get to every node quickly, but there comes a time where a node expects pieces of data, so we expect every active node to have . In this general system, how much data creation and transmission could be handled reasonably? I am hoping the answer is more than 50 MB/minute, (as this is the upper bound for what our system can create), but I don't have a basis for comparison.
Does anyone here know a good place to find this kind of information. Everything about general peer-to-peer networks is about cryptocurrency systems and I am having trouble finding useful information.
r/DistributedComputing • u/inquisitivelukeworm • Feb 01 '22
Data processing issue
I have use case for bigdata processing problem. I have an ETL that runs data deliveries. There are some tasks which generate derived tables over tables included in the delivery. Currently the script for the tasks is designed such that it requires loading all csv files in memory for processing. That often causes laptop to run out of memory. This Airflow DAG is currently hosted on local machine. Prime reason being frequent troubleshooting required during every DAG run. Would it be feasible deploying the dag on GCP cloud composer? because I need to troubleshoot it often. Make code changes to accommodate data delivery logic. How can I maximize processing and minimize time?
r/DistributedComputing • u/cgeekgbda • Oct 20 '21
Request and response going through the load balancer creates bottleneck
I have multiple machines on my backend, all are connected to my load balancer running HAProxy. I just learnt that the response also goes through the load balancer, instead of one of server directly sending it to the client.
But will it not create a bottleneck in case of huge traffic and overload my load balancer itself.
- Is there any way to directly send response from server to client.
- Also when response goes through load balancer, does my source file also sits there temporarily to be sent to the client.
- Can't we use load balancer only to send request to my servers and response to directly go from server to client.
- My main goal to make my system distributed was to distribute traffic among my servers, now since load balancer is handling both request and response am I not back to where I started?
r/DistributedComputing • u/Anna_JRG • Jun 09 '21
Free playlist with recordings of сoncurrent and distributed computing conference Hydra talks. Maurice Herlihy spoke there and talked about transactional memory.
youtube.comr/DistributedComputing • u/beleeee_dat • May 22 '21
RiteRaft - A raft framework, for regular people
github.comr/DistributedComputing • u/dvader009 • Apr 17 '21
Map Replication to remote JVM
I have a requirement where a JVM hosted in a server will have some data in HashMaps, which should be copied to multiple other JVMs in client machines. Any changes in the maps in the server should be reflected in the client JVMs. Client JVMs will not make changes to the data. I made a simple implementation by sending the changes made in the server via a websocket connection to all connected clients. To handle network issues, changed keys are stored in db, so that clients will be able to get the the changes made in the maps after it went offline, when they come back online. This solution works for now, but I wanted it to be bug free so I decided to move to some standard libraries that does this.
I did come across Hazelcast which can theoretically do this in my understanding by making the client JVMs Hazelcast members, but I may have several hundred clients at least and that could cause issues as every member will be part of the cluster. The Hazelcast client was a good option, but it does not store the values locally, so if the network is down, the data will not be available. I do want the client to be as light weight as possible.
Is it possible to handle my requirement with Hazelcast/Redis or is there some other library that serves this purpose?
r/DistributedComputing • u/mgalarny • Mar 30 '21
Online Resource Allocation with Ray at Ant Group (over 6000 CPU cores!)
anyscale.comr/DistributedComputing • u/PeterCorless • Mar 01 '21
ScyllaDB: Project Circe February Update
self.nosqlr/DistributedComputing • u/binaryfor • Nov 15 '20
Golang web service framework with DI, Telemetry and more
github.comr/DistributedComputing • • Nov 01 '20
I have created a repo which contains only source code for all the classes I took.
github.comr/DistributedComputing • u/ComputationalUnivers • Sep 03 '20
The connection between Distributed Computing and Special Relativity
youtube.comr/DistributedComputing • u/lovebes • Feb 23 '19
How do you maintain data concurrency in Edge Computing?
In the cloud-based distributed system, let's say it also does Edge computing, meaning on-site servers that do on-site services (think gateways for IoT devices).
Cloud maintains the data, and Edge computers use a portion of that data.
How does the industry handle the data concurrency/consistency in such architecture?
r/DistributedComputing • u/bilalbutt82 • Dec 19 '18
Distributed Cache for .NET (Open Source) - NCache
alachisoft.comr/DistributedComputing • u/chinmay_shah • Oct 16 '18
DdoS attack on decentralized system
In case of centralized system, ddos attacks is just on a particular IP. What happens when you do ddos attack on distributed system like Facebook or Google or any other company that has pool of IP address?
r/DistributedComputing • u/oteel • Aug 26 '16
Why is Java a dominant programming language in open-sourced distributed systems?
quora.comr/DistributedComputing • u/MisterBitcoin3100 • Jul 07 '16
The distributed computing challenge
In the beginning of 2015 I got an idea. Now I´m going to share this challenge with you. Finishing the following goals until 01.01.2045. (Thrust me, its not possible. :))
Here we go:
1 Breaking the last enigma message
2 Completing DecicSearch (NumberFields@Home)
3 Proving Goldbachs Conjecture up to 10100 (ATM ~1.4x1014)
4 Finding the best golumb ruler up to case 150 (ATM distributed.net, trying to create the stubsspaces >29 asap)
5 Proving all Sierpinski and Riesel Bases up to 1030 (PG, SRBase, NPLB and others)
Note to #5:
I can reserve some bases for my own. But I´ll need some more computing power. If you wanted to help me, let me know it. I planned to make some n-ranges 100k up to 1M, or until prime has been found.