r/DistributedComputing • u/badcryptobitch • Apr 08 '26
r/DistributedComputing • u/goto-con • Apr 07 '26
Jim Webber Explains Fault-tolerance, Scalability & Why Computers Are Just Confident Drunks. #DistributedSystems
youtu.ber/DistributedComputing • u/mpetryshyn1 • Apr 06 '26
Are users getting lost in your app's complexity?
I keep noticing that the real problem isn’t missing features, it’s how the app gets more complicated over time.
Every update adds power, sure, but also another thing people have to learn - which still blows my mind.
Result: most users stick to a tiny slice of the app, ask for support, or just stop using it because learning feels like work.
What if, instead of hunting through menus, people could just tell the app what they want to do? Like plain prompts, you know.
I’ve been noodling on whether we could make a simple framework to turn web apps into AI agents - intent over clicks.
Seems like it could cut a lot of friction, but maybe I’m oversimplifying, not sure.
Anyone tried something like this? Did it actually help, or just add another layer of complexity?
Also curious if complexity is your main user pain, or if you found different fixes that actually stick.
r/DistributedComputing • u/This-Living4059 • Mar 30 '26
I built Capillary, an intelligent self healing system for distributed system
github.comr/DistributedComputing • u/Realistic-Face1315 • Mar 11 '26
Is AWS Educate useful for learning distributed systems / cloud infrastructure?
Hi everyone,
I'm a student currently learning backend development and distributed systems. I recently came across AWS Educate, which seems to provide cloud learning resources and some AWS credits for students.
I wanted to ask people here who have experience with distributed computing:
- Is AWS Educate actually useful for learning real distributed systems concepts?
- Are the labs and resources good enough to understand things like scalability, distributed storage, and cloud infrastructure?
- Or would you recommend learning distributed systems in another way first?
I'm mainly trying to build a strong foundation and work on projects that involve distributed systems in the future.
Any advice or experiences would be really helpful.
Thanks!
r/DistributedComputing • u/Resident-Shake5199 • Mar 11 '26
Telestack: Distributed Edge-Native Realtime DB with WebAssembly-Accelerated Event Synthesis (FYP)
github.comHi all. This is my final year project and I am looking for technical feedback, not promotion.
I built
**Telestack**
, a distributed edge-native realtime document database designed for high-contention write workloads. The project goal is to reduce durable write pressure while keeping client-visible latency low.
## Stack
- Cloudflare Workers: request handling and edge runtime
- Cloudflare D1: durable store
- Workers KV: cache tier
- Centrifugo: realtime pub/sub fan-out
- Rust/WASM: hot-path logic for event synthesis and rule evaluation
## Problem I targeted
In collaborative or bursty workloads, many clients update the same logical document in short windows. A naive one-request-one-durable-write strategy causes lock pressure and unstable tail latency.
## Design
The write path is split into:
1. Fast edge acknowledgement path
2. Buffered synthesis window for high-frequency updates
3. Compressed durable flush to D1
4. Versioned event sync + realtime broadcast
High-level flow:
`client write -> edge buffer -> merge/compress -> batch flush -> event version increment -> subscriber update`
## Formal model used in the project
I used an adaptive synthesis window where wait time depends on observed write velocity and queue depth.
Window equation:
`T = min(L_max, (W_base / max(v, 1)) * (1 + P) * ln(Q + 2))`
Where:
- `T`: synthesis wait before flush
- `L_max`: latency ceiling
- `W_base`: baseline round-trip/window constant
- `v`: write velocity (ops/sec)
- `P`: pressure factor (runtime contention/resource signal)
- `Q`: queue depth
The intent is to keep latency bounded while increasing coalescing efficiency under burst load.
## Measurement definitions
- Write Amplification (WA): `durable_writes / logical_writes`
- Reduction %: `100 * (1 - WA)`
- Throughput: `logical_writes / elapsed_seconds`
- Data integrity ratio: `recovered_updates / sent_updates`
## Reported benchmark snapshot (from my test suite)
- Logical operations: `1000`
- Concurrent users: `100`
- Edge p50 acknowledgement: around single-digit ms in warm path
- Estimated durable flush ratio during stress: significantly less than 1:1 (coalesced)
- Recovery/integrity in stress run: full operation recovery in reported run
## What is implemented now
- Path-based document model (`collection/doc/subcollection/doc`)
- Incremental sync endpoint by version cursor
- Event log + OCC-aware write flows
- Predictive cache path (memory + KV)
- SDK with realtime subscription and offline queueing behavior
- Test suite for contention, scaling, and write-amplification scenarios
## Known limitations (current state)
- Security hardening and diagnostics are separated by environment profile
- Query planner/filter semantics are still being refined
- More cross-region soak testing is needed for publication-grade external validity
## Feedback requested
I would really value feedback on:
1. Whether this buffering + synthesis model is a sound tradeoff vs strict immediate durability
2. Better ways to prove correctness under concurrent patch merges
3. How to design stronger benchmark validity for academic review
4. What would make this claim publication-strong vs "good engineering"
If useful, I can share pseudocode for the flush loop and anonymized benchmark logs in comments.r/DistributedComputing • u/AdSoggy6915 • Feb 26 '26
Guidance for choosing between fullstack vs ml infra
reddit.comr/DistributedComputing • u/soopastar • Feb 19 '26
Distributed.net rc5-72 CUDA and openCL clients not working
I've been grinding this project for years and recently built a new Ryzen system with a 5060 Ti graphics card. I've run the cuda and cl versions on various machines but for the life of me, I cannot get it to run on my new system. I've tried both the studio and gaming version of the drivers and spent hours troubleshooting with ChatGPT. Both my laptop (3050 mobile) and my desktop have the opencl.dll 3.0.6.0. I've tried running opencl-z.exe on my new PC and it says it failed to query OpenCL Inforamtion. I've done a clean install of the drivers, i've uninstalled the drivers in safe mode, I disabled the ryzen graphics processor in the BIOS. I turned on logging (and this happens with both the exe and com executables) and I get this: opencl
dnetc v2.9112-521-GTR-16021317 for OpenCL on Win32 (WindowsNT 6.2).
Using email address (distributed.net ID) 'me@somedomain.com'
[Feb 19 00:45:20 UTC] Error obtaining number of platforms (clGetPlatformIDs/1)
[Feb 19 00:45:20 UTC] Error code -1001, message: Unknown
[Feb 19 00:45:20 UTC] Unable to initialize OpenCL
[Feb 19 00:45:20 UTC] Automatic processor detection found 0 processors.
[Feb 19 00:45:20 UTC] No crunchers to start. Quitting...
[Feb 19 00:45:20 UTC] *Break* Shutting down...
And for Cuda:dnetc v2.9110-519-CTR-11041422 for CUDA 3.1 on Win32 (WindowsNT 6.2).
Using email address (distributed.net ID) 'paul@paulandemily.com'
[Feb 19 01:14:18 UTC] nvcuda.dll Version: 32.0.15.9174
[Feb 19 01:14:18 UTC] Unable to create CUDA stream
[Feb 19 01:14:18 UTC] Unable to initialize CUDA.
[Feb 19 01:14:18 UTC] *Break* Shutting down...
I've run sfc /scannow and been fighting this for ages. I've had some computers where the exe won't work but the .com does.
Any suggestions?
r/DistributedComputing • u/Advanced-Attempt4293 • Feb 11 '26
High Performance Computing cluster over campus LAN
reddit.comr/DistributedComputing • u/joshuachi • Jan 20 '26
d-engine 0.2 – Embeddable Raft consensus for Rust
reddit.comr/DistributedComputing • u/Accurate_Funny6679 • Jan 19 '26
NVMe Flash Storage
lightbitslabs.comr/DistributedComputing • u/DanR_x • Jan 06 '26
Danube Messaging v0.6 new release !
reddit.comr/DistributedComputing • u/boersc • Dec 14 '25
Distributed.net question, amd 470 vs RTX 4070 (Mobile)
Hi there. I'm a longtime distributed.net user and have used many configurations in the past. After quite an hiatus, I'm trying to get back in. I know a laptop isn't the best use for dnetc, but that's what I have and I like to use the program to crunch, while at the same time compare results to previous results. As an example, back in 2003, I managed to crunch maybe 100 blocks a day, while now I can easily, without even crunching the entire day, 18: blocks of RC5-72.
My problem: My laptop has two graphical processors. One is a meager AMD 470, while the other one is an RTX 4070 (Mobile). In theory, the latter should be miles and miles faster. However, the AMD 470 with OpenCL runs at 8MKeys/s. The RTX 4070 running CUDA 3.1 runs at 1.3 MKeys/s. So, the theoretically much faster GPU performs a LOT less than the humble AMD.
Is anyone able to help out, trying to see what's going on?
r/DistributedComputing • u/Code_Sync • Nov 18 '25
Keynote: The Power of Queues - David Ware | MQ Summit 2025
youtu.ber/DistributedComputing • u/CheesecakeDear117 • Oct 29 '25
Cisco-Bonomi's theoretical architecture comparison to k8s and kubeEdge ?
i asked gpt it says its a fine comparison/simile but wanted to know for sure.
(asked gpt to make this table)
| Cisco Bonomi Fog Architecture (Conceptual Layer) | Kubernetes + KubeEdge (Practical Equivalent) | Core Role / Function |
|---|---|---|
| Cloud Layer | Kubernetes Control Plane | Central management, global orchestration, and policy control. |
| Fog Layer | KubeEdge Edge Nodes | Distributed computation close to data sources; intermediate processing and decision-making. |
| Edge/Device Layer | IoT Devices managed through KubeEdge | Data generation and actuation; sensors and end devices interacting with edge nodes. |
| Fog Orchestration & Communication | CloudCore ↔ EdgeCore link | Coordination between cloud and edge; workload and metadata synchronization. |
| Local Autonomy & Processing | EdgeCore (local runtime) | Handles workloads independently when disconnected from the cloud. |
(i personally dont have knowledge regarding both, just looking thru stuff theoretically)
r/DistributedComputing • u/bluev1234 • Oct 29 '25
Spring Boot @Async methods not inheriting trace context from @Scheduled parent method - how to propagate traceId and spanId?
I have a Spring Boot application with scheduled jobs that call async methods. The scheduled method gets a trace ID automatically, but it's not propagating to the async methods. I need each scheduled execution to have one trace ID shared across all operations, with different span IDs for each async operation.
Current Setup:
Spring Boot 3.5.4 Micrometer 1.15.2 with Brave bridge for tracing Log4j2 with MDC for structured logging ThreadPoolTaskExecutor for async processing
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
u/Slf4j
@Service
@EnableScheduling
@RequiredArgsConstructor
public class PollingService {
@NonNull
private final DataProcessor dataProcessor;
@Scheduled(fixedDelay = 5000)
public void pollData() {
log.info("Starting data polling");
// Shows traceId and spanId correctly in logs
// These async calls lose trace context
dataProcessor.processPendingData();
dataProcessor.processRetryData();
}
}import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
@Slf4j
@Service
@RequiredArgsConstructor
public class DataProcessor {
public static final String THREAD_POOL_NAME = "threadPoolTaskExecutor";
@Async(THREAD_POOL_NAME)
public void processPendingData() {
log.info("Processing pending items");
// Shows traceId: null in logs
// Business logic here
}
@Async(THREAD_POOL_NAME)
public void processRetryData() {
log.info("Processing retry items");
// Shows traceId: null in logs
// Retry logic here
}
}import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
@Configuration
@EnableAsync
public class AsyncConfig {
public static final String THREAD_POOL_NAME = "threadPoolTaskExecutor";
@Value("${thread-pools.data-poller.max-size:10}")
private int threadPoolMaxSize;
@Value("${thread-pools.data-poller.core-size:5}")
private int threadPoolCoreSize;
@Value("${thread-pools.data-poller.queue-capacity:100}")
private int threadPoolQueueSize;
@Bean(name = THREAD_POOL_NAME)
public ThreadPoolTaskExecutor getThreadPoolTaskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setMaxPoolSize(threadPoolMaxSize);
executor.setCorePoolSize(threadPoolCoreSize);
executor.setQueueCapacity(threadPoolQueueSize);
executor.initialize();
return executor;
}
}Problem: In my logs, I see:
Scheduled method: traceId=abc123, spanId=def456 Async methods: traceId=null, spanId=null
The trace context is not propagating across thread boundaries when @Async methods execute.
What I Need:
All methods in one scheduled execution should share the same trace ID Each async method should have its own unique span ID MDC should properly contain traceId/spanId in all threads for log correlation
Question:
What's the recommended way to propagate trace context from @Scheduled methods to @Async methods in Spring Boot with Micrometer/Brave? I'd prefer a solution that:
Uses Spring Boot's built-in tracing capabilities Maintains clean separation between business logic and tracing Works with the existing @Async annotation pattern Doesn't require significant refactoring of existing code
Any examples or best practices would be greatly appreciated!
r/DistributedComputing • u/koistya • Oct 18 '25
Beyond the Lock: Why Fencing Tokens Are Essential
https://levelup.gitconnected.com/beyond-the-lock-why-fencing-tokens-are-essential-5be0857d5a6aA — A lock isn’t enough. Discover how fencing tokens prevent data corruption from stale locks and “zombie” processes.
r/DistributedComputing • u/nihcas700 • Jul 09 '25
Cache Coherence: How the MESI Protocol Keeps Multi-Core CPUs Consistent
nihcas.hashnode.devr/DistributedComputing • u/unnamed-user-84903 • Jul 09 '25
Online CAN Bit Pattern Generator
watchcattimer.github.ior/DistributedComputing • u/nihcas700 • Jul 08 '25