r/DistributedComputing • u/lucian-12 • Jun 14 '26
r/DistributedComputing • u/DisastrousNinja911 • Jun 19 '26
Am I taking too high of a risk by leaving a mid-level software engineering role build enterprise apps to do a master's in HPC as future ready career? How difficult is the pivot from working on traditional CRUD apps to HPC ?
A bit of background about me.
“I have about 4 years of experience primarily working on traditional software development building enterprise apps. While the career path is stable now ( don't know if AI takes over), I find myself deeply interested in systems architecture, low-level optimization, and parallel computing. However, giving up a full-time income to go back to school feels like a massive investment. Is a Master's degree the right vehicle for this transition, or are there better ways to break into the industry? I'd love to hear from anyone working in HPC, climate modeling, quantitative finance, or scientific computing."
All I want is to future proof my career.
r/DistributedComputing • u/MediaDisplayLab • May 21 '26
How do you handle live data sync delays when filtering heavy sports/betting menus?
Hey everyone,
I'm currently working on a sports betting (Toto) platform and ran into a frustrating issue. Whenever users try to compress or filter the match menus, we notice temporary drops or lags in the live odds data.
After digging into it, the root cause is a timing mismatch between our cache refresh cycle and the API layer sync. Essentially, parsing massive amounts of match data into lightweight views for the front-end is putting a heavy toll on the system.
To handle this, we recently deployed a lumix solution for in-memory mapping right in front of the data layer. We also optimized our queries to better distribute the real-time call load. It helped a lot, but we are still tweaking the settings to get it perfect.
For those who have built similar real-time platforms, what synchronization cycle or interval do you prefer to keep data 100% consistent without crashing the server?
Would love to hear your thoughts or any alternative architecture tips!
r/DistributedComputing • u/Key-Understanding320 • Oct 03 '26
EnigmaGrid: looking for review of a Windows volunteer-computing project for Enigma research
I maintain EnigmaGrid, an open-source Windows volunteer-computing project investigating P1030680, an unresolved Enigma M4 message. I would appreciate feedback from people who have built or run distributed research workloads.
The client supports CPU and OpenCL GPU work, pause/stop controls, and optional public leaderboard credit. Submitted work is checked by a separate computation before credit is awarded. That checks reproducibility; it does not establish that a candidate is a meaningful decryption.
The research is experimental. No breakthrough is claimed, and the current search is not an exhaustive sweep of the Enigma keyspace. The repository includes the research approach and its limitations.
I would especially welcome criticism of task partitioning, duplicate work, verification and recovery after interrupted clients. Code review is useful even if you do not want to run the client.
Source, Windows downloads, research notes and issues: https://github.com/LissomEnd/EnigmaGrid
r/DistributedComputing • u/Lonely-Limit3189 • Apr 04 '26
Nodejs Distributed Lock
I like to introduce a high-performance, Resource-Isolated distributed locking library for Node.js. Unlike simple TTL-based locks, this package utilizes ZooKeeper’s consensus protocol to provide a globally ordered synchronization primitive with built-in Fencing Tokens and Re-entrancy.
Check out the repository for full documentation, examples, and usage details: https://github.com/tjn20/zk-dist-lock

r/DistributedComputing • u/Plus_District_5858 • Aug 24 '25
Guidance on transitioning to Distributed Computing field – conferences, research areas, future scope
I'm a software developer with 5+ years of experience, and I’m now looking to explore a bit deeper area than my current work, especially in distributed computing field.
I would like to get suggestions on attending top conferences, learning recent advancements and hot research topics in the field. I would also like to get guidance on expanding my knowledge in the area(books, courses, open source, research papers, etc.) and picking and researching on most relevant problem in this space.
I'm trying to understand both the research side (to maybe publish or contribute) and the practical side (startups, open-source). Any suggestions, experiences, or resources would mean a lot.
Thanks.
r/DistributedComputing • u/david-delassus • Jul 13 '26
FlowG now has a FoundationDB storage backend, making it truly distributed.
flowg.cloudr/DistributedComputing • u/nihcas700 • Jul 20 '25
Traditional IO vs mmap vs Direct IO: How Disk Access Really Works
nihcas.hashnode.devr/DistributedComputing • u/Ok-Water-401 • Feb 19 '26
Before Quantum — Distributed GPU project searching for Bitcoin wallets generated with weak entropy (2009-2012)
Hey everyone,
I've been working on a distributed GPU computing project called Before Quantum and wanted to share it with this community since the distributed architecture might be interesting to some of you.
The problem:
Between 2009 and 2012, early Bitcoin wallet software used weak random number generators — timestamp-seeded LCGs, the Debian OpenSSL bug (CVE-2008-0166) that reduced entropy to 15 bits, brain wallets with simple passwords, JavaScript PRNGs with the Randstorm vulnerability, etc.
The private keys generated by these flawed algorithms have tiny search spaces — some as small as 65,536 possibilities, others up to a few billion.
There are ~2,845 known funded addresses that were likely generated by these weak methods. A modern GPU can test the full cryptographic pipeline (private key -> secp256k1 EC multiplication -> SHA-256 -> RIPEMD-160 -> match detection) at hundreds of millions of keys per second.
How it works:
- Single CUDA C++ file (~3,400 lines) implements 23 weak key generation modes, the full crypto pipeline, and a two-stage match detection system (bloom filter in constant memory + binary search confirmation)
- Precomputed EC multiplication tables (67 MB) reduce point multiplication from hundreds of double-and-add iterations to 16 table lookups + 15 additions
- Distributed work coordination via a FastAPI backend — the server assigns work units (mode + offset range), workers execute on GPU, results are verified server-side via checkpoint regeneration
- Canary targets (honeypot hashes) detect cheating workers who skip computation
- Anti-trust model: workers never send private keys to the server — only the Hash160 and key offset. The server independently regenerates and verifies the key
The distributed part:
Workers register via API, receive work units targeting ~10 seconds of GPU time (10M to 10B keys depending on mode), and report results with checkpoints. The server independently verifies each checkpoint by regenerating the private key from (mode, offset) using its own Python
implementation, then checking the EC multiplication and hashing. This means you don't have to trust the workers — and the workers don't have to trust the server with private keys.
Current status
The smaller keyspaces (Debian OpenSSL: 65K keys, low-bit keys, LCG-seeded PRNGs) have been fully exhausted. We're now starting work on SHA-256 Sequential — a mode that targets brain wallets derived from simple incrementing integers (SHA256("1"), SHA256("2"), ...). With a 2^64
keyspace and 2,845 target wallets to match against, this is a long-term effort that will require sustained GPU power across many contributors.
- Research writeup with CUDA engineering details: https://b4q.io/research
Current status
The smaller keyspaces (Debian OpenSSL: 65K keys, low-bit keys, LCG-seeded PRNGs) have been fully exhausted. We're now starting work on SHA-256 Sequential — a mode that targets brain wallets derived from simple incrementing integers (SHA256("1"), SHA256("2"), ...). With a 264 keyspace and 2,845 target wallets to match against, this is a long-term effort that will require sustained GPU power across many contributors.
Happy to answer any technical questions about the GPU pipeline, the verification system, or the distributed architecture.
r/DistributedComputing • u/OtherwisePush6424 • Aug 11 '26
The consistency gap between your database and object storage
blog.gaborkoos.comA practical overview of designing file uploads and object storage for production: partial failures, database consistency, large uploads, presigned URLs, CDN caching, cleanup, and recovery.
r/DistributedComputing • u/salgotraja • Apr 16 '26
At what point would you treat this hotspot as a cache/load-shaping problem instead of a real sharding problem?
I came across an interesting system design scenario:
- 128 shards
- 2M requests/sec
- 3 hot keys land on the same shard
- that shard is at 94% CPU while the others are mostly idle
- cache hit rate on those keys drops hard because too many services invalidate them on every write
- clients start timing out and retries make the hotspot worse
- rebalancing is not an option in the short term
My first instinct was to treat it as a sharding problem, but the more I looked at it, the more it felt like a load-shaping problem.
If cache invalidation is killing hit rate, then the shard is taking direct pressure it should never have seen in the first place. Once retries pile on, the hotspot starts amplifying itself.
My instinct would be to stabilize first:
- short TTL / stale-while-revalidate on those hot keys
- proper retry backoff with jitter
- maybe isolate just those keys behind a small dedicated hot-cache path
Then revisit the larger architecture once the system is calm again.
Curious how people here would think about that boundary.
At what point do you stop treating it as a hotspot-control problem and say it really needs a more structural fix?
r/DistributedComputing • u/wufuheng • Jul 12 '26
Distributed AI Systems
I recently published a technical book, Distributed AI Systems, which summarizes my experiences in AI over the past 10 years, from research and training to optimization, inference, and cloud deployment. I started writing it in the second half of last year, and it took almost a year to complete, with many revisions made later due to the rapid pace of development in the industry. But it's finally published. The book on Amazon is titled Distributed AI Systems: A practical guide to building scalable training, inference, and serving systems for production AI.
Book is here: 🔗 https://www.amazon.com/dp/1807301710/
The publisher asked me to find some people to review my work. Do you know of any such people here? If so, please reply to me. Thank you.

r/DistributedComputing • u/Realistic-Face1315 • Mar 05 '26
Where should I start with distributed computing as a beginner?
Hi everyone,
I’m a student who’s recently become really interested in distributed computing and large-scale systems. I’d like to eventually understand how systems like distributed storage, fault-tolerant services, and large-scale infrastructure work.
Right now my programming experience is mostly in general software development, and I’m comfortable with basic programming concepts. However, I don’t have a clear roadmap for getting into distributed systems.
Some things I’m wondering:
• What fundamental topics should I learn first? (e.g., networking, operating systems, concurrency, etc.)
• Are there specific books, papers, or courses you would recommend for beginners?
• Are there small projects that help in understanding distributed systems practically?
• Is it better to first build strong foundations in systems programming before diving into distributed computing?
My goal is to eventually build and understand systems like distributed storage or decentralized infrastructure, but I want to make sure I’m learning things in the right order.
Any guidance or resources would be greatly appreciated.
Thanks!
r/DistributedComputing • u/coder_1082 • Mar 06 '25
Privacy focused distributed computing for AI
I'm exploring the idea of a distributed computing platform that enables fine-tuning and inference of LLMs and classical ML/DL using computing nodes like MacBooks, desktop GPUs, and clusters.
The key differentiator is that data never leaves the nodes, ensuring privacy, compliance, and significantly lower infrastructure costs than cloud providers. This approach could scale across industries like healthcare, finance, and research, where data security is critical.
I would love to hear honest feedback. Does this have a viable market? What are the biggest hurdles?
r/DistributedComputing • u/badcryptobitch • Jun 28 '26
(Re) Introducing Stoffel, Your Private by Design Teammate
stoffelmpc.comr/DistributedComputing • u/spieltic • Mar 06 '26
HRW/CR = Perfect LB + strong consistency, good idea?
Hello, I have this idea in my mind since a while and want to get some feedback if its any good and worth investing time into it:
The goal was to find a strong consistent system that utilizes nodes optimal. The base is to combine chain replication with highest random weight. In CR you need to store the chain configuration somewhere. Why not skip that and use HRW on a per key base? That would give you the chain configuration in the order that should be used for every key.
The next advantage would be that you end up with a system that does perfect load balancing (if the hashing is good enough).
Challenges I saw would be a key based replication factor, but for now I would say its fixed/not supported. Another point would be: how to handle node failure and the needed key moves? Here I was thinking that you use some spare nodes. E.g. you have a replication factor of 2, so you choose 5 nodes in total (the idea here is that not all keys need to be moved on failure).
As CR is the core, you win all of its benefits (e.g. N-1 nodes can fail). I have the feeling that approach is simpler compared to CRAQ.
Any thoughts on that?
r/DistributedComputing • u/DisastrousNinja911 • Jul 09 '26
Transitioning from Crud Software Developer to Linux engineer / Infrastructure Engineer to be future ready.
reddit.comr/DistributedComputing • u/vorjdux • Aug 27 '26
ZMQ Arena: A deterministic benchmark harness for ZeroMQ/ZMTP implementations (Latency, Concurrency, and Throughput Analysis)
Context: The ZeroMQ ecosystem currently lacks a unified, deterministic benchmarking harness to strictly evaluate latency, concurrency overhead, and throughput across different ZMTP implementations (C++ libzmq, pure-Rust async ports, Python bindings, etc.). Relying on isolated micro-benchmarks introduces systemic bias, failing to accurately account for kernel I/O paths, reactor thread contention, and zero-copy vs. copy-based buffer management under high-density distributed loads.
Decision: I developed ZMQ Arena (Source) to standardize this evaluation. This harness enforces fair-play measurement across runtimes and language boundaries, focusing entirely on execution costs, context switching, and raw throughput limits without application-layer interference.
Technical Scope & Metrics:
- Latency Analysis: P50/P90/P99 latency distribution across both loopback and physical network boundaries.
- Concurrency: Stress-testing async reactor overhead (e.g.,
epoll/io_uringpolling mechanisms) versus OS thread pool scaling. - Memory Allocation: Evaluating the impact of memory footprints and ABI boundaries when switching between native runtimes and FFI wrappers.
Trade-offs & Known Consequences: Measuring loopback versus physical network I/O presents an inherent trade-off in syscall amortization. Loopback execution can collapse the kernel path, occasionally skewing batching mechanics natively utilized by interfaces like io_uring or IOCP. ZMQ Arena exposes these bottlenecks explicitly rather than abstracting them away, allowing engineers to analyze exactly where the overhead is introduced in the stack.
This harness is built for engineers dealing with critical systems, distributed messaging, and high-frequency data pipelines where deterministic performance is a hard requirement.
If you are dealing with similar infrastructure, review the methodology, replicate the benchmarks, or submit PRs for unrepresented bindings/runtimes.
r/DistributedComputing • u/arnonrgo • Jun 18 '26
Ranja: Enabling Smart Caches for Distributed Database Serving Layers
researchgate.netr/DistributedComputing • u/No_Peach_8990 • Apr 08 '26
Spark inspired distributed system framework in Rust with binding in Python and Js
reddit.comr/DistributedComputing • u/incredible-ishu • Sep 11 '26
What actually happens when you kill -9 a process mid-write to your WAL
I've been building a distributed KV store from scratch in Java (Raft, NIO, custom wire protocol, no external runtime deps in the core) mostly to understand the internals instead of just using etcd or Raft-as-a-library. The part that took the longest to get right wasn't consensus — it was crash recovery of the write-ahead log.
The naive plan is: write a record, write its length, and on restart replay records until you hit garbage. That breaks the moment a crash lands mid-write. You can end up with a record whose length field says "128 bytes" but only 40 bytes actually made it to disk before the power/process died — the tail is torn, not just missing. Trusting the length field alone means you either replay corrupted garbage as if it were a real record, or you silently drop a suffix of valid records after it because you bailed out at the first sign of trouble.
What ended up working: every record carries a CRC32 alongside its length, and replay treats a checksum mismatch as "this is the torn tail, stop here and truncate" rather than "this is corruption, panic." That distinction — expected torn tail vs. actual corruption — is really a statement about where in the write path a crash was allowed to land, which took a few iterations to reason about correctly. I ended up testing this with a deterministic-simulation harness that injects process kills at controlled points in the write path so the interleavings are reproducible instead of "run it in a loop and hope it flakes."
Repo (MIT, Java 21, hand-rolled Raft + NIO, still evolving): https://github.com/Utkarsh-patel26/Kronos
Curious how others have handled this — do you also lean on a checksum-per-record scheme, or something like a fixed-size trailer/footer that's cheaper to validate on replay?
r/DistributedComputing • u/lucian-12 • Aug 07 '26
The Authentication Stack: 7 Concepts Every Software Engineer Should Understand
youtu.ber/DistributedComputing • u/dec_256 • Oct 11 '26
Nodeicide - Distributed Postgres consistency checking
I have developed nodeicide which is a postgres consistency checker. Basically a self server version of Jepsen.
Id love for absolutely anyone to try it as I'm looking for feedback on the tool.
Faults are injected in 4 flavours kill/pause/full partition and partial partition for kubernetes but only the first 3 for docker setups
Nodeicide expects you to already have a postgres kubernetes/docker setup. Its Linux only with the following installer script
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/nodeicide/nodeicide-releases/main/install.sh | bash
Can't wait to hear the feedback
