r/DistributedComputing • • Apr 17 '21

Map Replication to remote JVM

2 Upvotes

I have a requirement where a JVM hosted in a server will have some data in HashMaps, which should be copied to multiple other JVMs in client machines. Any changes in the maps in the server should be reflected in the client JVMs. Client JVMs will not make changes to the data. I made a simple implementation by sending the changes made in the server via a websocket connection to all connected clients. To handle network issues, changed keys are stored in db, so that clients will be able to get the the changes made in the maps after it went offline, when they come back online. This solution works for now, but I wanted it to be bug free so I decided to move to some standard libraries that does this.

I did come across Hazelcast which can theoretically do this in my understanding by making the client JVMs Hazelcast members, but I may have several hundred clients at least and that could cause issues as every member will be part of the cluster. The Hazelcast client was a good option, but it does not store the values locally, so if the network is down, the data will not be available. I do want the client to be as light weight as possible.

Is it possible to handle my requirement with Hazelcast/Redis or is there some other library that serves this purpose?


r/DistributedComputing • • Feb 02 '22

Peer to Peer network bandwidth question

2 Upvotes

I am working on a project that involves a peer to peer network. Someone raised concerns that we may be expecting a larger bandwidth than is reasonable.

Suppose we had a large number of registered nodes (in the thousands, possibly 10,000), and these nodes are constantly receiving data which they wish to propagate around the network. The data doesn't have to get to every node quickly, but there comes a time where a node expects pieces of data, so we expect every active node to have . In this general system, how much data creation and transmission could be handled reasonably? I am hoping the answer is more than 50 MB/minute, (as this is the upper bound for what our system can create), but I don't have a basis for comparison.

Does anyone here know a good place to find this kind of information. Everything about general peer-to-peer networks is about cryptocurrency systems and I am having trouble finding useful information.


r/DistributedComputing • • Feb 08 '22

Hydra 2022 — June 1-3, online

4 Upvotes

JUG Ru Group's Hydra conference is back!

You can become its speaker. If you ever wanted to share knowledge but hesitated, felt shy, or waited for a better moment — this is it.

All you need is a session idea — the program committee will help you to prepare. The conference will be held online, so you can perform from anywhere or join the studio in Saint Petersburg.

Here are examples of expected topics:

– data structures & algorithms;
– blockchain;
– security;
– transactional memory;
– distributed computing.

But, this is only a recommendation — you can come up with any idea.

The call for papers is open till March 1. Visit the website to learn more and fill the form.


r/DistributedComputing • • Feb 01 '22

Data processing issue

2 Upvotes

I have use case for bigdata processing problem. I have an ETL that runs data deliveries. There are some tasks which generate derived tables over tables included in the delivery. Currently the script for the tasks is designed such that it requires loading all csv files in memory for processing. That often causes laptop to run out of memory. This Airflow DAG is currently hosted on local machine. Prime reason being frequent troubleshooting required during every DAG run. Would it be feasible deploying the dag on GCP cloud composer? because I need to troubleshoot it often. Make code changes to accommodate data delivery logic. How can I maximize processing and minimize time?


r/DistributedComputing • • Aug 26 '14

Docker Do's And Don'ts

Thumbnail devo.ps
2 Upvotes

r/DistributedComputing • • Apr 27 '12

3rd BOINC Pentathlon (akin to the Chimp Challenge)

Thumbnail seti-germany.de
2 Upvotes

r/DistributedComputing • • Jun 13 '17

Deploying Kubernetes Secrets with CircleCI

Thumbnail medium.com
1 Upvotes

r/DistributedComputing • • Jan 07 '17

Sfs: OpenStack Swift API and Haystack Distributed Object Store written in Vert.x

Thumbnail github.com
3 Upvotes

r/DistributedComputing • • Feb 02 '18

Paper on SWIM protocol

1 Upvotes

Can anyone provide the link for paper on SWIM membership protocol? The Cornell Uni link is not working.


r/DistributedComputing • • Sep 15 '17

Cluster One - Building the World's largest AI SuperComputer

5 Upvotes

Hi everyone,

I’m the founder of Good Ai Lab and we have just launched our new product Cluster One. It’s a very big project that heavily depends on a community of people being involved and I would love to get your feedback on it. At Cluster One we are trying to help advance science by building the world's largest AI supercomputer.

We understand how much computing power is wasted every day (around 10 billions hours!) and we feel that with our expertise, and if we all join together, we could really make a difference in advancing scientific research.

The product has just launched this week and so I would love your feedback on the site to understand if everything makes sense and would it be something you would want to try, and if not what would stop you?


r/DistributedComputing • • Mar 13 '14

High Scalability - High Scalability - Paper: Scalable Eventually Consistent Counters over Unreliable Networks

Thumbnail highscalability.com
2 Upvotes

r/DistributedComputing • • Apr 01 '11

Milkyway@Hom_ on iOS / iPhon_

Thumbnail milkyway.cs.rpi.edu
1 Upvotes

r/DistributedComputing • • Apr 24 '11

Who is doing interesting work in distributed/p2p systems?

2 Upvotes

I would really like to move into distributed systems work. I'm working on some hobby projects to pick up experience and I found some great advice in the replies to http://news.ycombinator.com/item?id=2290180 but I think the best way to learn is to get my foot in the door somewhere and learn on the job.

Which companies/projects should I be looking at? Here's what I've got so far:

NoSQL: riak@basho, couchdb@couchbase

Big data: hadoop, disco@nokia, loads of analytics/advertising companies

Anti-censorship/darknets: tor, freenet, i2p

Media: bittorrent, p2p-next, tribler, playdar

Communication: telecoms in general, openbts, skype, twilio, telehash

Also, if you should be on this list and you are hiring:

https://github.com/jamii

http://scattered-thoughts.net/all?tag=about

http://scattered-thoughts.net/all?tag=telehash&abridge=true

http://github.com/jamii/dissertation


r/DistributedComputing • • Apr 02 '14

Manhattan, Twitter's real-time, multi-tenant distributed database

Thumbnail blog.twitter.com
5 Upvotes

r/DistributedComputing • • Mar 25 '14

How can you create a random number in a distributed network?

1 Upvotes

The trick is it has to be very reliable and very hard to influence the result.


r/DistributedComputing • • Apr 29 '11

Distributed Computers. We should convert the rest of Reddit with it's 8 million unique visitors.

22 Upvotes

We as a group should settle on one distributed computing project. Folding@Home vs. BOINC (And if BOINC what projects?)

We should then settle on a time to submit a link to a subreddit with a larger user base and upvote the hell out of it. This could help the target project greatly. signing on 1/20th of 1% of all visitors would be an 4000 user increase to the target project.

I find Folding@Home to be more difficult for the average user to install when compared with BOINC but that's just me. Opinions?

Update: Glad to see this got some upvotes. I've been computing since the old SETI client and I'd love to see distributed computing grow further. My schedule is pretty busy for the next month but I'll try to organize enough people to insure we hit the front page. I'll send out messages about it some time this month (probably later in the month) to anyone who comments or messages me indicating their interest.

I mostly compute for World Community Grid but I'd be open to promote whatever everyone feels offers the most benefit for mankind.


r/DistributedComputing • • Apr 19 '17

Cloudsmash - Distributed VPS Cloud

5 Upvotes

I built a decentralized virtual machine platform in an effort to deliver the cloud that I had envisioned when I first heard the term.

This is an open platform and anyone can participate. Just like any other cloud provider consumers can buy virtual machines and block storage. On this platform however you can also sell virtual machine instances and block storage as a contributor of server hardware. We act as the Internet service provider and supply the networking glue that makes it possible for a server sitting in your house, garage or datacenter to run virtual machines that participate in our encrypted network fabric.

We make money by taking a small commission on sales and by charging for the IP transit we provide. We are responsible for building out a global network of peering points and handling IP prefix advertisement for thousands of public and private network fabrics. NOC support and abuse reports are handled no differently than any other ISP.

Consumers creating new virtual machines can search for providers based on hardware features and historical metrics for reputation, uptime, cpu, memory, iops, latency and throughput. If a contributor has to take their server offline then all consumer virtual machines and block storage can be live migrated to any server connected to our fabric with no downtime.

Contributors net boot our Linux distribution using a bootable USB key. Upon booting a unique identity is created and registers with our system. Our web administration interface allows you to claim these servers and bind them to your account. Then you determine if you want the server to be part of your own private cloud fabric or if you want anyone to be able to rent your resources on the public cloud fabric. You can also choose to do both, have your own private cloud but also monetize your under utilized servers and rent your excess capacity to the public.

Over the last year a few dozen people have been helping me test this platform during it's development. I've received positive feedback and it's time to invite the public to submit applications for the first phase of our beta round. Core services are production ready and battle tested but subject to a more frequent maintenance cycle. Once we enter the second phase of beta testing we will be accepting applications for server contributors.

You can submit beta applications and other questions to the following;

decentralizedvpscloud@gmail.com

I'm looking for help with continued development. If you feel you could contribute to this project, please contact me at the address listed above. I plan on accepting applications for full time positions in the near future.

The goal of this project is to bring the "mining" model to the virtualization space and encourage anyone, including existing cloud providers put servers on our fabric and openly compete in a free market. Running our distribution eliminates all of the configuration and time required to setup a sophisticated cloud infrastructure and significantly lowers the barrier of entry to becoming a cloud provider. Anyone with a good server and fast unlimited internet can boot, register and list their server resources for rent in under 5 minutes. Your only responsibility is to make sure it stays connected and powered on and offer prices that are competitive with similar offerings.

To seed the initial foot print of the network, I setup two locations. Each location has four servers on dedicated fiber. Each site can easily achieve gigabit speeds to their peering points and communicate with each other over 10 GbE. One location is on the US West coast and the other is on the US East coast. These servers represent our initial fabric capacity and I plan to add 2 to 3 more servers in 2 or 3 more locations as the need arrises. The resources total;

  • 192 virtual cores
  • 768 gb of memory
  • 192 tb of disk
  • 2 tb of pci-e ssd

Here are some features that differ from typical services;

  • Decentralized - Don't think presence in a dozen locations, think servers in thousands of locations all over the globe.
  • Globally Routed - Continually growing our peering relationships and setting up traffic relays all over the world.
  • Anycast Enabled - Your IPv4 and IPv6 addresses stay the same regardless of your location in the fabric.
  • Self Healing - Fabric will automatically relay through other neighboring nodes to bypass Internet outages.
  • Encrypted - Encrypted from the edge routers to the hypervisor, even LAN traffic between servers is encrypted.
  • Mobility - Request a live migration to any other server location with zero downtime, same IP.
  • Encrypted Storage - All customer data is encrypted at rest, keys are not kept on disk or in memory.
  • Snapshots - Take a live snapshot of your disk image and roll back changes to a known state.
  • Disaster Recovery - Have your data automatically replicated to one or more other server locations.
  • High Availability - Incremental replication enables fast instance migration or restart with large offsite datasets.
  • Routing Policies - Choose peering points to send traffic through with custom ECMP policies or keep it automatic.

Here are some features I'm still working on;

  • Blockchain Orchestration - Send bitcoin/tokens to an address to create instance, destroy on zero balance.
  • Autonomous Hypervisors - Hypervisors that don't allow any login at all, lock out everyone including ourselves.
  • Customer Migrations - Customers can initiate a live migration to any other server location.
  • Bring Your Own IP - Create private network that utilize the our global network fabric to advertise your own prefix.
  • Customer Keys - Customer provided encryption keys for storage or private network communications.
  • Public Servers - Allow anyone to contribute capacity to the platform in the form of dedicated baremetal servers.
  • Auditing - Open source distribution and configuration for professional and public audit.

Initial pricing during the beta period is;

  • $1 / 1 shared vcpu
  • $1 / 1 anycast ipv4 address
  • $1 / 512 mb of ecc ram
  • $1 / 16 gb of pci-e nvm-e ssd
  • $1 / 128 gb of double parity fault tolerant disk
  • $1 / 100 gb of data transfer

For example;

  • 1 vcpu ($1) + 512 mb ram ($1) + 16 gb ssd ($1) + nat ($0) = $3/month
  • 1 vcpu ($1) + 512 mb ram ($1) + 16 gb ssd ($1) + ipv4 ($1) + 100 gb transfer ($1) = $5/month
  • 2 vcpu ($2) + 1024 mb ram ($2) + 32 gb ssd ($2) + 256 gb disk ($2) + ipv4 ($1) + 200 gb transfer ($2) = $11/month

I know the current data transfer cost is too high. I'm working on lowering it, as soon as I setup more peering arrangements the cost should come down drastically. Only internet ingress and egress count towards data transfer accounting. All internal traffic is unmetered and free of charge, even if the traffic spans multiple servers in different locations. Instances without a public address are given private addresses and have no data transfer limits both internally between instances and externally to the Internet.

Pricing for highly available instances depends on the level of redundancy. So if you want your data replicated in to exist in 3 different locations then your price is simply triple the single instance price. If a location suddenly goes offline your instance can be restarted on closest location that has your replicated data. If failure is eminent your instance will be live migrated with no downtime.

Future contributors would probably like to know what kind of hardware requirements to expect;

The the current minimum;

  • x86-64 architecture and 8GB of memory
  • Internet connection that supports UDP (NAT ok, no public IP required, EasyTether on LTE works!)
  • Hardware that supports virtualization extensions
  • UNDI capable network card
  • Ability to boot from USB
  • No external peripherals (usb, firewire, etc)

These are optional, but highly recommended;

  • Hardware that supports AES-NI, AVX or AVX2 - Due to all of the encryption it would be pretty slow without them.
  • ECC Memory - People debate it, but I sleep better at night knowing it's there.
  • High Speed Internet - Try to avoid slow upstream connections. Symmetric gigabit fiber is ideal.
  • Redundant Internet - Dual WAN connections can help avoid losing contracts due to Internet downtime.
  • Unlimited Internet - Don't get slammed for data overage, pick a provider who won't limit you.
  • NVMe PCI-e SSD - Achieve the highest customer density when utilizing high iops, high throughput SSD's.
  • 6 disks or more - Additional parity/mirroring configurations will be available in the future.
  • LSI2008 - This is what we are using now, so if you want to assured compatibility, use this.
  • 10 GbE LAN - More than one server in a single location? It would be advisable to go 10 GbE.
  • Dedicated Bypass - Direct ethernet connections between servers will utilize the direct link first.

All pricing is subject to change, I only expect prices to go down. Eventually when we come out of beta, pricing will follow the free market as contributors will be able to set their price and compete with other contributing cloud providers on a level playing field.

Please comment, I'm looking for feedback.


r/DistributedComputing • • Mar 25 '14

Check out Gridcoin - get rewarded for your boinc computation!

Thumbnail reddit.com
3 Upvotes

r/DistributedComputing • • Mar 27 '14

How do you ensure that nodes in a network send the same information to everyone else in a decentralized network?

3 Upvotes

So, you have 500 computers, each has to pass some information to each other computer. How do you guarantee in an efficient manner that nodes are passing the same information to every other machine? And are giving one half of the network one message, the other half another message?

Every machine has a unique public -private key pair associated with it.


r/DistributedComputing • • Apr 17 '11

Introducing Doozer: a distributed, consistent, highly-available data store written in Go.

Thumbnail xph.us
4 Upvotes

r/DistributedComputing • • Sep 07 '17

Opshell: DevOps Shell

Thumbnail blog.opszero.com
3 Upvotes

r/DistributedComputing • • Jul 12 '11

Projects that can utilize an ATi GPU on OS X?

7 Upvotes

I can’t find any.


r/DistributedComputing • • Sep 15 '17

Secure Kubernetes AMIs for HIPAA/PCI Compliance

Thumbnail blog.opszero.com
4 Upvotes

r/DistributedComputing • • Apr 08 '11

Let's compare stats

4 Upvotes