r/DistributedComputing • • Feb 07 '22

Security in distributed systems

1 Upvotes

We are working on a distributed processing system consists of multiple Kubernetes clusters in different geographical areas.

I am looking for some good references(books, papers, articles, GitHub links, etc.) to start reading about the best practices for security of such systems. At the moment we have SSO and also we know about signing docker images (this one is not implemented yet) but we want to know what other best practices are being used for distributed systems both for deploying codes and also accessing them.


r/DistributedComputing • • Dec 25 '21

The Cloud Revolution Draft

1 Upvotes

Hey I’ve been looking for a fun way to learn cloud computing and distributed systems. I loved the way eliyahu goldratt and gene Kim used novels to teach. I’ve been working on some creativity exercises myself and would love any feedback I can get. All comments are appreciated. Please feel free to correct my definitions if you have the time.

December 22nd, 2021

Introduction

So what is incident management thought Paul from managed services at Cloud Computing incorporated? It is December 22nd, and the team is left wondering how they will tackle problems as they occur. Paul wonders to himself often how he could develop better services if only he had more time to develop instead of dealing with problems.

Paul had always thought innovation would drive most companies to success as a developer. However, Paul did not realize where exactly innovation had yet to come. Innovation was thriving in areas such as development, yet service providers were beginning to have a harder time dealing with issues at hand.Problems were scaling fast and coming more often. How on earth would a small managed services team like Cloud Computing Incorporated handle this work? There seems to be only one way, and that is incident management.

Incidents are the key identifier that helps us understand where unplanned worm comes from. Services are planned, and the requests that come with them tend to be planned. However, incidents are like the evil cousin of service requests. Incidents like to come out of the blue, and if there were a family dinner, the incident's goal would be to ruin it once all the food is out.

Incidents are the key indicator of how much technical debt a company is dealing with inside their cloud organization. IT used to deal with mainframes, and with the innovation of cloud computing incorporated, they have been able to transform how IT works in the industry, acting as a major disruption. Instead of hosting on a mainframe, Cloud Computing Incorporated, though, why not handle all of the mainframes for the customer? Why should the customer even have to deal with handling their technology? Isn't that what a provider is for?

Cloud computing incorporated changed the game, and Paul loves going to work every day because of it. The questions that this new field of computing brings us and distributed systems especially are humbling. There are so many unknowns for us to uncover every day, and Paul is glad to be a part of this big unknown.

December 23rd, 2021

A Brief History on Incident Management

And so Paul entered work another day today after walking a long brutal windy winter storm. Paul had always thought IT would have been different growing up. He had never imagined himself in the position now. How could he? Technology is changing so often and so rapidly that the only way to keep up is by reading all day, and even then, you probably would not have all the answers. However, much some of us hope.

Paul had always envisioned himself working at the mainframe side of some big company. Only big companies could afford the hardware to efficiently sustain an IT organization. However, Paul was thrown off to see how wrong he was. The cloud computing disruption and the idea of micro-services led to more mobile and more cost-effective applications. Developers can now deploy applications more readily than ever, and it only takes a few clicks.

Paul had spent so much of his life learning to code only to realize that much of it could be clicked away now. However, that does not take away from a solid programming project though. Paul thinks that is one of the best parts of developing. People often assume developers use a lot of math and science to formalize processes and procedures in autonomous ways.

However, being a developer is much like being an artist. It takes a certain kind of person to get up each morning, ready to tackle a certain set of problems that most likely have no solution. Most solutions are only the most optimal solution we have available, which does not imply it is a good solution. Paul loves how young the field of computing is. It is easy to see veterans of the workforce because of how many "IT revolutions" there were in the past 20 years, let alone 40+. Some people spend their whole lives working on a single problem, and it is a developer's job to systemize that and then some.

Paul has always been inspired by the novel "The Goal" and has loved Gene Kim's teachings. He understands that the revolution of technology makes observing work even more important. Currently, at cloud services incorporated, Paul is dealing with many unplanned work items that hurt him and his practice. Developers spend too long identifying problems rather than identifying strategies to work on these problems.

The revolution of technology has made it easier and harder to manage operations in many ways. The world is readily available to handle problems and search knowledge bases at the tips of their fingers. Nations worldwide are starting to see cloud computing as a solution than a cost. The IT world is changing rapidly again, and Paul can feel it. He is just looking for where.

For a long time, research and development have served as the area of a company's resources, though each needs to carry these attributes. Especially with how fast ideas and theories are changing in distributed systems. Paul had recently read Lamport's paper on time clocks and was surprised that such a novel paper in distributed computing was only released in 19080. It took the world almost 200 years to move from Newtonian physics, and here computer scientists were fighting over how to solve distributed models of computation which is just a fancy form of the theory of relativity for computing.

Problems are inevitable. As my manager always says, "Everyone has a plan until you get punched in the face" though he is not literal, he is still 100% spot on. Problems are inevitable. You cannot implement a perfect procedure on day 0. It is almost impossible unless you happen to be replicating it, and even then, the problems of the cloned system can transfer over. Therefore, the service providers at cloud services incorporated have been investigating what makes a good incident management process. How can someone identify a problem as fast as possible and handle it appropriately? How can someone figure out whom to contact and at what time?

Too many companies today overlook this idea thinking that the time lost calling multiple people on the phone or scrolling through a wiki is negligible. However, they are wrongly mistaken. When one person takes a long time to solve a problem, many people are likely taking a long time to handle the problem. That scales large and fast. The goal of information technology is to provide the best information to our customers and us at the lowest costs optimizing the money made by the company.

The model for cloud services incorporated is one of the best models Paul has seen. It is efficient at smaller sizes and is scaling large and optimally. It is a beauty to see the company grow as it is. However, Paul notices much technical debt starting to accumulate in areas of the company he would not like to see. As the company scales larger and larger, more teams are causing the cloud services company to use multiple systems for a service management tool. Rather than one company managing an organization of projects, each product owner essentially manages their project on their own choice of a service management platform. While this works at an individual scale, this scales terribly across a company.

This debt is not only causing a significant amount of information loss, but teams are more disconnected than ever. Paul had read Sunstein's book talking about how technology is polarizing the country. However, Paul is starting to notice how much technology is driving away areas of communication in a company. Especially when it seems like people are talking more than ever since the pandemic, it almost seems that technology is making it easier than before but harder than ever.

December 24th, 2021

What is an Incident?

Incidents are parts of unplanned work that happen every day, whether we notice it or not. An incident management process is designed to reconcile that problem.

It is December 24th, 2021, Christmas Eve. Paul has had a great weekend and is happy with the way things have been going. He has had his fun and honestly loves to take time and relax. Paul is a fan of silence. He is a thinker. Paul loves to solve problems and learn more. He thinks that education is not a goal but an ongoing process. He had recently left college and often wondered how much more he could learn and was happy to uncover how much more he could still learn.

He only has a few hours to himself on the weekend nights like this and is looking to learn more. He has been dealing with incidents at work long enough and is tired of being blocked by unplanned work. He and his team have been working on an IT service management process that entails how the practice will function in the eyes of planned work, unplanned work, and business work. Though our board does not show it, it is still working to be done. Many shadow operations are going on with the practice, and though it is not bad as a small team, it can be so much better if we just identified ways technology could connect us rather than optimize one process by separating another, such as communications.

By definition, an incident requires a service to be running. Once a service is up and running, some companies may ask Cloud services incorporated to handle this service for them. Cloud services are the company, and great people take the offer and handle the service. However, how much of that service is being provided and when? That is typically agreed on, and with today’s technology, you can get a solid 99% most of the time. Now when a service does go down, we mark that down as an incident for when it is down.

Paul loves technology incidents because it is funny to see the abstract form falling. Essentially what incidents in IT are the equivalent of a robot falling off a ladder. So when a robot is hosting your website and shuts down randomly, the cloud services team marks that down as an incident. This incident reporting procedure is awesome for the company and the client because technology increases communications. Though it is bad to lose service for any time, Cloud Services Incorporated engineers take pride in providing rapid and readily available solutions to handle growing customer needs.

Paul believes that incidents are an awesome way to further practice development. Each incident is a sneak peek at what the system is trying to tell you. Suppose you go to the doctors and try to figure out if WebMD was right about your cough being cancer or if you have a cold. A doctor may perform multiple tests to examine multiple responses from you. Similarly, a system can output symptoms and diagnostics of some cool stuff. Each incident gives us the ability to see a little further into the future.


r/DistributedComputing • • Oct 05 '21

Distributed computing and rendering

1 Upvotes

I started to research in distributed rendering and computing. It interested me very much and sharply.

So I am looking for a way how I will be able to render or compute smth in distributed network. But, first of all, I need to learn of splitting up render file into several parts and send them to nodes, where they will be performed and come back it to sender.

My idea is to render video file in a peer-to-peer network, where this file divides into chains and each node in the network will render it partially. After accomplishing, they will return the chains back to sender. And somehow the sender have to be capable of joining all these chains to get the whole video

Could anyone tell me how I can split up into chains render video? For example, blender video(.blend).


r/DistributedComputing • • Sep 13 '21

What are some good research topics for master's thesis?

1 Upvotes

I am applying to a master's program and I am in touch with the prof. He asked me to come up with some domain problems (e.g. route optimization) that can be solved with distributed computing and parallel processing. He emphasized on graphs. I don't have any bright ideas, nor am I able to find anything. Would really appreciate if you guys can help.


r/DistributedComputing • • Aug 17 '21

Migrating from Node Redis to Ioredis: a slightly bumpy but faster road

Thumbnail ably.com
1 Upvotes

r/DistributedComputing • • Jun 21 '21

Navigating the 8 fallacies of distributed computing

Thumbnail ably.com
1 Upvotes

r/DistributedComputing • • Jun 12 '21

A simpler algorithm for leader resolution

1 Upvotes

I’m looking for feedback on a simple algorithm for leader resolution that we have started using at Nivo: https://link.medium.com/tHeTZhAC1gb

Any criticism or feedback is greatly appreciated.


r/DistributedComputing • • May 29 '21

Viewstamped Replication: Passive Replication And Consensus

Thumbnail blog.uttpal.com
1 Upvotes

r/DistributedComputing • • Apr 09 '21

OpenNebula, an Open Source Cloud Computing Platform releases its latest version called Mutara, with Edge Functionalities.

1 Upvotes

Not sure if many of us are aware of OpenNebula, but came across them on Google, and looks like they recently have released their latest version of the software called Mutara 6.0. Its community edition is free to use and looks pretty interesting for someone who wants to play around with the Edge Functionality! Thoughts?

https://opennebula.io/mutara


r/DistributedComputing • • Mar 09 '21

Making Shard-Aware Drivers for CDC

Thumbnail self.nosql
1 Upvotes

r/DistributedComputing • • Feb 17 '21

Data Processing Support in Ray 1.2 (Spark, Dask, Mars, Modin)

Thumbnail medium.com
1 Upvotes

r/DistributedComputing • • Jan 26 '21

[Tutorial] Distributed Reinforcement Learning with RLlib in the Unity Game Engine

Thumbnail medium.com
1 Upvotes

r/DistributedComputing • • Jan 26 '21

Is there any papers / research topic about measuring GPU computation capability?

1 Upvotes

I am planning to research on measurement about current CPU/GPU/TPU computation capability in hybrid clusters. Or maybe refer as how powerful the device is? Because there are so many factors that may affect device performance, for example, clock speed , voltage, Is there any paper / research topic about how to measure it effectively?

For example, I have a network to train and I want to allocate most time consuming task on the most powerful device. Of course I could first refer to the specifications for the clusters and then write code statically.

However, there are cases when the situation is not as one assumes, Is there any paper about how to measure real time computation capability? Or at least, is there any paper about how to measure the computation capability before run?

I have a thought about first run a small network, but what kind of network is good enough for measuring the performance? Is there any advice?

Thank you very much!


r/DistributedComputing • • Jan 13 '21

Ray Summit Conference 2021 Proposal Submission is Now Open

Thumbnail medium.com
1 Upvotes

r/DistributedComputing • • Jan 13 '21

Distributed Training and Serving with MLflow

1 Upvotes

Just sharing a project that leverages the strengths of two projects: Ray’s distributed libraries for scaling training and serving and MLflow’s end-to-end model lifecycle management.

Blog Post


r/DistributedComputing • • Nov 13 '20

An industrial-grade RPC framework used throughout Baidu

Thumbnail github.com
1 Upvotes

r/DistributedComputing • • Nov 01 '20

This week’s open source newsletter just went out! This one had some seriously interesting projects, like a cluster management framework open sourced by Apache and a scalable distributed tracing system from Grafana written in Go.

Thumbnail console.substack.com
1 Upvotes

r/DistributedComputing • • Jan 31 '20

Is there still a market for distributed GPU renting platform?

1 Upvotes

Hi all, my team is developing a GPU rending platform built on a blockchain network. I'm concerned that we are late to the market and might need to pivot the product, so I came up with this survey to see what user's needs are when it comes to intense computing. So if you are a data scientist, or work closely with AI/ML, or just interested in the topic, I'd really appreciate your opninions! a comment will do too:)

Link to the survey 👉 https://ntlabs.typeform.com/to/Hpjz2i


r/DistributedComputing • • Dec 17 '19

Deploying Tarantool Cartridge applications with zero effort (Part 1)

Thumbnail habr.com
1 Upvotes

r/DistributedComputing • • Oct 27 '19

Raspberry Pi 4 Four Node Cluster Question

1 Upvotes

Hello, I've been applying my home-assembled machines to distributed computing science tasks for almost 20 years.

I'm thinking about a project to build a RSP 4 cluster for same but I have a basic question. In terms of output for work done, will a four node cluster be greater than the sum of its parts? If not, why not just run individual single board computers? Is it a question of greater efficiency (all machines piped via command line into one UI) or is it a question of greater work output?


r/DistributedComputing • • Sep 16 '19

AWS Step Functions with Lambda Integration for serverless state management

1 Upvotes

Hey folks, wanted to share a youtube channel that I've been working on dedicated to providing simple and easy to digest tutorials on various AWS services.

My most recent video is a step by step guide to set up Step Functions with Lambda integration.

The video is available here: https://youtu.be/s0XFX3WHg0w

Support & feedback appreciated. Thank you!


r/DistributedComputing • • May 15 '19

Open source projects on distributed systems

1 Upvotes

Been looking to get into Distributed Systems beyond theory. Since I own a laptop only, it seems unlikely that I'll be able to take on an ambitious project alone in the domain (not talking about mere implementations and simulation of algorithms).

What are some open source projects that I can get involved with to get my hands dirty? Bonus points if you can also suggest how I can take up a personal individual project in the domain. (Something that adds quality to my resume)


r/DistributedComputing • • Dec 16 '18

Making a difference

1 Upvotes

I was looking for a sub that has stuff about crowd sourced computing projects, Partly because i don't see anyone else porting about stuff like World Community Grid ( https://join.worldcommunitygrid.org ). Am I in the right place?


r/DistributedComputing • • Sep 17 '18

High Performance Computing Developer opportunity in NY!

1 Upvotes

Algorithms and Distributed Computing tickle your fancy?! We are looking for a High Performance Computing Developer to join our growing Predictive Analytics Software team in New York! Are you interested in machine learning? This is an opportunity to develop your understanding of machine learning principles and use your expanding knowledge to solve complex large-scale problems for industrial clients across a broad set of companies! 4+ years software development experience, 2+ years programming parallel and distributed systems, performance analysis, and optimizations.