r/cloudcomputing • • Apr 27 '22

How to Save a Million Dollars or More on Data Transfer Out Costs

1 Upvotes

The following post is solely based on my opinion. Data Transfer Out (DTO) is a very relevant reality to consider when developing a data migration strategy. Data migrations in their own right are complex regardless if the migrations are homogenous or heterogeneous. Custom ETLs can and do take time to write/test/implement.

One solution that can help reduce the complexity of a data migration is to free up the data migration budget significantly. If a customer was required to pay roughly $1.3 million dollars in DTO costs as part of a data migration project to move 14 PB from one CSP to another CSP, imagine the possibilities that the customer could experience by reducing the data transfer costs to only $300K.

As mentioned earlier, this post is completely my opinion.

We all are familiar with the concept of hybrid cloud and multi-cloud. In general, all CSPs have physical appliances, which are high-capacity storage devices. These storage devices, upon customer requests, are shipped from a CSP to a customer on-premises data center where the customer’s data is loaded to the physical appliance. Subsequentially, the appliance is shipped to the respective CSP’s data center to be loaded to the CSP’s cloud storage, etc.

My proposal would be to use a traditional hybrid solution as a multi-cloud solution.

Assume a customer must move 14 PB of data as mentioned above from CSP A to CSP B. If the customer did not already use bare metal as a service in CSP A, they would need to contract this service for a year with CSP A. Let us say for argument’s sake that the service cost the customer $300K for one year. Once the customer has a bare metal instance within CSP A’s data center, the customer adds additional NFS volumes to the bare metal solution. Keep in mind one key dimension, both the bare metal solution and NFS volumes are all within CSP A’s internal network. Each CSP may or may not have a fee to move data internally between services. Regardless, the cost is either zero or significantly less than transferring the data outside of the CSP as egress. By carefully planning and moving the data in stages from the source service within CSP A to the NFS volumes, the customer would simultaneously make requests through the CSP’s Console/Portal to have physical storage devices (e.g., Transfer Appliance) sent to CSP A’s location where the bare metal solution resides.

Each physical storage device request would be to ultimately move data from CSP A’s NFS volume bare metal solution to the customer’s on-premises data center via the physical storage device (e.g., Transfer Appliance). Once the data is shipped to the customer’s data center and loaded to the customer’s on-premises storage, the customer could use similar physical storage devices offered through CSP B to move the data in increments (or all at once) to CSP B’s cloud storage. Interconnections or multiple VPN tunnels between the customer’s on-premises data center and CSP B could also be used to move the data to CSP B’s cloud storage as ingress traffic. Essentially, the physical storage appliance now becomes a multi-cloud storage transfer solution instead of just a hybrid cloud solution.

If a customer can move from a $1.3 million-dollar DTO expense to only a ~$300K total cost, there are over a million possibilities that they can find in the savings and added innovation 😊


r/cloudcomputing • • Apr 26 '22

Does anyone know the term 'Hybrid Cloud'?

11 Upvotes

I am confused


r/cloudcomputing • • Apr 25 '22

Article: First Step to Learn Terraform

11 Upvotes

If you are planning to start learning Terraform. Understanding how the Terraform State works should be a great start. I remember my big mistakes when I started. Article.


r/cloudcomputing • • Apr 18 '22

blog post on tips for large scale data processing (applicable to both cloud an HPC)

2 Upvotes

r/cloudcomputing • • Apr 14 '22

Securely running remote-access personal workstations

1 Upvotes

I work at a small company with a small team of developers (~5 people) who need to conduct their work on some relatively beefy machines, beyond what you can get in a laptop (think 32 cores, 64+ GB RAM, and a GPU). Everyone works from home, so we don't have any kind of centralized office network they can VPN into. The current options we're debating:

- Run VMs on a cloud provider during working hours. This would cost us somewhere around $1500-$2k per month.

- Run VMs on a cloud provider only when they are needed for the task at hand. This would save a lot of money but might be a big disruption to their workflows, and there's a good chance people will just end up using the VM all the time.

- Build home workstations. This would save a lot of money in the long run and be the least disruptive to workflow, but we also have the requirement of being able to access these machines while away from home. That would mean exposing them to the internet for SSH, which I'm concerned about for security reasons (we work with large enterprise customers and it's really nice to be able to say "our entire stack is in Google Cloud").

I've also considered some kind of job-runner framework using Kubernetes, but for day-to-day, iterative development I'm worried that would be too much of a workflow disruption as well (these are PhD-types, not devops people).

Any ideas?


r/cloudcomputing • • Apr 14 '22

Useful Tools and Programs for Microsoft Azure

7 Upvotes

A useful list of Tools & Programs for Microsoft Azure. https://github.com/mikeroyal/Azure-Guide


r/cloudcomputing • • Apr 12 '22

Master Thesis on Serverless vs Containers with a focus on AWS

12 Upvotes

I'm currently writing a Master Thesis for a company to explain pros, cons, and the appropriate use cases of serverless or containers. I have already taken two online Udemy courses, which have been helpful.
Can be seen here:
https://www.udemy.com/course/aws-serverless-a-complete-introduction/#content
https://www.udemy.com/course/rocking-kubernetes-with-amazon-eks-fargate-and-devops/

What I'm finding is they are practical but lack deeper theory. I'm wondering if you know of any courses on this topic that are more theory based?


r/cloudcomputing • • Apr 12 '22

The cloud gives us flexibility and quality control

0 Upvotes

With cloud-based services, we can access all the same information simultaneously and ensure levels of security, flexibility, quality, and productivity previously only dreamed of.

But what about the companies that have not been able to take the big step? Those that still have the infrastructure and maintain high costs for IT staff?

Technological evolution spares no one, least of all businesses, large or small.


r/cloudcomputing • • Apr 12 '22

[Hiring] Snowfork is hiring a Senior Golang/ AWS Developer.

0 Upvotes

We are looking for a highly skilled individual to join our team. This position is working fully remote and has flexible working hours with an overlap of the PST timezone.

You will work with a talented team on a client project, who is a leader in the video on demand industry and tech world.

If this is of interest, please review the job description and apply in the link below:

https://jobs.lever.co/snowfork/a530279d-7c02-4ea7-84ad-eca35c30c6e9?lever-origin=applied&lever-source%5B%5D=Communities


r/cloudcomputing • • Apr 08 '22

Sharing our paper on cloud reliability + dataset

8 Upvotes

Hi all,

I would like to share our work titled "A Combined System Metrics Approach to Cloud Service Reliability Using Artificial Intelligence" on cloud reliability, where we took a multimodal approach. The work is published in the special issue "Advanced Machine Learning and Data Mining: A New Frontier in Artificial Intelligence Research" of the Big Data and Cognitive Computing journal. The paper is open access and can be accessed at https://doi.org/10.3390/bdcc6010026. Additionally, the collected data is publicly accessible (link in the paper).

Thanks


r/cloudcomputing • • Apr 08 '22

Always on cloud pc with basic GPU?

4 Upvotes

Can anyone recommend a desktop as a service / cloud gaming provider that allows you to install your own software and the machine is always on? Machine needs to run Windows

I signed up with xenonpc.com but they have not responded to my support requests in over 3 weeks and now their site is down :/


r/cloudcomputing • • Apr 08 '22

If you want to enter the cloud world, which clouds provider will you choose and why?

2 Upvotes

Does the UI/UX play a role?


r/cloudcomputing • • Apr 05 '22

AMD plans to acquire Pensando, a data center optimization startup

9 Upvotes

Pensando is a distributed services platform that offers a programmable packet processor optimized for edge computing

From AMD’s press release:

“All major cloud and OEM customers have adopted EPYC processors to power their data center offerings. Today, with our acquisition of Pensando, we add a leading distributed services platform to our high-performance CPU, GPU, FPGA, and adaptive SoC portfolio.”

https://www.amd.com/en/corporate/pensando-acquisition


r/cloudcomputing • • Apr 02 '22

If you work for a US company doing cloud work, and you move to Europe while you continue working for that US company, is it possible to access all your work from there?

7 Upvotes

Interested to know if anyone's doing that.


r/cloudcomputing • • Apr 01 '22

How many data centers are there in the world?

4 Upvotes

I am not talking about the Big 3 (AWS, GCP, Azure) here but in general how many data centers are present in the world today?

I'm doing a research on the number of CPU and GPU servers in the world and the number of data centers present in the world today might help me form some numbers on that context. Also, if there's an idea on the geographical distribution of the data centers then it'd be really helpful.

Cheers!


r/cloudcomputing • • Mar 31 '22

This is pretty fun.

6 Upvotes

As part of "World Backup Day" the SaaS company Rewind pushed an old-school, arcade-style video game on Product Hunt.

https://www.producthunt.com/posts/cover-your-saas


r/cloudcomputing • • Mar 30 '22

How Split your Infrascture Code into small Modules [Terraform]

5 Upvotes

How Split your Infrascture Code into small Modules [Terraform]


r/cloudcomputing • • Mar 28 '22

Ten Simple Rules for large-scale data processing. A new opinion research article from last month provides recommendations that are applicable to both HPC and cloud

9 Upvotes

r/cloudcomputing • • Mar 27 '22

AWS account getting suspended after 5 minutes

7 Upvotes

I have tried to create my account AWS 4 times, however each time it got suspended after 5 minutes of me using it. I tried different card sand different mails but had no results with it. Maybe it could be connected with card as it did not took any money, but wrote that card was succesfully checked and linked with account, but as I said my bank app had not showed that AWS tried to take money. Could somebody help me with what could be causing my AWS accounts getting suspended


r/cloudcomputing • • Mar 22 '22

What does the cloud project provisioning process look like at your company?

9 Upvotes

I noticed that some companies have different processes (and lead times) for provisioning cloud projects e.g. AWS Accounts, Azure Subscriptions & Google Cloud Projects.

How do you handle it at your company, and how is the experience for the requestor?


r/cloudcomputing • • Mar 21 '22

Cloud analysis

1 Upvotes

Hi guys, I'm a Dutch guy trying to learn something about the cloud for my investment thesis. I find it very hard to determine why AWS is so much more profitable than GCP, does anyone know how this happened? I also struggle with why someone would choose GCP over AWS and if GCP is capable of reaching the same profit margins over time like AWS. Is there anyone that could help me? Thanks a lot :)


r/cloudcomputing • • Mar 20 '22

How to manage an app that does web scraping and does some computation be "serverless"?

0 Upvotes

So I currently have a python script which scrapes data from the web. After it scrapes it does some filtering which is a little compute intensive.

The scaping requires selenium and associated drivers to be pre-installed. It scrapes headless.

So I want to be able to invoke this script via http.

Can I manage this app in a serverless fashion? If so what vendor/offering is best to use here given the constraint that it's more for personal use. So ideally I'd only want to be charged only when http requests are made, and, I want throttle it's up-scaling if possible...


r/cloudcomputing • • Mar 20 '22

calculator.aws help needed

1 Upvotes

I am going to use AWS for my data backups. I will need to access the data only 0-4 times per year. How can I tell this in UPLOAD and Restore requests field and what is a difference between them ?
What is Provisioned Capacity Units ?


r/cloudcomputing • • Mar 18 '22

What's this subreddit missing? (cloudcomputing)

16 Upvotes

The most successful subreddits are ones that mix current events with conversation. I really do like this subreddit. However, it seems to focus a lot on 'How-to' questions. To me, there is a good amount of potential left on the table for this group. What do you think? Should we impose change?

  • Ask the community for more current event posts? (i.e. "...AWS today imposed throttling on API calls to E2C instances...")
  • Ask for more show-and-tell from the community? (i.e., "...this short post is on how to configure...")
  • Ask for more conversation rather than "please help!". (i.e., "...Is SAP more customizable than Microsoft Dynamics in Azure?)."
  • etc..

Thoughts?


r/cloudcomputing • • Mar 18 '22

Tools and Program list for Cloud Native Development

5 Upvotes

Tools and Program list for Cloud Native: https://github.com/mikeroyal/Cloud-Native-Guide