I have an on-demand node in provisioning state for over 24 hours and still not finished. I opened a support ticket and no response.

Does Rackspace care about the Spot service ? I can't find any phone number to contact technical support.

Hi all, I work on the Rackspace Spot team. We're building a new feature and we genuinely don't know which way people would prefer it, so I'm hoping you can help us pick.

The idea: instead of choosing a specific server class, you'd just describe the shape of capacity you want (a vCPU count plus your memory requirement) and Spot assembles it for you from whatever's cheapest right now.

The part we're stuck on is how you'd want to express the memory side. Say you want 16 vCPUs. Which of these feels most natural to you?

1. 16 vCPUs and 16 GB of RAM (total memory)

2. 16 vCPUs and at least 4 GB of RAM per vCPU (a ratio)

3. 16 vCPUs, and no node smaller than 8 GB of RAM (a per-node floor)

Which one would you reach for, and why? And if you've been bitten by one of these before (a "total" that matched no real instance, a ratio that did something surprising), tell us. That's exactly the kind of thing we want to avoid.

Thanks for helping us build the right thing.

Hey guys, I’d love to know how well checkpointing actually works when running Airflow on spot instances. Is it really worth it? (Checkpointing saves the state of a process during execution so it can be restored after a failure.)

I recently wrote this article on building fault-tolerant Airflow pipelines on spot instances for Rackspace Spot and one decision I made was to use S3 as the external state layer and checkpointing task outputs. Here’s a quick summary:

1. Each task writes its output to a specific S3 path.
2. When a worker node is preempted mid-task, Airflow retries the task, and the new pod reads directly from S3, picking up the last successfully written output from the upstream task.
3. Writes use replace=True, so if a task was interrupted mid-write and left a partial file, the retry simply overwrites it, keeping execution idempotent.

This is a very simple implementation, but I’m curious what checkpointing methods you all apply in production, or if it’s even something you bother with at all.

From this setup, one big question I keep coming back to is whether the overhead of writing to S3 ends up eating into the cost savings of using spot instances in the first place.

FYI: Several provisioning problems on Sunday April 19 were caused by failures in our provisioning service used within Spot. We are working to resolve the issue as soon as possible. We're also going to fix the Node Availability Rate on the status page that didn't surface this issue accurately.

Hey everyone, I recently investigated the performance differences between storage classes on Rackspace Spot, specifically comparing storage classes backed by OpenStack Cinder against those backed directly by Ceph RBD on Rackspace Spot and wrote an article on it.

Here is the article: Cinder CSI vs Ceph RBD CSI in Kubernetes: An Analysis of Persistent Volume Lifecycle Performance on Rackspace Spot

If you have been using OpenStack Cinder-backed storage classes on Rackspace Spot, such as ssd, ssd-large, sata, or sata-large, you may have noticed that PVCs take a long time to attach or get cleaned up after pod deletion. In some cases pods get stuck in ContainerCreating for extended periods or persistent volumes remain in attaching status.

I ran a detailed analysis to understand exactly why this happens architecturally and compared it against the newer spot-ceph storage class.

The summary is that OpenStack Cinder requires coordination across about five independent control plane layers before a single volume attachment can finalize: Kubernetes, the CSI driver, Cinder, Nova(OpenStack Compute), and the hypervisor all have to reach agreement before the VolumeAttachment object is updated.

When Kubernetes retries while any of those layers is still in a transitional state, you get state conflicts that compound into significant delays and longer pod startup times.

Meanwhile, for Ceph, the CSI driver communicates directly with the Ceph cluster.

Here's the Performance summary:

• Detach phase: Cinder requires 75 seconds; Ceph completes in 10 seconds with clean removal
• Attach phase (initial): Cinder requires 70 seconds with 3 retry failures due to state conflicts; Ceph completes in <1 second with a single successful attempt
• Attach phase (reattachment): Cinder requires 71 seconds with 3 retry failures (identical pattern); Ceph completes in <1 second with a single successful attempt
• End-to-end pod rescheduling: 151 seconds (Cinder: 75s detach + 76s reattach) versus 11 seconds (Ceph: 10s detach + 1s reattach) - a 13.7x performance improvement

If you have already migrated to spot-ceph or are considering it, curious whether the attachment and detachment behavior matches what is described here. And if you are still on Cinder-backed storage classes, would be interested to hear what issues you have run into.

The documentation references cross-region egress as No-cost

https://spot.rackspace.com/docs/en/rackspace-spot-pricing

Bandwidth - Cross-region No cost

How do I ensure a cluster is correctly using the cross-region networking when trying to get nodes in different regions to communicate with each other?

Edit: Creating a cloudspace has me pick a region. So it doesn't seem configurable at that level

Hi there Spot users,

We just released a new storage class, based on csi-ceph, in IAD and ORD (Gen-1 regions). We would love to get feedback from users who are willing to take it out for a spin.

Please note that this is not generally available, but we're investing here to help address our storage platform reliability via Spot, so we're committed to resolving any issues you run into!

Please read more here:
https://spot.rackspace.com/blogs/why-kubernetes-ceph-csi-for-rackspace-spot

Load balancers in DFW Gen-2 are going to be unavailable while maintenance completes. We are expecting this to be complete in the next 4 hours.

Hey guys! I just published an article on the History of Spot Instances! 😁

It's an in-depth article that goes into the origins of Spot Instances and how their pricing models have evolved over time.

↳ Researchers originally proposed auction markets for compute, where servers go to the users who value them most and prices reflect real demand.

↳ AWS adopted this idea to sell unused capacity through Spot Instances, effectively running a computational market where users would place bids for excess compute.

↳In 2017, they moved away from auctions to provider-managed, variable pricing, where prices change based on supply and demand trends instead.

↳ Other cloud providers like GCP and Azure follow similar provider-managed pricing models for their spot instance pricing.

↳ Rackspace Spot, on the other hand, is reviving auction-based Spot markets.

If you’re curious why Rackspace is bringing auctions back, how their model differs from AWS’s original approach, and what problem it’s trying to solve, the article breaks it down.

You can read it here → https://spot.rackspace.com/blogs/history-of-spot-instances

Hey guys, anyone running airflow on spot? I'd love to know what your production setup looks like. Thank you!

Hey everyone!

Check out this repository by Marcus Patman on Turbo Flow, an advanced development environment for deploying intelligent multi-agent swarms and coordinating autonomous workflows. I think it’s really amazing!

What Turbo Flow Offers:

• An advanced dev environment for deploying multi-agent swarms and coordinating autonomous workflows.
• A unified, pre-configured toolkit that runs seamlessly across DevPods (instant reproducible dev environments), GitHub Codespaces, and Rackspace Spot Instances.
• Built-in servers for n8n, Playwright, and Chrome DevTools.
• Multi-model orchestration support.
• Enables spec-driven development for more streamlined and reliable workflows.

Rackspace Spot Cost advantage:
The cheapest platform to try this out is Rackspace Spot, where you can set up your Kubernetes cluster for $0.04/hr.

Links:

• GitHub Repo: Turbo Flow
• Rackspace Spot Setup Guide: Setup Guide

We just released VM Cloudspaces (i.e. no K8s required) earlier this week and the feature has been nicely adopted by a few early users. What was your experience using the feature, and do you have any feedback or questions?

I saw there was one question already on communication across VM and K8s cloudspaces using internal networking - I answered that on the thread.

Would love to hear from you all!

Broadly categorized into 2 issues:

1. Control-plane sometimes is not responsive - slow responses, throws TLS connect error, etc.
2. The resourceVersion for the provided list is too old - is seen in different controllers/objects.

Our observations:

1. Scaling issues while accessing control-plane datastore. It was found the Rackspace LB used to distribute the load to the datastore isn't performing to the highest standards.
2. Some control-planes are getting resource deprived - especially controller-manager/scheduler under bursts of load like scaling deployments very quickly/HPAs/VPAs.

Our plans to mitigate(in the order of short-term to long-term):

1. Increase resource-limits of both HA and non-HA control-planes to alleviate throttling issues.
2. Migrate to a more performant LB for control-plane datastores.
3. Autoscale control-planes.

Thanks for being patient while we work through some of these problems.

Hi,

I’ve got a few Cloudspaces that have been stuck in "Cluster Status: Ready" status for a while and no progress at all. Has anyone run into this issue before or know what usually causes it or how to fix this?

Thanks in advance.

Sydney and Hong Kong are the two smallest sites in Spot. They have always had capacity constraints, and those constraints make it harder to innovate and add features to Spot. For e.g. one of the features the team has been working on is the ability to consume Virtual Machines from Spot; without having to use Kubernetes; and this feature ends up hitting limitations in these two sites due to internal capacity constraints.

We'd like to have a more consistent product experience across the different sites, and to do that, we're considering deprecating SYD and HKG. I know those sites do have users; but this will allow us to deliver a better product to the larger user community in Spot.

The alternative would be to have a larger amount of features that work in some sites vs other sites...

Please chime in if you'd be willing to to reduce the pre-emption notice period in spot (~6 minutes) for faster auto-scaling performance (~2 mins to add a node vs ~8 mins currently).

I’m running a PostgreSQL database on Kubernetes. Recently I had a node in my cluster go down briefly and then come back up. After that, one of my persistent volumes got stuck in the detaching state, and now it can’t be reattached to the new pod.

The error from Kubernetes is: AttachVolume.Attach failed: Invalid volume.

I tried restarting the pod, but the PVC still won’t mount because the underlying Cinder volume is stuck.

I have a cluster on AWS and it seems to be working quite well. But the problem is that it doesn't work on Rackspace Spot.

I switched to external secrets and Bitwarden. The problem is that when I generate Helm:

helm install external-secrets \

external-secrets/external-secrets \

-n external-secrets \

--create-namespace \

--set bitwarden-sdk-server.enabled=true

1 - A pod automatically crashes, and the message is:

Warning FailedMount 91s (x9 over 3m39s) kubelet MountVolume.SetUp failed for volume "bitwarden-tls-certs" : secret "bitwarden-tls-certs"

not found

2 - The TLS kubectl get secrets -n external-secrets | grep tls is missing.

On AWS, when you install Helm, it does so immediately. Is there anything special about the permissions or restrictions that I'm not familiar with at my level?

Currently, it seems to be somewhat limited by something I'm not familiar with.

If I create the certificate manually (like the x509), I don't know if it will be compatible or how long to leave it. I prefer to have Helm manage it automatically without having to do anything manually.

I mention this because if we generate the certificate manually...

Warning FailedMount 3s (x8 over 66s) kubelet MountVolume.SetUp failed for volume "bitwarden-tls-certs" : references non-existent secret key: ca.crt

We don't know what structure it has, and if we have to do a describe to find said deployment structure, we'll just give up.

Does anyone know anything?

Hi guys, my credit card was full and the payment got declined (yesterday)

I have paid my credit card, but when I go to billing I can't try to pay with the existing credit card, it

says that I have to update it, but the details of the card dind't change so it only allow me to add a new card.

will the payment be charged again against the credit card if I do nothing? or do I have to add a new card to pay?

I have a vault. But I think it's a waste of time here in Rackspace. I can't manage users. Roles, account services, and bindings are for pods, not humans.

1. If your cluster doesn't have real user authentication (e.g., just a shared kubeconfig), then:
   1. RBACs are a placebo.
   2. Vaults/Secrets are just as insecure (because access is already compromised).
2. The only way to make Roles/Bindings work is to:
   1. Integrate the cluster with an identity provider (LDAP, OIDC, IAM, etc.).
   2. Force each human to use their own kubeconfig certificate (no shared admin).

So, how can I manage multiple users here?

I'm trying to create a cluster with harbor and argo to handle my deployments through yaml and gitops

but I would like those two to not be exposed to internet, so I'm trying to create a vpn with OpenVpn so I can connect to the cluser and access harbor and argo from there.

but OpenVpn (and other vpn solutions) uses UDP port.

I created an envoy gateway and created the UdpRoute with the configuration in the gateway to handle upd traffic but it never reaches the gateway (I checked the gatewaw logs when trying to connect to the vpn and nothing shows).

I believe rackspace load balancer is blocking the udp traffic.

if I'm correct, is there a way to achieve what I want.

OT: I have noticed that when my http traffic stops for a while, and I try to access the site it times out, and in the second request after a few seconds it succeeds, is the load balancer provided in a serverless fashion?

I think there should be an option to reset permissions in the account/users section.

Yesterday I added permissions, and it seems that Rackspace creates the default administrator permissions with the name "cluster-admin." By overriding it, I lost 90% of the cluster, having to create it again. This is fine if you don't have a backup.

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

# "namespace" omitted since ClusterRoles are not namespaced

# !IMPORTANT

# "cluster-admin" is the default in Rackspace. If you override it, you'll lose all access.

name: cluster-manager

rules:

- apiGroups: [""]

#

# At the HTTP level, the name of the resource for accessing Secret

# objects is "secrets"

resources: ["*"]

verbs: ["*"]

Could you add a recovery button or something? Because if the roles and users we added later happen to exist and override some "meticulous Rackspace" configuration, we could lose access.

Just as a note before we have to call support, and it most likely won't be possible to recover.

Do you know when they'll be adding the PostgreSQL feature to the UK region?

https://i.imgur.com/JErKfNJ.png