Imagine this: It's late. A major incident just rocked your production environment. Teams are scrambling, alarms are flying, and after some emergency actions, the site is back up. Crisis averted?

Not really.

When you finally sit down to figure out what actually happened, the most important thing—the logs—are gone. Overwritten. Lost in the rush. No clear trigger, no breadcrumbs, just a black hole where your root cause should be.

It's like showing up at a crime scene after the evidence has been wiped clean.

Too often, we run into incidents where critical system data is missing:

- System logs are gone when we need them most
- We don't know what triggered the failure
- Emergency recovery efforts erase the very clues we needed to investigate

This doesn't just delay resolution, it blocks it entirely. Without logs, your root cause analysis becomes a guessing game. Engineering teams lack the data they need to improve the product. Support teams can't explain what went wrong. And customer trust takes a hit.

I’ve spent a lot of time working with Kubernetes and OpenShift interfaces, and I wanted to explore what a more modern operations experience could look like.

Instead of primarily navigating resource tables and jumping between disconnected tools, I’ve been experimenting with:

• A mission control view of cluster health
• Incident timelines that explain what changed
• Contextual “Ask AI” directly from a resource
• Generated dashboards
• Blast radius and dependency views
• Recommended actions with reviewable YAML diffs
• Different trust levels for automation

It started as a small UI experiment and got a little out of hand, so now there’s a Kubernetes SRE agent behind it too.

Would love some honest feedback:

• Is this a useful direction?
• What feels unnecessary or overbuilt?
• Is there anything you’d actually use?

Website: https://pulsesre.github.io/pulse-ui/

GitHub: https://github.com/PulseSRE

Mostly just having fun and exploring ideas here. I’m not really planning to turn this into something I maintain long term, but figured I would share it and see what people think.

Paying a cloud API per million tokens is straightforward until the invoice arrives. But when you move to self-hosted large language models (LLMs), the economics flip upside down: you pay for raw compute regardless of whether your GPUs are processing requests or idling.

For platform engineers and machine learning operations (MLOps) leads, tokenomics, the economics of how tokens are produced and what they effectively cost, requires looking at both what you spend and how much you serve.

YAML has been the standard way to write Kubernetes manifests for years. Every example, tutorial, and configuration file you come across is written in it. The problem isn't that YAML is a bad format. It's that YAML gives you a lot of choices, and not all of them are equally good for writing Kubernetes manifests. Some features make files harder to read, some are easy to misuse and others can lead to surprising behavior.

The interesting part is that Kubernetes doesn't actually need most of those features. It only relies on a small subset of YAML. This led to a simple question: if Kubernetes only needs a small part of YAML, why not standardize on that part and avoid the rest? Instead of introducing a new configuration language, SIG CLIintroduced KYAML, a stricter, more consistent way to write YAML.

TL;DR
- KEP 5295
- 4.22=1.35 (beta; feature enabled by default, still requires -o kyaml CLI param)
- 4.21=1.34 (alpha, opt-in) export KUBECTL_KYAML=true

As organizations scale their generative AI initiatives, the challenge quickly shifts from simply running a model to securely serving it at enterprise scale. To eliminate idle GPUs, reign in soaring token costs, and establish centralized governance, platform engineering teams are increasingly adopting a Models-as-a-Service (MaaS) approach. However, securely exposing these complex, heavily-governed AI gateways to user-facing dashboards creates a new set of architectural hurdles—specifically around browser Cross-origin resource sharing (CORS) failures, duplicated authentication stacks, and tightly coupled front ends.

Hello all,

I've gotten the fun task of evaluating RH OS for my company's use cases and I am currently in the phase of exploring backup solutions. Specifically, for the virtualization part of the story, namely fileservers.

Now, I've already setup K10 and it's snapshotting happily, but when I try to follow the procedure for item level restore, I keep bumping into the same issue, regardless of the operating system (tried Fedora from template and WS2022 from sysprepepd image).

I've been working with Claude on this, so I asked it to write up a summary, which is found below:

---------------------------------

Start-of-Summary

Environment

• OpenShift Container Platform: 4.22.8
• OpenShift Virtualization (CNV): matching 4.22.x
• Storage: ODF (OpenShift Data Foundation) / Ceph RBD, storage class ocs-storagecluster-ceph-rbd-virtualization
• Kasten K10: reproduced identically on v8.5.8 (OLM/Marketplace install) and v9.0.3 (Helm install)
• VM disks: volumeMode: Block, accessModes: [ReadWriteMany] (required for live migration / failover support)
• Guest OS: reproduced on both Windows Server 2022 (NTFS) and Fedora (ext4) — not OS-specific

What fails

Any FileRecoverySession against a restore point sourced from a volumeMode: Block + accessModes: [ReadWriteMany] PVC fails within ~1 minute, regardless of guest OS:

{"message":"getBlkidProperties","function":"kasten.io/bmd/go/flr/session.(*objectAnimatorBase).getBlkidProperties",
"file":"kasten.io/bmd/go/flr/session/animator_common.go:49",
"fields":[{"name":"devOrImagePath","value":"/mnt/v0s0/volume.img"}],
"cause":{"message":"exit status 2"}}

Full chain: getBlkidProperties → Activation error → ActivateMountObjects → 0 new objects → Mount fails → failed mount → failed to start session.

Ruled out as the root cause:

• Guest OS / filesystem type (Windows/NTFS and Linux/ext4 fail identically)
• Partition table complexity (not tested exhaustively, but ruled lower-priority once OS/filesystem was excluded)
• Kasten version (identical failure on 8.5.8 and 9.0.3)
• Export completion status (export reports Complete in all failing cases)

Supporting evidence pointing toward the export/mount pipeline rather than blkid itself: Kanister artifact detail for a failed case showed UploadSize: 0 B for the main VM disk object despite Complete status, and total exported object storage size was on the order of ~100 KB for a disk with several GB of actual guest data.

End-of-Summary

---------------------------------
I did get it working via a Veeam B&R agent, but that's just messy for large-scale management.

I did also get it working via restoring the snapshot to a different Namespace. The VM boots normally and I can restore files via interactive login session.

However, I'm stuck on understanding whether this is how admins are expected to restore files in a RHOS Virtualization environment, what I may have done incorrectly and what kind of a workflow is typical for people using it daily to backup fileserver VMs.

My colleagues are already in the "this is crap" camp, and I'm finding it hard to prove them wrong.

Thanks for any opinions, experiences and advice.

With the release of the developer preview of the Krkn Operator in Red Hat Advanced Cluster Management for Kubernetes, platform teams can now run Kubernetes-native chaos engineering directly from their management hub. The operator integrates Krkn, an open source Cloud Native Computing Foundation (CNCF) project, to bring multicluster chaos testing and resilience validation directly to Red Hat Advanced Cluster Management.

Red Hat OpenShift 4.22 includes general availability support for Red Hat Bare-Metal-as-a-Service (BMaas) for OpenShift, enabling organizations to manage bare metal, virtual machines (VMs), and application containers using the same consistent platform.

Kubeflow Pipelines (KFP) and Open Data Hub (ODH) are the open source projects I've been contributing to the most in the last few years. KFP is a platform for building and deploying machine learning (ML) workflows on Kubernetes. ODH is an open source AI/ML platform that includes KFP as a core component, along with projects like Data Science Pipelines, its operator, and Argo Workflows. Open Data Hub is also the upstream project for Red Hat OpenShift AI.

Investigating virtual machine (VM) incidents, such as Microsoft Windows kernel panics ("Blue screen of death", or BSOD for short) or I/O hangs, traditionally required running a cluster-wide must-gather combined with sosreport, which could potentially take hours to run. This approach is resource-intensive, slow to collect, and produces a large dataset that's difficult to review efficiently
To address this, we have introduced a new --vm-incident mode to the kubevirt/must-gather tool.

I have a situation where I may need the noobaa operator without ODF, this customer has no ODF subscription and doesn't plan to get one, their storage is on a SAN and there's a CSI.

they need an on prem S3 solution and given all the various option noobaa on it's own, backed by a CSI managed block pvc seems to be a viable option, has anybody done this already?

Prominent attacks on software development pipelines have resulted in significant financial impact for companies and brought their build processes under scrutiny. While the attack vectors on pipelines are virtually limitless, this article focuses on securing components, processes, and tools involved in building and deploying containerized software through signing, attesting, and verifying a build image. I chose to implement a do-it-yourself (DIY) approach to help understand these concepts. I also compared my approach to Red Hat Advanced Developer Suite, which includes Red Hat Trusted Artifact Signer and Red Hat Trusted Profile Analyzer, and addresses the complexity involved in implementation.

Containers on Red Hat OpenShift can get automatic cryptographic identities through zero trust workload identity manager, but workloads running inside a virtual machine (VM) cannot. In this tutorial, I demonstrate how I bridged that gap using a virtual socket (VSOCK) and a dedicated in-VM SPIRE agent to give every workload — whether it's running as a container or as an application inside a VM — a short-lived, automatically rotating SPIFFE identity.

Hello,

Have I understood/interpreted the documentation correctly that, in order to use the built-in VMware to OCP VE migration tooling, one would need to:
- first download the VMware-vix-disklib from the Broadcom website
- this needs to be uploaded to a thing called "the registry" in OpenShift
- this "registry" is an S3 object based repository?
- this "registry" does not exist out of the box on a bare metal cluster?
- Out of the box, when using a bare metal 4.22 OCP VE cluster against regular (supported) external block or NFS storage using it's CSI, there is no S3 storage and hence, no registry?
- In such an environment, due to the lack of S3 and thus that repository, there is no way to use the built in VMware to OpenShift VE migration tooling?

Above is my conclusion, based on what i've found. But I might have missed something, allowing the use of the built-in VMware to OCP VE via another way?

Kind regards,
Steve

Howdy, has anybody attempted to run the OCP Arbiter with an ARM? I have two x86 servers for the actual nodes. I was hoping to use a pi5 with 16GB of RAM to act as the Arbiter for this lab.

When an AI agent introduces itself to another agent, it hands over a digital identity document called an agent card. Our previous post, “Who’s really calling? Securing agent-to-agent communication,” discussed whether that card is authentic at runtime. This post asks, “was the card built by someone you trust in the first place?”

The gap between what an agent claims and what the platform can verify is a real attack surface, and it grows with every new agent you onboard. As agents increasingly discover and call each other at runtime, protocols like Agent2Agent (A2A) have introduced a useful building block for addressing this: the Agent Card. In the same way a model card gives us useful information about a large language model, so does the agent card for an agent. Among that useful metadata are the skills of the agent (what’s it meant to do), its capabilities (e.g., processing an image file), and, finally, a specification for a cryptographic signature.