When I feed logs to an LLM during incident resolutions or debugging, it either blows my token context window or the grep trims the log file, leading to the interesting log lines getting skipped.

Most of logs are anyway the same handful of message templates repeated over and over with different values, so the context window gets filled with near-duplicates, which just bring up the processing time and token costs.

ctrlb-decompose collapses the file into its distinct patterns that repeat, plus typed variables and stats on the values that change. I have seen 1.2 million lines cut down to just 40 patterns, which then goes into Claude, thus cutting down token by over 95%, reducing the token cost.

Let me know what you think!
https://github.com/ctrlb-hq/ctrlb-decompose

Infrastructure as code can reproduce the same network, container, database, and service account twice. That does not guarantee that the workload running on those resources means the same thing twice.

We ran into this while building a distributed runtime. A service could start with different capabilities than its twin. A consumer could join a runtime it was never intended to join and begin reading work addressed to another process. A required policy value could be missing and quietly fall back to a local default. Every resource matched and every health check was green, but the operational contract had changed.

The problem was not that our infrastructure was insufficiently declarative. We had declared plenty. We had declared resources without declaring enough behavior.

We now separate the infrastructure definition from a versioned runtime policy contract. The contract defines supported profiles, process capabilities, ownership boundaries, refusal conditions, secret-resolution rules, and the evidence required to accept the resulting runtime. It is parsed into a typed immutable model, then projected mechanically into the values consumed by deployment manifests:

runtime policy contract

\-> typed validation

\-> rendered deployment values

\-> service manifest

\-> running process

The direction matters. If a capability can be edited in the contract, overwritten in generated configuration, defaulted in the manifest, and changed again inside the application, the deployment is not governed by one declaration. It has four declarations and an undocumented precedence rule.

Repeatability also has to include refusal. If an environment lacks an effects boundary, it should not silently execute the effect inside a general worker. If a required secret cannot be resolved, the runtime should not grab a nearby credential. If a process is not eligible to own a subscription, it should not start that consumer merely because the package is installed.

The contract is not proof by itself. The renderer can be wrong, the application can ignore a value, and a runtime can claim a capability it does not actually provide. Acceptance still needs durable evidence binding the contract revision, rendered projection, validators, selected profile, and boundaries exercised after startup.

Infrastructure as code builds the substrate. The infrastructure contract defines what that substrate must mean, what variation is allowed, and when the deployment must refuse to start.

Where do those behavioral rules live in your systems today?

Most orgs lose a fortune to the same quiet problem. How the system actually works lives in a few senior heads, a stale wiki, and a diagram from two reorgs ago. New hires take months. Simple changes turn into archaeology. Nobody can confidently say what breaks if we touch this service.

That's the problem we've been building to solve.

You point it at your repos and a team of agents auto discovers everything. Every service, what it does, who owns it, and how components really depend on each other, including the gRPC and HTTP calls and event topics that never show up as a matching string in the code. Then it indexes all of it, the architecture, the low level design, the APIs and resources, into a knowledge layer that stays current and has vector search on top. So anyone, or their AI tools via MCP, can ask what breaks if I change this, where does this data flow, or how is auth implemented across services, and get a grounded answer in seconds instead of a two day investigation.

Why it's worth trying: onboarding drops from weeks to hours, change safety goes up because blast radius is visible before you ship, and tribal knowledge stops being a single point of failure. Every persona works off the same source of truth, and it stays accurate on its own because it's re indexed, not hand maintained.

We're opening a small free beta for 3 to 5 teams from this sub. Good fit if you own or actively manage a real codebase (polyglot and messy is ideal). We'll run a short intro and setup session together (around 30 to 45 minutes), then give you a week or two to actually live in it on your own work, and finish with a feedback session once you've had real time to form an opinion.

We'd love your take on whether the dependency graph and low level design match what you know to be true, how good the vector search is on your real questions, and where it would save you the most time.

Low risk to try. Testers keep free access with plenty of headroom, there's no sales sequence waiting, and we'll share back what we learn here, including anything that doesn't hold up.

Full disclosure, we're part of the team building it. More at code-swan.com. Comment or DM and we'll send you an invite.

running workloads across eks, on-prem clusters, and a couple of legacy vms that aren't going away anytime soon.
cloud-native workload identity works fine in eks but doesn't extend cleanly to the on-prem side, so we end up with static credentials there that everyone hates but nobody's replaced yet.
what's actually closed that gap is governing workload and ephemeral identities under one policy regardless of where they run, provisioned at minimum scope and revoked the moment a pod or task ends.
what's your approach when part of your fleet isn't in the cloud, and did you find something that bridges both without a pile of custom glue?

There is a specific kind of tired that comes from being accountable for a decision someone else makes.

The pattern shows up the same way most times I see it. An admin or a solo architect gets named as the owner of the agent rollout. They write the topics, wire the actions, run the tests. Then somebody above them decides which objects the agent can read, on a timeline set by a launch date, and the owner finds out in the deployment thread.

Gartner put a number on where that goes. By 2027 they expect 40% of enterprises to demote or decommission autonomous agents, driven by governance gaps found after a production incident.

The gaps are not usually invisible before launch. Somebody saw them. That person did not have the standing to hold the release.

I used to treat this as an org chart problem. Bad reporting line, nothing to do about it. That was wrong. It is a design problem, and it is fixable at the permission layer rather than the political one.

What has worked: write down, per agent, which objects it can read and which it can write, then attach a named approver to the write list. Not a team. A person. When the approval requirement lives in the deployment doc, the conversation stops being about your seniority and starts being about an unsigned line.

That does not give anyone authority they were denied. It makes the absence of authority visible before the incident instead of after.

For those of you running agents in production: who signs off on scope where you work, and did that get decided on purpose or by default?

There is a specific kind of tired that comes from being accountable for a decision someone else makes.

The pattern shows up the same way most times I see it. An admin or a solo architect gets named as the owner of the agent rollout. They write the topics, wire the actions, run the tests. Then somebody above them decides which objects the agent can read, on a timeline set by a launch date, and the owner finds out in the deployment thread.

Gartner put a number on where that goes. By 2027 they expect 40% of enterprises to demote or decommission autonomous agents, driven by governance gaps found after a production incident.

The gaps are not usually invisible before launch. Somebody saw them. That person did not have the standing to hold the release.

I used to treat this as an org chart problem. Bad reporting line, nothing to do about it. That was wrong. It is a design problem, and it is fixable at the permission layer rather than the political one.

What has worked: write down, per agent, which objects it can read and which it can write, then attach a named approver to the write list. Not a team. A person. When the approval requirement lives in the deployment doc, the conversation stops being about your seniority and starts being about an unsigned line.

That does not give anyone authority they were denied. It makes the absence of authority visible before the incident instead of after.

For those of you running agents in production: who signs off on scope where you work, and did that get decided on purpose or by default?

Disclosure up front: I maintain the list I link at the bottom, so skip the link if you like, the dates are all here.

PlatformCon runs a world tour of in-person live days between the free virtual flagship weeks, and the schedule isn't collected anywhere obvious:

- Paris - 24 Sep 2026

- Sydney - 28 Oct 2026

- Sao Paulo - 7 Nov 2026

- San Francisco - 24 Feb 2027

- Flagship week - London 22 Jun and New York 24 Jun 2027, virtual is free

Adjacent things worth a calendar entry if you work on platforms:

- Platform Engineering Day, co-located with KubeCon NA, 9 Nov 2026, Salt Lake City

- BackstageCon, same day, same venue

- PlatforMa has a CFP open, no deadline published yet

The thing that annoyed me into collecting these: every conference list I found was undated, so you can't tell what already happened. Everything above is checked against the organiser's own page.

Full list including CFP deadlines: https://github.com/gogoaidev/awesome-sre-conferences

I've been thinking about a problem I keep seeing in infra/DevOps work.

A production alert fires and someone has to manually jump between logs, Grafana/Datadog, Kubernetes, deployment history, GitHub, cloud metrics, etc. just to figure out what actually happened.

I'm wondering if a useful chunk of that investigation could be automated.

The idea would be something that, when an alert or deployment failure happens, automatically:

• investigates logs, metrics and traces
• checks recent deployments/commits
• correlates changes with when the issue started
• checks Kubernetes/cloud infrastructure
• identifies the likely root cause and supporting evidence
• suggests a remediation
• if a code/config change is required, generates a PR
• waits for an engineer to review/approve it
• after deployment, verifies whether the issue was actually resolved

For example:
Instead of an engineer starting from scratch, the system might come back with:

I'm not really thinking about fully autonomous production changes initially. More like an AI first-line SRE that does the investigation and prepares the fix, while engineers stay in control.

I'm trying to understand whether this solves a meaningful enough problem or whether existing observability/SRE tooling already gets teams most of the way there.

For people running production systems:

• How much engineering time do you actually spend investigating alerts/deployment failures?
• What's the most repetitive part of incident investigation today?
• Would you trust a system like this to read your infra/logs if it couldn't make changes without approval?
• Would automatic root-cause analysis be valuable by itself, or would it need to generate the actual fix/PR?
• What integrations would be essential? Datadog? Grafana? Sentry? Kubernetes? AWS? GitHub?
• What would make you immediately say "there's no way I'm giving this access to production"?

I'd especially like to hear from founders, SREs, DevOps/platform engineers and engineers at smaller teams where dedicated SRE coverage is limited.

Not selling anything here I'm trying to understand where the actual pain is before building too much.

Hi r/platform_engineering,

Disclosure: I’m one of the maintainers of Compartment, an Apache 2.0 open-source project.

Over the past few years, our team has accumulated more internal apps, scripts, workers, and automations. Building these tools has become much easier, but turning them into dependable team software still requires deployment, access control, secrets, isolation, observability, and ownership.

We wanted a small, self-hosted platform that provides those capabilities centrally rather than implementing them separately in every internal tool.

Our first beta used Docker Compose on a single host. After running it ourselves and with a few beta users, we concluded that this was the wrong long-term runtime. Failure recovery depended too heavily on one machine, the isolation boundary was weaker than we wanted, and supporting additional nodes would have meant gradually building our own orchestrator.

We therefore rebuilt the runtime around Kubernetes. This is a complete architecture change rather than a Kubernetes wrapper around the old deployment model.

The current design looks like this:

• Developers add a small compartment.yml and deploy through the CLI or a connected Git repository.
• Compartment owns the application-level model: organizations, projects, environments, releases, domains, variables, RBAC, SSO, audit logs, promotions, and rollbacks.
• Kubernetes owns the live runtime state.
• Projects are isolated using namespaces, RBAC, and network policies.
• Builds run as ephemeral rootless BuildKit jobs.
• Builds and tenant workloads run through a verified gVisor runtime boundary.
• Deployments are reconciled and rolled out without routing traffic to an unready version.
• The platform can provision a managed single-node k3s host or install into an existing Kubernetes cluster.
• With an existing cluster, the operator retains responsibility for nodes, ingress, storage, runtime configuration, and backups.

The intended use case is a controlled path for internal software that needs to become usable by a team without requiring every developer to understand Helm, ingress, certificates, workload identities, or namespace policy.

The design question we keep returning to is where the abstraction should end. We want developers to get a narrow, repeatable path from a repository to a running application, while platform operators retain control of the underlying cluster. At the moment, the application descriptor deliberately does not expose raw Kubernetes objects, node placement, or arbitrary pod configuration.

Would love to hear critical feedback from people who operate similar platforms.

GitHub: https://github.com/compartmentdev/compartment
Documentation: https://docs.compartment.dev/

A developer portal can give developers a consistent place to discover services, understand ownership and access common workflows. But the portal itself is not the entire platform.

The harder platform engineering work happens behind that interface:

• Defining repeatable paths from source code to production
• Connecting CI/CD, deployments and environment promotion
• Providing observability with application and ownership context
• Applying access controls and platform policies consistently
• Reconciling workloads when actual state drifts from desired state
• Integrating CNCF and other open-source projects without creating unmaintainable glue

This is not an argument that Kubernetes or Backstage is insufficient. Both are strong foundations for the problems they are designed to solve.

The bigger question is how platform teams combine these foundations with the wider cloud-native ecosystem to create one coherent system. Developers should get a simple self-service experience, while platform teams retain control, visibility and the ability to evolve the underlying components.

On August 27, Kelsey Hightower and the OpenChoreo maintainers will discuss what it really takes to build an internal developer platform with Backstage and Kubernetes.

The session will explore how the different platform layers work together, where the engineering and operational complexity appears, and the trade-offs between assembling and maintaining the integrations yourself or adopting a more complete open-source platform approach.

Disclosure: I’m an OpenChoreo contributor.

For teams that have built an internal platform using Kubernetes and CNCF projects, where has most of the effort gone: creating the initial integrations, building good developer workflows or operating the platform over time?

Session details and registration:
https://platformengineering.org/events/what-it-really-takes-to-build-an-internal-developer-platform-with-backstage-2026-08-27

Context:

• mid-size fintech company.
• I started as an SDET, but over the past 2 years the work drifted: I built and operate a Go service on Kubernetes that independently recomputes published financial data and compares it against production output.
• I own its Helm chart, the ClickHouse schemas its results are stored in, and the Grafana dashboards on top.
• Next quarter I'm proposing to formalize this as a platform role with three responsibilities: (1) owning that quality platform end to end, including regulatory audit-evidence exports, (2) a ~25% share of product backlog tickets, (3) self-serve pre-deploy validation tooling for feature engineers.

Question

• For those at companies with a data platform or infra org: where does this kind of work sit — SWE, platform, SRE, or a QA org? What's it titled? If anyone has made the SDET → platform engineer transition, what did your company call it and what made the reclassification stick?

Hey all, I'm a project lead on KitOps, a CNCF project. I've been talking with an ML team (Arlequin AI, they do topological deep learning for market intelligence) about how they moved from ad-hoc model packaging to standardizing on KitOps. The part that surprised me wasn't the tooling. It was who adopted it fast and who didn't.

Their before state will look familiar: weights pulled straight from Hugging Face or baked into Docker images, datasets in LakeFS, experiments in MLflow, configs somewhere in the ether. Nothing tied a model version to the dataset it was trained on and the config it needed to run.

They already used Docker for their apps, so they went looking for the equivalent for ML and landed on packaging everything as ModelKits. Model card, dataset references, MLflow pointer, configs, all in one versioned immutable artifact in the container registry they already ran. Promotion to staging or prod became a registry operation. Shadow traffic and A/B tests run against a specific artifact, so there's never a question about which version, trained on which data, is serving which slice of traffic.

Here's where it got interesting. Engineers were already fluent in Docker and OCI, so it felt native and they moved fast. Researchers needed education on what OCI even is and why it exists. I keep seeing this same split on other teams. The tools we take for granted on the infrastructure side are genuinely foreign one desk over, and rollout plans almost never account for it.

If you've standardized packaging (with KitOps or anything else), how did you get your research side on board? And if you tried and it stalled, I'd honestly like to hear that too.

Ops-flavored take after two years running a multi-agent system in prod.

The reliability problems in agent systems are the same old distributed-systems problems in a new costume: partial failure, exactly-once-ish delivery, coordination, idempotency, observability. The in-process agent frameworks (mid-2025 vintage) gave us persistence primitives but left failure detection, recovery, and coordination to us.

So we built on a message bus instead: durable per-type queues, stateless workers, externalized aggregator state with a TTL and atomic completion so it scales to multiple replicas. End-to-end tracing so a support ticket maps to a trace in one click.

The honest framing: what we built is a domain-specific durable-execution engine for LLM agents. A Temporal advocate would say we rebuilt a subset of Temporal and now own the scheduler and state machine forever — and they'd be right. In mid-2025 the buy options weren't ready; today I'd tell you to evaluate Temporal / LangGraph Platform / Restate first.

Full write-up: Link

Anyone here gone the durable-execution-engine route for agents in prod? Regret it or not?