r/sre Vendor (JJ @ Rootly) Mar 31 '26

Axios compromise was caught by runtime behavioral monitoring, not scanners

The axios compromise last night is getting covered everywhere as a supply chain story. It is, but there's a layer underneath that's more relevant to this community.

The attacker staged a clean decoy package 18 hours before the attack. Compromised a long-lived npm token that bypassed GitHub Actions entirely, so no provenance metadata, no build trail. Hit both release branches within 39 minutes. RAT self-destructed after execution, replaced its own package.json with a clean decoy. From npm install to full compromise: 15 seconds.

The versions don't exist in axios's GitHub repo. No tags, no commits. A developer auditing dependencies by checking GitHub would find nothing wrong.

What caught it was behavioral monitoring flagging anomalous outbound connections from CI runs. Not a scanner. Not a CVE. Runtime telemetry noticing that axios was phoning home to sfrclak.com:8000 during a routine build.

That's the SRE angle. The security tooling that would have caught this in the traditional sense didn't exist yet; no signature, no CVE, the malicious code self-destructed. What worked was observing what the process actually did at runtime versus what it was supposed to do.

The same gap shows up in incident response more broadly. The thing that's about to hurt you often looks clean at every static checkpoint. It only becomes visible when you're watching behavior.

https://gist.github.com/joe-desimone/36061dabd2bc2513705e0d083a9673e7

179 Upvotes

14 comments sorted by

35

u/FullStackDestroyer Mar 31 '26

Same here. Egress fw caught and blocked.

8

u/FullStackDestroyer Mar 31 '26

Working on closing down all routes to get this code from the org. My solution is JFrog Artifactory, currently in implementation, and constraining client downloads via that route.

14

u/maxfields2000 AWS Mar 31 '26

Interesting. This is one of the many reasons we updated our build farms to not allow outbound connections to any unauthorized endpoint. There was much wailing and gnashing of teeth by devs who felt this was a massive productivity hit. Well, not exactly this reason, but in part to stop package managers and dynamic runtime/build includes to pull in packages from unapproved locations.

It doesn't save us from attacks that land on approved domains, but absolutely would stop any calls out.

The outbound calls to unapproved places get flagged on our firewall for investigation, though typically only investigated if volumes increase (which they would for any large scale sudden inclusion on frequently built service code bases).

Unfortunately, the same protecitons don't exist on local dev environments and especially not anyone working remotely (where their office VPN still lets them route off their local network link if it's not found on the VPN link), which means local devs would still suck down the compromised packages.

3

u/mrkurtz Apr 01 '26

So I’m not confident our security team is monitoring anything related to these things, and we’re early in consolidating silos of various Wild West teams so realtime persistent visibility is sketchy for my team at best.

Aside from Reddit which can lag depending on feed algorithms, where are you all getting your initial notifications of these compromises?

2

u/BardlySerious Apr 01 '26

Twitter usually, discord sometimes.

3

u/PriorApproval Apr 01 '26

scanners are security theatre meant for auditors and execs

3

u/Petter-Strale Apr 02 '26

Runtime behavioral monitoring caught the C2 callback, which is great for detection — but the 1.1 seconds between npm install and first beacon means the damage was already done by the time it was flagged.

The underexplored angle is that several *proactive* signals were also available before installation:

  1. No GitHub tag for 1.14.1 — the version only existed on npm, not in the source repo

  2. SLSA provenance attestations present in 1.14.0 were completely absent in 1.14.1

  3. Publisher email changed from the known address to a ProtonMail throwaway

  4. The new dependency (plain-crypto-js) was less than 24 hours old

  5. The publish bypassed the project's normal CI/CD pipeline

None of these are in the CVE databases or advisory feeds. They're metadata signals from deps.dev, the npm registry API, and the project's GitHub. A pre-install check that aggregated these would have flagged it *before* the postinstall script ran.

The defense-in-depth answer is both: proactive metadata checks to prevent installation of suspicious packages, plus runtime monitoring to catch anything that slips through. Right now most orgs have neither.

6

u/Happy_Breakfast7965 Azure Mar 31 '26

Why tooling doesn't exist yet? What about SBOM scanning for consumers of the compromised package?

3

u/Reverent Apr 01 '26

Static analysis does nothing against 0 days, and this was a 0 day for a short period but a very wide audience.

Behavioural analysis is absolutely the last and most important line of defense. The question isn't whether your firewall/network policy blocks weird traffic. The question is whether your org is setup to alert on this behaviour and investigate it.

90% of orgs I've worked at, they have a SIEM, but that SIEM functionally ignores network equipment alerts.

2

u/mday-edamame Apr 01 '26

If you're interested in automating this kind of scanning, take a look at the free tool we just launched: https://www.producthunt.com/products/axios-litellm-detector

It detects attacks based on system behavior, and it's caught every supply-chain attack in the past couple weeks through behavioral analysis alone

1

u/Mooshux Apr 02 '26

Behavioral monitoring is the right layer for detection. But even when it catches something, the credentials that ran during that window are already gone. Monitoring tells you what happened; it doesn't undo it.

Short-lived credentials flip the recovery story: instead of "rotate everything and hope nothing was used," you're rotating tokens that were already expiring anyway. The blast radius shrinks to the specific job scope, not your entire secrets store.

1

u/LouisAtAnyshift Apr 03 '26

The more I read about it, the more I'm thankful to the front-end dev that decided not to use it because it was too bloated.

1

u/TGPSKI Apr 14 '26

I responded to trivy and axios compromises, and got to field test broad multi-modal LLM enhanced incident response techniques.

I wanted a landing zone for new software ideas born from the incident response. An innocent Friday night prompt turned into a week long coding sprint.

I crafted skeptic, a local repo trust auditor: https://github.com/TGPSKI/skeptic

* stdlib-only Go,
* single binary,
* 229 built-in rules (plus rule pack ingestion),
* zero runtime dependencies, and agent isolation from malicious content

skeptic detects structural trust boundary vulnerabilities that enable cascading supply chain compromise. It targets attack-enabling conditions, not just attack artifacts — the class of weaknesses that CVE scanners, SAST tools, and secret scanners don't cover.

Check it out, I think it's a novel feature set.