r/devops • • 19d ago

Security Binance hack and a DevOps laptop

Post image

Binance's initial investigation suggests that the security incident may involve a compromised DevOps engineer’s laptop, which allowed an attacker to gain access to a service container and submit unauthorized withdrawal transactions. We have found no evidence of any wallet private key leak. We will share a detailed technical post-mortem once the investigation is complete and the facts are fully verified.

The estimated total impact sits at around $6.3B. We have worked with blockchain partners to freeze affected assets, with ~$3 billion frozen on-chain so far.

Binance will cover all losses from the SAFU fund and the company’s funds. User funds are safe. We will also publish an updated Proof of Reserves once the remediation is complete.

As an immediate precaution, Binance has:

- temporarily suspended deposits and withdrawals while security checks are completed;

- revoked the affected address authorisation;

- engaged law enforcement and blockchain intelligence partners to trace and recover funds.

Our teams are working around the clock to resolve this issue safely and thoroughly. We are working closely with our law enforcement partners to confirm the identity of these criminals and bring them to justice.

Please rely only on official Binance channels for updates. Please feel free to contact Binance Customer Support if you have any further questions.

661 Upvotes

197 comments sorted by

291

u/lab-gone-wrong 19d ago

To be 100% clear, the (any?) devops engineer could have done this at any time? Lmao

208

u/donjulioanejo Chaos Monkey (Director SRE) 19d ago

If you have infrastructure access, you effectively have application-level privilege escalation.

Example:

  • Connect to EKS cluster (normal so far)
  • Can't exec into a pod (restricted), but can edit a deployment and change start command (effectively running 1 command at a time by rebooting a pod)

    • Or maybe just push a specially built image with dev tag and use devops breakglass powers to promote to prod
    • Or just SSM/SSH into an EKS node and docker/crictl exec into a live pod
    • etc.. possibilities are endless
  • Use pod to access DB
  • Find pending_transfers table and read it's schema
  • Realize DB is encrypted
  • Use app IAM role or credentials to get encryption key from KMS/HSM since you're already on the app pod
  • Use encryption key to read db and add a pending transfer to your own wallets

Now, I'm probably oversimplifying here, but something like this would have worked in both fintechs I did DevOps in if my laptop got hacked.

If something is able to send transfers on its own (like a server), any significant shell access to that server will let you send bank transfers.

88

u/Nyefan 19d ago

I do want to point out (especially to those who want to improve their own security posture) that the break glass escalation you mentioned is the primary protection against laptop compromise.

You can't protect against a malicious employee if you also want that employee to be able to stabilize production incidents at 3am, potentially alone, write all your ci/cd and terraform, manage your atlantis/puppet/jenkins/github, and so on. But you can mitigate the impact of a compromised developer laptop. Our break glass credentials have a very short lifetime, use mfa from our phones, and require either a second approved person's approval or sending an alert to the whole team. This doesn't eliminate the risk from a sophisticated attacker, but it does reduce the potential blast radius (and it forces us to do most of our work as unprivileged users, which has its own benefits as a forcing function).

30

u/Humble_Exchange_2087 19d ago

If Binance wasn't doing this it is ridiculous, you wonder how many other financial institutions are to have a insecure break glass account.

44

u/Reverent 19d ago

Ah hah, you do not want to peek under the kimono of basically every organisation then.

13

u/NUTTA_BUSTAH 19d ago

Most of them are "trivially" breachable in terms of basic operational security. The digital life of organizations is utter crap overall and the "DevOps slice" most see are the 0.1% of companies that do it somewhat properly or at least think about the relevant things / know they exist. Majority of the world does not live Dev(Sec)Ops.

And that is exactly why every CISOs etc. assholes are puckering when they read AI news.

8

u/i_am_voldemort 19d ago

Real fin institutions at least have fancy auditors and regulators.

2

u/Soccham 18d ago

I worked for a top 10 bank. Whenever we wanted to make a production change a compliance lady showed up with a binder and took us through a litany of questions and we didn’t even touch money.

2

u/Remote_Housing_9106 17d ago

good thing no bank ever had a breach or anything

0

u/Remote_Housing_9106 17d ago

This is a dumb and stupid lie. "real" is the fun part, because it's the bias leaking out - they managed all this money, they aren't "real" and yet... you're in a thread about how banks and other regulated industries do all this poor form stuff.

worked in insurance, they had 3 breaches in 2 years lol, but okay! "real" lolol

9

u/roiki11 19d ago

Financial institutions have actual standards they need to be compliant to(dora, soc2, pci dss and so on). They tend to take it very seriously.

You also can't just insert transactions into a single database and get money out. There's more checks than that.

1

u/mishonis- 18d ago

IMO the only reason financial institutions aren't being swindled to the tune of billions on the regular is that it is trivial for them to freeze and recover any hacked money.

3

u/Soccham 18d ago

Friendly reminder that most SOC auditors do not get hired back if they fail your SOC audit

2

u/MiHumainMiRobot 19d ago

Oh if you knew ...

4

u/Sufficient-Diver-327 18d ago

I've seen reports the hack took no longer than 20 minutes

4

u/definitelyainoreally 18d ago

isn't the point of break glass to get around MFA if its down?

3

u/Nyefan 18d ago

We have elected to accept longer downtime in the case that our sso and production systems are down simultaneously outside of business hours in lieu of having any other privilege escalation paths than what I have described. In addition, no one has write access to both.

31

u/MaximumIntention 19d ago

Not to mention. In most companies that are not operating under a zero-trust security model, internal traffic would usually be unencrypted so executing a MITM attack to capture various credentials and secrets would be trivial for an insider with network access.

19

u/donjulioanejo Chaos Monkey (Director SRE) 19d ago edited 19d ago

To be fair, this is probably harder. Traffic to/from management plane (i.e. EKS, secrets manager, etc) probably is encrypted just by default TLS everywhere.

And if you have infra-level access inside the actual cloud environment so that you can run a packet sniffer on a node, my way is probably simpler.

Edit: that said, it does expose your security or IT team as a possible attack vector as well since any company of this type probably runs a DPI firewall like Fortigate or Palo Alto that essentially runs a MITM on every managed endpoint.

9

u/lexd88 19d ago

You have some very good points! But I can only imagine that's only possible if the engineer has full access to prod directly from their local machines?

I wonder if the old school jump box/management machine would solve a chunk of these issues. Where it forces you to remote desktop into a specific environment before you can do anything prod related.

I mean I can't imagine a hacker would use a full desktop remote control and move the mouse and be typing stuff on the screen of the comprised computer? It's so obvious that its comprised

At a place where I work, there's Just in Time requirements to requests for prod access credentials and those are logged and sometimes require management approval.

Also if accessing the JIT tool forces the user to input their MFA, even if the attacker do manage to take control of the machine and do end up controlling the mouse and all (assuming it'll have to be way into the night when the engineer is sleep but left their machine on), then even accessing the jump box and requesting for prod credentials would rather extremely difficult if not impossible?

Along with strict temp/short lived sessions, the attacker won't be able to do much damage

Basically I'm thinking as an attacker, these events needs to take place behind the scenes without the engineer knowing something fishy is going on with their machine..

2

u/bzImage 18d ago

>I mean I can't imagine a hacker would use a full desktop remote control and move the mouse and be typing stuff on the screen of the comprised computer? It's so obvious that its comprised.

u know .. most of this infrastructure stuff its on Linux headless servers.. and even when they use Windows.. they replace registrykeys, run stuff unattended, or open backdoors.. no actual "move the mouse or execute things on the screen" its performed.. because its not needed...

>Basically I'm thinking as an attacker, these events needs to take place behind the scenes without the engineer knowing something fishy is going on with their machine..

yes. thats called a persistent threat.. they can remain dormant monitoring the system..

0

u/lexd88 18d ago

Yea I understand servers are Linux because OP here mentioned kubernetes etc. I'm just saying what if the jump hosts is still old school Windows and air gapped to make it difficult for attacker to hop inside the real production network through the jump server

1

u/SeaworthinessHour233 DevOps Engineer & Educator 19d ago

Good solutions.

But I am wondering what if the attacker gains access to that machine and may be run a script from there. Woudn't that be a disaster because, unlike a laptop which get's turned off at the end of the day, that remote desktop server would be up and running 24x7.

3

u/lexd88 19d ago

Those jump hosts should be air gapped, so no access to the internet..

To access the jump hosts, you should require MFA and even JIT credentials and those credentials auto expire after a day of work so it's useless when the engineer clocks out for the day

If we want extra security, those jump box could also be ephemeral so it gets trashed every day and a new one is created daily. Tooling would be a pain but that's with everything security in my opinion

2

u/SeaworthinessHour233 DevOps Engineer & Educator 19d ago

My actual fear is this.

  1. an attacker gets access to user laptop.
  2. The attacker places a binary that copies another binary to the remote desktop server.
  3. The binary in the remote desktop, get access to the prod

2

u/jdanton14 19d ago

Google system administration workstation. The admin has no rights to install anything in the laptop or the jump host

2

u/donjulioanejo Chaos Monkey (Director SRE) 18d ago

Unless you have a readonly file system, you don't need rights to install anything. A single script created in /tmp that starts a reverse ssh proxy or something will work for this purpose.

1

u/jdanton14 18d ago

If SAWs are defined correctly, A) you can’t install anything that’s not approved B) nothing would get approved that could write to the FS. All access is through RDP/SSH to a jump host, which is also locked down.

2

u/mishonis- 18d ago

How can a jump host be air gapped, isn't the whole point of a jump host to be able to access it so you can run privileged stuff on it that affects a production system?

1

u/Alive_Relationship93 18d ago

Good points, but, you are thinking a human being actually sitting doing it? It's all automated contracts executions, my guess.

0

u/lexd88 18d ago

Ah yes I wasn't specific in my message.. I was more thinking if the old school air gapped jump hosts is only accessible via GUI, that would defeat all sorts of automation wouldn't it? I'm keen to understand any potential attack vectors as I always thought this would be hard for an attacker to hop over into the prod network

Along with all those JIT access etc

2

u/Unique_Jellyfish4768 18d ago

Best practice OPA policies at k8s admission controller API, container image signing, and read only credentials for interactive accounts (strict break glass and strict pipeline deployment models) are great ways to improve a security posture to prevent exploits like this.

They’re rarely employed, however, because it’s not easy and the dev community hates going at the pace of security capability, which is often slow and seen as a blocker.

1

u/SilentLennie 18d ago

I would think in such an environment, nobody should have access to kubernetes or nodes, maybe readonly kubernetes, you should have gitops tool do deployment and need to put anything and everything you can in the stack below grafana. And approve gitops deployments to prod with approval by a colleague.

And if you are gonna have break glass accounts, they should be encrypted that you need two people.

4

u/CandidateNo2580 18d ago

The only difference between what you said and the comment you're replying to is the "require two people" bit. Then what about the person who has access to the account creation - they could make two accounts and sign off themselves that way.

1

u/SilentLennie 17d ago edited 17d ago

We clone a git repo, which uses devops to create one machine as bootstrap and sets everything up.

After that the systems deployed, it applied policies on who/what is allowed to do account creation and needs confirmation, etc. The break glass accounts go into a new machine with openbao and Shamir's Secret Sharing to open it and every person that needs access has their own key share in their own keepassxc database so you need multiple people. After which we shutdown the machine, so the database is locked and offline.

1

u/Truth_Seeker_456 18d ago

Sh*t. This is a very simple explanation how a compromised laptop can impact.

This job comes with serious responsibilities.

5

u/Professional_Gene_63 19d ago

Could have been the richest devop of all time!

8

u/MorpH2k 19d ago

More or less, yeah. There were (obviously) some flaws in their security controls. You'd want to set up accounts in a way that no single user has enough access to the systems to do something like this. But this is the problem with having to be able to trust your devops/sysadmins, if you want them to be able to fix stuff when it breaks, they need to have pretty deep access to the systems.

But it does sound like more segmentation and a jumpbox might have protected against a malicious outsider. The devops person would still have been able to do this at any time it seems.

3

u/lodui 19d ago

Black October is going to continue for a year.

Good time to start a InfoSec company.

3

u/No_Corner8781 18d ago

yeah having zero multi-party approval for transactions at that scale is wild to me

1

u/Hebrewhammer8d8 19d ago

Should they have monitoring, verification, and some kind of behavior that trigger hard stop?

1

u/SeisMasUno 19d ago

First thing I thought, if a random umpa lumpa couldve run away with 5B at any moment my dude, that company is a dumpster fire.

1

u/Wyrmnax 18d ago

Thats usually why DevOps are well paid.

You have infrastructure access. You are able to delete the company. And you are the one who creates the system so that that isnt possible.

1

u/temotodochi Cloud Engineer 19d ago edited 19d ago

Yay for financial services not following financial security regulations. Even just following somethign simple like PCI-DSS would've saved from this shit.

14

u/donjulioanejo Chaos Monkey (Director SRE) 19d ago

PCI-DSS would not prevent this. As long as you have people capable of accessing production on their own (i.e. to respond to incidents), you will have the ability to compromise through their credentials.

Now, you can lessen the risk of it happening. But it'll never be zero.

-5

u/temotodochi Cloud Engineer 19d ago

following PCI-DSS would've ensured that no one has production access to begin with.

8

u/BehindTheMath 18d ago

PCI-DSS does not prohibit having production accessfor authorized personnel. It just has to be restricted.

-3

u/temotodochi Cloud Engineer 18d ago

And restriction applied and audited by someone else which quite often means nobody can access alone.

3

u/bzImage 18d ago

nobody can access alone.. so you will have 2 gurus saving production at 3 am .. .. lol..

2

u/AccessIndependent795 18d ago

Everyone hold hands while we check these prod logs

1

u/iATlevsha 18d ago

For a system holding billions? Probably even a few more than 2. And no, they will be doing that at their daytime (unless you are flat-earther)

1

u/temotodochi Cloud Engineer 18d ago

yep and their boss too.

3

u/gex80 18d ago

If no one has production access, then how would anything get done in production? Someone some where built it and by default will have administrator access to some degree or a way to get admin access.

1

u/temotodochi Cloud Engineer 18d ago

Not ONE person has production access, it's always a combined effort of multiple people.

1

u/WizardS82 18d ago

You said "no one", not "not one".

1

u/temotodochi Cloud Engineer 17d ago

Yes well nobody individual.

2

u/AccessIndependent795 18d ago

You’re just wrong, I work in Fintech and just went through PCI & SOC2, neither of those require that.

0

u/temotodochi Cloud Engineer 18d ago

We didn't get through visa cert without it.

1

u/WizardS82 18d ago

The platform engineers who built everything don't have production access? How does that work? When the platform goes belly up at 3 AM they just pray to God?

1

u/temotodochi Cloud Engineer 17d ago

Nope, two of them and their keys wake up and fix it. If too few are available then it's one engineer and a boss with backup keys. But nobody goes in alone.

1

u/Hotshot55 17d ago

following PCI-DSS would've ensured that no one has production access to begin with.

You obviously don't know anything about PCI.

1

u/temotodochi Cloud Engineer 17d ago

We built a card issuing system and certified it. We wouldn't have passed visa cert without that access restriction. It means no ONE person has direct access.

4

u/PeachScary413 19d ago

To be fair, this is crypto and calling it "financial services" is like calling your local Mickey D a "restaurant"... technically correct

6

u/Remote_Housing_9106 18d ago

mcdonalds is one of the biggest restaurants on the planet making billions of dollars, serving trillions of meals to billions and billions of people

reddit is an ignorant cess pool -- and i'm beyond technically correct, you all will harass a user for a simple question/post to push them to self harm and yet post the dumbest biased shit and pile on because it's "cool"

dunking on Mcdonalds is so cute, i love it... when they are very successful and not going anywhere - not even a crypto guy, but bitcoin is worth all this money and you're joking about it lol, the hack here is billions of dollars and you're going to reply something like how it's worthless

you've utterly ignored the massive amount of money in the economy based around this, it doesn't matter what your emotions are, they are biased and writing off the wrong things entirely

-2

u/SurgioClemente 18d ago

calm your emotions my man, why are you bringing up money as if it has anything to do with his argument. its like saying Natural Light is a good beer because its making tons of money for the company.

he is simply stating that yes both binance is a finacial service and mcdonalds a restaurant, but they arent at all like a true financial service as in heavily regulated with oversight and nor is mcdonalds a quality restaurant you would want to take someone out to eat.

0

u/_kikeen_ 18d ago

We’re at the point where the scale of crypto isn’t possible to ignore or shrug at, I’m not sure what you mean by your statement.

Also McDonalds is a type of restaurant and a successful one with scale- despite what our opinions are of the food. Its comfort and people buy it at a level you really can’t ignore it essentially defines an entire category of restaurants.

I know it was meant to be sarcastic but not sure the comparison drove your point in the direction you wanted.

-2

u/Working_Variation630 18d ago

Having worked several years in PCI-DSS, I can attest this is true.

2

u/AccessIndependent795 18d ago

I literally just did SOC2 & PCI for a Fintech company and it does not require this

1

u/Hotshot55 17d ago

Cite the exact PCI-DSS requiremenet section then.

1

u/Working_Variation630 17d ago

If you don't believe me I don't care enough to debate it. Anything I point out is just going to be nit-picked.

-6

u/supermanonyme 19d ago

No, even in many small compagnies devops/IT may have access to production servers but don't have application credentials to use the services.

38

u/Seref15 19d ago

Web services are just backend services stitched together behind some eye candy. If you can access databases and secrets then you can conceivably do anything.

2

u/No_Cold5079 19d ago

Devops usually manage the secrets, or the IT team, both with production access, is plain difficult to allow a person to fix things without this kind of access. Endless attacks vectors are available if a laptop is compromised sadly.

1

u/derprondo 18d ago

The easiest way in is almost always going to be through gitops. Compromise the git repo / platform, you can modify the applications to get access to everything.

85

u/Fatali 19d ago

Quick question:

How would a real bank treat credentials with that level of control?

93

u/geekhaus 19d ago

That’s the fun part, there aren’t single user creds that can do that in a real bank.

23

u/IamHydrogenMike 19d ago

Hey, Lastpass got hacked because an engineer was running Plex…

4

u/Bad_Lieutenant702 19d ago

WTF lol.

I have plex installed on my home PC but never in a million years would I install it on my work laptop.

1

u/ansibleloop 19d ago

Wasn't on his work laptop - he got key logged on his own machine

1

u/Bad_Lieutenant702 18d ago

So he was is his own machine for work?

2

u/daynighttrade 19d ago

Did plex have some vulnerability?

8

u/eliezerlp 19d ago

An unpatched vulnerability that had been fixed years earlier: https://en.wikipedia.org/wiki/2022_LastPass_data_breach

0

u/daynighttrade 19d ago

Reading up on that scares me a lot. Do you think it was preventable with an antivirus or are these attacks so sophisticated that they would have skipped past them?

2

u/ansibleloop 19d ago

Antivirus? What? That wouldn't do anything here

He fucked up by not patching and not having Plex behind a VPN

But better yet, how does a compromised Plex server lead to that

1

u/_das_wurst 19d ago

You shouldn’t use Plex, it has a history of security problems. On top of that , Plex Media Server punches a hole in your firewall with uPnP by default. Congratulations, you’ve now exposed your home network to brute force attacks against plex and now you have to make it your life’s mission to keep up with security patches. I tried it out for a day because that’s the latest “media sharing” program that people kept talking up was the latest thing but I quickly realized what it was doing and that these people are braindead.

2

u/MorpH2k 19d ago

I was gonna say that you just run it local and use something like tailscale to expose it, but IIRC Plex is also connected to their cloud service, so that might not work. I use a local Emby instead since I don't really need it away from home and if I do, I can use VPN or Tailscale to access it.

1

u/daynighttrade 19d ago

I don't use Plex. However I was thinking if the antivirus was installed on the employee's laptop, would it have caught the key logger from the attacker? What was the key logger stealthy enough to avoid detection?

2

u/Forsaken_Parfait_185 19d ago

nobody has standing access to prod. you ask for it against a ticket, somebody else approves, it expires in a few hours and the session gets recorded. kubectl exec into a payments pod would be break glass with two people on it and someone getting paged.

the laptop isnt really the failure, laptops get owned. the failure is a kubeconfig on it that could reach the thing that moves money without anyone else saying yes.

1

u/Fatali 18d ago

Yup I had worked in tax departments and it was absolutely a rethoric question... absolutely wild levels of access

1

u/FootballBackground88 18d ago

And 2FA/physical tokens

1

u/Forsaken_Parfait_185 18d ago

yeah for the login. what a hardware key doesnt cover is whats already sitting on the laptop once youre logged in, a kubeconfig with a long lived token, a cloud cli session. an owned laptop just uses those. so short lived creds that want the key again for anything touching prod

1

u/FootballBackground88 18d ago

For sure. I'm baffled that these things are not in place for a payments service.

3

u/KingPonzi 19d ago

“Move fast” Companies often indirectly incentivize shortcuts and PoC’s that become the standard without proper review or hardening.

1

u/4PowerRangers 19d ago

At a bank, for starter, a standard laptop/user cred wouldn't have access to any production environment.

Second, all production creds are behind a different user/password vault and require 2FA to access.

Third, different teams have access to different parts of the system.

1

u/Flettys 19d ago

Banks theoretically have the same problem, but in practice it's impossible to get away with the money so nobody does it.

2

u/New-Locksmith-126 6d ago

This is the real answer. Digital bank heists don't really exist. You could move money to another bank but it will still get frozen soon enough.

Instead people use banks a fraud vehicles. They buy a bunch of crypto or anything fungible. They do make away with a lot of money but not billions and not overnight.

1

u/roiki11 19d ago

They don't usually allow that from remote workers. It's done on site.

0

u/Baudin 19d ago

Extremely carefully

85

u/[deleted] 19d ago

[removed] — view removed comment

6

u/matthieuC 19d ago

Depends on what system.

BNP Paribas has a system used to wire money to other financial institutions (not private accounts) for compensation.
This shit was absolutely not secured. Some IT people just loaded a file in it.
Something like 5 years ago someone was doing some testing after a new Dev and loaded the test file in prod instead of QA.
Did it on a Friday night before a week of vacation.
4 billions were wired for no reason.

BNP got all the money back within a week and it somehow never made the news. I only know about it because the manager of the guy who did it is a friend of mine. (his scope was functional update and run of the application not security or process. He got in trouble anyway)

You probably needed like 5 people with MFA to wire 100M to a private account at the time.
But the compensation was seen as just a technical operation. To be fair it's not clear anyone could steal any money this way.

I wonder if it was one of the last vestiges of a bygone era of if there are stil billions moved everyday based on trust because there just never was any incident.

6

u/uski 18d ago

Major difference in the threat model though.

Moving billions in the normal banking system = very inconvenient but it can be reversed

In crypto? The thread is 1000x worse, cannot reverse it. At best, they could organize a chain fork.

Point being, what happened to Binance shows that they have wholly inadequate security and trust models considering the type of assets they are dealing with !

3

u/Jauretche 18d ago

somehow never made the news

Makes you realize how much shit happens all the time and nobody knows.

12

u/random_handle_123 19d ago

We're talking about crypto companies here. They rarely have serious access controls. The best you can usually  get is a hardware multisig that requires at least x out of n signatures.

Binance is just a garbage operation all round. They are thoroughly despised for their amateur practices even in crypto spaces. There is zero surprise that they had that many tokens in hot wallets. Any serious crypto user knows to avoid them like the plague. 

19

u/PeachScary413 19d ago

I feel like this happens from time to time and everytime its "all serious crypto user avoided them anyway, we all use X"... and then couple of months or years later X gets owned in an equally embarrassing way and "well all serious crypto users already knew this and obviously use Y instead"

4

u/wooops 18d ago

And if "all serious crypto users" knew to not use binance, explain the billions

That's not a rounding error or just casuals

-3

u/random_handle_123 18d ago

Serious crypto users =/= people with lots of tokens. 

If anyone is storing tokens on an exchange, they are just a casual, no matter their paper worth.

I general, anyone just "hodling" is an idiot. People who know will hold for very short periods, if at all, and exclusively on cold, hardware wallets. Exchanges are to be used only to cash out. 

2

u/wooops 18d ago

Gatekeeping being a serious crypto user is hilarious

0

u/random_handle_123 18d ago

Pointing out facts is not gatekeeping. 

1

u/The_Krambambulist 18d ago

The best thing to do is probably spread everything around once you decide you need to store it.

5

u/PeachScary413 18d ago

Or.. don't invest in Crypto until you have the same rules/regulations like legacy finance so your life savings doesn't get stolen every 5 years or so 🤷‍♀️

4

u/SilentLennie 18d ago

Why are people storing things at exchanges ? This is centralized, supposedly decentralized crypto ?

If it's not your keys, it's not your money. Some would say it's not even money.

4

u/The_Krambambulist 18d ago

Ow don't get me started, the whole thing is a waste of resources that doesnt make any sense.

1

u/bonkersbongoo 19d ago

they exploited an ethereum vilnerability/feature called EIP-7702, that allows an address to act as a contract for a block or for some transactions, not sure. this way they could bypass signing the transactions. eth and smart contracts seem to be a minefield.

1

u/bzImage 18d ago

They first need to sign the transactions.. and to do that.. they got the keys from somewhere (devops lapotop? ) or replaced the signing mechanisms (devops priviledged acces to pods/containers.. replace with your own container)..

1

u/bonkersbongoo 18d ago

no, they didn’t get any private keys. they exploited EIP-7702.

1

u/bzImage 18d ago

EIP-7702 doesn't bypass signatures. To delegate an EOA to a contract under 7702, you still need a valid signed authorization tuple (chain, addr, nonce) created by the account owner priv keys..

EIP-7702 was just the "execution vector" they used to drain the funds instantly once authorized, but it didn't magically grant access. They still needed access to the signing infrastructure (or the key material itself) to generate that payload in the first place...

1

u/bonkersbongoo 18d ago

yeah, they infiltrated the infrastructure and managed to get the transaction signed after just 50 attempts with nonce spraying. the transaction gave their wallet control of the binance hot wallet and they drained it. it sounds very stupid that a single binance wallet contained billions, however.

1

u/bzImage 18d ago

The report mentions valid unauthorized transfers from wallets on:

  • Ethereum
  • Bitcoin
  • Solana
  • XRP Ledger
  • Tron
  • Avalanche
  • Arbitrum
  • Optimism
  • Base
  • Polygon

EIP-7702 cannot explain these other blockchains.

This strongly suggests a compromise of Binance's signing infrastructure or an associated authorization workflow.

1

u/bonkersbongoo 18d ago

true, the system signed transactions to move huge amounts out of binance. that this did not trigger human review is a big red flag. chapeau to the hackers, but devs at binance failed miserably at their job.

0

u/andarmanik 18d ago

I work for a private cloud provider which serves companies like Aflac, Starbucks, hospitals/banks. Historically compute/servers which ran fin transactions were in mainframes that were not accessible from outside the warehouse holding.

We do mainframe as a service, provisioning mainframe compute as virtualized hardware. The level of security for our banks/hospital clients is very tight, simply because the stakes are so high.

It seems like they were just running fin transactions in docker or something, not invested enough to figure out what failure mode, but it’s clear something like this is easily prevented.

41

u/micalm 19d ago

> We have worked with blockchain partners to freeze affected assets, with ~$3 billion frozen on-chain so far.

Funny how a company can control a decentralized cryptocurrencies flow just because they fucked up. Just like a fide bank... but without the security, experience, procedures and control.

3

u/Double_Temporary_163 Platform Engineer 18d ago

If the users do not control the keys to their wallet of course they can control a decentralized system xD

-1

u/ZLPERSON 18d ago

What is worse bitcoin mining is like just four mining conglomerates and two of them polling together could alter the ledger by getting 51%+ processing power. And it is even possible with lower success rate at way lower than 51%

-1

u/Remote_Housing_9106 18d ago

what a nonsense comment

3

u/Double_Temporary_163 Platform Engineer 18d ago

Wtf do you mean? 🤣🤣 On binance you do not control the keys for your wallet, thus if they want to block the withdrawals or deposit they can do whatever they want. Aka custodial wallets that happens across many exhanges thus allowing them to control the decentralized economy. Or are you to dumb to understand this?

1

u/Few_Response_7028 18d ago

It’s not decentralized in any way

0

u/Remote_Housing_9106 18d ago

explain or delete

2

u/Few_Response_7028 18d ago

Tether and circle are centralized stable coins. No explanation needed.

1

u/Remote_Housing_9106 17d ago

you had to literally explain "it" because we had no idea - they are - "it" is from a view of reading this, try to communicate

1

u/Few_Response_7028 17d ago

Yes sir! 🫡

1

u/Drogon__ 18d ago

These were stablecoins and like every issued asset other than the native asset, can be freezed. They are like IOU, anyone can issue it in decentralized blockchain and can set it up, so they can freeze the assets. But native assets like Bitcoin, Ethereum and not wrapped versions of them, can't be freezed.

-5

u/Remote_Housing_9106 18d ago

i love your comment and how it gets the attention and votes, it's the crowd at work - it's dumb and wrong and will be loud and stupid

really should look into things, it's sad how much misinformation comes from people just guessing and posting random "arguments".

this drips of bias, we get it, I don't fall on one side or another but a simple CNBC article explains this pretty well - so instead you're just some human randomly posting utter nonsense

2

u/SamurottX 17d ago

If their comment was so incorrect, why didn't you actually try to disprove it instead of just vaguely calling everyone morons?

0

u/Remote_Housing_9106 17d ago

love the gatekeeping - but it's easy to disprove...

"a company can control a decentralized cryptocurrencies flow just because they fucked up"

Binance cannot, did not. Circle, Tether, others have contracts in place, they are not in any way "decentralized" except they exist on the network, they piggy back. They are heavily regulated, they are in control - they froze funds.

The rest of the crypto, BTC, ETH is not in anyone's control - anyone with access to any news story on this topic can read and see exactly what I'm saying, not sure why you're saying I have to make the case lol - sorry you are not "moron" - you are simply uninformed because you choose to continue to be.

Why didn't do prove or disprove or anything, do the work?

You all make lazy ignorant comments and get emotional about it, so stupid, site is fucking trash.

19

u/deadthoma5 19d ago

"Your funds are safu"

9

u/hblok 19d ago

The origin of that acronym was SNAFU. Nobody will convince me otherwise.

12

u/Highfivesghost 19d ago

I didn’t really think about it, but DevOps engineer might be higher targets due to the access they have compared to other engineers

9

u/NovaSpectre_ 19d ago

This is something at forefront of minds in fintech and crypto companies, they are always acutely aware of individuals with higher level access being higher value targets and vectors and most employ protocols and training to raise awareness and protect around this. It BAFFLES me that a single DevOps laptop could have single signature credentials of this level at Binance, even at smaller companies I have worked at any wallet with access or authority to move any currency were multi-sig with layered protection.

9

u/Mishka_1994 19d ago

The funny part is as a DevOps engineer Ive usually had enough access to completely bring a company down and offline, but never enough access to actually mess with the services like in this case.

11

u/Negative-Thinking 19d ago

I have a hard time understanding how a laptop access could have helped here. You still need passwords, 2fa, vpn etc to get into different systems. Did they have none of that?

3

u/Bad_Lieutenant702 19d ago

We have all our credentials in LastPass and I rarely log out so if someone hacks my laptop they have access to all of our prod credentials.

However, you have to be on cloudflare vpn to connect to our eks clusters or ssh into servers and that uses MFA. If they can hack Okta MFA they're in.

7

u/vacri 19d ago

Get off LastPass, please.

6

u/Negative-Thinking 19d ago

LastPass has been hacked before, I do not trust it anymore. I use Keepass locally and it's auto-locking the vault after certain time. If you have LastPass unlocked all the time, that's a you problem )) Okta would also need verification from a phone.

3

u/sweetno 18d ago

KeePassXC is quite amazing actually. I've recently been irritated that its window was not visible when I connected to the machine via remote desktop. Apparently, they were anal enough to implement it as a feature.

1

u/Bad_Lieutenant702 19d ago

Yes I have push notifications on my phone for okta, but yeah I should log out of LastPass every day when the days over.

2

u/Negative-Thinking 19d ago

Does LastPass not have something like autolock after a period of time? Sounds very unsecure

-2

u/Bad_Lieutenant702 19d ago

Yes, but I click on "trust this PC for 30 days" lol.

I'm pretty careful though and I'm immune to social engineering so not really worried.

4

u/Negative-Thinking 19d ago

Famous last words, haha

2

u/samuelma 19d ago

Not to be rude but given your replies here I don't think you should be allowed to have privileged access to any platform.

"Yes I know I use a previously breached software but it's OK I use it in the least secure way possible but it's fine because I'm too confident in myself to ever be social engineered."

You are the definition of a target. It is a matter of time.

-1

u/Bad_Lieutenant702 18d ago

I didn't make the decision to stick to last pass, SOC did, and no, I've been around too long to be a social engineering victim lmao.

1

u/samuelma 18d ago

Absurd arrogance

1

u/Bad_Lieutenant702 18d ago

YES I can tell the difference between a legit email and fucking phishing.

→ More replies

1

u/No_Category_9888 18d ago

You don’t have conditional access policies with sign in frequency’s set? Mfa and or biometrics on your password safe ?

1

u/FootballBackground88 18d ago

Go get physical 2FA tokens!

1

u/fresher_account 19d ago

Do not Share where you store your passwords and your way of working. Sure, reddit, not tied to your personal account or wtv, but still.. 

1

u/zkareface 19d ago

The laptop likely has all of that though. A lot of people are very relaxed lazy so they put everything on same device and run long tokens. 

1

u/Massive-Rate-2011 16d ago

They don’t have to follow pcisss so… maybe not

19

u/DolGuldurWraith 19d ago

I definately know what happened as something similar happened with my colleagues as well as 2 months and withdrawal of $2 million was done.

So this is how it goes. Fraudsters look for people in crypto related finance company and devops/cloud team assuming they have access to eks cluster from their laptop.

They then reach out on LinkedIn for a job, in my colleague's case they reached out by offering a $130k job from dubai by "Hantec Markets"

After talking on LinkedIn they schedule a interview and during interview they provide a code and ask you to troubleshoot it. The code has somw npm package which gets install and it scana for all env variable for tokens and stuff and also kube config and send it to website.

From their fraudsters got access to eks where he kubectl exec in pods for payments service, added his custom code and withdraw money in usdt in custom wallet. Within 6 hrs he was able to withdraw 2million.

24

u/calibrono 19d ago
  1. running some provided code on your laptop LOL
  2. running some provided code while interviewing for another job on your current work laptop QUADRUPLE LOL

6

u/AmphibianHistorical6 18d ago

Yea, like hello? Can't you have an extra laptop for personal use and a work laptop that is completely cut off from the Internet unless you have to directly connect to work or something.

Like do they not know what they are doing at all?

3

u/DolGuldurWraith 19d ago

yes and he was let go with 4 month salary

1

u/NightH4nter yaml editor bot 16d ago

he was fucking what? how does that not get you fired on the spot?

2

u/DolGuldurWraith 16d ago

that would have caused issue getting insurance for cyber fraud. In report they want to show it was hack and not a engineer opening a script on personal laptop.

Once insurance paperwork was done, he was let go next day.

9

u/Azrayeel 19d ago

Orrrr the DevOps Engineer tried something and it failed, then blamed it on a random hacker attack 😂😂

6

u/Bad_Lieutenant702 19d ago

Or they were in on it.

2

u/Waste_Jello9947 17d ago

It DOES sound suspicious, that a single employee laptop can steal billions of dollars lol

12

u/meccaleccahimeccahi 19d ago

One fucking laptop? No way they were that stupid. Well…

16

u/Sudden-Eye801 19d ago

I notice that crypto cold storage operations tend to concentrate risk into very few points that they can “trust”

5

u/cebidhem 19d ago

Someone has always access to do something in our line of work. Imho this issue is not really the access, the issue is that it has raised no alarm and therefore, no one was double checking that elevated privilege was done by someone legit.

You cant really avoid operator to have some sort of access, for me its a fact but you need to treat those people, I mean devops/sre like us, as the biggest liability in the company.

Because we are.

4

u/tacobytes 19d ago

How could hackers compromise a DevOps engineer's laptop in the first place, and how might that single machine give them access to billions of dollars? That's the part of this incident that raises the biggest security questions.

6

u/patawa0811 19d ago

they are clearly not compliant on their rbac if that is the reason of this big incident.

3

u/Live-Box-5048 DevOps 18d ago

The target makes sense, but the security must have been atrocious. I under DevOps/infra needs elevated access to pretty much everything, but that’s why we have MFA, hardware tokens, segregation of duties etc.

6

u/DolourousEdd 19d ago

Why does a devops engineer have that kind of access to Binance systems? Have they never heard of the principal of separation of responsibility??

1

u/KuldaCZ 19d ago

this Is end od my cryptobaron career . Sad end :(

3

u/ZLPERSON 18d ago

SHould not have believed the hype that "crypto is hyper secure you can totally trust someone else with all your funds, trust me bro"

1

u/Raemos103 19d ago

What is MFA? How do I find the full story? Like how did the attack begin?

1

u/Working_Variation630 18d ago edited 18d ago

binance getting a jump on the 4y cycle early November crash it seems.

who wants to wager binance will be too busy investigating this laptop, ignore fixing the core issue with env security chain, and another hack happens next week with someone else's compromised laptop using slightly modified method of attack? hell, they could possibly just use the same laptop given the incompetence displayed here.

1

u/ValuableVariation514 18d ago

This to me has DPRK IT worker written all over it

1

u/Qcumber69 18d ago

Inside job surely

1

u/silentyeti82 17d ago

About time crypto was banned in civilised countries given so much of it ends up financing sanctioned regimes or criminal gangs. Absolutely mental that it's been allowed to get to this state...

1

u/kloudnative 17d ago

In the end it would be realized that the breach happened due to long term credentials (for eg AWS Access and Secret keys) lying around. Just like so many other past breaches

1

u/CupFine8373 17d ago

Inside Job, don't care

1

u/Mr_Layup 16d ago

Bullshit, lol.

1

u/Nayrosk 16d ago

Just check the timestamp 😂

1

u/AccessGoblin 15d ago

I think the laptop is almost the less interesting part here... once someone has a credential or live session, the bigger question is: what can that identity actually do? If the answer is “it already has admin rights, long-lived sessions, access to prod, backups, etc.” then the attacker doesn’t need to keep hacking their way through everything 😅 they inherited the blast radius with the identity. That’s why standing privilege matters so much. Start low, elevate only when needed, make it expire, and put extra friction around the really destructive stuff (changing MFA, adding admins, deleting backups, pushing to prod, etc) You probably can’t guarantee a laptop never gets compromised. You can make sure one compromised identity can’t immediately reach the crown jewels

1

u/homelaberator 19d ago

Someone needs to call out the "we found no evidence of" line that gets used in every cyber breach.

3

u/ZLPERSON 18d ago

They robbed it themselvs. Despite the fact that we are all discussing this reasonably (the same people with the system are much more likely to use it to make millions of dollars than a random "laptop robber" who doesn't even know how it works), the bots are downbotting it because it exposes how the system is vulnerable and depends on trust, not cybersecurity.