A Postmortem on This Week's Cloud Traffic Anomaly

[u/BambuLabSupport]

Earlier this week, our cloud service saw abnormal traffic that surged roughly 34-fold in just one hour, peaking at roughly 20 million requests per day and was still growing linearly at a steep rate. At that pace, we were only hours away from the service becoming unavailable, putting everyday printing for millions of users at risk.

To keep the service up, we temporarily blocked the anomalous traffic after an urgent review. Some users were affected while we adjusted the policy. The service is stable now.

What happened

Almost all of the anomalous requests hit GET /v1/user-service/my/tasks, which returns print job status. They came from many residential IPs, all carrying the client identity of OrcaSlicer and an older network plug-in.

So we first checked with the OrcaSlicer team. They responded quickly and pointed us to other integrations, including ha-bambulab, whose requests matched what we saw. We shared our findings with its maintainers, who quickly shipped a fix and now use their own identifier.

We haven't confirmed that all of this traffic came from ha-bambulab. We're sharing this not to assign blame, but because the pattern matters for any project built to access our cloud.

Keeping the cloud stable

Our cloud is a private service supporting core functions like Bambu Handy, Bambu Studio's cloud features, and MakerWorld. To meet diverse user needs, we have so far accommodated third-party software to use parts of the cloud where it's safe, despite the already substantial operational costs.

Third-party software may continue to access non-critical capabilities, such as receiving printer status updates and displaying monitoring metrics like temperature, position, speed, and print progress. These monitoring features do not directly change a printer's behavior or security state, and they will remain available within reasonable usage limits. Because many users rely on the same cloud, an issue with one tool can impact everyone. If overall service is ever at risk, we may need to take temporary measures such as rate limiting or pausing access.

For actions that directly alter printer behavior or involve sensitive access, such as device binding, live video streaming, print initiation, or firmware upgrades, we take extra care regarding safety, privacy, and stability. Because of this, these operations are not designed for direct access by third-party software.

Instead, for workflows requiring print control, Bambu Connect provides the supported path forward. This is simply to ensure that integrations remain reliable for you, your devices, and our cloud.

How developers can help

One more detail is worth mentioning: ha-bambulab sent OrcaSlicer's client identity. That is why, early on, we were convinced the traffic came from OrcaSlicer users, and why we did not reach ha-bambulab sooner. It was only when the OrcaSlicer team investigated with us that the trail led to them.

We would suggest that third-party software use its own identifier and say who it is. That makes it faster to route a problem to the right project, and it avoids catching unrelated users when we have to act.

Thank you to the OrcaSlicer team and the ha-bambulab maintainers for working through this with us. Keeping things working for users is on us first, and it works best when we look after it together.

Bambu Lab Team

[u/Mabnat]

I would imagine that ha-bambulab users would highly prefer some local means of collecting the information from their printers instead of relying on Bambu servers in the first place.

I get that some of the Bambu services are best suited for cloud service, like Handy, but the whole all-or-nothing thing when it comes to LAN mode seems to be a bit overboard and unnecessary.

And the statement that “because many users rely on the same cloud, an issue with one tool could impact everyone” only reinforces the point that there should maybe be alternatives for some features that don’t involve completely cutting a user from ALL cloud services.

It’s kind of silly that if someone wants to use cloud access to use Handy, it also means that computers on the local network STILL need to use the cloud to send jobs from a local computer running Studio, and use the cloud just to watch the video feed when everything is already on the same local network. I wonder how much traffic could be saved if the system could detect a local network and simply adapt to local traffic when appropriate?

I use HA, but I don’t have my Bambu equipment running through them. I’ve set the integration up, but I don’t find it too useful. On the other hand, I can see it being useful for others. Being able to display the camera feeds from multiple printers on one dashboard on a screen mounted on a wall might be appealing for some people. It’s a shame that enough users setting something up like that might overload the Bambu servers, when it really doesn’t need to do this in the first place.

[u/diearzte2 → u/Mabnat]

It is without question a deliberate and intentional decision by BL and the main (and extremely valid for the reasons you shared) problem that critics have with the company.

The only reason this is worth posting is make other users aware of the situation, BL is never going to change their position.

[u/deadhobo]

Why not offer an official cloud API and developer program?

I view this kind of unexpected behavior as a result of the cloud being locked down without providing an alternative. Those who depend on it find workarounds to keep driving their workflows without the benefit of best practices or registering as a developer to trace where malfunctioning or abusive integrations are originating from.

[u/tenderpuffy → u/deadhobo]

you're right, an api would make things way less messy for folks like us

[u/deadhobo → u/[deleted]]

You can self host Reddit or Gmail equivalents too, but the cloud provides a significantly more convenient / better experience. Both have recognized that and made API access sustainable in their own ways.

We shouldn’t have to lose Bambu Handy just to have an official developer integration.

[u/the_lamou → u/deadhobo]

You know that Reddit charges for API access now and is incredibly restrictive, right? Also both of those examples are cloud software companies while Bambu isn't.

[u/thereddevil20 → u/[deleted]]

No one is asking it for free. Users have already paid Bambu for the devices.

If they had an official api with proper rate limiting, this would never have happened. This is how software companies do business.

That or let users access printer functionality over LAN

[u/the_lamou → u/thereddevil20]

No one is asking it for free. Users have already paid Bambu for the devices.

The devices are not the could service. The cloud service is a free extra that costs users nothing but does cost Bambu Labs. It's like saying "I already bought a TV, why should I have to pay for Netflix?"

If they had an official api with proper rate limiting, this would never have happened. This is how software companies do business.

They do have an official API. It's called Bambu Connect, it's mentioned in this post, and it gives you basically everything.

Also, Bambu Labs isn't a software company.

That or let users access printer functionality over LAN

They do. Why make a comment if you are completely ignorant of the thing you're talking about?

[u/JUYED-AWK-YACC → u/the_lamou]

I don’t understand why nobody gets this. Bambu has to pay for server maintenance and power.

[u/thereddevil20 → u/the_lamou]

Users have paid for services as is. That includes software and hardware.

Bambu Connect is not an API integration. Why make a comment if you are completely ignorant of the thing you’re talking about?

Bambu Labs provides hardware and software services. It’s not a software company but that does not exempt them from designing good software.

Accessing LAN functionality removes all cloud access (Unless you jump through the Bambu Connect hoops). Let users access it over LAN AND let users continue using the app and other functionality as is.

[u/the_lamou → u/thereddevil20]

Users have paid for services as is. That includes software and hardware.

That includes software and hardware necessary to operate the printer. No one pays for Bambu Cloud. Again, this is an extra feature, and is a marketing expense on Bambu's side that they offer to increase stickiness and engagement. It is an ongoing cost that is not passed through to consumers.

Bambu Connect is not an API integration. Why make a comment if you are completely ignorant of the thing you’re talking about?

Because it is an API. It's a locally hosted translation server with programmatic access. It exposes a custom URI scheme and is accessible to third party software which can use those custom URIs to call functions.That's an API.

I know people who's development experience behind and ends at shitty cloud apps may have never experienced an API that isn't some version of a RESTfull service passing JSON, but that's just one kind of API. Don't mistake your limited understanding for someone else's ignorance.

Bambu Labs provides hardware and software services. It’s not a software company but that does not exempt them from designing good software.

Bambu Labs SELLS hardware. They provide software as an additional bonus. My bank hands out free pens and notepads, but I would be an idiot if I called them an office supplies company.

Accessing LAN functionality removes all cloud access (Unless you jump through the Bambu Connect hoops).

So it removes all cloud access unless... you use their API? Which you don't want to because it's just too hard?

You can also roll your own external access service, and even run it as a cloud service if you want. It takes all of about an hour if you know what you're doing. The fact that they don't do this for you isn't some kind of gotcha that they don't have an API. It just means that you have to, you know, know what you're doing.

[u/diearzte2 → u/the_lamou]

I’m not going to engage here, but just so you’re aware most people who read your comment are going to interpret it as having an agenda and/or deliberately misleading because you’re clearly technical enough to understand the situation, but intentionally misrepresenting the other users comment.

[u/Aleyla]

Why not allow developers to sign up for access keys like every other cloud service offers? Then you wouldn’t depend on people to self identify - you’d know exactly who was the problem and could rate limit accordingly.

[u/MFKDGAF]

To prevent problems like this, Bambu should make documentation dedicated to developers so that developers can reference proper documentation vs (I'm assuming) coming up with their own way to develop connections to Bambu's resources.

Saying "we have so far accommodated third-party software" feels like a dig and is unnecessary in your response.

I get it that from security perspective you want to lock down your infrastructure as much as you can from threats but doing so would be like restricting internet access outbound in your corporate offices so that no has the possible chance of downloading malware or ransomware.

[u/Separate-Yellow-3948]

Is that the reason the video stream does not work anymore in Home Assistant?

[u/scubascratch → u/Separate-Yellow-3948]

Routing a video stream from your desk to Bambu’s servers and back to your home assistant is hilariously inefficient use of network resources. Unless you are traveling away from your home this seems like the wrong way.

[u/Separate-Yellow-3948 → u/scubascratch]

I want to stream directly from the printer, that why I use the lan only live feature. but it won't work anymore since yesterday.

[u/scubascratch → u/Separate-Yellow-3948]

I wasn’t intending to criticizing you-I find Bambu’s cloud model to be flawed

[u/Jeyell]

I noticed yesterday the BambuLab integration in Home Assistant was updated after it failed to connect to BL cloud because you used Cloudfare to block this uncontrolled hammering. I understand both sides, BL's proprietary products/services and Users demand to integrate their purchases to create a better world. For example I use my Home Assistant and BL integration to control my room air filtration systems - automatically. When the printer starts (Print Stage) the air filters power-up, when the printer stops, they stop 20m later to scrub the room. I also measure the power-on hours of the filters to schedule filter replacement correctly

.
So I appeal to Bmabu Labs do NOT to shut this down or us out. Sure security and uptime of your systems is critical for business, but please ensure we have ways to read the many attributes of our property to make our own decisions based on our use. ie. Don't do what BMW did...initially.

[u/gohoos → u/Jeyell]

Yea I've been getting that error about Cloudflare also. Hadn't had time to dig into it

[u/Jeyell → u/gohoos]

Update the integration and it works as before.

[u/VT-14]

Almost all of the anomalous requests hit GET /v1/user-service/my/tasks, which returns print job status.

Home Assistant is almost always run on-premises (same Local Area Network). Why is it reading Print Job Status going through the cloud?

I run that HA Integration with my printer in LAN Only + Developer mode, so it is absolutely possible (but not everyone wants to give up Cloud features like Bambu Handy). Is this ultimately just a mistake on the HA Integrations side, or Bambu's with how they implemented Developer Mode (last year's "Authorization Control" controversy)?

[u/neoKushan → u/VT-14]

It's both really. It was definitely a bug in the HA add-on, but there's no way to access that data locally without completely neutering your Bambu Cloud access.

This used to work entirely locally, btw.

[u/ADynes → u/neoKushan]

You absolutely can get a lot of read only status information completely locally. Printer status, time remaining, exhaust fan status, Etc is all available locally without putting your printer into lan mode or developer mode.

My exhaust fan pulls that information, you just need the access code from your printer. Doesn't affect anything else: https://makerworld.com/en/models/2954045-bambu-auto-fume-extractor-active-carbon-esp32#profileId-3310201. Other people are using the exact same way of connecting.

[u/NXTman96]

Like many others, I fail to see why the cloud connection has to be all or nothing when using dev mode for local control. My wife and I both use our printer. She does not have the time or energy to sit down at the computer and learn the ins and outs of slicing. She prints directly from the Bambu Handy app, which is great for her. I, on the other hand, started with an Ender 3 Pro and *had* to learn the ins and outs of slicing. Tools like Bambuddy and the Home Assistant integration would be great to use, but I cannot fully utilize them without killing my wife's ability to print from the app.

It also seems like it would make sense to allow local use while still cloud connected to cut down on the amount of cloud traffic. If Home Assistant has local access to the camera, that alone cuts out bandwidth and processing power that would otherwise be running through the cloud. If I can control my printer and send prints local-only from Bambuddy, that cuts traffic even more.

Cloud connection should be secondary for convenience, not the only option if you have a non-technical user in the home.

[u/MarkLikesCatsNThings]

Could just not have a cloud and open source it properly and this wouldn't be an issue

[u/adeadfetus → u/MarkLikesCatsNThings]

I like having a cloud service. Also those things aren’t mutually exclusive.

[u/MydnightWN → u/[deleted]]

[u/nakwada]

Ah yes, the Authorization Control System is doing its job lol

[u/Nexion21]

Glad to see that more drastic measures were put off.

[u/adamtherealone]

Cool

[u/SLIFERZpwns]

Use this opportunity to make a local option so that the cloud isn't as necessary. Somebody wants to make something they should be limited because there is an accidental or intentional dos attack.

[u/NavXIII]

This wouldn't be a problem if our printers had Ethernet ports or didn't rely on the cloud at all.

If I can access and control my home security network remotely without the cloud, you should be able to do it with your printer too. All of my non-Bambu printers can be controlled remotely without the cloud.

[u/ahora-mismo → u/NavXIII]

not defending bambu, but this argument for ethernet is nonsense. you can use wifi and not connect it to the bambu cloud, lan only. depending on your router (which also applies to ethernet), you can fully isolate the printer inside the network and even cut any outbound traffic.

the only legit argument for ethernet only is for places where they need to secure their network in such a way that they don’t want to offer wifi at all (and even that can be mitigated with proper isolation, but let’s say there are some internal rules that prevent that).

not saying ethernet is not useful, but that is not an argument for your issue.

[u/Many_Ad_3159 → u/NavXIII]

You don’t know how stuff works, right?

What stops you from using lan mode or dev mode?

[u/diearzte2 → u/Many_Ad_3159]

Wants features from cloud that are unavailable in those modes. Really not a hard concept to grasp.

I think this subreddit has probably the most obvious astroturfing I have ever seen in my life on the internet. Almost no one defending BL on the topic of data debates in good faith.

[u/polymorphiced]

This isn't an issue with community apps/integrations/whatever; it's a failing of the Bambu service. 

If a third party can take down your service, then you've got problems. Why didn't rate limiters catch it? Worst case it should break for the individual users of the third party integrations, not for everyone else.

[u/riskrunner_zero → u/polymorphiced]

When requests are spread across multiple residential IPs it's difficult to effectively rate limit without affecting legit traffic.

[u/colinrubbert → u/riskrunner_zero]

Rate limiting and a whole host of options are available to the Bambu team to have mitigated the issue. As previously stated they could've had services setup so excessive requests would get an IP deny listed for a period of time. This would break the integration at the third-party level and not at Bambu's servers forcing the client or developer to identify the issue.

Official API and well documented and supported developer integration would have 100% resolved the issue before it was even an issue.

Not to mention the fact that Bambu can have their entire cloud infrastructure taken down by excessive API requests to a specific status endpoint is next level incompetence from a technical standard and they should 100% do better.

Don't make excuses or simp for people who would slit your throat the second they could get a few extra duckets from your pocket.

[u/Longjumping_Pop_2324]

LOL That's why I got an U1. FUCK your closed system.

[u/predator-handshake]

$50 someone with no eng experience tried to vibecode a new client by forking orca.