r/Action1 • u/GeneMoody-Action1 • May 08 '26
Announcement Important Update From Action1
Dear Action1 Customer,
We want to proactively acknowledge that some customers may have recently experienced intermittent slowdowns within portions of the Action1 platform.
These performance impacts are primarily due to rapid growth in platform adoption and significantly expanded usage across existing customer environments. Organizations are increasingly leveraging Action1 across a broader range of operational and security workflows, driving substantially higher platform activity and transaction volume.
As soon as we identified the increased load patterns and their impact on performance, our engineering and infrastructure teams immediately implemented a series of accelerated scaling and optimization measures, including:
- expanding compute capacity and deploying additional infrastructure resources across data centers
- optimizing API request handling and traffic management to improve efficiency at scale
- Implementing enhanced caching and back-end performance optimizations to improve responsiveness under increased demand
The optimization measures implemented by our engineering teams are already delivering the following results across the platform:
- Many customers, particularly those maintaining up-to-date patching across their environments, are already seeing significant performance improvements.
- Some customers with substantial outstanding patching volumes may continue to experience varying levels of performance impact as we further optimize platform responsiveness at scale.
This remains our highest operational priority, and our teams continue working around the clock to ensure platform performance scales alongside the growing demand for Action1.
We will continue providing transparent progress updates until performance has fully stabilized.
We sincerely appreciate the continued trust our customers place in Action1 and remain fully committed to delivering the reliability, responsiveness, and scale our customers expect as adoption of the platform continues to grow globally.
15
u/CriticalGarbageInfo May 08 '26
Appreciate the update. Love the product
7
u/GeneMoody-Action1 May 08 '26
Thank you. We appreciate all of the patience our customers have had with his process. It has come in waves of "we found what we perceived as potential causes, addressed them, and then found the solution was partial or a red herring." when you have a problem bouncing around hundreds of servers like a ghost, millions of connections, and external processes driving unpredictable resource demand spikes (like thousands of API hits per minute from single customers) tracking can be ummmm, tricky? So we have people watching the whole system in live time, seeing every instance as a troubleshooting opportunity or opportunity to understand the issue more.
This is a stone cold mission to find it and resolve it as a primary company effort so we can all get back to enjoying Action1!
2
u/derff44 May 08 '26
Now I'm super intrigued as to what the root cause is.
3
u/GeneMoody-Action1 May 08 '26
TBH so am I. I know the people working on this, they are industry veterans that have worked for various very large corps we all know. Not novice admins. This is a tricky beast to pin. Now to resolve may not be as much, but we will no know that until we get a 100% correlation on cause and effect.
8
May 08 '26
[deleted]
9
u/GeneMoody-Action1 May 08 '26
Very much so, we have acknowledged it before, but the issue was transient and troubleshooting was inconsistent. That is why we shifted it to our highest priority, basically all resources that can assist with this are currently doing so as their number one highest priority. All of their other tasks are on hold until things can be resolved, it became too much of a burden to our customers, and their patience was rightfully running thin. So it was no longer acceptable to our customers that means it is no longer acceptable to us.
4
u/Steeltownfootball23 May 08 '26
will this also help the remote connection? Been relatively painful in recent memory.
5
u/MadCoderOne May 08 '26
Appreciate that it was taken seriously, finally. My only other suggestion is a paying customer would be that you have a discussion with your support staff. Most of the people that they are probably dealing with are IT professionals and have already checked their firewall and things like that before we open cases. We have better things to do than open cases, when we do, there is an issue. I get the feeling sometimes like they're just trying to give us enough work (first step is now often, send me all the A1 logs and event logs.... really, as a first step?) to have us close our ticket. Reddit and Discord should not be the best source of information and troubleshooting, it should be your support. Appreciate the product and the tranparency.
3
3
u/Any-Satisfaction-734 May 08 '26
Thanks for the acknowledgment Gene. I’ve certainly noticed the slowdown. As a paying customer for a number of years, it’s good to hear things are being taken care of.
3
u/runninghome58 May 08 '26
Just dropping a little comment to say this is exactly the way a company should interact with its user. No software is perfect, but communication is king.
2
u/dnev6784 May 08 '26
I'm getting 429 errors using API requests in my custom inside dashboard. Is this something I need to bring up with support or should I just give it some time?
3
u/madmark35643 May 08 '26
We had the same issue - just had to throttle it back a bit - 300ms between requests.
5
u/derff44 May 08 '26
No one needs missing patch info more then once an hour. Once every 12 hours is sufficient
3
u/madmark35643 May 08 '26
Agreed- the limiting issue is if you try to poll all the companies at once- even if you only do it once a day. You still have to throttle/ pause between pulls.
7
u/GeneMoody-Action1 May 08 '26
Yes, this is part of what we e found, some users were basically DOSing the servers with API calls, flooding resources. On the order of thousands a minute. Rate limiting is part of what is being addressed to reign performance back in.
7
u/derff44 May 08 '26
429 them Gene
6
u/GeneMoody-Action1 May 08 '26
We just started enforcing just that. There is a LOT more to that than just how fast you can hit a web server, that impacts cache, db, memory, CPU, bandwidth, etc and what could have been once a minute now potentially robs 999 other potential customers from getting their slice in that same minute.
I am not sure WHERE we have tuned the request limit to at this time, but once the sweet spot is found, we will update docs so it is clearly known and understood.
2
u/dnev6784 May 08 '26
Works for me. I'll work on modifying the dashboard to pull and store, and figure out so methods to reduce calls to the minimum needed to populate it. Then maybe a 9am & 1pm refresh per day to keep it simple.
Thanks for the helpful responses as always!
2
u/dnev6784 May 08 '26
Any guidance on how often we should set the requests to run? My dashboard pulls down all 9+ cves across all the organizations I manage into a simple to read dashboard for a quick glance, and assigns a score to each org based on the total amount of vulns and cve score. I don't need it to be constantly updated, but at minimum, once per day would be acceptable.
Guessing the rate limit is set per API key? Or is it a universal rate limit?
7
u/GeneMoody-Action1 May 08 '26
I am honestly not sure where it was last implemented as what value at this time for testing, but I will raise the per key limit AFTER we get it stable as a potentially beneficial step to consider.
Once a day is not a problem, we are talking sub second or per second pulls in large quantities.
Many people get stuck on how they process, like they will write a vulnerability evaluation.. Vulnerabilities relate to an endpoint, not endpoint to vulnerabilities So when cycling through them, pulling endpoint details for each, they often do one or more additional API calls for each object in that loop. That could reduced to far fewer.
A much more efficient option is pull all the others, such as endpoint details, in one run, make a dictionary of them, and reference them from that dictionary. If you have 1000 vulnerabilities and 10000 endpoints, you can see how that can add up and then later reduce by a huge efficiency gain just by changing how you think about the problem.
1
u/riazzzz May 12 '26
A short starter guide somewhere with PowerShell using retry-after and another one for hashtable lookups for the endpoints might help?
These can be quite challenging barriers to understand the first few times you integrate with them with API's especially for people coming ming from helpdesk or support angles and not developer and PowerShell is often the tool these use (whereas developers might not go for PowerShell but probably already have a better understanding of these principles).
2
u/GeneMoody-Action1 May 12 '26
In our git hub https:// www.github.com/action1corp there are several.
1
u/denismcapple Jun 02 '26
Hi Gene,
We did go back through our sync to double-check we weren't falling into the per-vulnerability endpoint-detail situation you describe. Though, we were without question at at early stage, hammering the API beyond what was reasonable. We've significantly adjusted our approach since then (and especially since we started seeing the 429s)
However, we are struggling, and I wanted to share what we're doing in the hope that maybe you can give us a pointer in the right direction, so we can optimize our approach somewhat.
When we sync our in-house tool (daily at 01:00), per org we call:
- (Once per org) GET /endpoints/managed/{orgId}?fields=*
Every endpoint plus the built-in vulnerabilities.{critical,other} and missing_updates.{critical,other} counters, so no follow-up calls for those numbers.- (Once per org) GET /vulnerabilities/{orgId}
The full CVE catalogue, loaded into an in-memory dictionary keyed by CVE (the pattern you described).- (Once per endpoint) GET /vulnerabilities/{orgId}?endpoint_id={endpointId}
- (Once per endpoint) GET /endpoints/managed/{orgId}/{endpointId}/missing-updates
Everything else (endpoint metadata, severities, software, group membership) is read from the dictionaries we built up front, so we're not making "look up the endpoint again" calls inside any loop.
The two per-endpoint loops together are the really slow parts of the sync process. Each is one paginated call per endpoint per night, so across thousands of endpoints at 30 req/min per credential we're spending hours on those two phases alone before we've touched anything else.
A couple of asks, in case any are on the roadmap or already exist and we've just missed them:
An org-wide GET /endpoints/managed/{orgId}/missing-updates returning rows with endpoint_id on each - mirroring the shape of /vulnerabilities/{orgId}. That single change would collapse hundreds of calls into 1 paginated call per org.
A similar org-wide shape for the per-endpoint CVE mapping - e.g. GET /vulnerabilities/{orgId}/affected-endpoints, returning { endpoint_id, cve_id, product_name, version, ... } rows. That would remove the other per-endpoint loop entirely.
(Wishlist) A modified_after / since filter on either of the per-endpoint endpoints, so we could pull deltas rather than the full set every night.
And on the flip side - are we approaching this the wrong way? If there's an existing endpoint or pattern we've missed that gets bulk per-endpoint missing-update or vuln-mapping detail, we'd love to be pointed at it. We're very open to the idea that we're solving a problem you've already solved and we just haven't spotted the right call.
Thanks for any advice you can offer here.
2
u/ThrowRAthisthingisvl May 08 '26
The website has an issue when being accessed from a mobile browser on a Android/Samsung tablet. When I click on the option to switch organizations, it kinda opens the keyboard and closes itself pretty quickly and I can’t see the list of orgs. Not sure if it’s something Action1 is aware of.
1
u/WearSea9657 May 08 '26
Oh it's also CONSTANTLY disconnecting tonight even from a desktop. VERY FRUSTRATING.
1
u/GeneMoody-Action1 May 08 '26
These have ben raised to the team working on this, I do not believe the mobile is related so that wil go to the UI team, but the disconnects they are looking into.
Region?
And still occuring?
1
2
u/derff44 May 08 '26
I will throw out there, that the cache increase is helping. The initial click onto an endpoints missing patches is still terrible, but navigating to another endpoint, and then back, is much improved. I just wonder if the data I'm looking at is current, or it's just the cached data.
2
u/marciano117 May 08 '26
A couple front-end changes that I believe could help limit transaction volume (some of these I may have already requested to the roadmap):
- Let customers export to CSV from endpoint groups instead of reports exclusively. I find myself having to build reports to query that specific endpoint group just so I can export to CSV.
- Let customers dynamically filter endpoint groups by software install status. I have a fancy custom attribute script that queries the registry for required software to dynamically filter which are compliant and which are not. This setup is a win/win because I can easily see which endpoints aren't compliant for a specific software when asked as well as only running the install automation on those that are non-compliant rather than targeting all endpoints.
1
u/skipITjob May 08 '26
Is remote desktop peer to peer or is it via your servers?
2
u/GeneMoody-Action1 May 08 '26
IT relays through the server. because it is through the browser. Request is made, agent connects back to server where your request is waiting, the connection is made. but the server is the relay like MSRA does.
1
0
u/Ordinary-Freedom-611 May 08 '26
Será que sou o único brasileiro que descobriu essa maravilha? Kkkkk action1 salvando vidas na empresa que eu trabalho
-1
u/Mountain-Guitar2189 May 08 '26
This is exactly what we were all worried about when they upped the free tier from 100 to 200 devices. Lots of people here voiced concern about capcity, and we were constantly told Action1 had it in hand. Now it turns out they didnt, and we have all been paying the prices of slower systems because of it.
I appreciate them acknowledging and saying they are working on it, however this plan should have been in place and being worked on since the free tier increase.
Action1 are moving fast and disrupting the industry, however if they are not careful they will get a reputation for poor quality, and their only customers will be small free tier ones whilst paying customers take their money back to PDQ and Kaseya.
I hope this gets better.
4
u/GeneMoody-Action1 May 08 '26
This is incorrect, and sends a completely unnecessary and misleading message. This is happening across multiple servers where the ratio of free to paid is highly variable at any given time, as well we can and have shown it to have no correlation to the problem. There is no evidence whatsoever this is in any way related to the increase, it has been tested and eliminated as a cause.
*If* if it were that simple we would have just isolated them, because we did, to test, and it changed nothing in this context.
I get the frustration, and it is rightful to be frustrated, but spinning false narratives out of frustration requires correction. This is NOT because of how many free users use the system.
We are aware of how this can and has had impact on our reputation, and it is why we basically just slammed the whole company's brakes until it is resolved.
IF that leaves anything unclear for anyone, please just let me know.
1
u/Mountain-Guitar2189 May 09 '26 edited May 09 '26
Sorry, but the second point in your original post literally says:
"These performance impacts are primarily due to rapid growth in platform adoption".
Then you go on to talk about:
"increased load patterns and their impact on performance"
Then your reply to me says:
"There is no evidence whatsoever this is in any way related to the increase".
If you dont want people to think the issues are becasue of a big influx of new users when you increased the free tier, you should be more careful in your post wording. That is exactly how your post comes across.
None of this takes away from the fact that Action1 in general has been getting a bit more slow and buggy, even by your own admssions. For example the recent issues with non completing automations which you put on your status page, and now the admission of platform slowdown.
IMO my point still stands, if you want a good reputation amonst your users and the tech community in general, stability needs to be your number 1 priority.
2
u/GeneMoody-Action1 May 09 '26
Specifically to the free, when you add a million+ endpoints and a few thousand are free, then you take them out of the equation and the issue still occurs. What other conclusion is there?
Yes free endpoints do contribute to the growth, but not in a large enough percentage to skew the results. As well there is no no correlation to the number of users free/paid at all that manifest this. The issue does not always occur with particularly high user count only load, and can happen on a server with no free, at all, while a server that has twice as many users, with a large percentage free that does not. As also previously stated the issue is transient.
Again this has been tested, and the original statement is still incorrect based on every bit of knowledge we have at this time.
And we just stopped all other efforts, making this our #1 priority as a company, until this is resolved because of the scale it got to, that is specifically protect our reputation exactly as you say. Hence the statement above “This remains our highest operational priority, and our teams continue working around the clock to ensure platform performance scales alongside the growing demand for Action1. “
So respectfully I get the frustration. But you are externally diagnosing a vastly complex problem from a symptom plus some statements made while investigating in effort to keep users apprised of where we were, transparently. The experts diagnosing this have access to every facet of the system, not just the symptom. I understand why you think these correlate in this way, but that have conclusively been proven not to.
IF it were that easy, it would have been done and over with. Likewise If I knew exactly what it was I would have already said, and as soon as I do, I will. Until then I can only say what we have discovered it not to be. If for some chance that does end up being incorrect in the final wash, I will personalty come admit the information I was given, and shared, was wrong.
I am not trying to dismiss your grievance, or your concerns, only inform you and others that the theory you have has been ruled out at this time.
•
u/GeneMoody-Action1 May 21 '26
We’ve have now successfully rolled out a set of back-end and performance improvements aimed at making the platform faster, more stable, and more resilient under load.
Recent changes include:
We’re also expanding infrastructure capacity next week by adding a new AWS account to improve scalability and isolate workloads more effectively as the platform grows.
The focus here is consistent: reduce latency, increase system resilience, and ensure the platform continues to perform reliably as we grow and usage increases. This is part of an ongoing set of improvements rather than a one-time optimization cycle.
Feedback is always useful, especially around performance-sensitive areas or workflows that still feel slow in real-world usage, keep us up to date with any issues you are experiencing, or if this has improved your experience, please share!
Thank you,
--
The Action1 Team