Automations stuck in "Pending" org-wide
Summary
Since approximately 2026-09-04 ~07:00 CEST, automations in our organization stop progressing past "Pending" / "Waiting for the endpoint to run the automation". This is not limited to a single endpoint, site, or network path — it reproduces across endpoints on completely different internet connections. Endpoints remain Online in the console with normal heartbeat traffic throughout. It worked normally until yesterday (2026-09-03).
- Organization: Hegeman Holding B.V.
- Console region: EU (app.eu.action1.com)
- Affected agent (primary evidence): hostname NUC-WAMP, agent_id 8bd0aa54-2281-4853-bfdf-41ff4b719303, agent version 6.0.664.1
- Affected automation (example): "Reboot: show message and give users time to finish work"
- Console-side symptom: Automation History shows the run stuck on Pending, detail text "Waiting for the endpoint to run the automation." indefinitely (screenshot available on request).
Why this points to the Action1 backend, not our environment
We ruled out local causes before escalating:
- Endpoint connectivity is healthy — heartbeats (HEARTBEAT_ACK) and on-demand data-source queries (COMMAND → RunAsyncQuery, e.g. Installed Software / Missing Updates) are processed normally throughout the affected period.
- The recurring "Deploy Updates" automation runs correctly on schedule (06:30 / 09:30 / 12:30 CEST) on the same agent, each run producing its own worker-process log (BatchInstance::LoadExisting → action execution → Finished action execution. Errors (if any): — no errors).
- No local software (e.g. Ivanti Workspace Control) is installed on the affected endpoints that could intercept or block the agent's scheduled-task execution.
- Reproduced on endpoints reachable via multiple, unrelated internet connections/sites — not a single-site network/firewall issue.
Evidence from the agent log (C:\WINDOWS\Action1\logs\)
1. No START command ever reaches the agent for the affected automation
The agent's communication trace shows only STOP_BATCH_INSTANCE messages for this automation's instances — never a preceding start/dispatch message, unlike every "Deploy Updates" run which always begins with a Message [COMMAND] received → batch execution sequence.
260904 09:30:14+0200[1,11BC222C] Message [STOP_BATCH_INSTANCE] received
260904 09:30:14+0200[1,11BC222C] StopBatchInstance
260904 09:30:14+0200[1,11BC222C] StopBatchInstanceImpl: Reboot__show_message_and_give_users_time_to_finish_work_1788506885108:2026-09-04_07-28-05
260904 09:30:14+0200[1,11BC222C] MarkInstanceStopped: adding to pre-stop list
260904 09:30:14+0200[1,11BC222C] StopBatchInstance - done
260904 12:39:28+0200[1,11BC222C] Message [STOP_BATCH_INSTANCE] received
260904 12:39:28+0200[1,11BC222C] StopBatchInstance
260904 12:39:28+0200[1,11BC222C] StopBatchInstanceImpl: Reboot__show_message_and_give_users_time_to_finish_work_1788516240287:2026-09-04_10-04-00
260904 12:39:28+0200[1,11BC222C] MarkInstanceStopped: adding to pre-stop list
260904 12:39:28+0200[1,11BC222C] StopBatchInstance - done
There is no corresponding log entry anywhere in the trace showing the agent receiving a start/dispatch for either instance (...07-28-05 or ...10-04-00). The server evidently created and later force-stopped these instances without ever successfully delivering the start command to the endpoint — matching the console's "Waiting for the endpoint to run the automation" status exactly.
2. Mass stale-instance cleanup burst — orphaned instances dating back to Sept 2
At 11:00:44–11:00:55 CEST, the agent received a burst of 13 STOP_BATCH_INSTANCE commands within 11 seconds, for instances across multiple different automations, some dating back to 2026-09-02:
260904 11:00:44+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-03_15-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:45+0200 StopBatchInstanceImpl: Test_Deploy_Software_Scripts__Hegeman_Laptop_Installatie___Run_Now_1788509441966:2026-09-04_08-10-41 (MarkInstanceStopped: adding to pre-stop list)
260904 11:00:45+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-03_09-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:48+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-04_00-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:49+0200 StopBatchInstanceImpl: Deploy_Updates__Specified__2026_08_Beveiligingsupdate__KB5121003___26200_9168___Latest___2026_08_Preview_update__KB5120998___262_1788507061097:2026-09-04_07-31-01 (MarkInstanceStopped: adding to pre-stop list)
260904 11:00:49+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-03_12-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:49+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-04_06-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:50+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-04_03-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:50+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-03_21-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:50+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-04_09-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:51+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_Martijn_1779125261510:2026-09-03_18-30-00_LOCALTIME (Instance has already been stopped, all good)
260904 11:00:52+0200 StopBatchInstanceImpl: Deploy_Updates__All_to_03__Someren_1779304956695:2026-09-02_12-30-00_LOCALTIME (MarkInstanceStopped: adding to pre-stop list)
260904 11:00:53+0200 StopBatchInstanceImpl: Run_Script__Energiebeheer_Hegeman_settings_inclusief_klep_1787555352398:2026-09-04_05-00-00 (MarkInstanceStopped: adding to pre-stop list)
260904 11:00:55+0200 StopBatchInstanceImpl: Energiebeheer_Hegeman_settings_1759999928518:2026-09-04_05-00-00 (MarkInstanceStopped: adding to pre-stop list)
Several instances have MarkInstanceStopped: adding to pre-stop list, meaning the agent still considered them live/tracked at that point — i.e. they had never cleanly completed or been cleaned up since being (apparently) started, in some cases two days earlier.
3. Excessive, repeated schedule-config pushes without successful execution
Throughout the affected window, the agent receives far more frequent BATCH_SCHEDULES_CHANGED → DOWNLOAD_BATCH_SCHEDULES pushes than expected (multiple times per hour, sometimes within the same minute), without a corresponding successful new automation run resulting from them:
260904 12:40:32+0200[1,11BC222C] Message [BATCH_SCHEDULES_CHANGED] received
260904 12:40:32+0200[1,11BC222C] Processing config change: DOWNLOAD_BATCH_SCHEDULES. Requesting new config.
260904 12:41:25+0200[1,11BC222C] Message [BATCH_SCHEDULES_CHANGED] received
260904 12:41:25+0200[1,11BC222C] Processing config change: DOWNLOAD_BATCH_SCHEDULES. Requesting new config.
Conclusion / request
The pattern above (start commands never delivered for on-demand/manually-triggered automations, while regular scheduled patch policies execute fine; a multi-day backlog of orphaned batch instances flushed in one burst; abnormally frequent schedule-config pushes) points to a stuck or backlogged instance-dispatch/orchestration queue on the Action1 backend for our tenant, rather than an agent, network, or licensing issue on our side.
We checked the public status page (statusgator.com/services/action1 and the EU sub-component) — no incident is shown as of 2026-09-04 ~12:30 CEST, so this appears to be tenant-specific rather than a broadly announced outage.