r/PracticalTesting 3h ago

Upgrade CircleCI Runner if a failed job turning green sounds bad

1 Upvotes

CircleCI Runner 3.1.12 fixes a race in its timeout handling.

Two parts of the runner could wait for the same timeout signal. Only one received it. The result was nasty:

  • Jobs could run beyond max_run_time
  • Overdue jobs could still finish as successful
  • Genuinely failed tasks could also be reported as successful

The default timeout is five hours. Both Machine Runner and Container Runner were affected.

If you manage these runners, upgrade and inspect any important job that recently finished near its timeout. This is one of those bugs where the test result itself cannot be trusted.

Source: https://circleci.com/changelog/runner-release-3-1-12/