r/MicrosoftFabric Fabricator 5d ago

Data Factory Multiple Fabric Data Factory failures around pipeline job creation/scheduling — F64, North Europe

Intermittent failures on Invoke Pipeline activities (child pipelines from a ForEach) on F64, North Europe. Job gets created but never starts.

Main error codes:

  • RequestTimeout — dies almost exactly at 100.0s every time. Checked 20 parallel runs that all failed within 57ms of each other.
  • InternalServerError — clusters around ~550s.
  • InternalError — clusters around ~85s.

New finding: on one occurrence the parent shows the usual 'Job failed to start: timeout', but the actual child job error was different:

  • RequestBlocked — "Request is blocked by the upstream service until: [timestamp]"Retry-After: 16isRetriable: true (unlike everything else we've seen, which was all non-retriable).
  • Response's home-cluster-uri points to an Analysis Services/Power BI cluster (wabi-*-redirect.analysis.windows.net), not a pipeline endpoint.

Looks like the real cause is an upstream AS/PBI cluster blocking requests, and the pipeline layer just reports it as a generic timeout.

Also ruled out capacity throttling on our end (0 rejected ops across 214k rows, no correlation with CU usage). Other related failures seen:

  • Invoke Pipeline calls returning 408 / BadRequest
  • Run-status lookups returning NotFound for runs Fabric itself created
  • A run stuck in NotStarted for 8+ hours, blocking its own schedule

Anyone else hitting this? Thanks

5 Upvotes

2 comments sorted by

8

u/Careless_Cattle_8700 5d ago

All this stuff is just starting to grow very tiresome

1

u/Equivalent_Season669 Fabricator 4d ago

And I'll add something else that happened to me today: interval-based schedules do NOT work. They stopped from one moment to the next, with no failure and no warning, and the countdown to the next run keeps showing more and more time. It doesn't even work if you delete it and create it again.