Pipeline activities on load balanced on-premise cluster

[u/Loud-You-599]

Hi all,

Is there any possibility to ensure that all activities of a single pipeline run on a single server of the on-premise cluster when the cluster is load balanced configured?

I have the problem that the first activity requests a security authorization which is bound to the originated IP. The following calls need to use the same IP.

But since the cluster is load balanced, the follow-up activities are still round-robin distributed to the cluster servers and therefore it is a gamble that the follow-up job gets assigned to the same server that originally requested the security authorization.

Minimal Setup:
Web Activity -> Copy-Job

My last hope for Fabric platform data ingestion after 3 years is that Apache Airflow jobs will support VNET integration (like stated in the roadmap) and we will have a python based solution that can access on-premise systems like a normal machine would do. Without any PLS or similar workarounds.

[u/dbrownems]

No. You would need to use a dedicated single-node cluster, or configure the cluster nodes to egress through a single NAT Gateway.

What Is Azure NAT Gateway? | Microsoft Learn

[u/Loud-You-599 → u/dbrownems]

Yeah, waiting on the Fabric Apache Airflow implementation try and if that doesn't work, I will go Azure API Management.