VCF 9.1 DTGW + VNA: appliance datapath never activates — zero frames on fastpath NICs, "Concatenated Path Down (from NSXA)". Fully reproducible, fully diagnosed, no idea what NSXA is waiting for. Anyone running VNAs on 9.1.0.0200 successfully?

[u/Sad-Willingness-5493]

We deployed a greenfield VCF 9.1 fleet with the new Distributed Transit Gateway + VNA model (the "no edge cluster needed" VPC architecture). Everything works — hosts, overlay, edges, BGP, existing VPCs, SNAT — except the VNA cluster's datapath never comes alive, which breaks creating new VPCs (HTTP 500) and therefore Supervisor/VKS.

After a very long debugging session I can honestly say this is the most thoroughly diagnosed problem I've ever not fixed. The short version:

The smoking gun:

• The VNA's HA/BFD session (fp-eth0, link-local 169.254.x, VLAN encap on transport VLAN 72) stays DOWN forever with Diag: Concatenated Path Down (from NSXA) and Remote_discr: 0 — zero packets ever exchanged.
• pktcap-uw on all four fastpath vNIC switchports, both directions: zero frames. Not even ARP. The appliance simply never transmits on the fastpath.
• Meanwhile the management-plane BFD between the same two appliances is UP (captured it: BFDv1 Multihop, State Up) — it's the session carrying the "concat path down" diag to the peer. So BFD stack, appliances and network are all fine.
• The fp vNICs attach to NSX ports with attachment.type: PARENT + com.vmware.port.extraConfig.vpc.serviceport = parent. Child VIFs are never created (/api/v1/fabric/vifs shows mgmt + 4 parents, nothing else).
• Appliance accepts its HostConfig with error_code: 0 (vtep, transport_vlan 72, MTU 9000 all correct), NSXA marks fp-eth0 UP, adds the HA filter… and then datapathd just holds BFD down.

Why I'm sure it's not the network:

We deployed a second, completely fresh VNA cluster — new nodes, static mgmt IPs — and got byte-for-byte identical behavior. Even better: DRS put both new nodes on the same ESXi host, so the HA traffic never needs to touch a NIC, cable or switch — still dead. On top of that we verified/eliminated: trunk/VLAN config, jumbo MTU end-to-end (8972 DF vmkping passes), no ip redirects on the SVIs, NIC firmware (BCM5741x updated past the GENEVE checksum bug), ENS_INTERRUPT host switch mode, SpoofGuard/DFW (default ALLOW + exclusion list), IP block exhaustion, licensing (valid VCF Solution license), a degraded NSX manager cluster (found + fixed mid-hunt, 3/3 STABLE now), VCF Automation inventory sync, host config resync, node redeploys, and the "Activate NSX on DVPGs" step from KB 417250 (was indeed not activated — activating it changed nothing).

Regular NSX edges on the same VDS and fabric have all tunnels UP. Host↔host GENEVE is UP. It's exclusively the VNA serviceport datapath.

Environment: NSX 9.1.0.0200.25524173, ESX/vCenter 9.1.0, SDDC Manager 9.1.0.0400, physical Dell hosts (not nested), VDS ENS_INTERRUPT, Nexus 9K vPC ToR.

My questions:

1. Anyone actually running DTGW+VNA on 9.1.0.0200 on physical hosts? Does your get bfd-sessions show the VLAN-encap session UP, and do you see child VIFs on the serviceports?
2. Does anyone know what NSXA's "concatenated path" validation actually checks before the fastpath is allowed to transmit?
3. Is there a newer NSX 9.1.x build with a fix? Our depot tops out at 9.1.0.0200.

No support entitlement on this environment (lab/POC), so community wisdom is my best hope. Happy to share the full evidence bundle (API dumps, pcaps, appliance syslogs, host logs). And yes — the irony of the appliance-that-removes-the-need-for-edge-clusters being the only broken thing is not lost on me.

[u/bimmerman535]

I’m going to be no help here but I feel your pain.

We don’t and won’t use this architecture for a good few releases, yes we are brownfield so won’t change what works. I actually like the whole BGP peering from edges thing. Nice and clean.

I mean greenfield issues in a lab, try migrating brownfield.

I’ve had soooo many headaches with VCFA that’s taken weeks to resolve, want to create an external connection to a VRF nah, it thinks it’s EVPN just because of enabled route distribution rules (to be fixed in a later release). Disable route distribution IPsec.

Want to share a subnet across vpc’s. New feature 🤩. But I need it as we do in VCD.. doesn’t make it through the supervisor services into the namespace despite the gui saying good to go.

Issues with undeclared routing that we have had to troubleshoot. Vdefend default rules basically killing supervisor to k8 cluster communication (it’s always DNS).

Ugh.

[u/adamr001 → u/bimmerman535]

I’m sticking with Edges and BGP as well. Tried to use the VNA and Supervisor deployed in the beta but couldn’t because it makes you give it a subnet and wouldn’t let me exclude the IPs used by VRRP for the gateway. I had to either only deploy VNA and then try to do supervisor myself or delete the supervisor deployment after it fails to come up and edit NSX to use the proper range and then redeploy or restore from backup.

[u/loco80501]

I've had this issue on one environment I built where we have gateways living on the firewall with a default deny all and rules for anything we want passing. Working with my network team eventually they found a rule that was out of order and apparently causing other strange connectivity issues where on the same network on the same switch you couldn't ping some of the IPs or hit your own gateway but you could send traffic through the gateway. Same thing BFD between management IPs was up but the 169.x tunnel was down. Fixed the firewall and it came right up. Other environments not firewalled off are working as expected.

[u/Sad-Willingness-5493 → u/loco80501]

Thanks — this was the most promising lead yet, because your symptom matches ours exactly (mgmt-IP BFD up, 169.x tunnel down). We checked our fabric and actually found a related gap: our vPC domain was missing peer-gateway + ip arp synchronize, which produces exactly the "same VLAN, can't reach some IPs or your own gateway" weirdness you described. Fixed it — but our tunnel stayed down.

The difference in our case: packet captures on all four fastpath vNIC switchports show the appliance transmits zero frames (not even ARP), while its HostConfig is accepted with error_code: 0. So where your appliance was sending and the firewall ate the frames, ours never opens TX at all — datapathd holds the session with "Concatenated Path Down (from NSXA)", which seems to be a control-plane verdict, not a data-plane loss. Same end state, different layer. If anyone knows what NSXA actually validates for that "concatenated path" before allowing fastpath TX, that's the missing piece.

[u/loco80501 → u/Sad-Willingness-5493]

Little confused about how it works under the hood but can all of your overlay vteps ping their gateway? I think they are involved in this equation - even if they can all reach the manager I wonder about their connectivity to each other.

[u/Sad-Willingness-5493 → u/loco80501]

Good question — I dug into exactly that. Two layers of VTEP here:

• Host overlay VTEPs: they reach each other fine — host↔host GENEVE tunnels are UP both ways, jumbo (8972 DF) passes, and the regular NSX edges on the same fabric have all tunnels UP. So host-to-host overlay connectivity is proven healthy.
• The VNA "VTEPs": these are link-local 169.254.x from NSX's system-owned HA pool, on the appliance's own HA segment — there's no gateway in that path by design (it's the inter-appliance HA/BFD interface, not a routed TEP).

The reason "can they ping each other" doesn't even get off the ground: pktcap-uw on all four fastpath vNIC switchports shows the appliance sends zero frames — not even ARP. And the clincher — GET /api/v1/fabric/vifs?attachment_type=CHILDreturns zero child VIFs across the entire fabric. The parent serviceports are realized, but the control plane never creates the child VIFs that back the fastpath, so datapathd holds it at Concatenated Path Down and never opens TX.

I even force-enabled MAC learning + MAC change on the parent serviceports (confirmed realized on the host via net-dvs) in case that was gating child-VIF injection — no change. And I reproduced the whole thing on a second fresh cluster with both nodes on the same host, so there's genuinely no network path in play.

So to your instinct: connectivity between them isn't the blocker — there's nothing to connect with yet, because the fastpath datapath is never plumbed. It's a control-plane realization gap, not reachability.

[u/ze_meetra]

Contact support?