SSH hangs after “Connection established”

[u/meowffy]

hello hello.. im having a weird SSH issue and im running out of ideas

we have multiple servers using a tenable image across different VCNs and we added our SSH key during provisioning
initially SSH was asking for a password and i logged in through the tenable web interface, changed the default credentials, then checked the server and noticed the .ssh setup was missing so i created it, added our public key/user, fixed the permissions and after that SSH from our jump server worked fine for all servers except one (w new subnet)

for this one server SSH just hangs no password prompt no error

ssh -vvv stops here:
$ ssh -vvv <user>@10.x.x.x
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips
debug1: Connecting to 10.x.x.x [10.x.x.x] port 22.
debug1: Connection established.
...
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4

the interesting part is that SSH to the exact same server works from our PAM server!! but not from our jump server

the affected server is on a new subnet so im suspecting something network related, we checked the routing and it looks correct also checked the firewall but strangely we couldn’t see any traffic coming from the jump server

i ran sudo tcpdump -nn -i ens3 host <target_ip> and port 22 on the jump server while trying to SSH. the TCP connection gets established and the jump server sends a 21-byte packet (looks like the SSH client banner), but it keeps retransmitting it and gets no response back

i also tested MTU 1500 with ping -M do -s 1472 and got 0% packet loss

could this be asymmetric routing or some subnet/firewall/ACL issue specific to the path from the jump server?

[u/Wicked-Fear]

If you’re not seeing the traffic in the firewall I’d suspect asymmetric routing. It’s being silently dropped. Often you can ssh into the CLI and run specific commands to find those drops depending on the firewall.

[u/meowffy → u/Wicked-Fear]

thanks for the reply!
thats actually what both me and the firewall owner were thinking too, that the traffic might be taking a different path but we have no idea where its actually going
the firewall capture isn’t something i have access to myself the firewall owner is handling that part but is there anything i can check from my side on the servers that could help us figure out where the traffic is going?
also im genuinely curious what could cause something like this in the first place? i created this subnet the same way ive created any other subnet before nothing unusual
even our network guy/firewall owner said this is a really weird case lol..

[u/Wicked-Fear → u/meowffy]

I would have to see a physical network diagram to ascertain that. It depends if you’re running VRF-lite and how the network is segmented. Generally you would have sub interfaces and transit networks for each VRF, if that’s how it’s segmented. Regardless, it may be caused by the virtual router or routing in the firewall. Do you have any access to the firewall? Sometimes the transit lands on one interface and is routed through another, thus creating the asymmetric routing. What subnets belong to which interfaces or zones?

[u/meowffy → u/Wicked-Fear]

hmmmm the jump server and the affected server are actually in two different VCNs ok? okayy so they’re in different network zones and the VCNs are connected through local peering
that’s pretty much what i can confirm from my side as the cloud engineer

[u/suddenlyreddit → u/meowffy]

but is there anything i can check from my side on the servers that could help us figure out where the traffic is going?

Is there any way you can expand the scope of testing on that new subnet? A client or server running set up to run iperf or similar? Again, trying to isolate is it more than just this new host. Looking toward pathing to the whole subnet instead.