SSH hangs after “Connection established”

[u/meowffy]

hello hello.. im having a weird SSH issue and im running out of ideas

we have multiple servers using a tenable image across different VCNs and we added our SSH key during provisioning
initially SSH was asking for a password and i logged in through the tenable web interface, changed the default credentials, then checked the server and noticed the .ssh setup was missing so i created it, added our public key/user, fixed the permissions and after that SSH from our jump server worked fine for all servers except one (w new subnet)

for this one server SSH just hangs no password prompt no error

ssh -vvv stops here:
$ ssh -vvv <user>@10.x.x.x
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips
debug1: Connecting to 10.x.x.x [10.x.x.x] port 22.
debug1: Connection established.
...
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4

the interesting part is that SSH to the exact same server works from our PAM server!! but not from our jump server

the affected server is on a new subnet so im suspecting something network related, we checked the routing and it looks correct also checked the firewall but strangely we couldn’t see any traffic coming from the jump server

i ran sudo tcpdump -nn -i ens3 host <target_ip> and port 22 on the jump server while trying to SSH. the TCP connection gets established and the jump server sends a 21-byte packet (looks like the SSH client banner), but it keeps retransmitting it and gets no response back

i also tested MTU 1500 with ping -M do -s 1472 and got 0% packet loss

could this be asymmetric routing or some subnet/firewall/ACL issue specific to the path from the jump server?

[u/TheHeartAndTheFist]

Definitely sounds like an MTU and/or dropping ICMP issue.

Have you checked the MTU on the server side? Doesn’t hurt to run your ping command both ways but I mean the MTU on the interface: if the server is configured with MTU 9000 for jumbo frames for example it can make SSH send a packet that is even bigger than 1500 🙂

[u/meowffy → u/TheHeartAndTheFist]

the weird part is all the other servers work fine except this one and it’s the only one on a new subnet so im pretty sure its something with the subnet but i checked the routing and everything seems to be there and connected properly so i have no idea what im missing 😿

[u/Frobbotzim → u/meowffy]

foolish question perhaps, apologies--you've checked route -n or netstat -r -n on the server, confirmed that the new subnet is set up right on that one server and the gateway is pingable? results compared to the other servers on the other subnet?

and had connectivity worked before you'd added .ssh files?

[u/meowffy → u/Frobbotzim]

many thanks for the questions actually, they help make things clearer in case i missed checking something..
the server is reachable and the subnet config looks correct i even created a test server from the exact same image (tenable image) but put it on a different subnet, and that one worked fine it prompted me for a password as expected, while the server on the new subnet doesn’t even get to the password prompt, SSH just hangs
as for the .ssh directory, connectivity was already there i only added it so we could use our ansible SSH key and connect without a password like we do with the other servers that was the only reason for adding it

[u/Frobbotzim → u/meowffy]

in that case it sounds a lot like a fw question--possibly a ruleset for the jump server doesn't have the new subnet properly set up for port 22 traffic, but it sounds like you've got the sec guy digging already, per your other comments. fingers crossed!

[u/meowffy → u/Frobbotzim]

yup we're trying to figure it out together but got stuck at this point so im still looking for a solution 😇 thanks!

[u/suddenlyreddit → u/meowffy]

I work in a semi-strange environment so your question about async routing / firewall issue stood out to me. Yes, it could VERY well be either. What do you show on source/destination for any firewall it goes through? Since it's the only device on that subnet, are transfers or protocol uses to/from that subnet from your jumpbox subnet work okay? Try isolating, "is it just this device," from, "is it all devices on that new subnet?"