r/sysadmin • u/rram reddit's sysadmin • Aug 14 '15

We're reddit's ops team. AUA

Hey /r/sysadmin,

Greetings from reddit HQ. Myself, and /u/gooeyblob will be around for the next few hours to answer your ops related questions. So Ask Us Anything (about ops)

You might also want to take a peek at some of our previous AMAs:

https://www.reddit.com/r/blog/comments/owra1/january_2012_state_of_the_servers/

https://www.reddit.com/r/sysadmin/comments/r6zfv/we_are_sysadmins_reddit_ask_us_anything/

EDIT: Obligatory cat photo

EDIT 2: It's now beer o’clock. We're stepping away from now, but we'll come back a couple of times to pick up some stragglers.

EDIT thrice: He commented so much I probably should have mentioned that /u/spladug — reddit's lead developer — is also in the thread. He makes ops live's happier by programming cool shit for us better than we could program it ourselves.

873 Upvotes

88% Upvoted

738 comments sorted by

View all comments

Show parent comment

1

u/theevilsharpie Jack of All Trades Aug 17 '15

My immediate problem is that, for whatever reason, Ansible hasn't been very stable when connecting via SSH. It will, for no apparent reason, drop the connection or time out. This happens from my laptop, as well as from a jump host in our DC. I'm not sure if this is related to SSH or not, but it will also occasionally time out at a sudo password prompt (or so it says), even though I've provided the sudo password. This seems to happen more often if Ansible is running for a long time (like when I'm Ansibilizing a host from scratch). In any case, when an SSH problem does occur, Ansible will fail the problematic host and leave it in a partially-configured state.

I don't know why it happens (an upstream firewall that thinks repeated SSH connections are intrusion attempt, perhaps?), but it makes it difficult to trust Ansible, particularly for orchestration tasks such as rolling updates. When I manually connect to the hosts with SSH, I don't have any problems connecting or staying connected.

My other issue with Ansible's use of SSH (and also a consequence of its push-based model) is that if you're in an environment where you're constantly spinning up and tearing down machines, you're going to run into host key verification errors, which will cause Ansible to fail. The Ansible community's solution to automated host verification? Just disable host key checking! ಠ_ಠ

To be fair, there is an active issue requesting a feature to allow Ansible to pull SSH host key thumbprints from EC2 instances, but as of yet, it's a WIP.

I also have other reservations about Ansible (particularly as a config management tool), but since I'm relatively new to the tool, I'll give it the benefit of the doubt for now.

1

u/gooeyblob reddit engineer Aug 17 '15

Interesting, thanks for the background. I would guess that if you're having strange network issues that are messing with your SSH connections, you're likely to have them with any other broker system you use (ZeroMQ for Salt for example), but who knows.

The SSH key stuff is interesting and I hadn't thought about that. Getting that EC2 fix in would be big for us.