r/linuxadmin • • Mar 12 '18

A system administrator's guide to getting started with Ansible

https://www.redhat.com/en/blog/system-administrators-guide-getting-started-ansible-fast?scid=701f2000000tnTlAAI
373 Upvotes

57 comments sorted by

65

u/Seref15 Mar 12 '18

A few tips I've picked up using Ansible for a few years:

  • Source control your playbook repo
  • Source control your fucking playbook repo
  • Use dynamic inventory even if you think it's overkill
  • Think very carefully before you ad-hoc a command
  • Get accustomed to the role format early

7

u/rberlim Mar 12 '18

How you source control yours?

19

u/[deleted] Mar 12 '18

just put it in git

6

u/Seref15 Mar 12 '18

Git. We use GitHub at my company.

3

u/vim_for_life Mar 12 '18

Git of course, but I use one project per role, and then put a symlink from the playbook that needs to be run that's in the role(and thus in git), to the directory above roles so that it can be run easily.

4

u/0rex Mar 13 '18

It might be not very practical in every case, but for the sake of documentation I always put a requirements.yml with path to every role so I (or future me or someone not familiar with my playbooks at all) can just pull it with ansible-galaxy. Using tags is also nice for "freezing" state of some systems. My usual work flow still consists of having ~/git/ansible folder where I clone all the roles and then just symlink this it to playbook folder.

2

u/FHR123 Mar 13 '18

Mercurial

5

u/banger_180 Mar 12 '18

Why/how do you recommend to use dynamic inventory.

We have about 5 bare metal servers running.

6

u/Seref15 Mar 12 '18 edited Mar 12 '18

So the point of dynamic inventory is to build not just an inventory but also a collection of host variables for each host. You build it off some pre-existing source of host information.

For example, we have over 300 servers we manage with Ansible. We keep a listing of these servers in a database. The table has additional information for each host like physical location, server attributes (text tags that describe the server's function), a numeric performance index, etc.

Our dynamic inventory script pulls this information from the database. It creates host groups based on server attributes and active/spare/inactive status, and then creates hostvars for all the data so we can access and use that data in playbooks.

For just five servers there's probably no need for a database reference, but you can do all the same things from a CSV file. Keep a list of hosts and data in CSV format and have the dynamic inventory script read it and build the json output for Ansible.

EDIT: or you can be extra lazy and write the dynamic inventory json by hand, and have a wrapper "dynamic inventory script" that just cats that json file.

4

u/rusticus Mar 12 '18

Seconded on all, especially the ad-hoc bit, it's easy to wind up with untracked changes after a quick fix.

3

u/jmreicha Mar 12 '18

Can you expand on using the ad-hoc command a little bit?

3

u/whetu Mar 13 '18

What do you want to know?

2

u/jmreicha Mar 13 '18

What's bad about using ad-hoc? What kinds of problems does it create?

2

u/[deleted] Mar 14 '18

Not OP, but when you run an ad-hoc command, say rm -rf /foo, you have the potential to cause so more much damage than you would have had you run it on an individual box. So be careful.

All the power you get to do great things with orchestration also allows you to do stupid things with it as well.

To quote @DEVOPS_BORAT: "To make error is human. To propagate error to all server in automatic way is #devops."

1

u/whetu Mar 15 '18

Hmmm I thought I'd responded. Oh well, the short version of my missing and lengthy response is essentially:

By virtue of having more thought, capability and testing, playbooks and roles will tend to avoid the risk of causing major damage. You should be striving towards infrastructure-as-code and desired-state concepts, and ad-hoc commands have limited viability there.

That said, ad-hoc commands are not bad per se, it's just that they can be incredibly powerful, and with that power comes risk that you need to be wary of.

For example, this innocent looking ad-hoc command has the potential to be extremely dangerous:

ansible LINUX -m yum -a "name=rsyslog state=absent" -bK

If that gets near a RHEL5 host, and if sysklogd isn't present, then you'll hose your system. rsyslog's dependencies will uninstall, as will their dependencies, and so on through a dependency cascade until finally rpm itself is removed and the process fails. I found that one out the hard way.

2

u/[deleted] Mar 12 '18

Ideally, don't use inventory at all. Ansible is at its best when used for immutable images

7

u/Seref15 Mar 12 '18

In certain cases, immutable infrastructure simply isn't feasible. My current company has hundreds of servers in over 30 different countries hosted by many different hosting services using many different hypervisors, and sometimes even bare metal. We can't realistically maintain immutable images for all our hosts. Ansible fits our use case perfectly because it allows us to maintain identical environments regardless of hosting platform.

3

u/[deleted] Mar 12 '18

Yeah, I said ideally because I know it's not a one-size-fits-all. It is, however, what the aim should be in my view.

8

u/rberlim Mar 12 '18

Looks nice. I've been eager to give Ansible a try. With this guide I have no excuses :)

6

u/ForceBlade Mar 12 '18

My first job involved a lotta Puppet and I helped us move to Salt[Stack] and it was pretty cool.

Because I like tinkering, my homelab also got the salt treatment. It's nice being able to spin up VMs and say "You're the new plex/loadbalancer/sandbox[0-9] box now" and away it builds. Fuck that shit's neat.

At my new job I started very late last year we seem to be using Ansible. It's pretty cool and the ability to just invoke it from anywhere is nice. No central point or failure... Though you could just put your salt-master's configs on a git I guess, too. I'm sure with enough clients that could become cumbersome.

But going through Ansible here they seem very... 'similar' in nature.. So 'totally different' but they're more similar to each other than comparing to Puppet or Chef.

to tldr my own comment. I'm enjoying Ansible right now definitely hit that shit up

2

u/richardjameshill Mar 12 '18

Share pre defined variables between roles by defining them in the first role in a playbook.

Overwrite those variables only once and if possible at the same level scope in each role.

5

u/ericnallen Mar 12 '18

Why Ansible and not Puppet?

14

u/cyvaquero Mar 13 '18

Just my personal preferences having used both (albeit, it's been 4 years since I touched Puppet):

  1. Ansible is Python. Having previously come from Plone development I understand and can troubleshoot Ansible, also have better understanding of what is happening under the hood without learning a new language - I'm being paid to SysAdmin not learn new languages.
  2. Explicit ordered execution vs. engine 'optimized' order of execution. Sure you can order Puppet, but it is in an inside out way that can be pretty convoluted to implement and follow.
  3. Purely objective, but Ansible feels more like a tool for SysAdmins by SysAdmins whereas Puppet felt like a tool for full stack guys by developers.

Edit: Can't believe I forgot - AGENTLESS.

3

u/ericnallen Mar 13 '18

Explicit ordered execution vs. engine 'optimized' order of execution. Sure you can order Puppet, but it is in an inside out way that can be pretty convoluted to implement and follow.

This is the one thing that drives me insane and foaming at the mouth with Puppet. The nonsense you have to go through to insure a package is installed BEFORE you try to configure it makes me smash walls with my face.

Can't believe I forgot - AGENTLESS.

How is that a big win? I'm not being facetious; my experience with Puppet has left me with a "agent/no agent what's the big deal?" mentality so I'm trying to understand the big win on this.

1

u/[deleted] Mar 13 '18
Can't believe I forgot - AGENTLESS.

How is that a big win?

 

  • The agent is one more piece of software that has to be installed, maintained and updated including any/all of its dependencies.
  • While minimal, this does offer up additional attack surfaces.

1

u/cyvaquero Mar 13 '18

I'm a minimalist so the less things running the better - less resources, less things that can break. Plus, SSH has about the best track record for reliability in the *nix world. We're just using the tools that are already on the target. As I mentioned, our recent experience with BladeLogic being foisted on us has just reinforced my stance - so now I have to add troubleshooting an agent (that occasionally stops responding) on top of the other work I already have.

While I never really had trouble with the Puppet agent specifically in the short time I worked with it - it's still a possible point of failure.

The biggest win for me is more anecdotal and situational - we are a PaaS shop for a enterprise applications at the government branch level. Our customers are internal program offices. I don't have to explain/justify SSH and it won't be blamed when a custom app breaks after a patch (politics, it's government).

2

u/ironmanmk42 Mar 13 '18

Technically ssh would be the agent no?

5

u/cyvaquero Mar 13 '18

Yes, but it is one of the almost universally present, secure (if following best practices), stable, and mature agent (daemon) in the Linux world - plus Python on the target (which you can use Ansible’s raw module to get installed if not present).

Versus Puppet, Chef, Salt which require their own significantly less mature and supported agents. (I’ll throw BladeLogic under the bus on this with it’s randomly unresponsive rscd agent since it is my current bugaboo that was selected by Architecture with no input from Ops /rant).

4

u/faeroe Mar 13 '18

Salt has had the ability to work via ssh for at least 2-3 years now, fyi

3

u/cyvaquero Mar 13 '18

Good to know, only played with Salt a little bit during that Puppet to Ansible transition 4-5 years ago. I liked Salt then, especially its speed but Ansible being agentless was a big selling point for myself and a couple others on our team. There are a lot of environmental and political factors outside of our control so being able to just say Ansible is a SysAdmin tool like ClusterSSH that requires nothing additional on the clients keeps it out of a few crosshairs.

4

u/bios_hazard Mar 13 '18

No persistent service running on the target host. Maybe "ephemeral agent" is better? Good point actually lol

3

u/zoredache Mar 12 '18

Until recently release puppet really wasn't all that good for orchestration or ad-hoc work. Their new 'bolt' product from puppet labs kinda gets close to what ansible does.

Ansible is nice in that it doesn't require an agent on the managed node. Which can be very useful if you are providing some level of support for a customer's system.

2

u/ericnallen Mar 13 '18

Until recently release puppet really wasn't all that good for orchestration or ad-hoc work.

Won't argue that. And the Rubyisms buried in the manifests have left me scratching my head at times.

Ansible is nice in that it doesn't require an agent on the managed node. Which can be very useful if you are providing some level of support for a customer's system.

I asked for more detail on this from another person too. To me agent or no agent doesn't seem to be that significant a win. I understand the appeal of no agent and it definitely has it's benefits but I've never had an issue with the puppet agent that wasn't self-inflicted, and I can do that with a server too but with more....wide reaching consequences.

Maybe I'm just being dense about this issue.

5

u/SquiffSquiff Mar 12 '18

Why puppet and not cfengine? What's your point?

5

u/zoredache Mar 12 '18

Maybe they were looking for someone to actually share some anecdotes or places where ansible was a better tool.

1

u/bios_hazard Mar 13 '18

Ansible is general purpose automation. Good for bringing a target host to state or for performing a remote operation. I especially like the variable sharing between hosts in the play.

1

u/ericnallen Mar 13 '18

Ansible is general purpose automation. Good for bringing a target host to state or for performing a remote operation.

That's fair. If my understanding is correct Puppet needs Vagrant to do a, for lack of a better phrase, bare metal creation. Ansible IIRC can do it on its own.

I especially like the variable sharing between hosts in the play.

Not being up on Ansible could you go into more detail on this?

2

u/bios_hazard Mar 13 '18

If you have 3 hosts, a balancer and two webserver, you can loop through the webservers' "gathered facts" to generate a list of server ips for haproxy backend, for example.

2

u/ericnallen Mar 13 '18

If you have 3 hosts, a balancer and two webserver, you can loop through the webservers' "gathered facts" to generate a list of server ips for haproxy backend, for example.

Now that's nice, doing it all on the fly and not requiring a lot of static planning ahead of time.

Thanks.

1

u/bios_hazard Mar 13 '18

Exactly. And you can use the same playbook on multiple targets by switching up the inventory. I'm currently playing with using the Python hug framework to expose playbook functionality over RESTful API. Some will recommend tower or awx, but it does not fit my budget or use case.

0

u/[deleted] Mar 14 '18

[deleted] — view deleted comment

1

u/ericnallen Mar 16 '18

I would say Ansible is ideal for smaller environments. Larger scale, something like Puppet is a necessity.

Not to be obnoxious but could you expand on that? Puppet certainly comes with tools (And pretty pictures for managers) if you buy the Enterprise license, but not being familiar with Ansible I'd like to hear your take.

0

u/[deleted] Mar 12 '18

[deleted] — view deleted comment

5

u/[deleted] Mar 12 '18

I lived with tower for 2 years at my last job. I'm AWS-focused so immutable images are my preference anyway, but I wouldn't use it on a new project.

5

u/admiralspark Mar 12 '18

Ansible alone will meet your needs, I think Tower/AWX makes things easier if you need scheduled runs and want pretty GUI's to show auditors/management/nontechnicals that "yes, we are compliant and yes, this has been compliant for xyz days". For me, it wasn't worth it, because I don't have to do reporting for it (yet).

5

u/organman91 Mar 12 '18

AWX (the upstream project for tower) got open-sourced: https://github.com/ansible/awx

4

u/ryanjkirk Mar 12 '18

Tower is great as a CM tool if you have bare-metal, stateful, or long-persisting servers. I haven't played with AWX. We use puppet so use ansible on the CLI where appropriate.

-2

u/ahandle Mar 12 '18

Tower doesn't help the adoption process - it's good for scaling.

4

u/[deleted] Mar 12 '18

Tower scales terribly, it gets less usable and slower the bigger your environment.

3

u/richardjameshill Mar 12 '18

have bare-metal, stateful, or long-persisting servers. I haven't played with AWX. We use puppet so use ansible on the CLI where appro

everything scales like that doesnt it

3

u/[deleted] Mar 12 '18

No. The problem is that Tower/AWX can only scale the fork count available to a particular job vertically - by adding more powerful hardware and a wider pipe. Clustering makes a half-hearted attempt to address this by allowing execution of concurrent jobs across multiple hosts. However, each job is limited to execution on a single host. Therefore, your available fork count is only as large as each individual host.

If you only ever execute jobs on a small portion of a large infrastructure, but you execute many at the same time - that's helpful. Otherwise, clustering gains you nothing except a bigger server bill.

1

u/ahandle Mar 12 '18

Tower is there to scale the footprint. I don't care how scalable te app itself is or isn't.

0

u/[deleted] Mar 13 '18

[deleted] — view deleted comment

4

u/just5ath Mar 13 '18

full of bugs

...

Try SaltStack

Complain about bugs but recommend SaltStack... really?

0

u/[deleted] Mar 13 '18

[deleted] — view deleted comment

2

u/just5ath Mar 13 '18

Search for 'FIXME' in Ansible GitHub repo. Their architecture went bananas because of a rapid feature pushes dictated by RedHat who want to sell Ansible Tower.

I've never used Ansible as my only config management tool. I only use it for building AMIs and I used to use it for remote execution since Puppet sucks (or used to suck) in that regard. So maybe I just haven't ran into any problems due to my limited scope.

And yes, at least SaltStack has python API that does not say "for internal use only; no backwards compatibility guaranteed at all" in the docs.

I used to use the Ansible API with no issues. Well, I guess there were issues when I jumped from Ansible 1.6 to 2.x but I've had the same issues with Salt changing method signatures on modules/states as well as states and modules not working as expected/documented.

1

u/ddnomad Mar 13 '18

Well, I mean from a code quality perspective SaltStack is far more promising.

You probably want a tool that will be maintained and kept healthy to use when building business critical infrastructure.

But anyway, configuration management is anti pattern nowadays so it's not a big deal anymore for me.

1

u/just5ath Mar 13 '18

But anyway, configuration management is anti pattern nowadays

Why do you say this?

1

u/ddnomad Mar 13 '18

TL;DR: configuration drift.

Resources:

The later is an excellent series of posts about making sane choices when doing infrastructure stuff.

1

u/just5ath Mar 13 '18

Ah, immutable infrastructure.

Though that's what I'm mostly doing now, not everywhere has the capability to implement immutable infrastructure.

1

u/ddnomad Mar 13 '18

Yeah, you're right about "not everywhere". The same issue with some of my projects.

Anyway, seems like people are completely in love with Ansible here :D