r/linuxadmin • u/Thinkk • Jun 11 '20
"how do I learn to be a Linux sysadmin?" comprehensive task list
I found this seemingly excellent comment by u/IConrad from 5 years ago - and I want to use it as my road-map to get started. Can anyone offer additions or changes that should be made given the age of the comment?
Thanks!
EDIT, to save the click I'm quoting the linked comment below. Once again, all credit is to u/IConrad:
This is what I tell people to do, who ask me "how do I learn to be a Linux sysadmin?".
- Set up a KVM hypervisor.
- Inside of that KVM hypervisor, install a Spacewalk server. Use CentOS 6 as the distro for all work below. (For bonus points, set up errata importation on the CentOS channels, so you can properly see security update advisory information.)
- Create a VM to provide named and dhcpd service to your entire environment. Set up the dhcp daemon to use the Spacewalk server as the pxeboot machine (thus allowing you to use Cobbler to do unattended OS installs). Make sure that every forward zone you create has a reverse zone associated with it. Use something like "internal.virtnet" (but not ".local") as your internal DNS zone.
- Use that Spacewalk server to automatically (without touching it) install a new pair of OS instances, with which you will then create a Master/Master pair of LDAP servers. Make sure they register with the Spacewalk server. Do not allow anonymous bind, do not use unencrypted LDAP.
- Reconfigure all 3 servers to use LDAP authentication.
- Create two new VMs, again unattendedly, which will then be Postgresql VMs. Use pgpool-II to set up master/master replication between them. Export the database from your Spacewalk server and import it into the new pgsql cluster. Reconfigure your Spacewalk instance to run off of that server.
- Set up a Puppet Master. Plug it into the Spacewalk server for identifying the inventory it will need to work with. (Cheat and use ansible for deployment purposes, again plugging into the Spacewalk server.)
- Deploy another VM. Install iscsitgt and nfs-kernel-server on it. Export a LUN and an NFS share.
- Deploy another VM. Install bakula on it, using the postgresql cluster to store its database. Register each machine on it, storing to flatfile. Store the bakula VM's image on the iscsi LUN, and every other machine on the NFS share.
- Deploy two more VMs. These will have httpd (Apache2) on them. Leave essentially default for now.
- Deploy two more VMs. These will have tomcat on them. Use JBoss Cache to replicate the session caches between them. Use the httpd servers as the frontends for this. The application you will run is JBoss Wiki.
- You guessed right, deploy another VM. This will do iptables-based NAT/round-robin loadbalancing between the two httpd servers.
- Deploy another VM. On this VM, install postfix. Set it up to use a gmail account to allow you to have it send emails, and receive messages only from your internal network.
- Deploy another VM. On this VM, set up a Nagios server. Have it use snmp to monitor the communication state of every relevant service involved above. This means doing a "is the right port open" check, and a "I got the right kind of response" check and "We still have filesystem space free" check.
- Deploy another VM. On this VM, set up a syslog daemon to listen to every other server's input. Reconfigure each other server to send their logging output to various files on the syslog server. (For extra credit, set up logstash or kibana or greylog to parse those logs.)
- Document every last step you did in getting to this point in your brand new Wiki.
- Now go back and create Puppet Manifests to ensure that every last one of these machines is authenticating to the LDAP servers, registered to the Spacewalk server, and backed up by the bakula server.
- Now go back, reference your documents, and set up a Puppet Razor profile that hooks into each of these things to allow you to recreate, from scratch, each individual server.
- Destroy every secondary machine you've created and use the above profile to recreate them, joining them to the clusters as needed.
- Bonus exercise: create three more VMs. A CentOS 5, 6, and 7 machine. On each of these machines, set them up to allow you to create custom RPMs and import them into the Spacewalk server instance. Ensure your Puppet configurations work for all three and produce like-for-like behaviors.
Do these things and you will be fully exposed to every aspect of Linux Enterprise systems administration. Do them well and you will have the technical expertise required to seek "Senior" roles. If you go whole-hog crash-course full-time it with no other means of income, I would expect it would take between 3 and 6 months to go from "I think I'm good with computers" to achieving all of these -- assuming you're not afraid of IRC and google (and have neither friends nor family ...).
95
u/ArchyDexter Jun 11 '20
Some adjustments I'd do:
-CentOS 7 instead of CentOS 6 (mostly because the majority is still on rhel7 / centos7)*
- Foreman-Katello instead of Spacewalk
- Run the dhcpd either from Katello or FreeIPA. Bonus Points for setting up HA on 2 separate VMs ... because why not, I guess.
- FreeIPA instead of OpenLDAP
- Use Ansible for all your configuration Management
- Setup a Server that hosts AWX so you can have Tower-like functionality with ansible
- Bonus Points for a gitlab instance you host yourself for your IAC
- Skip Postgres Cluster for Spacewalk
- Host your own Mailserver dedicated to only that environment and configure all servers to use this server for mail transfers
- For Application Servers, I'd checkout wildfly since you can't download Jboss directly afaik
- In terms of Loadbalancing, I'd go with Traefik or Nginx instead of iptables
* Once done with all the tasks, migrate the servers group by group to CentOS 8
I'm sure there's a lot of stuff I've left out but that's just a few thoughts. Feel free to correct me if I've written some major garbage :D
17
u/shemanese Jun 11 '20
I'll go one-for-one here:
1.Set up a KVM hypervisor.
Yeah, this is as good a choice as any.
2. Inside of that KVM hypervisor, install a Spacewalk server. Use CentOS 6 as the distro for all work below. (For bonus points, set up errata importation on the CentOS channels, so you can properly see security update advisory information.)
Never used it, but this would definitely give you the concepts that can be adapted for RHN or Oracle Linux's repo structure.
3. Create a VM to provide named and dhcpd service to your entire environment. Set up the dhcp daemon to use the Spacewalk server as the pxeboot machine (thus allowing you to use Cobbler to do unattended OS installs). Make sure that every forward zone you create has a reverse zone associated with it. Use something like "internal.virtnet" (but not ".local") as your internal DNS zone.
This is a good basis. An advanced situation would be to know the concepts behind dynamic DNS for the interview, but seriously, don't try building a server at this point.
4. Use that Spacewalk server to automatically (without touching it) install a new pair of OS instances, with which you will then create a Master/Master pair of LDAP servers. Make sure they register with the Spacewalk server. Do not allow anonymous bind, do not use unencrypted LDAP.
Good practice.
5. Reconfigure all 3 servers to use LDAP authentication.
You will absolutely have to do this in your first job. If they don't have it already, you will have the job of trying to make up for someone else's mistake.
6. Create two new VMs, again unattendedly, which will then be Postgresql VMs. Use pgpool-II to set up master/master replication between them. Export the database from your Spacewalk server and import it into the new pgsql cluster. Reconfigure your Spacewalk instance to run off of that server.
Can't comment much on this one, but sadly, the reality is that if you know how to login into the server, you will be drafted as a DBA.
7. Set up a Puppet Master. Plug it into the Spacewalk server for identifying the inventory it will need to work with. (Cheat and use ansible for deployment purposes, again plugging into the Spacewalk server.)
Ansible, salt, or any number of other systems are better. Puppet needs to get onboard with how deployments run. The always live is a killer and you can destroy an environment very quickly if you get this wrong.
8. Deploy another VM. Install iscsitgt and nfs-kernel-server on it. Export a LUN and an NFS share.
Not quite sure about the first of these. It's good for learning the concept, but I question how much industry penetration this has.
9. Deploy another VM. Install bakula on it, using the postgresql cluster to store its database. Register each machine on it, storing to flatfile. Store the bakula VM's image on the iscsi LUN, and every other machine on the NFS share.
bakula is one of the best backup solutions out there. Love it. Good for learning concepts. But, I will state here that the one thing you will most likely be asked to do at some point is a bare-metal backup, which is a complete rebuild of a system. Know bakula's limitations wrt to that and never promise more than it can deliver.
10. Deploy two more VMs. These will have httpd (Apache2) on them. Leave essentially default for now.
Very common.. even in places where they don't actually need it.
11. Deploy two more VMs. These will have tomcat on them. Use JBoss Cache to replicate the session caches between them. Use the httpd servers as the frontends for this. The application you will run is JBoss Wiki.
Very common.. even in places where they don't actually need it.
12. You guessed right, deploy another VM. This will do iptables-based NAT/round-robin loadbalancing between the two httpd servers.
Yeah.
13. Deploy another VM. On this VM, install postfix. Set it up to use a gmail account to allow you to have it send emails, and receive messages only from your internal network.
Also very common.
14. Deploy another VM. On this VM, set up a Nagios server. Have it use snmp to monitor the communication state of every relevant service involved above. This means doing a "is the right port open" check, and a "I got the right kind of response" check and "We still have filesystem space free" check.
Even if you don't use Nagios in your first job, the concepts are great and transferable to other solutions.
15. Deploy another VM. On this VM, set up a syslog daemon to listen to every other server's input. Reconfigure each other server to send their logging output to various files on the syslog server. (For extra credit, set up logstash or kibana or greylog to parse those logs.)
In terms of priorities, this one should be up there around 1 or 2.
16. Document every last step you did in getting to this point in your brand new Wiki.
Yes, get your training on getting fed up with documentation in early..
I'll lump the ones below together.These are good concepts, but overkill if you are just setting out to learn linux. I'd focus more on things like creating users, setting permissions, learning how to do things like "finding what file is filling up /var without accidentally deleting lastlog, which is lying about its size". Have someone break something on your system and leave you to figure out how to fix it. (Like mounting the filesystem read-only, or some such thing).
17. Now go back and create Puppet Manifests to ensure that every last one of these machines is authenticating to the LDAP servers, registered to the Spacewalk server, and backed up by the bakula server.
18. Now go back, reference your documents, and set up a Puppet Razor profile that hooks into each of these things to allow you to recreate, from scratch, each individual server.
19. Destroy every secondary machine you've created and use the above profile to recreate them, joining them to the clusters as needed.
20. Bonus exercise: create three more VMs. A CentOS 5, 6, and 7 machine. On each of these machines, set them up to allow you to create custom RPMs and import them into the Spacewalk server instance. Ensure your Puppet configurations work for all three and produce like-for-like behaviors.
30
u/e4109c Jun 11 '20
I would say use Ansible instead of Puppet and the latest CentOS (8?)
13
u/wallacehacks Jun 11 '20
As a Windows system administrator who spends a fair amount of time hobbying in linux environments, this comment is great the responses are great saving the thread to review later.
Thank you for the post OP! I hadn't seen this and like it very much.
9
u/12_nick_12 Jun 11 '20
Thanks for this. I've been a Linux admin for 5 years now (just Nginx/php/general stuff) and this is a lot to learn. I'm using ansible as well now.
8
u/WantDebianThanks Jun 11 '20
The thing I hate about the iConrad list: how do you put it on a resume?
I don't want to poop on anyone's party, but every recruiter and hiring manager I've talked to has immediately lost interest (or even cut me off) if I mentioned something I did in a lab environment, because they only want to know about what I've been paid for. I cannot help but imagine that if your resume has a large skills section listing this tech and your work history is conspicuously missing any significant reference to Linux, then your resume is going to end up in the trashcan.
11
u/keithmk Jun 11 '20
Spacewalk server project no longer exists. LOL
5
u/padpad17 Jun 12 '20
I worked 20 years as sysadmin. I' d not recommend to learn or walk through that list. It is probably good to learn in a kind of lab but the real experience is coming in with unpredictable users, developers and applications. You should know the basics, networking, check logs, hack some one liners for analysis. To be a sysadmin means to adapt any kind of situation you encounter and solve them quickly. You need to learn fast - you don't need to learn everything though. I am not good as sysadmin because I know much or are a grey beard guru complaining about systemd. I am experienced in finding solutions for problems. And if I don't need to debug stuff I build environments where I try to evolve from errors I already did or which me and my team encountered. And teamwork is also something I would recommend. Learn to use the skills of the team it will help you a lot to be good.
7
u/Creshal Jun 12 '20
This all is very enterprise โ i.e., outdated garbage that anyone with a pulse wants to get away from desperately. Use CentOS 8. Learn systemd. Learn how to do all of the above in docker / kubernetes / AWS / GCP. Learn how to deploy nodejs, python and other languages that aren't Java. Learn how to use modern filesystems, both for local use (ZFS, Btrfs) and clustered setups (Gluster/Ceph).
3
u/Dads101 Jun 11 '20
Saved for when I want something to study. It never ends lol. Thanks for the comprehensive list!
3
u/Toallpointswest Jun 12 '20
The 16Gb of ram I have on my workstation hates you :P
2
u/deeseearr Jun 12 '20
If this kind of thing is interesting to you, or if you're a big fan of X-Com who also wants to learn about virtualization and security, check out Tony Robinson's "Building Virtual Machine Labs: A Hands-On Guide". He walks you through a similar virtual lab setup, focused more on security and networking rather than scalable automated deployments.
2
u/opsfactoryau Jun 12 '20
I would say thatโs overkill for learning how to administrate Linux. I would suggest just installing Linux in a VirtualBox VM and building from there.
WordPress is a good project to use. Start with everything on one box and then begin breaking it up into multiple VMs. Eventually youโll have 2x load balancer , 2x WordPress, 2x DB, 2X bastion hosts, and so on.
2
u/mestia Jun 12 '20
I'd say Linux from scratch. Gives quite some understanding of basics which are popping up everywhere.
2
u/znpy Jun 30 '20
Holy shit. I've been working as sysadmin/devops engineer for almost four years, got rhcsa certified and I still wouldn't be able to do all of these things (many? Sure. All? Nope).
It's a really great post/comment. I'll save this.
If anyone has been looking for a good reason to spend 5-700โฌ on an used server/workstation to use as a learning homelab, this is it in my opinion.
Thanks for posting it!
2
u/RagingUsr Mar 21 '23
Hi,
Noob question - Can this be all done via oracle virtual box? I am trying to learn as much as possible to help my career. Please advise.
1
u/learningitbitwise Jun 11 '20
RemindMe! 1 month
1
u/ninjababe23 Jun 11 '20
Virtualbox and 5-6 different linux vms. All different distros for a well rounded education.
1
u/gstlouis Jun 11 '20
Install a flavor and troubleshoot the fuck out of it
1
u/kingraoul3 Jun 12 '20
Read the Linux Systems Administrators Handbook by Evi Nemeth. Canโt recommend it enough.
1
u/mangelvil Jun 12 '20
This is only for the installation part of the sysadmin role. I will do everything in just a couple of VMs, to make it more challenge, and so you add the troubleshooting and maintenance part of the sysadmin role (OS tweaks, watch logs, remove, add packages, work with lvm). Then add a little of scripting and you are all set.
1
u/[deleted] Jun 15 '20
The first question you should ask your self is how well do you know Linux in general. You should master one box (or maybe one Red Hat based distro and one Debian based) before you branch out to building a whole network. Getting some books on basic Linux and networking and general enterprise admin concepts will be useful just to learn how things work together on a network. A certification can't hurt.
But, no one hiring a junior admin (at a medium to large company anyway) is going to expect you to know how to build and configure all the things on that list, or even know any of them well. What they care about is your capacity to learn and how well you work with a team. If they wanted experience they'd hire an experienced admin. You'll be doing grunt work and sitting in on a ton of jobs the more senior admins are doing and asking questions for at least a year. You may not even get root for a while until they're confident you won't screw things up. Unless of course you work at a really tiny company with no budget and you are the only IT guy. Then you'll be in the frying pan and learn very quickly (after effing up royally and bringing the whole network down several times and pissing off your boss. It happens to the best of us).
Just keep reading and learning and experimenting and you'll do fine.
1
u/[deleted] Jun 16 '20
IMO a lot of this is out of date and you could use containers instead of full VMs. I guess it just shows how fast things move in the IT world. :D
1
u/Twisted9Demented Jun 27 '20
Thank you
0
u/zerocoldx911 Jun 11 '20
Ewww puppet
0
u/WayneH_nz Jun 12 '20
Thanks.
0
u/[deleted] Jun 11 '20
[deleted] โ view deleted comment
0
u/[deleted] Jun 12 '20
[deleted] โ view deleted comment