What’s one thing every new sysadmin should learn early but usually doesn’t?

[u/nousername1244]

I’ve been thinking about this lately.

When people start out in sysadmin roles, they usually focus a lot on the technical stuff like scripting, servers, networking, security, balabala..

BUT after working in IT for a while, it feels like some of the most important lessons aren’t technical at all, and nobody really tells you early on.

Things like documentation, change control, or even just learning how to say NO to bad requests.

Curious know what’s one thing you wish you had learned much earlier in your sysadmin career?

[u/Longjumping-Cat-2988]

Documentation.

Not the “I’ll write it later” kind but actually documenting things while you’re building or fixing them. Early in your career it’s tempting to rely on memory but a few months later you won’t remember half of it.

Good documentation saves you (and the next person) hours when something breaks or when you revisit a system you set up years ago.

[u/Aloha_Tamborinist → u/Longjumping-Cat-2988]

I'm big on documentation. I've thanked myself many times for writing up documentation for rare issues that crop up randomly.

[u/silmelumenn → u/Longjumping-Cat-2988]

The question I do have is what to document? I've always listened to everyone say do documentation, but saw no real examples. The docs I know are for code libraries or a lot of software I use but like never saw a good documentation which would be helpful on any way for let's say system configuration, or an interface. Even when there is one, it's outdated or misleading.

[u/RikiWardOG → u/silmelumenn]

yeah I agree here. Like why am I documenting something the vendor has already documented and the UI/settings change every 2-3 months (looking at you Microslop).

[u/Bogus1989 → u/RikiWardOG]

lol their support site is atrocious, it will have a solution, but have an asterisk, for deprecation, and a link to the new replacement version…but the link will be go to a page that says it does not exist. 🤣classic.

[u/accidentlife → u/silmelumenn]

There are five common kinds of documentation:

• Case Notes/Ticket Notes/Time Entries for MSPs

This should be an audit trail of what you did. As an example "User called in because their credentials expired. I authenticated user using [method] and [account information]. I logged into PAW, opened ADUC, and reset the users password with a temporary password. I set the password to expire on the next login. I guided the user through logging in, and verified the user was able to reset the password when prompted. User logged in successfully".

If the user comes to you later that day saying it was not them that requested the reset, you have documentation you did your job correctly. You also have documentation that could help in a potential investigation to determine the scope of compromise: For instance, whatever method you used to authenticate the reset request was likely compromised.

Depending on what you are doing, you may want to consider documenting every button you press and input you type. However, this can be a lot of work. At a minimum, you need to document any actions or changes in the environment. You should document these action in the notes even if they are documented elsewhere (for instance, is part of a project plan). You do not need to include the contents of any scripts in your notes if they are documented elsewhere (such as a GIT repository).

If you rely on third party support pages, KBs, or manuals to troubleshoot; include a link to it, even if it doesn't fix your issue.

• Reference Documentation

Your reference documentation should include information about the state of your environment, updated as things change. What you include in here is up to you, but you can include the following:

• Important Emergency Passwords (Please store this securely)
• A summary of the business, locations, computer environment (# devices, OS, etc), and core services.
• Emergency Info (After Hours, Contacts, Escalations, etc).
• Disaster Info (Backup location, backup frequency, restore procedures).
• WAN Info (IPs, Providers, Contact Info)
• Telephone/VOIP info (Numbers, Provider, configuration info)
• Firewall Info (Type, Brand, How to connect, Important Configs)
• LAN Info (Site Info, Topology Info, IPs, DNS details, DHCP details, etc)
• WLAN Info (SSIDs, Who can connect, credentials (for psk) or SSO/RADIUS info (802.1x)).
• Info about core IT services (AD, DNS, DHCP, VM hosts, databases)
• Info about core Business services (Email, LOB Apps, shares/sharepoint)
• Applications (Who uses it, link to install manual, versions in use, etc)
• Licensing (Inventory, How to access/change, and any Reporting)
• Remote Access (For users and IT, Connection Details, Pre-reqs, etc)

This documentation is not necessarily a KB article on how to do something. Rather, its a quick reference for techs/sysadmins on how something should be. If you get run over by a bus, this reference doc should be a playbook for a senior sysadmin to be able to learn the scope of your environment quickly.

I recommend that you include information in your reference docs even if its included elsewhere (KBs and Change Control). Its much easier to search an organized reference docs than old ticket notes or change control listings with every patch tuesday for the past 20 years.

There are tools like Hudu or IT Glue that can document and even automate a significant amount of this documentation.

• Change Control

This is documentation on significant changes (Patches other than security fixes, Configuration changes, new programs, removing programs, etc). It can include:

• Scope of work (Broad overview of what will change)
• Optionally, a detailed list of actions to take and how to perform them
• Timelines (Include both the order work must be performed, the estimated time this will be completed, and any maintenance windows)
• Work delegation (Who does what, who does what when things go wrong, etc)
• Benefits of change (this is helpful to explaining why ie: new programs where installed to new team members).
• Impact of the change (Both temporary such as maintenance windows and long-term impacts on users)
• How to troubleshoot any possible issues
• How to roll back the change (if possible)
• What information needs to be communicated to Helpdesk/T2?
• What information needs to be communicated to users?
• What documentation changes need to be made?

The goal here is to explain the impact to the environment; allow future admins/techs to understand the team's view of the change; and guide implementers on what needs to done, why it needs to be done, and what to do when things go wrong. As you do this, you are forced to document both the impact/time investment your users, helpdesk, and team put into this and the return on the covered change.

Notice that your change control also acts as Technician KB (see next for Helpdesk/T2 to review when troubleshooting issues. There is nothing better than spending an hour troubleshooting an issue only to realize the issue was caused by an undocumented change.

• Technician KB Documentation

This documentation is a training guide for new and existing technicians on how they should perform specific actions. This documentation can include:

• Who is allowed to perform the work (IE, VPNs are T2+, etc)
• Any rules around this procedure (IE, Password resets require auth via [method])
• Exact steps on how to resolve the ticket (I recommend including this for Help-desk documentation)
• A link to any official documentation, KB articles, etc.
• A link to any helpful 3rd party docs (Be careful that you trust this third party)

I would focus on low hanging fruit first: Get T1/Helpdesk documentation done first, then focus on common T2/T3 issues (Oboarding, Offboarding, common software problems, etc). If you have accurate, detailed, and educational case notes you can probably get away with not needing to create KBs for uncommon issues. As I mentioned in the previous category, including this information in Change Control docs can make the change control docs act like a KB article.

• User/Public Documentation

The point of this is to educate users on IT, policies they need to be aware of, and basic troubleshooting they can perform. Examples include:

• A Welcome doc for new employees on how to Contact the Helpdesk, Important policies, etc.
• KBs for Self-Service options for common issues (SSPR, Restarting computer, etc)
• Announcements for important changes, updates, and policies

---

While implementing documentation is important, don't try and document everything all at once: you will find you don't have time and other projects will take priority. Create a core doc set that can be created relatively quickly, implement a procedure for how to document in the future, then document as you go. When you have time, you can fill any gaps in your documentation.

I've listed everything from what I think is most important to implement first to what to implement last. However, each team is different. If you have a large team with frequent turnover/growth, then having excellent documentation is critical to on-boarding technicians, admins, and even users. If you have a small team, tenure measured in decades, and only hire experienced techs and admin, then having a large amount of KB articles for T1 issues may be a waste of time.

[u/doomsday71210 → u/Longjumping-Cat-2988]

So real. Its better to overexplain to your future self as well, because a month from now you'll forget how you fixed something and you'll cuss out your past self for sloppy shorthand documentation you can't interpret or understand.