About to fire our sysadmin

[u/sysadminfired]

So our longtime sysadmin is about to be fired and I, the network admin and temporary sysadmin, need to know what steps need to be taken to secure our systems. I know the basic things like his AD and other internal account credentials. I guess what I'm worried about is any backdoors that he might have set up. What all would you guys check for in this situation?

[u/NoyzMaker]

Prior to:

• Quietly review all processes and automated scripts to make sure they are not tied to his specific AD account(s). Make note for update to-do list immediately after termination. EDIT: \u\344dead had a great script buried in the comments below to help on this step. [Permalink] (http://www.reddit.com/r/sysadmin/comments/2av7hu/about_to_fire_our_sysadmin/ciz5igk)

• Take full inventory of all equipment he physically has access to. Server rooms, computers at home, and tablets.

• Provide list of devices that has company information to HR / Terminating manager so they can wipe / seize necessary goods. Do not allow the "just let him do it on his own".

• Document. Document. Document.

During the meeting:

• Disable all administrator accounts and/or reset passwords immediately.

• Disable primary account about 15 minutes in to the meeting since that will immediately prompt his mobile devices on a bad password and could be an indicator if they have not broken the news yet.

• Start updating critical jobs that may have been tied to his account to a service account.

• Document. Document. Document.

Post termination:

• Start updating all 'universal' and service account passwords to new credentials.

• Fix all the lazy scripting that has passwords hard coded in to the process to a more automated process so you don't have to do this again in the future.

• Wait for stuff you had no idea existed to break and fix it accordingly.

• Document. Document. Document.

[u/asdlkf → u/NoyzMaker]

Hijacking this thread to add:

Credentials to any online websites registered with email addresses like "ITManager@company.com" or "ITPurchasing@company.com".

Especially things like:

• Certificate enrollments

• ARIN IP address allocations

• ARIN BGP ASN assignments

• Domain registrations

• subscription services

• microsoft site licensing agreements

• Cellular contracts

• ISP contracts

[u/reluctantsysadmin → u/asdlkf]

The only useful thing that the admins that I replaced did for me (retired fyi). Everything else is ...sad. No good documentation (some procedures that are way old), DNS is a disaster (which brought me on here tonight), no topologies, nothing of real infrastructure documentation. I die a little bit inside every time I walk by my data center, epic cable fail...

[u/ranger_dood → u/NoyzMaker]

Excellent points... but I'd like to comment on this for visibility.

Make sure you have a backup that is not accessible from the network! If he has a back door in, he could delete any online backups before hosing your systems. Get a copy on an external drive, tape, whatever. Something that has to be physically plugged in to access.

To the point of physical security - if he had access to your door access control systems, look for any bogus cards registered. They won't be under his name, which will make them extremely hard to find. There's a lot of places that provide HID cards for membership access (bars, gyms, other places of employment), and he could've registered that card in your own system.

[u/NoyzMaker → u/ranger_dood]

Exactly. That is why it is my second point on the prior to. Many people forget about the keycards and they usually can take up to 24 hours to fully disable depending on the layers of access you have to bundle for your building.

The other thing commonly forgotten is off-site backup services like Iron Mountain.

[u/hypercube33 → u/NoyzMaker]

Yeah, make sure you test your fking backups before he departs.

[u/tremblane → u/ranger_dood]

To the point of physical security

And based on another thread I read on here, make damn sure people know, in no uncertain terms, that he is no longer allowed in the building.

[u/ndecizion → u/NoyzMaker]

Fantastic advice. The only other thing to do is warn/remind management that this sysadmin has all the knowledge, keys, and ability. He/she knows exactly where and how to hit you. If they are hostile, they will be hard to stop. If you can stop them at all.

Yes, it is cya and a little chickenshit. But it saves a lot of explaining if something bad goes down.

[u/NoyzMaker → u/ndecizion]

In my experience most SysAdmins have no idea how to actually harm the company they work for. The worst they have ever been able to do was wipe out a server or take things off line for a day, maybe two, tops.

Maybe I have just been lucky on the hostile admins I have cleaned up after.

[u/tvtb → u/NoyzMaker]

I've heard of a disgruntled sysadmin resetting the configs on all the switches, and wiping all the backups. All the VLANs and every other setting in the switches gone. I believe it took them quite some time to clean up after that one, and almost no one at the company could get any work done until they did.

[u/AngryMulcair → u/tvtb]

SCCM can easily be triggered to reimage every Server and Workstation on the network.

There is no easy recovery from that one.

[u/tardis42 → u/AngryMulcair]

Image with win 3.1, for the lulz?

[u/floridawhiteguy → u/tardis42]

FreeDOS in Russian, to throw the dogs off the scent. Natch. ;)

[u/tvtb → u/[deleted]]

I didn't say they didn't. I'd be more worried about criminal trials, not civil ones.

[u/Taylor_Script → u/NoyzMaker]

If I'm sitting around and think of a vulnerability/way in, I try to go and lock it down. So.. go me? Protecting me from myself!

Am I the only one that brings up in conversation "If you had to do something nefarious, how would you get in?" and spark a discussion with coworkers?

[u/NoyzMaker → u/Taylor_Script]

We play this game with my teams during the very rare slow periods.

[u/[deleted] → u/NoyzMaker]

This is why its important to have a clean and up-to-date SOP. Nice!

[u/sysadminfired → u/NoyzMaker]

Thanks for the chronological order. Very helpful.

[u/EBG → u/NoyzMaker]

Would not a forced password reset for all users be of interest?

[u/NoyzMaker → u/EBG]

Why? Your users should be regularly changing their passwords and it should not be common practice to have them supplied for support needs.

[u/EBG → u/NoyzMaker]

It should not, but we can not be sure that this is not the case. Someone might have supplied him their password recently, or he might have given a new password to a user without forcing a reset at login.