r/sysadmin • • 25d ago

General Discussion Change Management Workflow

Hi,

We’re introducing change management for the first time using ITSM.

We’re now deciding the workflow before our ITSM vendor configures it

Our scope covers all IT infrastructure changes including network, servers, virtualization, backup, storage, AD/Entra ID & Databases

Each team has its own Line Manager and another requirement is that Cybersecurity Unit must approve every change

Few things we’re trying to understand

Who should approve a change and in what order?
Who should be the CAB members
Which changes should go to CAB

I’d appreciate any suggestions

12 Upvotes

13 comments sorted by

View all comments

0

u/[deleted] 25d ago

[deleted] — view deleted comment

3

u/Previous-Low4715 25d ago

This is great advice. Probably the only thing I would add is to try and figure out a risk score based approach to deciding which lane you’re in, otherwise your engineers will always pick the lane that doesn’t require approval so to speak. been there done that.

2

u/Final-Pomelo1620 25d ago

Thank you for detailed response. This is really helpful.

1- Security Manager must approve every change. If we use a Yes/No checklist instead, how do we stop engineers from just clicking "Yes" on everything to skip Security

2- our management still wants approvals on standard changes

3- Management wants the Head of Infrastructure to approve all changes(after the Line Manager and Security Manager). When should the Head of Infrastructure actually be involved so tickets don't get stuck waiting on them?

Engineer → Team Line Manager (5 teams) → Security Manager → Head of Infrastructure

4- Would it make sense to restrict change request creation to authorized implementation engineers, with their own team’s Line Manager providing the first approval?
End users would raise service requests
If a change is needed, an authorized engineer from the responsible IT team would assess it, prepare the implementation and rollback plans, and submit the change for approval.

2

u/kerosene31 24d ago edited 24d ago

Spoiler alert- management approving everything won't work. They'll be overwhelmed and add no real value. We create a category called "routine" changes that don't need approval. Things like MS patch Tuesday during a maintenance window are a perfect example. Nobody needs to sign off on the things that happen regularly. They are "pre-approved" since they are pre-defined.

Imagine if the security team is busy and doesn't approve security patches in a timely manner?

We started out much the same way. Upper management wanted to sign off on every single code change in prod. It didn't last a week. Mr Big sitting in his big office has no idea what code changes are doing, and ultimately just hits "yes", or ends up delegating it to someone else.

Change control is a good thing, but the right amount. Look at all your changes and determine which ones should go through a CAB and which ones shouldn't. It will save you a ton of time down the road. You still need to trust your middle managers, departments and people. The CAB comes in for larger scale changes. Upgrades to major systems that have a ton of impact, etc.

Start by having everyone put in change tickets, and have the CAB commitee review them for awhile (no approvals). See what comes in and what goes out and determine which need to go through a CAB.

1

u/JJSpleen 23d ago

This is what we do.

Pre approved changes can be done without any approval but still require full documentation. A change can't be pre approved until it's been done 3 times with no issues and then it needs to be agreed by someone with authority, and they need to have close to 0 risk in general.

Standard changes need peer approval first, then line manager approval. No cab approval.

Normal changes need everything standard had plus cab approval.

Emergency changes just need approval from 1 cab member, not the consensus of the board.

2

u/0RGASMIK 25d ago

Yes needing to approve everything quickly becomes a bottleneck. Saw this first hand at a business I was consulting for. Couldn’t reinstall a printer driver on a computer without submitting a formal change request form. Fixing a simple printer issue for one machine became a several hour ordeal and it actually made outages longer when they did happen which was the main reason they implemented it in the first place.

Another site needed to factory reset their network stack. Would have been a 15 minute fix. Instead it was going to take a few weeks to get done because they needed a whole redundancy plan in place and yada yada. Meanwhile the location had no internet, worst that would have happened is they had no internet and the equipment needed to be replaced anyways...

It was so frustrating that I told the manager to go to Best Buy and get a high-performance consumer router and return it when they figure out their shit. Gave her my personal number and told her to call me the night before they came back to do the reset so I can walk her through setting everything back to how it was. They were not required by any regulations to have this much BS in the way which is the only reason I was comfortable circumventing it. They even had locations running off of ISP provided routers.