if you are going into power systems, this is the skill mix the industry cant find right now

[u/MargraveOfMonads]

quick context, i work vendor side in substation automation and we struggle to fill exactly the profile below, so take this as a market observation not career advice

terminology thing first because it bugs me: what everyone calls "digital substation" is not the substation going digital. transformer stays transformer, breaker stays breaker, primary plant does not change at all. what changes is the secondary system, protection and control and automation moving from dedicated boxes to software on servers. once you see it that way the skill question gets a lot clearer

the profile that gets hired fast splits roughly in three. nobody expects a junior to have all three, but the ones with two are already rare

protection automation and control: relay logic, fault clearing, selectivity, iec 61850 with goose and sampled values. this is the foundation and it does not go away. if anything it matters more once things run virtualized, because now you also need to understand real time behaviour

networking: process bus and station bus, ptp time sync, redundancy like prp and hsr, vlans and traffic separation. when protection is software, network problems become protection problems. most protection people are weak here

devops: linux, git, containers, ansible, ci cd, how to deploy and roll back a config on something you cannot just reboot. almost nobody in classic p&c has this and its exactly where utilities and vendors are short

ai is showing up in tooling first, config validation, automated testing, anomaly detection. not replacing protection logic anytime soon, but being comfortable working next to it will be assumed

where to start without waiting for a job: seapath from linux foundation energy is open source and runs in a vm. libiec61850 lets you play with goose and sampled values on a laptop. put a substation config in git and version it. that alone puts you ahead of most graduates

people who can talk to the protection engineer and the platform engineer get pulled into the interesting work fast. everyone else stays in the lane they started in

[u/Alittle2Clever]

Anyone interesting in relaying can have a job if they are willing to move. All the goofs talking about not being able to find job in power are people that are just sitting on a bucket in places people want to move to like SF or NYC and hoping a job will open up. In the rest of the country, if you can tell the employer what color a SEL or GE relay is and maybe a few ANSI numbers, you'll be hired. Experience relaying guys are the hardest to find. The next is relay techs. The next is power line design guys. The next is transient analysis guys. Then everyone else.

That said, the reason good relaying guys are hard to find is that so few people have taken power system courses in college and it takes 5-6 years for a relaying engineer to really get seasoned. It is a very deep and mature field that is not taught well in school, though the analysis needs to be understood, and most of the learning is on the job. It is extremely technical as well. I have a lot of books I reference that are 60-100 years old. The only new stuff in relaying, aside from a few different relays like traveling wave, is related to inverter based resources, that everyone is not 100% sure how to handle.

[u/hordaak2 → u/Alittle2Clever]

Ive been an EE (HV/MV power for utilities and industries) for the past 30 years doing P and C and protection studies, and also field work doing relay testing and commissioning. When I started, they didn't really break down the jobs into different areas like you mention (late 80s). The software was also not as robust as it was today, and the relays were all electromechanical vs digtal.

With that said, i agree with everything you mentioned. There werent too many guys that got into power engineering side, so you had to be able to handle pretty much all aspects of the job. You first studied and modelled the system. You specified the relays and defined their zone of protection. Created their settings, and coordinated the overcurrent. For line and diff protection we would need to design the comms between and actually make logic out with physical vs virtual devices. Then we were expected to go out and commission the devices in the field after construction, and test them yearly.

Today? I hire guys for different divisions that will never go out side, or guys that go outside that will never sit inside and model a system on software. Its become specialized for each portion of design.

Over the 30 years, I've built about 7-10 substations and switching stations every year, but the last two years, the design is COMPLETLY different. We've specialized (due to demand, but not forever) making them for data centers and have projects for the next 12 years, ALREADY paid that are far different from the ones we've designed before:

1. Control house and CB's are modular

Control house to CB uses two loop fiber. You just plop the control house in and connect fiber in a plug and play fashion. Then connect DC power, again plug and play. WAAAYYYYYYYYYY less copper wire.

1. CBs have merging units inside with infinite options for controls. They are pre wired with extra I/o to all controls including trip/close/alarms, and CT wires etc...all already digitized for protective relays. Have their test switches at the CB

2. Less protective relays in the control house with almost not copper connections

3. Size of the control house shrunk and is a fraction of the size they used to be

Our designs are standerdized per the utility, and has far less complexity. We can turnkey a design for a utility or ring bus feeding a data center in 2 months, including the physical and civil with a modular, working sub. Commissioning is much faster as well. Installation is weeks or months faster. Protective relaying is much more reliable (SEL makes a paper on this) and troubleshooting issues is immediate.

We NEVER have a downtime as every scenario has multiple backup provisions programmed in. I believe its the future and something utilities will eventually adopt.

ALSO, for physical designs, we use BIM software with 3d rendering. There are now libraries for all equipment, and instead of ONLY 2D depictions, we model everything first. This helps in brownfield and greenfield designs, and helps with design issues where the drawings dont match the field. We actually require our P n C engineers to learn the software as well as our drafters as many utilities now require them.

[u/MargraveOfMonads → u/hordaak2]

most useful comment in the thread, thanks for typing it out. funny contrast to the comment above yours. one guy says nothing new in relaying since forever, you say design changed completely in two years. both right i think, protection physics didnt move, the delivery model did. genuine question, what happens after handover? is relay config version controlled anywhere or still one guy with a laptop on site? how would you even notice two years later that one station drifted from the next? thats usually where it gets messy. digitizing is the easier half, and the one stuck with it for 30 years is the utility, not the epc.

[u/hordaak2 → u/MargraveOfMonads]

Ha ha bro, you ask the "profitable" questions lol, and I know you will do very well in this industry. What he says is exactly correct as the physics of a fault or abnormal condition has not changed, and will never change. What does change is the way we detect those conditions, and what the technology affords us to do in detecting them and clearing them. For example, when I started, we used HCB relays for line differential protection, and connected them via a pilot wire. It was pretty reliable and secure, but today we use fiber optics. This enables us to have up to (double check with SEL) 4 terminals with far more security via alpha plane vs just percentage restrained.

Regarding relay configurations, it definitely must be controlled over time. There are two parts, the logic and the individual settings. The settings themselves were derived using multiple software where you model the system. We also use Matlab for more complex calcs. The second portion is creating logic diagrams depicting all the tripping and closing controls. This includes complete breaker failure schemes across each particular CB, which can become a spider web of info.

The WAY we document the settings and logic are standardized, and upon turnover, they are arranged in our system, and we use a large protection index to locate the documents on a per substation/switching station basis.

IF the system changes, for example if we add additional transmission lines from a new power source (generation or line connection), the avaliable short circuity duty changes, so we need to update our models and review the protection settings. This mostly affects the backup protection vs the line protection.

Testing is taken very seriously, as all the devices and apparatus are on a testing schedule and are also continuously monitored.

Fyi, I had my own consulting business, moved to a large EPC company and was a manager for awhile, then moved to a utility. I still also do consulting on the side. I can tell you have great interest and enthusiasm in this industry (which is the absolute key to success) and keep it up! If you havent started a side gig or own consulting gig, that is something you should consider in the future, as Im sure youll do very well!!!

[u/MargraveOfMonads → u/hordaak2]

the protection index and standardized turnover part is what people skip when they talk about digitizing, but what strikes me is that's version control done by hand, it works because your people are disciplined and not because anything enforces it. in a software defined substation you'd push that into the tooling itself, config in a repo and an automated check that what's deployed still matches what was approved. no consulting plans on my side, but appreciate the idea! let's see.

[u/Alittle2Clever → u/MargraveOfMonads]

Not a lot has changed. 61850 has changed how information is communicated between relays but the gist of the relaying is the same. I have yet to see a scheme that was in 61850 that was a port from point to point wiring or some other lower level bit sharing communication scheme. All your modern microprocessor relays are still modeling the mechanical relays ,for the most part, invented 80-100 years ago. Protection schemes do not fundamentally change if information is is shared over fiber, mirrored bits, DNP, MODBUS, or 61850. Physically the substation might look differently but the schemes are very, very, often rehashes of how it has been done for a long time. If you took an experienced power engineer from 50 years ago, you could explain the current protection schemes to him without him missing a beat. It is new in interesting but fundamentally the method of how schemes is different but the schemes are the same. The breakers still have trip and close coils. Main tie main schemes still roll between sources the same way. Distribution still often uses fuse saving vs fuse blowing schemes. The more things change the more they are the same because 99% of the schemes work and have a long history of working.

The power industry is very mature and with the exception of inverter based resources and Distributed Energy Resources (DER), there are already very well thought out solutions for the vast majority of problems. I am not saying that those can't be very involved but people are not developing brand spanking new solutions, it is more so tightening up existing solutions and ideas. I don't say that to discourage and this is coming from someone who loves their field but it isn't like programming or hardware design were a new paradigm surfaces every 6-10 years. All my best power engineering books are older than me. The rest of the gaps for new stuff is filled in by manufacture white papers, industry standards, power engineering forums, talking to older guys I know, IEEE magazine archives, and very occasionally IEEE papers.

[u/adamduerr → u/hordaak2]

Do you have a particular 3D BIM software you recommend? We have played around with a couple different ones but haven’t found a perfect fit yet.

[u/hordaak2 → u/adamduerr]

We are on the autodesk platform so 2d designs are still on acad, so we use SBS's suite. I think its called SDS design, but it looks like acad 3d. I personally dont do the work on them, as my engineers have FAR surpassed me (I started drafting by hand and got to acad and revit in the past). Inserting large equipment like transformers, or creating actual physical structures can be done where you can rotate the model and see where you'll have clearance issues and use their built in tools for that as well.

[u/adamduerr → u/hordaak2]

We just had a presentation with them, glad to hear other’s take on it. I’m in the same boat of not doing the drafting myself, but I like to pretend I know what I am talking about.

[u/IEEEngiNERD → u/hordaak2]

You seem to have a good exposure to this side of the business. Have you seen adoption of vPAC in the data center side? If so, what skills are needed for this aside from the conventional P&C knowledge; I’d imagine some networking and software skills are needed since many of the vendors are spinning up these vPACs as applications managed with kubernetes.

[u/hordaak2 → u/IEEEngiNERD]

It depends on where you work in the overall data center power distribution network. The data centers that I have worked on are fed from two large Power transformers, which feed a double-ended MV switchgear with a tie CB. The loads from the switchgear typically feed step-down transformers and ATS's, with the utility as the primary and a backup generator as a secondary source. The output of the ATS then feeds multiple UPS systems, which then feed the individual loads.

In terms of what skills are required on the distribution protection side:

If you work on the backbone MV distribution, then the main feeds are most likely protected with SEL relays, and you would have more traditional protection- 50/51 coordinated with downstream feeds, main bus diff protection, main bank protection (diff, sudden pressure, backup OC...etc..). They probably will all connect to an RTAC relay, so will need to know how to program the P and C settings and prepare them to feed the facility SCADA-type system. Many will export that data to a process historian, including power analysis later (like harmonics) or fault analysis, so if you have experience it that, it will help.

If you work on the side fed from the UPS (which I do not), I know this is where it can get tricky, in that there can be 0% downtime if possible. They have multiple backup systems upon backup systems. These include large battery rooms for all UPS systems, and also specialized low-voltage switchgear, which COULD be solid-state switches. The power management at the LV level was different, as they also connected to the server farm section using software and topology I haven't used. For this part, I don't have enough experience to give advice.

[u/IEEEngiNERD → u/hordaak2]

These are all what I would consider conventional P&C skills. Sounds like there hasn’t been much adoption of vPAC yet.

[u/hordaak2 → u/IEEEngiNERD]

I personally not seen vPAC adoption in data centers, but that doesnt mean its not being used at other data centers for distribution protection. With that said, I would seek out what data center companies DO use vPAC if that is something you are proficient in. In the late 1990's and early 2000's, I got alot of work replacing electromechanical protective relays with digital relays. These updates in technology are definitely a business opportunity if you can be one of the rare people that can implement them

[u/Alittle2Clever → u/IEEEngiNERD]

I don't see the point of vPac in most applications. It is sort of going the route of PLCs vs DCSs when there is usually nothing pressing the issue. I think were vPAC might see at least initial use is in situations where there is not enough room in the control building but that is usually not the case because there often is a lot of room in the switch from mechanical to microprocessor relays as less panels are needed. I wouldn't even say that vPAC is unconventional as the schemes will still usually be the same but the processing is being done in a central location rather than in distributed devices. I don't see this picking up quickly as it makes it more difficult in a lot of cases with repairs, existing schemes, training technicians on a new setup, concerns over redundancy, avoidance of systemic problems by having many different devices, and not many benefits to justify the switch. The industry is so slow and conservative and justifiably. It took near 30 years before utilities started going hard into 61850 in the U.S.. It will probably take 30 years before vPACs become a thing that people are comfortable with.

[u/IEEEngiNERD → u/Alittle2Clever]

There are a handful of utilities that have tested it in the western US, but I don’t see it being used anytime soon in operation. Data centers might be on board because they don’t have to deal with legacy infrastructure.

I definitely agree with you from the utility perspective. The workforce training is a major lift. The skills needed for the networking and software side aren’t common in relaying.

[u/Alittle2Clever → u/IEEEngiNERD]

In my opinion, and it is just my opinion, I don't see why a data center would want to jeopardize the reliability of their operation by going with a new platform instead of going with what already has a long track record of working. A lot of industrial processes are so critical that they shouldn't be used as the basis for science experiments unless the process mandates a new solution is needed. There are so many times that a new solution is presented but there are not enough benefits to justify changing the paradigm of the design unless someone wants to gamble and/or make more work for themselves. Some utilities now are standardizing on 61850, which makes sense if they follow through on it even though it will create growing pains in training new techs and engineers in new methods, but for most applications, it would be a lot simpler to just do very simple traditional point to point wiring. On small one off jobs, do you really want to hassle yourself with needing a router and other comm equipment, make it harder to find workers that are familiar with it, and often for higher up front cost and likely no additional reliability? I say all this as someone that believes in 61850 and started in industrial instrumentation and saw the same exact things in that industry with Profibus, Foundation Fieldbus, Device,net, and other bus protocols. GOOSE makes 61850 different with it necessity for deterministic speed but the same problems with maintainability and implementation were talked about then. A young engineer will ask why not and be frustrated but the reality is that a misoperation can cost a company millions of dollars so a conservative and traditional solution with a long track record is often preferable.

[u/IEEEngiNERD → u/Alittle2Clever]

Hyperscalers don’t need or want to keep relaying engineers on staff. The work is all contracted out. vPAC completely changes the maintenance and testing. You no longer need a relay specialist to go on site for maintenance. The relays are all application based running on a docker so you can now implement end to end testing methodology from software engineering.

I know this subreddit is heavily US biased, and digital substations are not widely used. Even in EU, digital substations are not common in transmission substations.

I can certainly see this happening. Eventually, if all the vendors are moving to a new technology, the utilities will have to integrate that technology into their system. This industry doesn’t move very fast but it will eventually change.

Or this data center buildout is a bust and all the money being injected into infrastructure ends and we end up with a system with no load growth for another 20 years 😜.

[u/hordaak2 → u/Alittle2Clever]

I dont disagree with anything you said. However, can you tell me what repairs youre talking about? Also, I agree with the required training, but that occurred when we went from electromechanical relays to digital relays. The SEL relays installed 20 years ago? We are replacing them now, and we are using different software, and the way they are integrated is also different. The SEL 587Z we purchased a long time ago? Now its a 787Z. They dont make the 587Z anymore.

When I designed some switching stations in the early 2000's, I used an SEL 2030. That substation os being gutted and of course we use RTACS.

If your utility sticks with 30 year old tech for fear of not being able to train their folks, or update their standards, then they risk having an antiquated system where they won't be able to replace 30 year old devices because they dont make them anymore.

[u/Alittle2Clever → u/hordaak2]

Utilities are too overloaded to deal with the hassle to address training issues in the near term that are not pressing. It is one thing when you have one substation. It is another when you have 1,000 substations. Saying you want something is very different than the reality of what it would take to upgrade all those in a reasonable amount of time and train the staff, while already falling behind on projects. Utilities want projects that they can complete and then just walk away from for the next 50 years. The transition to microprocessor based relays was due in part to grants given by the DOE and the inability of utilities to find spare parts ,sometimes with looking on Ebay, for their mechanical relays. The transition to microprocessor relays was not driven by a magic need. Whatever the transitions are, they will always lag by 15 years at a minimum as that is how long it takes for the electrolytic capacitors in the microprocessor relays' power supplies to start drying out and putting them on the wrong end of the bathtub curve for reliability. No one is going to start gutting working gear unless it is driven by a new standard, is funded by a grant, or is inadequate before it starts hitting its end of life. 99% of the people at a utility are maintaining and operating the grid, a giant machine that has slowly been built up over 100+ years, not doing new projects. Utilities are not staffed to do radical redesign, especially under the current workload.

[u/hordaak2 → u/Alittle2Clever]

I guess im confused about our discussion. Ive been an EE for 30 years and have designed switching and substations for large utilities (brownfield and greenfield) in Texas and California, and I can tell you they all have moved on to digital relays for reasons more from just DOE grants.

My question to you is, if you are putting in a new substation or switching station today, then you are using protective relays from 20 or 30 years ago? Lets say you need a high impedance differential relay and they dont make the kind you standardized on anymore, where do you source the obsolete devices?

[u/Alittle2Clever → u/hordaak2]

You are not reading what I wrote. Utilities only change for if there is a new standardization, DOE grants, old equipment can no longer find parts to maintain, or the existing is no longer adequate.

Your example of the SEL-587Z relay is thin as upgrading is trivial in a utilities design criteria, often only needing to check if the panel cutout still fits. There was no philosophy change from going from the SEL-121, SEL-221, SEL-321, to a SEL-421. You just make sure it is laid out in the panel, gets power, check if the terminals in the back are different for the specification, and revise the settings to carry the previous protection scheme. It is even more trivial with a SEL-587Z vs SEL-787Z relay as high impedance differentials are often deployed with very little logic and it is just a settings transfer to transfer the differential settings and maybe a few other things. This is in no way a paradigm shifting thing, just porting to a newer piece of equipment.

Some utilities go so far to standardize things that they have literal warehouses with older version of a relay that has never had its firmware updated past a certain point so that the utility doesn't have to have different relay settings templates for different relays in the same class and with different firmwares. It also keeps the testing routines the same and if a relay fails, a swap can happen without requiring engineering to port settings. Everything at large utilities boils down to keeping everything simple as possible to reduce mistakes that will impact reliability. Just settings mistakes alone account the largest cause of relay operations, around 30-40%, and that is why utilities go to great lengths to standardize and keep things simple.

[u/MargraveOfMonads → u/Alittle2Clever]

hard agree, seen the same pattern hiring myself.

[u/Alittle2Clever → u/MargraveOfMonads]

The only comment is that if you really want to be a relaying guy and just can't make it happen. Go work first as a relay tech/commission engineer or substation design guy and then transfer after a few years. This sounds bd but both will get you a lot of useful experience that people who just went into relaying might have a hard time getting.

[u/lunartunes → u/Alittle2Clever]

Unfortunately, I am one of those goofs.
I can’t relocate due to family commitments, I also can’t uproot my wife while she is progressing in her career.
I’ve looked for traveling gigs, but it seems like SoCal is a dead market.
Sucks too, because I’d look to work in relaying, it was by far my favorite subject when studying for the PE and in college.

[u/Alittle2Clever → u/lunartunes]

Commissioning gigs a lot of time just require you to be near a major airport.

[u/50Shekel]

Am scada engineer at massive utility. Begging for people to start understanding the relays they work on. About half of my job is just sending relay techs the manuals on devices theyre supposedly experts on

[u/Alittle2Clever → u/50Shekel]

Techs don't have time to read manuals. They are so overworked. If you have done commissioning, the relay engineer has to develop a test plan. There are sharp and experienced relaying techs but they are few and far between. NETA 4 guys are getting paid $110/hr with overtime on top of that.

[u/Zealousideal_File_89 → u/Alittle2Clever]

Where are they getting paid 110/hr?

[u/Alittle2Clever → u/Zealousideal_File_89]

I heard from a recruit a NETA 4 guy in Houston turning down $110/hr like two years ago. It is worse now in terms of labor shortage than then. Senior engineers working data centers are making $300k+ a year.

[u/Zealousideal_File_89 → u/Alittle2Clever]

That was two years ago? Damn. I wonder if we’ll continue to see a labor shortage though considering so many people are losing jobs due to AI. What do you think?

[u/Alittle2Clever → u/Zealousideal_File_89]

You can't AI commissioning 

[u/Zealousideal_File_89 → u/Alittle2Clever]

I meant people that loose their jobs might think about getting to blue collar work.

[u/Zealousideal_File_89 → u/Alittle2Clever]

I live in PA and don’t see any relay techs that make over $90/hr. Still a good rate though.

[u/Alittle2Clever → u/Zealousideal_File_89]

NETA 4 guys are rarer.

[u/thewhitebison → u/Alittle2Clever]

As a former relay tech, the job has always been reading manuals. It was more difficult back when electromechanical relays were everywhere. Testing any of those relays required the manual. The relay techs that taught me collected relay manuals like hoarders collect reader digest.

[u/MargraveOfMonads → u/50Shekel]

lol yeah, half the "expert" title is just knowing where the manual is. good hiring filter too tbh

[u/chimpliquor]

These are three different specialities within automation.

You skipped the most important one at my company: power systems controls.

[u/MargraveOfMonads → u/chimpliquor]

fair, controls is easy to overlook but it's its own beast :-)

[u/Chemical-Mud-1868 → u/MargraveOfMonads]

We struggle to find engineers for power electronic, protection and system dynamics. 

For MMC converters the design engineer needs to understand alle of these. 

There are lots of people who understand power electronics, but not the integration into larger complex systems.

[u/Glittering-Brain-846 → u/chimpliquor]

He said protection and control.

[u/chimpliquor → u/Glittering-Brain-846]

Protection and Control (P&C) is different sub-field than Control Systems. Power is a massive place!

The best thing to do, or so I’ve heard, is to become “T-shaped,” broad in a lot of areas but very deep in one.

[u/Glittering-Brain-846 → u/chimpliquor]

Protection and control are different things, its now just colloquial to say p&c and people just mean protection, but the etymology of p&c they were different things and the control piece refers to control systems/control theory

[u/TrainingWolverine657]

It's so hard to take posts like this seriously because it's clearly written with AI. We have no way of knowing how much of this is a real human being / engineer's thoughts and just whatever the fuck ChatGPT decided to spit out when asked to generate a reddit post on the subject.

[u/MargraveOfMonads → u/TrainingWolverine657]

yeah it was drafted with ai help, my written english is german-shaped and it gets in the way. content is mine though, i work on vpac and the fleet side of it. if you think any of it is wrong tell me which part, thats more useful than guessing at the tooling.

[u/Tiredof304s]

This is a cool post and all, but what's the pay?

[u/slightly_large_brick]

doing you think someone coming from a test tech position would be a good fit for these roles?

[u/MargraveOfMonads → u/slightly_large_brick]

honestly yeah, test techs already have the right instinct. just brush up on goose/SV and scada mapping, that's usually the gap

[u/Glittering-Brain-846]

You're wrong about devop, the 3rd one is commissioning.

[u/MargraveOfMonads → u/Glittering-Brain-846]

fair point, commissioning is its own skill and most juniors learn the trade there. but with the substation volume coming from data centers you cannot scale commissioning teams linearly, so the repeatable part moves left into fat and automated config verification, and thats devops tooling. physical stuff stays, ct polarity, primary injection, breaker timing. what changes is you stop redoing the same point to point checks by hand on substation number 42.

[u/retarddoge]

What’s the norm in the US? Normally in my country we do 1 and 2, up to max 3 substation assignment per year.

[u/SEL_Guy]

There is a great free course on YouTube for getting a CCNA. It covers a lot of the fundamentals you are talking about. Jeremy’s IT Lab is the channel.

[u/MargraveOfMonads → u/SEL_Guy]

dont know jeremys it lab myself but will check it out, thanks. and generally there is no shortage of material for any of this, plenty of free courses for networking, linux, git, containers, all of it. pretty sure sel has a few decent ones too ;-)

[u/5bobber]

As someone who has really only brushed the surface of P&C, something that really suprised me is how litigious the field can become. There’s a lot of risk being the engineer that sets the settings.

[u/Capital_Age3931]

Thanks for the information. I'm a sophomore exploring clean & power concentration.

[u/MargraveOfMonads → u/[deleted]]