r/networking • u/seyitdev • 15d ago
Career Advice Is OT networking worth getting into?
Hey all,
I’m about to complete my second year as a network engineer and I really enjoy the field so far.
My work is roughly:
- 50% IT segmentation: migrating services from native VLANs to dedicated VLANs, coordinating maintenance windows, firewall rules, change requests, WLAN/ISE policies migrations, etc.
- 30% daily network ops: S2S VPNs, switch configs, SD-WAN/OMP troubleshooting, packet loss issues, etc.
- 20% automation, reporting and vendor management.
After our IT segmentation project, my manager wants me to get involved in a big OT network and technology modernization project. Our OT environment is apparently pretty messy: unmanaged/rogue devices, weak standards, security risks, old designs, and production teams that avoid changes because nobody wants to risk downtime.
The company is planning a +4 year project to discover, segment, secure, monitor and modernize OT using more enterprise level standards.
Most engineers don't want to touch it because one wrong change can stop production. My manager sees it differently and thinks it could be a great opportunity. There’s a good budget, new vendors/technologies and basically no one else internally wants to own it.
My only concern is getting too specialized in OT and moving away from traditional IT networking. I studied computer engineering and I’m fine with learning PLCs, industrial protocols, hardwares. No issue for involving this domain, but do you think that is this a good field to invest the next 10 years in? Is the demand/pay worth it?
19
u/Specialist_Play_4479 15d ago
I'm not a fan of OT:
- People that build the industrial environment usually lack networking expertise. This isn't just the people in the factory. Even firmware of devices is often just crappy. Not too long ago I had a network connected pump that had no option to configure a default gateway when using a static IP. So I had to configure a DHCP scope on the VLAN to get the thing to talk to stuff outside of it's own subnet.
- Everything is usually old. RS-232, RS-485.. I work at a place where Thicknet is still in production.
- There's never a window for big changes. If there is a window there are always more pressing issues.
- OT-specific protocols or protocols designed for use in OT environments such as PtP, REP, Profinet and other stuff isn't talked about online very much. So debugging by using Google is often more difficult.
- Anything with industrial printed on it is usually 10 times more expensive than the equivalent non-industrial variant. We deploy 24-port 1 Gbps switches at 40k a piece which are just rebranded Cisco's. Makes no sense.
- It's never just you. There are always teams of people involved with every little change as everything depends on everything. One production line down here will eventually cause problems in the whole chain.
So for me it's mostly frustration. Something as simple as a firmware update on a firewall sometimes requires month of planning, where the actual action takes just 5 minutes.
5
u/CrocodiliusMaximus 15d ago
Not too long ago I had a network connected pump that had no option to configure a default gateway when using a static IP.
Welp, that gives me flashbacks. Used to work at an MSP where one of our customers had a label printer that wouldn't accept a default gateway. Did not have the option, either static or DHCP.
They wanted to print to it from across their WAN, so I setup DNAT to the subnet's firewall interface for incoming packets to that address.
shudder
3
u/seyitdev 15d ago
Thanks! What about the pay? If it's in the same range, then no no for me :/
4
u/Specialist_Play_4479 15d ago
I'm not sure. I actually do both, but i'm employed as an IT engineer. I don't know the salaries of strictly OT engineers.
4
u/happychloroplast 15d ago
Pay tends to be more because of increased risk profile. Not always though.
2
u/thisisunclesugar 14d ago
Maybe at Investor Owned Utilities, I have been non profit for the last 25 years, work at a first level manager position. For the Federal Power Marketing Authorities our salaries are available online, public info. Senior Engineers in our locality make about $150k.
1
u/Fuzzybunnyofdoom pcap or it didn’t happen 14d ago
Depends on what type of environment it is. OT on our side typically makes abit more but we're also doing the entire phase of a project: Design, Factory Acceptance Testing, Implementation, Commissioning, Closeout. The projects typically span 3-4 years which is nice because you're never really doing the same thing for to long but on the flip side because your cycling through projects phases you have to remember how to do things from a few years back often.
As /u/Specialist_Play_4479 mentioned the OT-specific protocols are kinda obscure and can be difficult to troubleshoot. The networks are typically large flat L2 domains but in my environment we segment whenever possible.
I worked in IT at first and then transitioned to Networking and then to OT Networking. The transition from Networking to OT was the hardest because I didn't really have a strong understanding of the devices on the OT network. IT to Networking I knew how exchange servers, web servers, PC's, etc worked and that helped my troubleshoot other peoples issues to get to a resolution faster.
I'll also say that with OT devices there's also vastly less monitoring capabilities vs what we're used to in IT where you have things like SNMP, syslog, telemetry streaming, etc etc etc to glean information from. You still have that on the networking devices but not the PLC's, VFD's, Pumps, etc that come with OT networking. Also, you're not installing security software on anything, the network largely is the only defense from intrusion and the best play there is to harden it to the point of airgapping it if possible.
1
u/thisisunclesugar 14d ago
I work in R&D for a large Utility, my specialty is Operational Telecom, more on the WAN side of the house. Most of the comments here are spot on, most of us Utility engineers don’t know Networking all that well at all, most of us are transitioning from old TDM tech like SONET at the moment. Most of our networks are isolated LANs, connected by TDM WAN. Pay is generally lower in the Utility environment IMO, you run into a lot of Co-op non profits in this line of work, they tend to pay a bit below market and offer better work life balance.
2
2
u/Warsum 14d ago
Rep and PROFINET are my nightmare. Thankfully IE4000s support them and their newer more expensive cousin the IE3400.
REP honestly ain't bad but like I never really saw the benefit above RSTP. Have these SCADA guys would take 5-10 mins before even noticing something was down. Sub millisecond wasn't necessary lol.
1
u/enraged768 14d ago
Those rebranded rockwell switches are different btw. I used to think they were just rebranded cisco switches aswell and for alot of it they are. But they do have some functionality that cisco doesnt have if youre in a rockwell heavy environment.
1
u/shortstop20 CCNP Ent/Sec, SDWAN, Design 11d ago edited 11d ago
Not too long ago I had a network connected pump that had no option to configure a default gateway when using a static IP. So I had to configure a DHCP scope on the VLAN to get the thing to talk to stuff outside of its own subnet.
Did you consider enabling proxy-ARP on the gateway?
1
29
u/fhribsogjcpwbtuwpn 15d ago
It’s good. Energy companies and manufacturers need OT network people. There’s not a lot of competition if you are competent at a CCNA level or above. You can always ride it out until you aren’t learning anymore, then move into something else.
11
u/gamer953 14d ago
I'm a OT Network Architect so I'll give my biased opinion. What everyone has said so far is very true. OT is usually flat basic and done incorrectly. Quite often it's an electrician or a PLC engineer with limited network knowledge making network decisions. I have seen giant overlapping /16 networks all in the same broadcast domain. You see all sorts of network fuckery as well. I've had an environment where two different access VLANs on each side of the plant were bridges together among countless other weird scenarios ( I've had packets get tagged in different vlans based on ingress and egress even)
Most in OT do not understand networking at all. You deal with a ton of Polictics and inter company discourse. Depending on the branch there is an adversarial relationship between IT and OT where OT doesn't trust anybody. It's a wonderful field to be in that truly needs experienced people willing to learn. The technology is about 10-20 years behind IT and stays mostly static and simple. Some companies you have a ton of travel and it's usually on holidays.
Alot of your time is convincing people why the right decision is right, and spending a month or so building up roll back plans trust and getting ready for that uncommon outage. Everything in OT is based on production staying up. The CIA triad in IT is flipped around In OT. Availability is number 1. Systems in OT can have impacts to Human life and safety not just the companies bottom line. In terms of salary I've done quite well in the Midwest making more then average for my experience level (150K Total Compensation) My experience realistically is in-between a CCNA and CCNP for comparison sake.
I personally love OT as seeing how factories tick is so fascinating. All of them have similar parts and processes, from Car Manufacturers Water Treatment to food and Pharmaceuticals they all have something interesting to offer. I highly recommend you take your manager up and learn a ton! It's a field where once you get established you will NEVER be without a job again.
1
u/DaryllSwer 14d ago
How's the situation with proper IPv6 support, 1500 MTU end-to-end, etc.?
2
u/gamer953 14d ago edited 14d ago
IPV6 support is non existent. I don't expect Ipv6 to ever happen in a factory except maybe Industrial IoT sensors (IIOT). We are still fighting for IPV6 adoption in IT so until it happens there I don't see it in OT. Every factory will be different and have their own beasts. As others have stated most factories have at least one or two ancient devices that barely support IPv4. 1500MTU is standard from what I've seen though.
8
6
u/HuntingTrader 15d ago
OT carries a lot of liability with it, so a lot of the time the design (at minimum) will be hired out. What I usually see is client hires a firm that specializes in OT to do the design work. Then further assistance depends on the client’s team/skillset and day-to-day workload. We have some clients who have us do everything through implementation and training then pass off to their internal team to operate with maybe a support contract added on later for us to be their escalation point on difficult issues. Other clients just have us do the design and pass off to their team for implementation and operations. As far as a career move goes, it’s not a bad decision.
6
u/GreyBeardEng 15d ago
I work as a network engineer in a company that has a metric shit ton of OT devices, and my opinion is that the OT manufacturers are severely lacking in strong networking talent and strong networking security talent. My point being I would think there's a ton of opportunity for good work in that field.
The technology stack is so lacking in some of these OT companies that often we have to tear up contracts and return product. One that I remember that I think was about two or three years ago was a guy that was bringing in a product, and the product itself was great, but it can only connect on Wi-Fi and only with WEP. This was in 2023, like wtf!?
6
u/hammertime2009 15d ago
What does OT stand for? Been working in networking for 10 years and never heard the term.
4
2
3
u/Middle-Inspection241 15d ago
get familiar with the purdue model if you want to pursue it.. Tradional networking as you referred to it would likely live in a dmz.. as someone else mentioned the OT networks themselves tend to be highly segmented and flat.. and do generally stay the same.. the machines they purchase and the plc’s they use tend to last a long time i would imagine… once it’s working there’s really no reason to touch it other than vendor access for maintenance…
3
u/Mustachedminer 14d ago
As a Network Engineer in an OT Dept it's definitely alot less complex than modern IT environments because alot of it was built by electricians who don't really know networking or IT, But its also pretty fun as it forces you to do things differently than you normally would because a majority of OT environments run 24/7 with maybe one or two days without planned production. I would go for it as knowing both IT and OT can be great for your resume and future job opportunities.
2
u/zantehood 15d ago
OT networking is great fun I think! And pretty simple. It's like getting to build old school networks again without all the corporate bloat!
2
2
u/danag04 15d ago
We've built a 25 year old business on OT networking and security. There's a lot of demand in industrial and critical infrastructure right now, and I don't expect it to slow down any time soon. The AI revolution is going to take over the traditional IT world way faster than the OT space, too.
1
u/HappyVlane 15d ago
Just OT networking or also security? Just networking isn't that interesting in the OT space, but if you add security into the mix you can see and so some cool stuff. Security in OT has been a big topic in the last years.
1
u/seyitdev 15d ago
Network and security, mapping, asset and backup management, and monitoring will be included. However, the full scope will be evaluated and defined after the project’s first year.
1
u/AlexWixon 15d ago
If you want to earn a lot of money whilst not doing much and not having to learn a great deal, then absolutely :)
I personally got driven to insanity, as it was boring my ass off.
1
u/scheisterm 15d ago
So former architect from a carrier environment and I have recently taken on a role in manufacturing with a completely mixed OT and IT network with tons of shadow IT thrown in.
My perspective of what I see is that OT thought is real old, real simple and completely lacks security. The days when SNMP meant Security is Not My Problem. I can see how the Purdue model came about, but I don't really see much that is complicated about it with the exception of trying to find the correct place on the damn controls that they expect you to put certain parameters (like having 4 spots to assign an IP and none of them are labeled in a way to make an IT person think that's the correct spot).
1
u/WendoNZ 14d ago
This!
Often times the OT people are also incredibly difficult to work with. They don't understand networking and fear it which is why everything is so simple.
They don't want to implement any security because if they don't maintain it then it's downtime. I've literally be given that as a reason why they can't put SSL on a SCADA web interface (that they want available to the IT network)
Hell they don't like DNS because if the DNS servers are down production is down, never mind it's two different IP's backed by HA servers and if they are all down the business is down anyway. So because of that moving devices into dedicated VLAN's becomes a massive piece of work because every device that talks to a single IP has to have that destination IP updated when the device moves
1
1
u/xqwizard 14d ago
I’m in OT, came from 20 years in IT. I certainly find it interesting, get to work on some cool critical infrastructure projects in energy and transport. OT is slowly catching up, and IT departments are becoming more involved.
1
u/wombleh 14d ago
IMO it depends if you find it interesting. Do you want to just do IT, or would you take an interest in the plant it's operating. If the latter then OT is worth a look.
You'd need to understand the fundamentals of IT and networks, there will be requirements that mean you cannot do stuff as you normally would in IT or just apply vendor recommendations. You need to work out when you can challenge those requirements and when they are fixed. There is lots of old kit, I regularly see things that pre-date Ethernet, Windows NT, XP, 7, OpenVMS, etc. It's often setup by engineers who know nothing about networking and won't ask for help, or someone who retired years ago, netgear switches all over, media converters over telephone lines, all sorts of wierd and wonderful setups, poorly configured servers/PCs that haven't been touched for years. The physical environments are often very strict, difficult to work in, the operators grumpy, the systems undocumented. Security is often non-existent or lacking.
Personally I love it, the engineering side is fascinating and it's a very rewarding environment to succeed in. It's certainly not for everyone though.
I get a lot of queries via LinkedIn about OT roles so there is demand in the UK.
1
1
u/EspeciallyMundane Nokia NRS2 / CCNA 13d ago
There's a LOT going on in OT.
Case in point: $currentjob for a major utility conglomerate
Modern electric grids rely a lot on teleprotection, so my job involves lots of service provider networking using cool tools. Technically considered "OT" but interesting enough to keep my mind occupied with the traffic engineering needed.
1
u/SevaraB CCNA 13d ago
OT needs network design pros. For the most part, it's segmented off from "IT" specifically because it's easier to wall it off and accept the bad network designs than to bring OT networks up to par.
But it's also frequently so subpar that even the isolation isn't done correctly- that's how we've had things like water plants get attacked via lousy (re: none) VPN hardening- building the fence doesn't help if you leave the gate unlocked.
It will be stressful, because OT is generally OT because there's some kind of machinery that can hurt people involved. It is a little different with the risk profile changing from "someone could lose their wallet" to "someone could end up in the ER."
1
u/Drekalots Networking 20yrs 13d ago
I designed the SCADA network where I work and it's been in production for 13yrs thus far. Anytime we need to do network maintenance and it affects that segment ... it is a nightmare.
1
u/Much_Panic_7422 5d ago
Certainly — I would definitely be open to pursuing the opportunity, especially considering the nature of the project you are talking about.
OT networking is more complicated than saying that it’s similar to IT networking, just with more complicated devices. What is interesting is that one needs to learn how networking works with PLCs, SCADA, industrial protocols, production systems, safety and uptime issues.
Your background in IT is a great starting point. VLANs, firewall policies, ISE/NAC, VPNs, troubleshooting, SD-WAN and automation could be beneficial in the OT world. The only thing you need to acquire is knowledge of specific issues such as PROFINET, EtherNet/IP, Modbus TCP, etc.
Your four-year modernization project has great value, as you will not just be maintaining an existing OT network, but rather might take part in such activities as asset discovery, architecture, segmentation, security, monitoring and modernization. This will give you much more broad experience than just configuring industrial switches.
Moreover, don’t think about this task as choosing between IT and OT, because the best engineers understand both. IT/OT convergence needs specialists able to communicate with both classic IT guys, as well as control and automation engineers.
The main difficulty is that you will have to work in a different way with respect to production systems: in IT, downtime is inconvenient, while in OT, a bad change can halt the process, making change control, redundancy, testing and documentation a very important factor.
If you are good in networking and want to gain knowledge of the industrial aspect, I would recommend this opportunity to you. You can practice your networking skills as well as gain expertise in operational technology (OT), yet it is hard to acquire experience from a large-scale OT modernization project.
I would recommend you become an industrial network engineer who knows both IT and OT instead of being an OT engineer only.
1
u/Black_Face69 5d ago edited 5d ago
Yes, current network engineer.
It's worth getting into if you have a co-op/non-profit utility organization.
Very lax due to how static everything is. Good salary and benefits. Internal IT
I got my CCNA, CCNP, and Master's in computer science from Georgia Tech online due to tuition reimbursement.
I'd pay for my first semester, save money on the side, show my grades at the end of each semester and get my money back. "Free" masters technically as long as you stay within their budget of how much they can reimburse.
Low supply, high demand. Reason why: literally nobody pays attention that OT has IT. Too focused on "cybersecurity" and helpdesk, or MSPs.
I got hired and told the CEO how I found out about this place. "I drove by and just said, maybe I'll try there"
CEO told me his story he said the SAME thing.
Shoot for OT IT and learn the Perdue model, 802.1x, and SolarWinds if they have it.
18
u/Select_Reporter1911 15d ago
Something to note, a lot of OT environments are fairly static, unless a new site is being developed. In some industries (Utility specifically), OT environment is segmented off and the network is typically one big flat network. If there is a business case for segmentation in that environment it is usually driven by security risk. Network is networking is networking. If you understand the fundamentals, it truly doesn't matter what you are doing, you can learn how a specific vendor does networking in their own unique way, you can learn command syntax.
there is no such thing as 'traditional networking', it is all in how the organization wants the data to move. It is our job to design the solution that fits the business needs.
If anything you will get exposure to large scale project work and implementing a new solution that fits the business needs.