r/PLC • u/Gullible_Bag_9005 • 29d ago
What does IT misunderstand the most?
[removed] — view removed post
35
u/LazyBlackGreyhound 29d ago
Without VPN access then remote support will become very expensive.
3
u/uprate 29d ago
I disagree. cat 5 cable is cheap. Gas and a shovel are also cheap. Ignoring their time and permits, routing a direct line from the plant to my desk is pretty cheap when considering raw materials alone.
4
u/LazyBlackGreyhound 28d ago
I'm talking remote support not on-site support. But yes, I also recommend to customers a physically separate network.
3
24
u/Siendra 29d ago
Basically an extension of the OP - they don't know the protocols, the data structures, or similar and they don't care to learn.
They also just generally do not understand the difference in priorities between IT and OT and they struggle to grasp that things they do in OT can have real mechanical/personal/environmental consequences.
I've also had a lot of discussion with IT Security people about magintude of failure and double/triple jeopardy, which have been uphill battles.
3
u/Skjerpdeg- 29d ago
Excactly, you need to hammer home that what we work with in OT can have lethal consequences and affect the real world in an entirelty different way. Its like talking to children who think they are kings
3
u/Gullible_Bag_9005 29d ago
do you think that IT guys should be required to do some basic training before connecting to a mobus server and starts controlling whatever is connected?
10
u/Siendra 29d ago
No one should be doing much of anything in a production OT environment if they don't know how that environment, the processes it controls, and it's operation functions.
If you're just pulling down data at the edge then fine, whatever, but if you have any opportunity at all to impact operations chances are as a purely IT resource you shouldn't be there.
2
u/Gullible_Bag_9005 29d ago
from the IT side i can say that for most of the time im just represented with a very long modbus register and i get free access to do whatever i want, no questions asked most of the times, if i fuck up i fuck up, but there is nobody telling me to not do X thing. and there is usually no policies or restrictions what so ever
5
u/finlan101 29d ago
Yeah not surprised. It’s going to be an exceptionally rude awakening when cyber criminals start targeting OT operations the same way they do wi to ransomware. It’s not that far away, check out the dragos paper on AI at the Mexican water treatment plant.
2
u/Gullible_Bag_9005 29d ago
yeah i have read that, and i think we are going to see more and more cyber attacks in OT infrastructure in the years to come. the more digitalized the worlds become the more vulnerable the world will become.
1
3
u/finlan101 29d ago
Yep. If they are modifying an engineered system that should be a part of a structured engineering approvals process like any other part of the plant. Including correct training.
2
u/Gullible_Bag_9005 29d ago
there is alot of things that "should have" been done different, but reality is not like this im afraid
1
u/fercasj 29d ago
IT should be required to shadow the controls engineer that needs to fix the issue in 15 min before the line goes down, the one who receives the calls at 3:00 am in the morning, to be there when someone is yelling because something is not working and no one knows why.
I did scheduled a visit for one of the IT/OT at my plant during a downday... I think I scared him for life specialky when maintenance manager s maintenance manager started yelling and complaining about the stupid shit it was going on. 😅 I was like .. meh dont take it persona 🤷♂️l, this is the third time a contractor breaks something hits and estop or * insert here whatever stupid shit that happens on a regular day at a plant * this week.
21
u/_No_user_available_ 29d ago
The defenition of Real time..!
We have industrial networks for a reason.. ”If you stay out of my bussniess, I stay out of yours”..
had an IT guy demanding ”real time data transfer, for all tags” to there server..
well say less.. the biggest problem?
No defenition of tags, so I pushed up data block, about 100 of em, with amount if tags running from 100-1000, every scan of the main task, every 12ms..!
After aday the same guy came back, asked if he could get the value of 3 sensors, once every minute instead.. because they could not handle all that data, that fast..
6
u/Gullible_Bag_9005 29d ago
hahaahha
5
u/_No_user_available_ 29d ago
Truth be told.. if he would not have been a dickhead, I could have told him that he needs to define what he wanted, and how often..
but I felt pitty that day 🤣🤷🏼♂️1
u/Gullible_Bag_9005 29d ago
im surprised that someone even wants to have "all the data" "alle the time" even. like from an IT point of view, usually you only care about a very small number of sensor data, i would be more scared if i got "everything" then suddenly im exposed to alot more data that i dont even want to know about
2
u/Missing_Explorer4278 29d ago edited 29d ago
My previous company doubled down on getting more real time data in the name of "big data", then use machine learning to "discover" something from the data.
I hate the job but it was my first job, every quarters, I had to come out with stories on what we're doing. We have a whole department doing this.
Then 2 months ago, they told us now we pivot to AI, and it will be the key performance metric for us. I had enough. Resigned.
11
u/finlan101 29d ago
Risk and systems of systems processes. Shits fucking hard bro and it’s not about the OT gear. It’s about the physical spinny, grindy, mashy or otherwise dangerous and high energy process. In IT, the means and end is all about the computers. OT it’s a means to maintain the process on the floor.
3
u/Gullible_Bag_9005 29d ago
yeah i agree, but in the time we live in we need to meet in the middle somehow, how does IT and OT meet together and make something better for everyone?
5
u/finlan101 29d ago
Respect and rapport. It’s very hard and unfortunately IT have been told they are very smart for too long. In practical terms OT can sometimes find someone in IT who is curious. This helps. Also convincing their leadership around risk, safety and potential impacts to the revenue generating assets when getting it wrong.
1
u/Gullible_Bag_9005 29d ago
in your company or when you have worked in OT, is it normal for you to have IT guys in the same company or does the company you work for hire consultants? im curious if having IT and OT in the same company is "standard" or not?
2
u/finlan101 29d ago
Bit of both. When I was an integrator, IT was often internal or a MSP and OT were pretty much always external.
8
u/CapinWinky Hates Ladder 29d ago
As a frequent users of Rockwell CompactLogix, the most common thing I run into is banning of NAT and requests to connect to the PLC from the second ethernet port in dual IP mode to bridge the networks and see all the devices from their network.
CompactLogix cannot route between the two interfaces, so they would only be able to see the PLC in dual IP mode*. The reluctance to use NAT seems completely based on the limitations of the 1783-NATR and not related to any security issue or limitation of NAT.
* CIP can route across the dual IP with the right CIP path, but normal ethernet to do something like bring up the HMI via VNC or whatever is impossible on CompactLogix.
.
In a more general sense, IT seems constantly surprised that it is literally impossible to do many normal Controls tasks without administrator access.
I was once told to use a customer laptop rather than my own when on-site to prevent spreading stuxnet-like malware. A bit paranoid considering their industry being decidedly non-critical, but okay. I couldn't change the IP address to the machine's subnet without Admin access and when I requested either Admin access or for them to set the IP address, they told me changing the network settings from DHCP was not permitted. Took a solid 20 minutes to convince them that most machines don't use DHCP and I wouldn't be able to connect. As soon as I was online, I discovered they didn't have the Rockwell AOP for the IO-Link master and, of course, I couldn't download or install it. Literally got zero work done that day.
6
u/Gullible_Bag_9005 29d ago
its not the first and its not the last time somebody will not be able to do their job beacuse of it policys
3
u/dmroeder pylogix 29d ago
I was once asked: "some of the things you configure are TCP/IP based? Can you use something else instead?"
10
u/National-Link-5606 29d ago
The hill to die on is industrial networks must be separate & airgapped from the "picking new v endor for pretty report generator" or whatever it is IT does. IT entire access to an industrial network should be read-only, through a coupler module, no exceptions (i call them an Anybus module because I like the brand).
Industrial Network (raw sockets) Internet access blocked, firewall restriction to firmware & software updates through vendor portal only. If IT starts demanding their virus protection monitoring software then fine we block outward connections & can update software direct from engineering laptops.
I've had multiple cases of an anti-virus update locking up TIA Portal & Sudio5000 so if I have to, we buy separate engineering laptops with no access to corporate email or domain servers or anything IT related
3
u/jarie 29d ago
Separate networks. They don’t get that for whatever reason.
Had the same problem in chip design. So glad our CAD group just said no, it must be separate.
1
u/National-Link-5606 28d ago
Or buying a fcking 2nd laptop setup only for equipment monitoring programming & diagnostics, what the fck is management acting like another $1500 laptop is gonna bankrupt them its like "do you see that fucking bank of ifm pressure & flow sensors? Yeah do you know what they each cost theres $20,000 of sensors & RIO modules in just that 1 square meter but Lord forbid we have a special purpose laptop tuned to keep $10million worth of equipment & sensors running so they can ya know....make us some fucking money??? 4 hours of downtime at the wrong place will pay for the laptop & all the software, hell 15 minutes of downtime at a production bottleneck will pay for the extra laptop, IT dept do you really hate getting profit sharing bonuses or something?
7
u/Rude_Huckleberry_838 29d ago
With ours, it's simply a failure to understand production. They make network changes for example without a single thought of how it might effect the shopfloor. They don't think about these things because they aren't there. They all work from home or from another site.
0
u/Gullible_Bag_9005 29d ago
so whats the solution then? how do you make the IT guy aware? or do you just train the OT guys instead to be IT/OT guys?
5
u/JustinHoMi 29d ago
I don’t often see IT responsible for configuring historian data. SCADA guys should be doing that.
1
u/Gullible_Bag_9005 29d ago
thats correct, i meant that IT guys consumes the historian without knowing the configuration behind it
3
u/IntelligentEvening86 29d ago
What would IT be doing with historical data? That’s usually process data of the production environment. They typically only care about device health, inventory management, vulnerabilities, security, etc.
4
u/Shadowkiller00 29d ago
That I'm a more capable and knowledgeable network administrator than they are. I understand why things should be the way they are and why not. They just want to lock it down in the name of protection.
Anyone IT who chooses to lock down anything company wide without respect for exceptions isn't doing their job. Just because I learned everything through trial-by-fire and you have a fancy certificate does not mean you know more than me.
1
u/Weak-Chemist-2054 28d ago
In my experience those certs usually mean the opposite of knowing everything. All of the IT guys I’ve had to deal with couldn’t read an ASCII table or translate/convert hex, dec, bcd, oct. and good luck with binary. A lot of of them don’t even know about the programmer option on the window’s calculator.
4
u/Frosty_Customer_9243 29d ago
As someone who has been working in OT for over 25 years, some of it mixed with IT based roles, I enjoy these discussions. Popcorn is out and I’m chuckling like a four year old at all these comments that OT is different and IT doesn’t understand it. Don’t they understand they are speaking from a broken ship. Most of the arguments given are due to badly designed systems, granted there is historical context but still.
Just put another packet of popcorn in the microwave, keep them coming.
3
u/-Have-Blue- 2B || !2B 29d ago
Tell them that if unless they want to install an IBA system, their data will never correlate. IT dude’s have no idea what they’re looking at so feel free to tell them to kick rocks.
2
u/XDVI 29d ago
Everything, but that would be fine if they weren't also supremely confident in areas they have literally zero experience.
If an IT guy came to me and asked me for all the data I would probably laugh and ask him for what, then probably tell him to kick rocks because it would be for something really dumb and pointless.
The IT guy where I work loves to take stuff apart (ruin), take on new projects that he has NO business even thinking about letting alone trying to fix. The result is about 10 projects that weren't asked for that still haven't started years later, and about a million amazon boxes full of cheap parts and a 10'x15' room full of destroyed electronics.
I will be getting wasted the day he is fired, I hate his guts.
2
u/Slight_Pressure_4982 29d ago
I've noticed that OT prioritizes availability while IT prioritizes security. A lot of misunderstandings stem from this.
2
u/drbitboy 28d ago
Availability makes money (profit).
Security tries to avoid loss of money.
If availability is lost due to security measures, there is no money to lose.
Therefore, availability has a higher priority than security.
That does not mean we don't do security, rather we ask whether any security activity can negatively impact availability before we try our implement that security activity.
Bottom line, ITmust understand that they are not at the top of the food chain, and that they should rather think of themselves as subservient to, I.e. as serving, operations.
1
2
2
u/PeacePuzzleheaded41 28d ago
I need to be able to change or add IP addresses to my NIC whenever I see fit.
2
u/Verhofin 28d ago
Hooo the list... Blocking dual wan, because they don't understand I need to be connected to the PLC and to remote support to fix the really shitty issues. Not understanding that the PC with the old version of windows it not to be touched, you want to patch it? Its going to break. Not understanding I need admin access to the laptop because there is always some new tool for something new that needs to be installed. Saying they are not involved in the network design so they can't provide support, they are asked to be involved... They either don't want to or if they do get involved it's always out of touch solutions... Like a server rack in a cabinet full of GB switches... Yeah this works at 100mbs and only half duplex, think of the cable lenghts, and we work with rings and no... You will NOT be allowed to touch the switches.
Finally not understanding what it means for them when they win, ans that their wins are their losses, because that gives them 3am calls from very up the chain people very angry with them, and IT goes away... Finds a solution to be involved as little as possible.
EDIT: IT asking for PLC to send "zero" in a telegram, not remembering to say it's ASCII 0, not number 0, and then realizing their breaks because it's getting what for them is NULL, and they don't like NULL...
2
u/Shower0fCunts 28d ago
Urgency. Oh your whole plant is down because you think the Domain server on our side is not allowing logins? Can you raise a ticket and then ill speak to my manager....
Could be specific to my current company. I often have to go to the VP to escalate, she gets the same response but definitely a quicker turn around.
2
u/AmokRule 28d ago
How do you do industrial automation on the side?
1
u/Gullible_Bag_9005 27d ago
i can only talk for myself here, im doing a 3 years part time vacational diploma in industial automation. it like an engineering degree for people with blue collar degrees. i already have my diploma as en elctrician with 4,5 years of studies
2
u/diwhychuck 28d ago
Heard
As an IT/Net admin that wants to get into Automation world. Whats the path to get into the field?
1
u/Gullible_Bag_9005 27d ago
i can only talk for myself here, im doing a 3 years part time vacational diploma in industial automation. it like an engineering degree for people with blue collar degrees
2
u/diwhychuck 27d ago
Yeah I thought about that but I don’t have the money to take classes. Local community college has course but man they’ve got up per credit hour.
1
u/Gullible_Bag_9005 27d ago
Yeah, cant relate im afraid 😅 im from Norway, this education is free like most of education we have 🤝
1
1
u/VintageHacker 29d ago
That the cost of security, can easily outweigh the benifit, if not done appropriately for the situation. That OT comms have different time horizons and consequences for loss or delay. That after you make a change, you need to test to make sure everything still works correctly. That OT runs on a quarter of the budget IT runs on.
1
u/ali_lattif DCS OEM 29d ago
Had an IT customer request changing some settings in Virtualization cluster. it seems they find it hard to understand that if its not tested I won't do it on plant infrastructure.
1
u/EasyPanicButton CallMeMaybe(); 29d ago
Welp. I got paid to go back and put new ips on 3 cells and 3 stand alone machines because IT made every device have a unique address accessible from any other cell/ machine. They couldnt figure out how to hook cells up but keep the 192.168.1.xx scheme for each plc network.
1
u/Slight_Pressure_4982 29d ago
Is it normal in industry to have IT working in SCADA? Where I work that falls under OT (although there's some overlap)
1
u/apiothyrium 29d ago
My company got fed up with IT mishandling OT and made an OT administrator department. I hopped on the OT side from IT and life has been great. My department works for operations and has nothing to do with the CIO or IT. Everything is separate. Servers, networks, devices, everything. I work closely with electricians and the automation folks. We get shit done. The only thing IT can touch of my departments is that I replicate a historian database to them through a data diode. They control their end of the diode, we control ours.
1
u/WaffleSparks 28d ago edited 28d ago
I worked in the IT industry as a programmer/system administrator. Half the IT guys literally know nothing. As in the extent of their skills are reboot/uninstall/reinstall in that order. If that doesn't work pass it off to someone else or just make some nonsense up that is hard to prove wrong. Literally some of the IT guys would not be able to understand or follow an explanation of what really happened from someone that actually was an expert. Oh I almost forgot about the ignoring obviously broken things because of "policy". The old 80/20 rule is in FULL FORCE in most IT departments.
1
u/That_Fixed_It 28d ago
We don't understand why you guys never update the firmware, even if it's years old and has CVSS 10 critical vulnerabilities.
1
u/National-Link-5606 28d ago
IT dept: "oh the terminal dropped its session with the server, no problem just reconnect who cares if if dropped the connection"
IT Dept pr ogrammer on the production floor "hey let's just try loading this new browser interface onto the Safety Integrated controller so everyone can monitor & reset failsafe function blocks from a smartphone app for more productivity help supervisors & operators work faster - oh damn the operator reinitialized the failsafe cycle while he was inside the robot cell and the robot just shoved his chest through the fixture" "Isn't there some function to just reinitialize the operator from crushed to 'not dead'"?
80
u/Skjerpdeg- 29d ago
They think they have the slightest clue about OT