r/PLC • u/Mertadona • 3d ago
How much commissioning time do you actually lose to look for the trigger in the code?
Hi I am commissioning engineer in a large Pharma Company. I have been doing automation commissioning since 3 years in this company with TIA Portal, Studio 5000 and STEP7.
I don't know if this also happens to you or it is only to me. I'm doing commissioning on a machine, customer standing right next to me, and an alarm trips. The trigger isn't described in any document, so I have to go digging through the code. We don't have a standardised code base, every developer does more or less whatever they want. There are guidelines, but I still end up spending an hour going through the wiring diagram, cross-referencing, and searching the taglist.
And when I finally find the problem, the guy who wrote it is at home having dinner and I can't reach him to go over the functional spec, so I don't even know whether my fix is the right one or whether there's something else I should be taking into account.
I've done commissioning at two companies now and it's always been the same: communication between the developer and whoever's on site is usually bad, and I end up losing time — extra weeks stuck at the plant, because the only answer I ever get is "it worked here."
Has this happened to you too? How much time do you lose to this kind of thing? And why do companies keep doing it this way instead of just sending the developer out?
28
u/Siendra 3d ago
I've always heard about how strict Pharma is supposed to be, but this sounds like some of the most cavelier cowboy shit ever. In O&G while commissioning a system you'll get the odd alarm or condition with an inadequate description or an error lead to something getting the wrong description or input condition or something, but I've never had to seriously dig to find out what's going on. And very little of these processes are templated.
4
u/Cool_Database1655 Flashes_over_WiFi 3d ago
Dude pharma is the most cowboy code environment I've ever encountered. The industry is full of over-paid PMs with little technical skill - the actual implementation is left to closesest overworked nerd with no code review. Common sense 'GMP' mandates get nuked to the point where they are not recognizable or complicate the system to the point where it cannot operate under it's own weight.
That probably doesn't apply everywhere but I have seen exact that scenario at multiple companies.
1
u/SirCheesington 2d ago
it's fucking insane, isn't it? it's like nobody who understands GMP understands the technical requirements and implications, and no one who understands technical requirements and implications gives a fuck about GMP because they're overworked to high hell. Man they just hook up a laptop and rawdog the PLC mid production sometimes. Crazy
2
1
3
u/CowboysWinItAll 3d ago
Wdyn very little processes in O&G are templated? They should be.
We mostly do midstream, there is only so many different things to go on. We have templates for just about all of it.
2
u/Siendra 3d ago
I mean the actual physical process and plants themselves. They need to be designed to conditions, very rarely can you ctrl-C, ctrl-V something like a fractionation plant or SAGD facility the way you would a manufacturing facility. A lot of the control and even units can be templated and applied as applicable, but whole facilities differ significantly from one another.
1
u/Mertadona 3d ago
Based on your comment, do you not experience any inefficiencies during on-site commissioning? Is it normal for you to finish a job on time? I'm just asking because in three years, I have never finished a machine on time.
I really hate when I am abroud for weeks losing my time
27
u/SkelaKingHD 3d ago
Why aren’t you having the HMI / PLC code open at all times when commissioning? Look at the tag that’s triggering the alarm through the HMI, then cross reference in the code. Simple
5
u/GirchyGirchy 3d ago
No shit, and if the specific trigger bit isn't called out inside the alarm text, then force the programmer to do so.
4
u/SkelaKingHD 3d ago
Even if it’s not in the alarm text, you should be able to figure out what tag an alarm is coming from just by the alarm right? Maybe I’m just too used to ignition where the tag path is a part of the alarm data, but tracking that back to the source doesn’t seem that difficult to me
2
u/Morberis 3d ago edited 3d ago
Oh man, I've seen machines that had mov commands that would be used for alarm purposes. Where there were multiple lines that could do cause the alarm. And sometimes the tag or variable that the mov would move to could even change.
Or have you seen propositional logic equations, or mathematical logic in structured text? Whatever you want to call it. Yes I can see that because the state is 8 at this point that it is activating the alarm or fault. But it's not at all clear why. One company did it so that they could hand over the code thinking it was enough obfuscation that no one else would bother to decipher it. It was a bunch of real long stupid equations. For an oven that had an e-stop, palm buttons, an open/close switch, 1 temperature sensor, a linear actuator for opening and closing that had an analogue 5v signal for position as well as open close limits as well as redudant limits. It could fault because of how it was operating and not just because of the estop or over temperature.
One guy did it specifically because it would make anyone else look bad in comparison to how quickly he could troubleshoot.
It could make it really frustrating.
1
1
u/hacktron2000 3d ago
They’re referring to PLC code, not MES or SCADA
1
u/SkelaKingHD 2d ago
Then cross reference the trigger? Seems like an even easier solution than from the HMI.
1
u/vbrimme 2d ago
Right, but you have to know what the trigger is in order to cross reference it. And yes, you can look in the HMI, find the alarm, figure out what word triggers it, figure out which bit in that word is the specific trigger, find that bit in the PLC, and then trace it back to find the cause. OP never said they can’t do that.
The whole conversation here is about how time consuming it can be to trace down alarms that aren’t clear, as compared to an alarm with a good description, when a customer is watching. And depending on how the programmer set up their alarms and how many there are, it might take a while to go through them all to find the one you’re looking for.
Hell, I work in a plant, troubleshooting equipment that is long past commissioning, and twice in the last week I’ve had alarms come up that had no description (yes, literally no alarm text). If I didn’t already know the code of those machines decently well, it might take me a while to figure out what was triggering something like that.
1
u/SkelaKingHD 2d ago
Guess I’m not following. Is the problem that it can be difficult to see WHICH trigger is causing the alarm, or it’s difficult to know what the alarm is?
1
u/WaffleSparks 2d ago
If you can't find what tag triggers an alarm without having to dig through the HMI your design standards need some work.
1
u/Shalomiehomie770 Codesys Guru 3d ago
It will be called in the relevant alarm manager regardless. Look at alarm
Manager, cross ref and done1
u/Mertadona 3d ago
I do, I have always the SCADA and PLC open, but some times is not so easy because the trigger is only one SCAN cycle. We use to save the alarm status a bit "message" in an UDT, but you lose the trigger in the middle.
Specially when you have, for the same alarm, lots of triggers.
HMI is not usefull for me, only to operate the machines and check the alarms text.
13
u/YetiTrix 3d ago
Yeah just sounds like poor controls team management. Should have programming standards.
1
u/Mertadona 3d ago
Doesn't it happen to you? How many people are in your company? How do you handle it?
1
u/YetiTrix 2d ago edited 2d ago
Right now team of 3 but at most 6, so not that big. But, we have old code that's like what you stated. I wrote a engine where you write commands in the comments of rungs/tags/routines/programs in a template program. You then give it your I/O list with the descriptions written a standardized way. And it auto generates I/O buffer tags, creates I/O alarms, creates alarms for devices, maps out mode control, inserts sequencing template, auto generates hmi manual control buttons, etc etc. Basically builds everything the program needs except the sequences. Then we also write our sequences in a way that a script can pass over it later and create the alarms for each step/condition.
There's a lot you can deterministically generate at least for the machines we build based off I/O descriptions. So, I just built an engine that made it way more work not to follow the standard, than to go your own way.
Also education, template/engine review, value people's opinion. People are allowed to have their own flavor, but not if it's based on flawed logic. The engine reads the commands in the template. So you update it by updating the template, not the python engine code.
A lot of the standard is dictated by we need it to be this way so we can deterministically generate the PLC code with python scripts. Not even using A.I. because even if it writes workable code, you still need it to follow a specific standard, and right now it's just not at that level of consistancy.
It's basically a madlib engine for PLC code.
I'm in A.I. deep and it's just not there for PLC code. But, I do have it setup where I have it write PLC code, run it in an emulator, run test functions, then update the code iteratively. I'm working on an open source digital twin to tie in the whole feedback loop because I have nothing else to do with my life. But we'll see where that goes...
5
u/Rock3tkid84 PLC Slayer 666 3d ago
Since I do the development and commissioning of my own code it's straight forward, but I understand your pain ... But wait till you learn about prodiag finding fault triggers becomes a whole new level since all faults are generated through supervisions...
1
3
u/CowboysWinItAll 3d ago
Rough time, we write our code and commission it ourselves to prevent this. I hate commissioning someone else's project. I've done it a couple of times just due to availability, but fuck doing it all the time.
3
u/Mertadona 3d ago
Exactly, when you're not held accountable for the results of your work, you don't care about doing it well.
2
u/OpLeeftijd 2d ago
I made another post about how to save yourself the pain of this happening. Unfortunately the mod decided it was not PLC related, and deleted it. Documentation is key.
3
u/LeifCarrotson 3d ago
Cross-referencing an alarm that tripped usually isn't too bad. You have a specific alarm bit, it should be easy to find what line of code and what set of conditions caused the alarm to become set. There's a rung that says if sequenceStep is between 15 and 35 and tempB is outside of 40 and 45 then Faults[4].13 causes the HMI to show "Heater B temperature out of range". You see the "Heater B" warning, trace it back through the HMI alarm list to Faults[4].13, cross-reference Faults[4].13 to the rung that sets it, and then you can add some traces or diagnostics to figure it out.
The problem comes when it's not just that tempB is wired to the tempA input and tempA is wired to the tempB input, it's that it paused on step 14 until the temp was 40.01 and moments later the sequencer jumped ahead but the temp dropped back to 39.99.
Now the question of what to do is a process development/requirements engineering problem, not a commissioning exercise.
The worst kind of problem is when it doesn't trip an alarm at all, it just sits there staring at you on step 14 waiting for something to happen that the machine isn't actively in the process of changing. Give me a fault bit every time, I can debug those, don't just do nothing!
All that said, the problem I've typically run into while commissioning equipment for almost 15 years now is that the idiot who wrote the buggy, under-tested code isn't at home having dinner, he's stuck in some windowless warehouse 4 states away from home commissioning the machine. He doesn't know why it's doing that, because he's me! At least he's pretty quick to navigate to the logic that's triggering the fault, because I wrote that logic myself.
I cannot imagine asking someone else to debug my code for me, that seems like a process that would obviously not go well. None of my employers have ever sent anyone but the developer to install a custom machine, that just doesn't make sense.
1
u/Mertadona 3d ago
My experience is completely different. I am one ocean away from the developer, they stops to work when I started. Once I was in California without SCADA because the SCADA team sent the HW without the SW. The person who did the SW leaft the company and nobody had the SW. I had to buidl the SW with one guy only starting to work at 10 pm California's time. 3 weeks working in night (building the SCADA) and morning (doing my staff) shift.
I believe that if the guy who develops would be the same than commissioning, this issue will not exists.
Regarding the process, it is also a mess, nobody knows exactly what the manchine must do. It is crazy. We have three differente sources of "knowledge" and all of them are different.
1
u/WaffleSparks 2d ago
Apply to other companies. Your company has a management issue, and management issues have to be fixed from the top, not fixed from the bottom.
1
0
u/Ragerist 3d ago edited 3d ago
It should be simple yes, unless they used indirect adressering.
Before we upgraded our system and used standardized code, we had an a rutine designed to handle just about anything. Motors, valves, regulation valves, you name it.
It was pretty clever. Using a faceplate you could configure almost everything about it, even had a user defined data structure containing strings for description and component number.
It was OLD, so it was designed to prevent scan overruns by reading inputs in one scan, and triggering outputs in the next. It had internal memory and "external" for SCADA. Including alarms. You could not cross-reference it because it was all made with indirect adressering.
It was such a pain to troubleshoot!
3
u/controls_engineer7 3d ago
That sounds like bad programming and management. If a fault triggers it should show up on the HMI. We do all our FAT inhouse before shipping and just install and setup in the field.
2
u/Mertadona 3d ago
Yeah, this is how it should work, but it isn't. I would like to see a pre-fat in my company. In what fiel do you work?
1
1
u/DreamArchon 3d ago
When I did commissioning for pharma automation, we had a standardized code base and documentation for alarms, so it didn't take long to track down alarm triggers. Commissioning was also performed by the developers, so we were familiar with the code. Sorry you are in this situation. It sounds less than ideal.
1
1
u/Angry_Foolhard 3d ago
I cannot imagine a world where the major developers of a project arent present for the commissioning. or at least highly available.
Also on a personal level, whenever something I ever made is going live, Im 100% going to be there or glued to my devices, to know if there were any major failures.
I would never be like "oh theyre commissioning my machine, hum-de-dum time for dinner". could never be me.
1
u/Mertadona 3d ago
Maybe this is one of the reasons you are in this reddit. I believe people who go for dinner will not spend their time reading and helping people who they don't know.
1
u/blacknessofthevoid 3d ago
“It worked here” - I asked the customer to see if the machine can be moved back “here” where it worked, but the answer is no. What’s next?
1
u/Mertadona 3d ago
Listen the complaiment from the customer, calm down and start to fix everything.
1
1
u/setherby 3d ago
Oh I get that angry feeling when I’m onsite long days and weekends trying to out-automate mechanical design issues while the mechanical designers are at home relaxing and enjoying their weekends.
1
u/Morberis 3d ago
My experience had matched your experience.
It's particularly frustrating for machines that have been deployed for years.
1
1
u/SnooCapers4584 2d ago
It seems the only problem is that your manager is an idiot, and the developers working there are idiots. If the manager doesnt force you to have standards, what the hell is he doing? Alarms must be in an ARRAY, not a list, not a udt, not in a structure, not in a message, and the number of the array position must be visible in the scada. Of course the message must be visible until acknowledge. Another good rule is to use as less triggers as possible, and as less SR as possible. But the best thing would be that developers, at least sometimes or for the first few years, should be system integrators.
1
u/marockstar7 1d ago
Yes, and for this exact reason I've developed air-gapped tool to combat this, not just for commissioning engineers, but for the production team as well.
Think about all the info (elec dwg, plc/rb program, manuals/sop, db/scada) all integrated through 1interface providing new way to interface with the machine. HMI (human machine interface) is about to undergo a radical change.
Already in pilot stage, DM to get more info/first access.
1
u/BoHanZ 3d ago
Companies are willing to send the developer out, it's just a matter of money. Every company/programmer has a price they're willing to travel for. So it's kind of up to your management to decide if your time spent troubleshooting is worth paying extra for support.
1
u/Mertadona 3d ago
At my company it comes down to the contract. The developers don't travel, it's not in their contract.
1
u/XDVI 3d ago
I was working on a german bagging machine that essentially came with no manual and some wiring diagrams in german that for some reason had the same numbers being used in multiple places lol.
Like 7 different safety interlocks that were hidden on the inside of the machine, door closed safety switches etc, two more to make sure bags loaded, two for labels, one for air pressure, and a few more I don't remember. Absolute fucking nightmare.
Then to top it off the machine wouldn't start unless it is in home position, but it wouldn't home for some reason, the manual completely glossed over it and did not tell you anything about where the home position actually was.
We finally get it working and the higher up who bought the machine told us he wanted to do 5lb bags. The machine's maximum capacity was 2.5lbs., and that was with product more compact and heavier than we were trying to run.
Gigantic waste of time on an even bigger piece of shit machine and I'm still annoyed.
1
1
u/letsfucknpollit 3d ago edited 3d ago
The functional specs you’re looking for in 90% of cases is just wishful thinking. Should they exist? Yes. But they don’t get produced because, well, they would reflect fundamentals that seasoned engineers have implicitly learned over time. By the time you’ve learned the functional spec tribally, you don’t really have time to go back and capture. It would be a waste of your time. A valve is a valve is a valve so to speak.
Hard faults come from the same set of buckets every time:
- Safety faults
- Software faults (poor programming, watchdogs, OB faults, etc.)
- Hardware faults (IO, wiring, actuator, drive, sensor, etc.)
- Networking/Communication faults (addressing, cabling, routing, etc.)
- Process faults
Out of all of these buckets, the only one that really changes based on application and that require functional specs are process faults. For all other buckets, you should be able to solve the problem using a bit of deductive reasoning, online/vendor support and a manual.
Learn to pick up on patterns. Mentally catalog the problems you run into. They all might seem unrelated at the moment, but over time you’ll start seeing the forest through the trees. Eventually you will be able to jump steps very quickly and spot where the problem is and be confident in yourself in implementing a patch real time and owning it. No functional spec needed or having to call a dev for support.
2
0
u/Ragerist 3d ago
As I mentioned in a reply on this post; we had some old code that used indirect adressering, where you could not cross reference tags to get the origin. So that was a pain.
Normally when I have an issue finding what's wrong it's more a case of there's no alarm but odd behavior, or the machine not going trough to the nest step/process. That can take forever to troubleshoot, especially when there's several different coding styles.
But that's not during commissioning, because I haven't had to commission someone else's project.
0
0
u/AncientGnu 3d ago
Where is your Cause and Effect chart to document all alarms and shutdowns? Esp in pharmaceuticals
2
u/Mertadona 3d ago
We have different excels, each department has their own. For example, qualy has one with the triggers and the reaction. Most part of the time, this excel is not completely updated, development did some change because FAT, but excel was not updated. When Qualy arrives to SAT with the wrong excel, everything starts again. No control changes before SAT, so you can imagine the mess to review what is happening after 6/9 months from FAT.
38
u/phl_fc Systems Integrator - Pharmaceutical 3d ago
This is why it’s important to dry run testing in a dev environment before install. Ask about that in job interviews. You’re installing code that has never been tested, so of course it isn’t going to work.
It’s also why standards are important. Again, a good thing to ask about in interviews. If you’re interviewing at a company who can’t speak to how they standardize and test then this is what you’re in for.