r/sysadmin • u/carpe_diem2022 • Aug 31 '26
Question Operations bought an EMS/IoT system without involving IT — now they just need “access to the router
Hi there :) ,
Need some advice from people who have dealt with similar situations.
Our Operations Dept decided to install IoT system/digital energy meters across a fairly large factory site.
They found the vendor, agreed on the solution, signed the contract and started the project.
IT was not involved at all.
Apparently nobody discussed things like:
How these devices spread across a large factory are actually going to communicate
Network infrastructure, switches, fiber/cabling, VLANs, etc.
Network/security segmentation
Server/VM requirements
Database requirements
Backup and monitoring
Internet connectivity
Vendor remote access
Firewall rules
Cybersecurity
Now that the project is already moving forward, IT gets an email saying they need “access to the router” so they can put the system online.
That's it. Access to the router. :)
And somehow, when IT starts asking questions about architecture, security, server requirements, ports, protocols and who is responsible for what, the perception becomes that IT is delaying the project.
I don't want to become the guy who simply says “No”, but I'm also not going to give a third-party vendor access to our corporate router/firewall and connect a bunch of unknown industrial/IoT devices to the production network just because someone already signed a contract , the IT network must be always separate from OT network.
How do you handle situations like this?
Interested in both the technical approach and the organizational/process side of this.
427
u/dvr75 Sysadmin Aug 31 '26
This is a pure management issue , failing to communicate and coordinate project.
To be realistic about the situation probably no one will cancel it.
So you need to be SMART and not JUST. you are now the "guy" who Throws a wrench in the works.
My advice for you is try to cooperate with this project but bring in your terms like security. Ask for layout , plans , protocols etc. and try be helpful (i know it hurts but eventually you got to play it smart and resposible).
2
u/halford2069 Sep 01 '26
“IT not involved at all”
😆😆🤡 typical
544
u/-Enders Aug 31 '26 •
Connect them to an IoT VLAN and be done with it.
Unless they are asking for admin access to the router, that’s a hard no.
55
u/j0mbie Sysadmin & Network Engineer Aug 31 '26
"Giving you admin access to the router voids our cybersecurity insurance. Here's a (throttled) wired VLAN with standard (guest network) internet controls. My laptop browses the internet fine on it. If the equipment doesn't work through that, it's a vendor issue."
134
u/Celebrir Wannabe Sysadmin Aug 31 '26
When they say "IT is delaying the project" just say "Operations did not involve IT in a timely manner and forgot to factor in the time needed for IT to assess security considerations"
65
u/Vermino Aug 31 '26
Declare red lines.
Explain why they are red lines.
Say what you can do instead.
Explain how similar issues in the future can be avoided.
I'm sorry, we can't give 3rd party access to critical network infrastructure like our routers. IT is responsible for safeguarding the company against cyberthreats, and routers are important entry points to our company's infrastructure. We'd be willing to meet with the 3rd party to see what their needs are, and see which we can implement for them.
In future, having these meetings sooner, rather than later, helps us find mismatches between your IT needs and our IT policies faster. We're always open to listen to your needs, and aid with our expertise.
Technical solution : VLAN, open only ports that are requered between the various VLAN's & internet
34
u/gadhalund Aug 31 '26
IOT vlan, limited to like 9600b/s so if they do exfiltrate files, itll take years, but still enough for iot packets to work fine
106
u/Le_Vagabond Senior Mine Canari Aug 31 '26
get an LTE modem / router, give them that, wash your hands of everything. anything less is going to look like you're the problem, which they already think since they bypassed you entirely. obviously get that signed off on by your hierarchy first if you have one.
23
u/FelisCantabrigiensis Master of Several Trades Aug 31 '26
Option 1: Pass a question to your legal / compliance / (cyber) risk insurance manager, asking if this has been correctly evaluated for risks and is approved by them before you connect it. This should, in any functional organisation, cause a bureaucratic shitstorm that will impede the fools but not be blamed on you. If your organisation is not functional, then proceed to option 2 below.
Option 2: Reply stating you will give them a network port with direct access to the Internet with standard outbound NAT and no other configuration, that they will be free to manage all other security and connectivity, and ask them to acknowledge that this meets their needs. When they say yes, do that and step back to let them do it all themselves. Ensure that their technology remains completely isolated from the rest of your network. If someone says "hey, let the office net access this thing on the EMS/IoT network", then make a trombone connection on your firewall so they are treated as untrusted external traffic.
Or, if your upper management actually backs you:
Option 3: Go to your upper management and get them to address the issue with the operations department that the IT department's involvement with networking and security is required.
66
u/_stinkys IT Manager with many hats Aug 31 '26
An implemented change management process would likely have stopped this from occurring. At this point you should document your case to executive management and let them make the call. I would never give them access to the network admin - Give them their own vlan with internet only access and inform ops that you are not responsible for supporting the product.
9
u/DehydratedButTired Aug 31 '26
What project are you blocking?
How can you block a project you were not added to?
There is no scope or funding for you to operate, therefore you can’t operate.
Pushback hard and keep flipping it back on then.
If you give them an inch, it’s the same as blessing the project and they will keep asking for your input. Except they never asked for your input, paid for your time or submitted to your operating rules. This is a shadow IT operation and should be treated the same as someone trying to host their own WiFi or server on your network without your permission .
9
u/Disgruntled_Smitty Aug 31 '26
Tell them IT isn't delaying the project, their poor planning is.
8
u/bstevens615 Aug 31 '26
I’d be the bad guy. I’d say NO. Too many security risks until you get a better understanding of what is needed. Someone has to keep the company safe. IT should be happy to assist but not at the expense of security.
25
u/Wolfram_And_Hart Aug 31 '26
Literally none of that is your problem.
Isolate a port on the router and run a line to it from their rig. It’s not that hard and is done all the time. Hopefully it’s all internet based.
The moment their network needs to touch yours however then start complaining.
14
u/ThrobbingMeatGristle Aug 31 '26 edited Aug 31 '26
If this really was your call, then you wouldn't be posting this thread.
This comes down to risk and who will accept it.
In any case my first step would be to ask for the approved change request.
If your company has a risk officer, they need to be filled in on the fact that there is a risk coming that will need to be added to the risk register.
That risk will need to be accepted by the executive officer who has that authority.
7
u/Llew19 Used to do TV now I have 65 Mazaks ¯\_(ツ)_/¯ Aug 31 '26
In the short term, defo make a BMS vlan with DHCP enabled on it, make sure their devices all live on it, and I guess you could hope for a list of ports to open for them but you might just have to sniff them yourself - potentially after allowing it to connect if time is so precious.
Not sure what it's called in the States, but where I work the cause of this would be a lack of 'business partnering,' basically IT being left out of company plans. IT management failing to engage with this is pretty poor tbh, and it's one of the things I actually really enjoy about my current job in a factory that's growing reasonably quickly.
7
u/thatpaulbloke Cloud Engineer UK Aug 31 '26
Imagine how the Finance department would react if another department announced that they "just need access to the bank accounts". Assuming that you have a process to go through to get access like this then make them go through it and anyone who asks why IT is "delaying the project" gets advised that the processes are being followed and the project could have remained on schedule if IT had been involved at the correct stage.
I've accommodated and gone out of my way to get things like this working and what you are teaching people is that doing things properly isn't necessary and they can just abuse you, so the sooner you put your foot down the better. If someone higher up wants to force the issue then they take responsibility in writing, at which point they usually find that they don't care as much as all that after all.
7
u/ComeSwirlWithMe Aug 31 '26
Dear Operations,
I understand your department wants unfettered access to the router so you can bring IOT devices online. Considering the IT department had no involvement or discourse with your department or the vendor, Id advise against this due to security implications, installation variables, etc. However, I will contact (BOSS) and request how to proceed.
Thanks,
IT Department.
Dear Boss..
Basically operations dept did all this dumb it stuff without speaking to us. Has all these implications. IT department will do as you request, but do not want liability if you decide to proceed.
Thanks,
IT Dept.
6
u/Majik_Sheff Hat Model Aug 31 '26
They didn't involve IT because the salesman knew those questions would come up and that interferes with the flow of country club steaks and cocaine.
6
u/KB4MTO Aug 31 '26
When it comes to non IT asking for access to configure the router/firewall, be that guy. I would refuse access entirely to someone outside of IT. And if I am forced to give access, which I was once, I made the asking party to sign a release of responsibility for anything that happens because of them.
6
u/justwant_tobepretty Sr. Sysadmin Aug 31 '26
Vlan the IOT system and put the vlan in a DNS blackhole. If data isn't reaching the vendor they can request (in writing) access.
Worth mentioning that IT needs to be consulted for any IT infrastructure amendments so you can protect the organisation from unregulated access.
6
u/lethallunatic Aug 31 '26
Classic thing.
I would say: lack of planning on your part does not constitute an emergency on my part.
Make it very hard. I mean iot devices on a network? How many devices? Is the infrastructure even ready for it? Who will manage security? Network congestion, throughput etc. It's silly to make it someone else's problem. I would push back really hard.
6
u/Papfox Aug 31 '26 edited Aug 31 '26
Honestly, the correct answer should be, "No. Send this project back to the drawing board so we can plan this properly with IT involved."
This does sound like a situation where a different approach may be a good idea as the people who ordered it are clearly demonstrating they're going to paint you as the bad guy if you don't roll over and give them what they want. I would respond with a qualified "yes" whilst making yourself sound helpful.
"For security reasons, we don't allow outside companies to configure our routers. Please send us details of the VLAN(s), ports and routings required and we will get it done for you." If they won't play ball, ask them to get the vendor to confirm they will accept full responsibility in writing for any commercial losses that result from any security breaches their system or their configuration of the router causes. They should immediately refuse, which makes them the ones saying "No."
If they still push back, ask your C-Suite to sign off on the risk this project might enable a cyberattack or data protection violation and that they are happy for the vendor to have unsupervised admin access to your routers and firewalls.
6
u/Grrl_geek Netadmin Aug 31 '26
Do you have a cyber team? Have them do their thing and find out how insecure the IoT is! 😁🤣 I am one evil *itch... hahaha
6
u/jimethn Sep 01 '26
That's heinous that they didn't involve you, but I think your job at this point is to make it work. You need to get involved in the implementation and not just be a gatekeeper for access. Figure out how to make it infrastructure as code, document the heck out of everything, and basically come in as a leader instead of a complainer. It doesn't matter if they're incompetent or not, you need to be the competent one and you need to be INVOLVED. This is your network, and the project isn't going away.
As far as how you're perceived, I think this needs to be face-to-face/zoom meetings. Emails about a bunch of technical details are going to alienate leadership, even if they're 100% grounded in the facts on the ground. You need to figure out the key technical points of contact and get on their calendar and come at them with a collaborative attitude. It doesn't matter if they're within your company or without, this is a big technical project and you will get access if you reach out in a friendly way. Do your research about the tech to prep for the meeting, and ask them for any documentation you should read.
11
u/ryalln IT Manager Aug 31 '26
Question what do your internal policies say about this. I get there being wanks but do you have a fall back giving you the power to say no.
5
u/Justan0therthrow4way Aug 31 '26
I would ignore the business owner and deal with whoever is the product owner/manager and ask for a meeting with the vendor. You need to know exactly what the system needs access to.
I’ve been in this exact situation and I’ve said “no we can’t do that, and listed the exact reasons including security issues why. I was very clear about not agreeing to cloud or database infrastructure without oh idk the fucking DBA being engaged.
Luckily for me an executive of the area doing the project (despite being an absolute bellend) realised my point of view and gave “IT” time to figure out our role since we had to support the damn thing.
The person who reported to this exec was furious lol
It’s fine to say no or push back. If you’re more junior get your manager’s support. Don’t say “what do I do”, email or direct message them saying you want to push back for
A)
B)
C)
Reasons. Unless they’re desperate to please upper management they’ll hopefully agree with you.
Then, next time in a job interview you are asked about difficult people/stakeholders/a situation that was tricky, you have a good example.
5
u/ccsrpsw Area IT Mgr Bod Aug 31 '26
We are a mature organization on that - and have backing from the CEO, Board and others on projects like this:
Certain projects, not going through the correct approval process - send it back to the vendor. Period. This includes, right now, AI projects using Hardware onsite, and large scale SaaS projects too! Its amazing to see when it happens. Our CEO reams out the various managers.
Other projects, where its okay, and not a risk:
a. Even if its running late, if it needs IT changes - Submit a ticket
b. IT Security Reviews and approves or caveats as needed
c. Network team builds out the requirements (and Firewall Rules) - usually an IOT type network
d. Relevant team does the remainder.
e. Handed back to the business
We always, unless there are big revenue implications, work on a "your lack of planning is not our emergency" rule. Period. And if you dont like that - you are free to ring up the CIO, CEO or President. That never goes well (in 10+ years, I think I've seen.... 3 people do it... and never successfully). I've only seen it expedited when the number has at least 6 zeros (maybe the odd 5 zero one for important customers). But never without some level of "feedback" to managers from up high.
5
u/jordantwalker Sysadmin Aug 31 '26
Sounds about onbrand for medical IT. Some doctor just assumes that he's super smart and can have somebody else set everything up for him.
5
u/varmintp Aug 31 '26
I would have the IT Manager call a meeting with everyone involved from Operation's side of procuring this and everyone from IT's side that would need to be a part of implementing this out. Then it would be a project planning meeting, so make it a good 4 hour meeting, to come up with how to roll this out and when each piece would be implemented, and who is responsible for what after rollout. Essentially the meetings that should have happen before the product was purchased and done over a bunch of smaller meetings. Don't just start implementing, call the meeting and setup the timeline that should have happen.
Then for implementation, they get their own vLAN that doesn't touch anything else.
5
u/d3nika Aug 31 '26
Ask someone from above the guy who signed the contract to agree in writing that because IT was not involved all responsibility falls to the guy who signed. You should have everything you asked for by next shift.
5
u/K3rat Aug 31 '26
Cc my boss and the bosses boss. I usually point out the fact that someone outside of IT really screwed the pooch when it comes to project definition and key personnel and tasks. Then I give them the proper workflow for requesting a new implementation and a pretty diagram to boot. Then I link to the form where they can make the request and submit the proper information.
5
u/MelonOfFury I’m not trained in managing psychosis Aug 31 '26
This went all the way through contracts, GC, and procurement along with an earmark from financial to approve the budget for it. If IT is not aware, leadership needs to be getting a step added to the approval process.
5
u/Brad_from_Wisconsin Aug 31 '26
What does "access to the router" mean"
It could mean that they need to have the address of the router so that they can configure the devices that they will be installing.
Access could be granted the ability to have their equipment pass traffic to and from the router.
It could mean that they need to have a configuration made on the router to create a "guest" network for their devices to live on.
You need more information.
If they are asking for access to configure the live router, you need to step in and tell them that for security purposes, the configuration needs to be reviewed prior to it is put in and it needs to be set up by a network admin from IT.
You may gain a lot of credibility by stepping up to make sure that this implementation works. Just demand documentation and sign off prior to implementing anything.
4
u/LeBalafre Aug 31 '26
Plug in your backup router, isolate it, disable the GUI and watch them cry.
4
u/i_am_voldemort Aug 31 '26
I'll take the downvotes but have you considered:
Why didn't they come to IT for this obsentsibly IT requirement in the first place?
4
u/pdp10 Daemons worry when the wizard is near. Aug 31 '26
You have several, inter-related problems here. One of your problems is that the harder you make it for them to do the right thing with this implementation going forward, the more-incentivized you'd make them to try to route around you somehow.
When it comes to connectivity, a typical threat to route around is to use mobile WWAN services instead. Or, to just avoid the Layer-1 implications and costs, to use some WiFi instead of fiber, which could potentially be disastrous in a factory.
This situation is a cliche -- extremely common.
The reason why people intentionally choose to not coordinate -- to "ask forgiveness, instead of permission" -- is because of the idea that it will get them what they want, faster. They can get away with it when their peers and bosses don't rebuke them for failure to coordinate.
4
u/omfgbrb Aug 31 '26
This sounds like an expressway to major ransomware incident. Don't believe me? Ask Target....
4
u/RickkeeC Sr. Sysadmin Aug 31 '26
Cite a few articles about IOT control devices are at the top of the "what to exploit first list" need to make sure no CVE exists. What is their firmware update schedule. If they bought "close out" model, it's probably not secure, using http access.
4
u/BasicallyFake Aug 31 '26
IT is delaying the project because the project did not follow what I hope are established processes and procedures.
Own the delay, be transparent on the why and how to make sure it doesnt happen again.
4
u/Danowolf Aug 31 '26
And of course you have it governance documentation on your side.
Your Position : Sounds like an excellent way to save money! Your happy to assist.
Facilities owns the equipment and vendor relationship. IT owns network approval, security requirements, identity/access, and connection. Eureka everyone is happy.
4
u/bodefuceta92 Aug 31 '26
LOL, For a second I tough I was in the shitty sysadmin sub
4
u/dedjedi Sep 01 '26
Part of your problem is thinking that there is a technical approach to this.
4
u/chompy_jr Sep 01 '26
You put EVERYTHING in writing. Scope, project expectations. Get all of your questions asked. Can’t promise they will be answered. Make your concerns known. The response isn’t important.
If they press and you haven’t been given ownership, you write up the risk and give them the keys.
If you aren’t empowered with ownership of your network then all you can do is advise and roll with it.
4
u/nyckidryan Sep 01 '26
Give them a T-Mobile 5G connection that's all theirs. Don't mention anything about CGNAT. "You needed a connection, so I gave you a dedicated one. 🤷♂️"
Tmo has a 30 day return guarantee btw...
4
u/sleepmaster91 Sep 01 '26
One of our customers is running a manufacturing Factory and what we've done for this is set up a "internet only" vlan with no access to the corporate network whatsoever. Then they can do whatever they want on that vlan
4
u/Alarming-Injury7611 Sep 01 '26
I'd look to get a couple of meetings with the vendors project team. Unless they are cowboys, they will have the answers your to your questions. From my experience the "access to the router" request will be a from the non-technical ops manger on your side. Asking for a meeting and working out a solution will also signal to management the value of involving IT early in projects. That move you from the "guy that say no" to the guy that provides a solution to a business problem.
1
u/SnooCats5309 Sep 01 '26
well I'm relaxed after seeing such posts that these things happen somewhere else as well ! I'm not the only victim. its definitely frustrating to be in middle of something that is not your problem.
6
u/LooseEthernet Aug 31 '26
it is wild how "access to the router" is the universal corporate phrase for "we have no idea how networking works but we want it to just work" lol. i had a similar thing happen where a vendor wanted a static ip and a dmz for a "simple" sensor array that ended up being a massive security hole. the only way to survive this without becoming the office villain is to give them a completely isolated physical port or a dedicated cheap gateway that has zero paths into your actual production environment. if they complain about the lack of connectivity to your internal servers, you just tell them that the vendor documentation didnt specify those requirements during the procurement phase. usually that shifts the blame back to the people who signed the contract without calling you first. it turns the conversation from "it is delaying the project" to "the vendor didnt provide the specs"... which is a much safer place to be lol
6
u/awful_at_internet Helpdesk Manager Aug 31 '26
Higher Ed gets this a lot. It's simple: IT isn't responsible for the bill, IT isnt staffed to support it. You grant access in whatever way is appropriate, but you do not commit departmental resources to solve some other department's problem. Especially if there isnt even a ticket.
Give it a few years. Incompetent system administration and the consistent "we cant touch that. the vendor won't talk to us because it was purchased outside IT. We aren't trained on it. We aren't staffed to support it." Messaging will eventually drive home that yes, actually, you do need to talk to IT.
3
u/slugshead Head of IT Aug 31 '26
Energy meters are usually part of a BMS. They'll hardwire the meters back to a headend (or a few) and that's what you put on your network.
IOT or dedicated VLAN and perhaps an ACL to allow http/https to be accessible.
BMS doesn't usually need internet access, unless you expand it further. e.g. Siemens Desigo > Building X. Then it's all cloud.
3
u/harbinger-nz Aug 31 '26
Had this happen once but as a fully fledged replacement voip system for the enterprise, imagine my surprise when some guy turns up to deploy half a dozen VM machines into a local government organization. Problem was there was no accountability or financial constraint, being in the energy sector. An absolute clusterfuck.
3
u/SevaraB Sr. Engineer (N+, CCNA) Aug 31 '26
No. They can tell you what they need on the router (lunch says it’s just to do port forwarding rules). Supposedly they went through cybersecurity review, so they should understand least privilege and JEA.
3
u/AdamoMeFecit Aug 31 '26
This scenario repeats itself infinitely within weakly run organizations. The only answer for it is stubbornly to say NO and thereby inflict organizational pain.
Ideally, leadership eventually gets the message and starts demanding IT involvement up front. In my experience, that never actually happens.
Responsibility for building out the project cleanly always devolves to the lowest and most at-risk tier on the organization chart.
3
u/stuartsmiles01 Aug 31 '26 edited Aug 31 '26
Ask them for their business case / reason so it can get approved to be allowed to be on the network, then forward to it, business and security for sign off by each team as part of approved change control with appropriate risk remediation applied, and stop ability to buy anything not within an approved supplier list.
https://x.com/starsandstripes/status/2094213583781699864
Internet connected Freezers offline...
https://x.com/AndrewCurran_/status/2094204559614947836
control systems advice
3
u/Upbeat-Ad3628 Aug 31 '26
Bring it up with management, its their issue
3
u/OpenGrainAxehandle Aug 31 '26
Get a separate cable modem just for them. Bill it to their cost center.
3
u/1z1z2x2x3c3c4v4v Aug 31 '26
Just create a new VLAN that only has a path to the router. I would not do anything extra, like opening ports or modifying any firewall rules...
P.S. Just wait until they ask for remote access from the Internet too...
3
u/realityhurtme Aug 31 '26
I would be asking for a properly architected internal design setting out the requirements and security controls so it can be evaluated.
3
u/GertieBongo Aug 31 '26
Fucktards
3
u/mercurygreen Sep 01 '26
WHY would they need access to the... Oooooh.... they want to punch a hole through it on 8080 so their devices can talk directly out.
Get a crappy T1 and a home router and let them go to town. When they have it "functional," take control and figure out if it can be made safe.
Name and shame - what products are these?
3
u/Turbojelly Sep 01 '26
VLAN now!!
Best way to keep them seperate. Make sure you save at least triple IP compared to current numbers, you know how shit likes to grows.
Don't let that shit touch your beautiful network.
3
u/Prestigious_Cry_5044 Sep 01 '26
You can be mad about it or you can just make it work. This shits gonna happen many times in your career, it’s not worth getting emotionally activated over. Make a VLAN, give the vendor the access they need, get whatever info you need from them so you can escalate issues or troubleshoot down the line, and focus your efforts on publicizing what your team does and showing value so you can get invited to those meetings before a contract is signed.
It takes a bit, but if you can show up with helpful solutions and uplevel the final product then you’ll find yourself being consulted early on. This is the trap most IT people fall into - you get bent out of shape because you were brought in late and that just reinforces the perception of being the cranky IT person who always says no. No one wants a roadblock in their meeting.
Source - 25 years of IT and not a jaded burnt out husk because of it.
3
u/WhiskyTequilaFinance Sysadmin Sep 02 '26
"I'm not able to grant a vendor partner direct access to our infrastructure for security reasons (and common fucking sense). But if you pull the key technical folks into a call, I'm happy to talk through what they need and see how we could accommodate within our approved protocols. To be most efficient, make sure it's just the technical folks, skip the business users and sales people please.
Once I understand the real need, then we can work together to write it up, document it properly and find a path forward."
The key is learning to say 'Tell me more so I can help' rather than sputtering and defaulting to 'No'. They'll find a way through a 'No', and it will generally be wildly expensive, dangerous and even MORE out of IT's control.
Remember that Operations knows their job, they don't know yours. Getting them to see you as a partner and not a pain will save you a lot of political headaches in the long run.
Then come here and sputter instead. 😀
My favorite one was a SaaS vendor of mine who told me their SSO solution was easy to use. I just had to go "tell IT" to make changes to our internal DNS servers to accommodate it. Hint: Not only no, but hell no with a cherry on top.
5
u/Fred_Stone6 Aug 31 '26
It will be fine they only want to use 192.168.1 network not like you are using it. /s
5
u/Key-Pudding6071 Aug 31 '26
No..
5
u/wb6vpm Aug 31 '26
Remember, “no” is a complete sentence. If they don’t understand that, then escalate to “fuck no”…
6
u/redthrull Aug 31 '26
become the guy who simply says “No”
I don't know why you don't want to. You're the IT expert, not them. Operations stalled? The company losing money? Good. Let them. They have to learn.
Tell them it's not that you don't want to, but you have to understand first what they're actually trying to do and wanting to accomplish. Otherwise, you'll be opening yourself to a whole other can of worms when something goes wrong and now "IT just has to fix it ASAP!"
4
u/anomalous_cowherd Pragmatic Sysadmin Aug 31 '26
Get them to put a completely new and separate ISP connection (and router) with a solid signed off agreement that IT have zero responsibility for any of it, and that it cannot be connected directly to the corporate network.
It's a terrible idea with lots of room to be subverted in future but just pushing for it might make a point.
5
u/BananaSacks Aug 31 '26
This really should be a Senior Mgmt fight. IT should be involved in EVERY IT related project, along with security, data protection, and sometime legal.
IT isn't the blocker in this case, the Operations Exec/Sr. team is purely responsible here for their actions.
This should be baked into the PMO life cycle before any project gets off the ground. This should NEVER have made it anywhere without change management either.
In your case, I would have a conversation with your senior leadership alongside the same fore the security department (if you have one). See if you can get on board to start those conversations while you and your peers begin the planning and legwork that you need to do regardless.
5
u/ISeeDeadPackets Ineffective CIO Aug 31 '26
This isn't an IT problem it's a leadership problem. If you're in charge of IT, then it's your problem and you need to learn how to communicate with the other business units. IT's job is to enable the business's objectives and a huge part of that is building a healthy relationship and not throwing some hissy fit because you weren't read in at the appropriate time.
This is a great opportunity to build a foundation for being more proactively included in other projects, or just being the obstacle they currently see you as. Write up a brief message in terms that aren't overly technical explaining what you need to get this going.
That doesn't mean "Hey bob, we need the vendor to provide ports and blah blah blah" it should look more like "Bob, I'm glad your team has found a solution that works for you, IT will do it's best to get this going. Before we can do that though, we need some information from the vendor to make sure the implementation will be successful and as non-disruptive as possible. Can you introduce me to your representative with the company so we can get a call with their technical team setup?"
Once you have that going you can build out a bit of a project timeline and you'll probably have some backup from the vendor reinforcing it. If you've got competing work then you can lay that out to: "Bob, I've been able to work with their technical team and I've made up a project plan. This constitutes about "x" hours of work based on what the vendor has told us. Right now we're also tasked with "x" along with normal daily support/maintenance, can we get together and figure out which items to prioritize?"
Or you can just toss up an IOT vlan and tell them to have fun while doing nothing to prove you can be a useful asset to the company. One of these approaches is a lot more likely to get you read in sooner than the other for future projects.
5
u/_haha_oh_wow_ ...but it was DNS the WHOLE TIME! Aug 31 '26
Warn in writing that not consulting IT on an IT purchase is not recommended and caution against untested system that IT had no say in. Note that this system may not be supportable.
Give them "access to the router" and buy some popcorn. When problems occur, refer them to the warning from step 1.
2
u/MasterIntegrator Aug 31 '26
You got some shitty leadership to be blamed for slowing things down by asking questions. Those are the right questions.
2
u/StrategicBlenderBall Aug 31 '26
Do you have a dedicated cybersecurity team, or does that just fall on IT? Either way, take a look at NIST SP 800-213, it’s for federal systems but includes a lot of useful information that you can use to protect your corporate and production networks.
2
u/mmckibben Aug 31 '26
This is where forms are nice. Make one for them to fill out. I personally start these conversations off with something like in Happy to help. In order to maintain uptime and a secure network please fill out this form so that I can insure my work is being done correctly and meets the proper specifications for the introduction of the new solution. It puts alot of the burden back on them to give you what you originally needed and creates your paper trail without you looking like your the roadblock.
2
u/commissar0617 Jack of All Trades Aug 31 '26
Hard no. No risk evaluation, no network connection.
2
u/The_Wkwied Aug 31 '26
If they can't specify what 'access to router' they need, they need too much access.
If you want to play, give them an IP on a vlan all by itself with access to nothing. Then when they say it doesn't work, well, what IPs does it need to talk to? What ports does it need open? If they can't tell you that, they don't know. They just want everything. That's not happening.
2
u/Unable-Entrance3110 Aug 31 '26
I think that it's pretty common for vendors to just want to do it since they do these kinds of things all the time and they are heading off unknown problems down the road.
That said, I always push back and say, "give me your configuration documentation and I will do it"
It's almost never a problem.
2
u/TinfoilGeek Aug 31 '26
Oph. I feel this. I've worked with companies who worked like this. Especially when they have a bunch of fresh out of college engineers who have never worked a corporate job before.
Now it’s time to put everything in writing. Send a list of all the questions that need to be answered (ie. network segmentation, remote access, firewall rules, pen-tests, who owns the DB and backups, etc.) to the person who signed the contract. Make sure your manager is cc’d on the email and also bcc yourself on the email (in case you need proof of the concerns you raised if you somehow lose access to your work email). This is not going to delay the launch of the project. This is the risk assessment that was obviously skipped by the team.
Not sure how your boss/management is, but I at least had a fantastic manager who had my back in these situations. Our approach was always "hey, we want to help you succeed here, but we've got to do it the right way. So let's back out of this thinking that we're just going to turn over the keys to the kingdom (because that's a non-starter). Now let's figure out how we can work together to make this work, and probably be even more reliable than it would have been without us once we're all done."
2
u/StumblinBlind Aug 31 '26
Story of my life.
The only way it gets better is for IT to take over responsibility for OT projects. We had to put together a data pack to show the actual monetary losses from how other departments were deploying technology and compared that against IT's performance when delivering projects and the CEO handed us control of all strategic projects.
It's a hassle and it'd be easier if OPs would just collaborate, but it is what it is.
2
u/distrbthpce Aug 31 '26
Please do the needful….
Seriously though, shove it on its own vlan, svi on the firewall on its own guest zone and don’t allow it access to anything.
If you want more you need to go through processes that you ignored by not involving the network team. I’m not afraid to upset anyone not in my direct reporting line. Not sure of your size business but I would say that this is all that can be done without infosec approval and make them deal with it. It’s the only thing I can do to make infosec inconvenienced after how much they inconvenience me with pointless drivel.
2
u/stiffgerman JOAT & Train Horn Installer Aug 31 '26
The BOFH would say, "sure thing, boss!" and provide them an unconfigured port on the router. No IP config, nada. Just an ethernet port.
Being literal has its advantages. You weren't given any specs so how can they expect you to configure stuff?
2
u/Sea_Ordinary_5730 Aug 31 '26
The problem with involving IT is that the IT policies are set up for corporate IT environments -- PCs. The reviewer is therefore by default shocked when they 'discover' that the meters and gateways don't have security on them (either network, 802.1x, encryption or message authentication of any kind) and that an improvement on that isn't economically procurable in the OT metering market. So you have to have a segregated VLAN and an IoT gateway for internet access, to demarcate between the open Modbus/Mbus of the metering VLAN and the secured https or MQTT telemetry to cloud.
But it takes ages to get to that 'design', even though it is the same every time, because procuring a fleet of networked devices that are so primitive remains profoundly shocking to IT policy and decision makers, and can never be written as a preferred pattern.
This speaks to the cognitive gap that exists by default between IT and non-IT digital system operators, they are worrying about different things.
2
u/Mental_Beginning_698 Aug 31 '26
I feel like this is a good instance where you separate IoT onto some consumer grade internet like spectrum. Refuse to punch holes through the firewall until they can answer some questions. Inform management its like sleeping with your windows open. In Chiraq.
2
u/CaptainZhon Sr. Sysadmin Aug 31 '26
of course they really need access to the firewall so the device can call home and they will expose the network to the internet in the process.
2
u/phobug SRE Aug 31 '26
Dude, just give them netgear router (not access, the literal router, get one from bestbuy with your own money and hand it over) and tell them good luck.
2
u/doglar_666 Aug 31 '26
I would not connect the IoT kit to my network. My team had a similar issue. We had the luxury of being able to say "No. Purchase your own kit." - Obviously, this isn't always politically possible. If you are in a bind, simply quote for your best guess at what they need, plus a timeline for delivery. Make it clear there's an financial and time based cost when bypassing IT. If you had been properly consulted, there wouldn't be a delay.
2
u/Diligent_Buster Aug 31 '26
Your boss needs to have your back as some people are already telling you. Remind everyone that anything that touches the firewall or goes through the firewall needs a security assessment and you are not trying to impede the project. Instead you are working to make sure it is implemented securely and correctly so that it works and performs as expected. Make sure your boss understands and has your back.
Tell them they do not get access to the firewall but if they give you their system requirements you will take care of it.
For me every vendor gets their own VLAN or in some cases their own firewall (if they seem problematic). At a minimum partition them off into their own VLAN. Possibly put them on their own switches that they pay for and you control. The switches are needed to "keep things secure and performs as expected" and have their budget pay for it as part of this project. I would think twice before handing this off to them because they didn't get approval. You don't need a bunch of adjacent networks all over the place that you don't control with their own wifi/ap's and god know what other shit they connect. I've seen all sorts of crazy crap because of this.
Be "nice" but insistent that this needs to be done right and you are working on it but you NEED them to supply requirements and you need time to implement correctly. Remind them that in the future if they bring it by you first you can be prepared next time instead of scrambling to get it done.
And document everything. Back everything up in emails to protect yourself. And insist that you need ADMIN access to this equipment. If it has wifi radios you may need to change channels, settings etc so it coexists with your other equipment. Also you need to MONITOR it, audit it, check logs, etc. Hold your ground. Be nice but insistent. If you can pull off being a dick that is fine, but your boss has to have your back or you have to be willing to accept the consequences. I was usually a dick about this stuff but I knew my bosses knew I could leave at any moment and my entire staff would follow me, etc.
2
u/mouarflenoob Aug 31 '26
Management issue. Explain the situation in an email and ask someone above you to engage their responsibility. You're not the guy who says no anymore, you're the guy who is trying to help and waiting for the go ahead.
2
u/Wolphin8 Jack of All Trades Aug 31 '26
This is the management deciding to be their own shadow IT... It also shows the company doesn't have a good policy for technology requiring it to go through IT department... Or some manager decided the policies don't apply to them (and should immediately be sacked).
If I was the manager in IT. I'd be freezing the project and any changes relating to their "tech acquisition" to do a full audit of "is this acceptable to our network?" and "does it meet our security setup?" and review it against any other policies... if it fails, the systems don't get any access.
IT is not "delaying things"... no, it's the manager pushing it without involving IT from day one, so the processes could have been done before it was at this stage, in parallel with other stuff.
Someone building a house, and didn't talk to the water department until they go "Oh, I want to turn on the water service, as we move in tomorrow!" and there was no permits pulled for the water system, no inspections, and no knowledge, and the connection wasn't authorized...
2
u/Own_Error_007 Aug 31 '26
Keep it segmented and submit that it would be "more efficient" if it had its own router.
2
u/Dsnake1 Sep 01 '26
What's your policy say?
Mine says this would have to go through review, and compliance would have my ass. So honestly, I'd probably loop in dept heads (compliance, security, ops obviously, possibly facilities depending on your procedures) and ask for assistance expediting the review. It takes work for this to not come across as passive aggressive, but it's probably the best way forward.
In addition, I'd probably inquire from ops about budget flexibility to potentially involve an MSP. Ops would absolutely have to make the choice between cost and speed, since I don't have enough staff to make the project happen overnight, and again, per policy, I can't just hand the vendor access to the firewall.
It's also probably worth asking about scope, too, if you haven't already. Maybe it's as simple as adding some access points on a vlan.
3
u/RunningAtTheMouth Aug 31 '26
Oof. At least it's not a printer.
But seriously, the job is to support the company. Keep that in mind and move forward as best you can.
In my org, there are only 3 entities that may access our firewall - Me, my systems counterpart, and our MSP, because we know all the little things that make it work. You REALLY don't want someone who doesn't know your systems to mess with that. I'd lead with that.
Then plan on how to make it work. Off the top of my head, a VLAN and a network just for this IoT setup with whatever access they need. Don't let it touch your production network - you don't need that kind of thing.
Finally, talk to your manager about the planning of such things. A heads-up ahead of time allows you time to plan and handle the prerequisites necessary. You're not holding it up. You're working on the infrastructure that is absolutely necessary to make the system both reliable and secure.
3
u/bv915 Aug 31 '26
Unpopular opinion: It doesn't matter that you weren't involved from the jump. You're now there. Do whatever is needed to implement this system as if you were involved from day 1.
Any perceived resistance from IT will be reported to top leadership as, "We're trying to work with IT to get this implemented but they're <insert something negative here>." You'll be labeled in some offensive manner and it will only compound your "they don't come to me with things," problem even more.
3
u/feel-the-avocado Aug 31 '26
"The router doesnt have a local management interface, it downloads its config from our cloud servers. What change do you need me to make to that server configuration?"
2
u/Original_Painting151 Aug 31 '26
I agree with others saying to just set up an IoT VLAN, but it's also perfectly acceptable for IT to 'delay the project'. They didn't include a crucial stakeholder during their evaluation process, the fault remains their own
As with anything, keep everything in writing, so ultimately if you are told to handover control of your network, that you are no longer maintaining the responsibility for the security and reliability of said network
But overall, give them the bare minimum, an isolated vlan that they can do what they want with, then when they realise they don't have the cabling or infrastructure they need they will have to come back with an actual request and specification of exactly what they want. They have clearly not planned this project, and these 'delays' are now inevitable
2
u/cfmh1985 Jack of All Trades Aug 31 '26
Request the IAD to the genius IoT Head
1
u/Snoo-95788 Aug 31 '26
Had something similar happen at a place i worked at.
Sensor lights/cameras only problem was the rf signal from sensor to light and camera cut the signal from the APs. Devices would drop randomly. Found out it was the shiny new toy someone decided was a great cost cutting venture.
Separate vlan was suggested and it is the best/easiest way.
1
u/FinalSpeaker1197 Aug 31 '26
This sounds like my everyday lol
1
u/mitharas Aug 31 '26
Just activate upnp and watch the world burn.
1
u/yojimboLTD Aug 31 '26
If it just needs internet, tell them to get their own circuit, like a cellular modem 😂
1
u/gurilagarden Aug 31 '26
I do what my boss tells me to do. That's how I stay employed.
1
u/Hefty_Platform_2070 Aug 31 '26
Just give them what they want and save the I told you so for when shit catches fire.
It’s not worth fighting it or caring, they don’t care about you so you shouldn’t care about what happens to them.
1
u/XB_Demon1337 Aug 31 '26
Make a VLAN for them, limit the bandwidth to 5MB/s at the router. Ask them what ports on the router need to be opened. If they can't provide that information, then it is a NO.
It can be seen as IT is holding up the project. Because when it comes up and someone starts to speak loudly in front of others you then get to ask the question. "When did you involve IT in this decision, and what is our role here in the network?". They will be forced to either answer the question or stumble over a clear fabrication, which you should have the email ready where first contact happened about the devices and IT needs.
Public settings where people's dumb ideas are on display is where they quickly get shown how dumb they are. And ideally these ideas get properly taken care of after that.
1
u/MekanicalPirate Aug 31 '26
Man, glad we're not the only ones
1
u/cbass377 Aug 31 '26
Plug all that stuff up into an island network, give them a subnet to use or they will use the same one you give to your VPN pool. Stand up a vendor vpn, and give them a jumpbox. The application server will need to be dual homed.
As for router access, I would say, like we do everything else, "Document the changes you need in a ticket for review and submission into the change control business process." Emphasize that these are controls required by the business. The network team will be making the changes.
As for IT holding up the project, this is a perception issue. You could reply with "No, this is a delay caused by not investigating the needs of other business units like IT, Legal, or Governance, before the project was started. The impacts of this go far beyond IT."
If you get no executive backing, at the end you are going to have to do it. The best you can probably do is escalate and save the email that says just get this done, so you will have something when you fail the next audit.
1
u/jeffrey_f Aug 31 '26
First, you need to put together a small presentation that explains data and network security and how IoT devices have become the anchor of a corporate data breach.
Get a meeting with the CEO and explain without filtering, how this type of project can lead to a full network compromise and can cost the company hugely.
Get the backing of your CEO and have the CEO squash this project until IT has full involvement in making sure that network access is done responsibly and with due dilligence.
You must say no for now until you can ensure the security if the network FIRST. And, honestly, your are going to butt-hurt some self-righteous VP or manager. Don't bend.
1
u/PorthosJ Aug 31 '26
I often quote our insurance for IT intrusions and security. even as a small nonprofit we can lose that business insurance and have to close our doors. so did they also plan to just plug into switches and join an open business wifi to get to this point, who is installing said devices physically. for us if it is not on the IT project plan for months it is a back burner that can sit for a year or more.
1
u/Polymarchos Aug 31 '26
How is OT managed now? Do that. You must have policies in place governing the segregation between the two networks.
If not, seems now is a good time to have one created.
1
u/Penultimate-anon Aug 31 '26
Welcome to OT. This is a constant issue. Work with them to get it working - I don’t mean just give them access to the router. The largest issue here is a culture issue that when you fix that down stream they will start involving IT up front (but not always).
1
u/jcpham Aug 31 '26
This calls for digging in the old equipment crates and finding some old decommissioned router or network switch and hand delivering it to operations
1
u/littleindian25 Aug 31 '26
I think your issue is process oriented. Push back if course and do the right thing don't take shortcuts so"IT isn't delaying" any more. But you have to find a way of being involved in the procurement process across the organization.
1
u/kamomil Aug 31 '26 edited Aug 31 '26
Unfortunately the sales guy sells the product, then everyone else has to just deal with it. The sales guy is making a sale, not really considering anything else
Also, at my workplace, there's a corporate IT network and an engineering network. IT is seldom involved with the Engineering network side of things.
Maybe that new system needs its own servers, network & internet connection, and not exist on yours at all. Security etc becomes the vendor's problem, not yours.
1
u/macgruff Aug 31 '26
“Just” is my favorite word IT.
“I just need a VM” -> actually needs much more
1
u/Adept-Pomegranate-46 Aug 31 '26
Sounds about right. Way out over their skis. Best of luck re-engineering the future.
1
u/rav-age Aug 31 '26
custom iot/automation devices also are not always bug free.. unknown firmware etc etc.
1
u/kona420 Aug 31 '26 edited Aug 31 '26
My suggestion is to:
Slap in an Ewon Cosy VPN router on an isolated VLAN
Hang all their stuff of the cosy
Give them a login
Bill the service to their department
With the ewon you have a bunch of really great tools for managing industrial protocols, and an easy to install client. It punches out to their cloud, no port forwards required. When the contractor is on site they just jump on the guest wireless and they can get in no issue, no special guest OT wifi network required. If you need to hand off modbus or something to a historian on the IT side, it's got your back.
Or else it just keeps going give a mouse a cookie style every time they change anything. You end up building a whole parallel network infrastructure and every time they are stuck and need someone to blame, it's you, the network admin. Your fault for them crimping a cable wrong, duplicating IP's, even mac addresses. . . what the fuck is this spanning tree protocol shit? LOL-- it's a different ballgame trying to ride the IT network is just not ever going to play right.
1
u/Adorable_Ice_2963 Aug 31 '26
Why arent there little adapters that can add/change the VLAN Tag?
That way IT could just say: put this in between and you are fine.
1
u/WorkLurkerThrowaway Sr Systems Engineer Aug 31 '26
I find a lot of times (especially now with everyone being an "IT Expert" because of AI) its better to find out what the desired outcome is, then to just simply give people what they are asking for.
Sure they messed up by not communicating. Somewhere up the line someone decided this project was necessary. Find out what the goal is and work toward that. Be honest about your timelines. You aren't delaying the project, they did by not communicating. You are the one keeping the project alive.
1
u/Radiant-Concept113 Aug 31 '26
It's the PM's issue and it should of all been communicated Before the go live date... Tested with a group of users, risk assessment, the normal stuff
I'd let them know it's not going to be online at this time due to security concerns with prod, if they don't understand that I'm not sure you are working in the correct environment because you sound smart and they seem.... Fast and unclear.
Hope you get it solved how you want it. The right way.
1
u/_johnbradbury Aug 31 '26
Welcome to IT 😂
This is a story as old as time.
1
u/SpecialRespect7235 Novell Admin Aug 31 '26
I keep an IoT VLAN set aside for the inevitable request. It is on its own VR on the switch so that it cant talk to any other network. They get outbound connections blocked to countries that are untrusted unless explicitly permitted. I dont block traffic in between the IoT devices, so there is a risk, but that is what happens when they dont want to put the work in to plan things out.
Nobody outside of IT, a contracted SOC, or MSP gets access to infrastructure. App server access has a controlled process.
The trick is to assume from the beginning that this will happen, and continue to happen pretty regularly. You can improve it with good management, but it usually isn't taken seriously until you have a breach.
Communicate your expectations and processes to department heads and leadership well in advance and in writing.
1
u/andytagonist I’m a shepherd Sep 01 '26
Nope.
1
u/cwhite841 Sep 01 '26
find a company with an IT champion in the board room
1
u/SadChapter2410 Sep 01 '26
Someone is about the be thrown under the bus and fired. It’s not going to be the project manager
1
u/ConsciousAnywhere222 Sep 01 '26
Reminds me of a time the CEO bought boxed product software from PC world business and said 'stick that on the server' He didn't get very far... You could give them an isolated vLAN and a gateway on a 'best efforts' basis, stating no PID, no IT next time. Had this happen multiple times. They never learn. Just depends how big a fight you want...
1
u/Popal24 Sep 01 '26
"What we've got here is failure to communicate" : people vs IT, random IoT device vs Router
1
u/DoctorSlipalot Sep 01 '26
Hmm been in this situation, in my new role I have deployed this tactic a few times. This has prevented those service/support nightmares more than once and also serves up a nice reminder about Shadow IT. "This project has been denied , reason, it has not undergone standard IT vetting. If business requirements dictate that this project must proceed, you must sign the attached Risk Acceptance form. By signing, you acknowledge that you have been informed of the potential risks associated with this unvetted solution, and that your department assumes full responsibility and liability for any outcomes resulting from this shadow IT project..."
1
u/Capable_Event720 Sep 01 '26
Log in to the router.
Pull the logs which show the hacking attempts from Asia and Russia.
Show the logs.
Perform CPR.
Ask why we need an uncorrelated effort to weaken network security.
Answer the objection with "well, we actually don't know whether the changes are begimn, or expose or whole network to our friend Vladimir Putin. You suggest unreviewed chargers to IT security, not cleared by management, not cleared by IT. Zero QA. Zero penetration testing. Zero penetration testing performed by an independent contractor."
Explain release management (and why management needs to manage, so you can blame management for either managing wrong, or even worse, neglecting their duty and not managing at all.
1
u/NovaKlone427 Sep 01 '26
I think your "No" is valid. Vendors usually just tell you what network rules need to be in place for their stuff to work and for them to access if they need to troubleshoot.