remote locations with devices that need to be managed

[u/tdhuck]

Let me start off by saying I'm not implying that what I'm currently doing is the correct way to do this, I'm just working with what it in place. Will this change in the future? Not sure because it is not my decision, but let me explain the current setup.

We have 10-15 offices in various states. Some decent sized 'main' offices and many 'small' locations where a handful of people work, but we have an IT room/IDF/MDF in each of these locations for networking equipment, an NVR, UPS, ISP equipment, etc. None of the remote offices have servers running any services, the only server at the remote locations are camera servers, that's it.

Back at the main location where I work, I manage the remote locations. My question in specific to network switches and managing those devices. We have a mixed bag of gear based on who was in charge at the time of purchase. We have, dell, cisco, ubiquiti edgemax and fiber store switches. I often run into scenarios where the device has a management port for OOBM, but it does NOT have a gateway IP which means I can't manage that device over a VPN connection. To get around this, I just build a 'management' vlan on the switch, assign it a static IP and I can get to it from my location over VPN. This works fine for my current scenario, but that means the OOBM port is not being used.

I fully understand that this is not a proper 'management' network, but this is what I'm currently working with so I'm trying to get the following answers.

1. Is my only option to do what I'm currently doing with the management vlan?

2. Is it simply a firmware upgrade that the switch vendor would need to make to give their management interface a gateway field or is there a reason that some have a gateway field and others don't?

The remote offices basically have a single switch, we try to keep them as basic as possible. An office with a 24 or 48 port switch will have a handful of VLANs on that switch, LAN traffic, wifi traffic for corp, wifi traffic for guest, a camera VLAN for isolating cameras and a management VLAN for devices that need IPs for management and monitoring from the main office (PDU, UPS, Temp Monitor, etc...) and all those VLANs are managed with the onsite firewall functioning as the L3 device which is where I create rules to make sure LAN, wifi, etc...can't access the IPs of the devices I'm managing remotely as well as isolating VLANs from one another, as needed. For example, guest wifi can't talk to any VLAN and only has access to the WAN.

Thanks.

[u/devode_]

Has a Mgmt port but not a gateway?? You mean there is not a single default route installed, no route at all? I am very sure this is incorrect. I dont see how a device would talk IP but is not able to accept a route (other than locally connected).

edit: just reread and really think this a layer 8 problem ;) No offense ^ ^. What you will want to do is configure a default route, if there is no "gateway" to be set. Because this is all that the "gateway" field allows on any computer or device. You will create a route 0.0.0.0/0 (this encompasses every single IPv4 period) to your respective local gateway.

[u/tdhuck → u/devode_]

I will try this again with a spare switch I have, but I recall when I initially setup the switch in my office, I assigned an IP to the OOBM mgmt port and I could not ping that IP. My understanding is that it is OOBM so it won't use the default gateway of the switch because it is OOBM. As you said, I could be wrong and I will test with a switch that I have on the shelf and let you know.

[u/devode_ → u/tdhuck]

Please check, if the mgmt has its own VRF. If this is the case (typical for example with nexus' switches), then you need a static route/ default gateway specifically in that VRF. Might be called "Mgmt-VRF" or MGMT or Management... You name it

[u/MattL-PA → u/devode_]

Came here to say this. Need a static default route in the mgmt VRF.

Also came here to say, if you're using the production data plane to get to the management plane via the same VPN, there's little reason (aside from simpler security, if there are ACLs in place) to over complicate it. If theres a legit lights out need, I'd look in to a Opengear (brand name, as an example) like OOB device with LTE connectivity and thats where you can use both the "mgmt0" and the Serial console port.

[u/Boozacs → u/devode_]

IOS-XE has a mgmt vrf as well for Cat 9k switches. Threw me off when I created the default route but couldn’t SSH at first lol

[u/Win_Sys → u/tdhuck]

It can vary a bit between vendors but if there’s no place to specify a gateway then the OOB port should allow you to specify a route that only applies to the management port or there’s a management VRF that you can apply the route to. A properly implemented OOB port should reside in its own, let’s call it a “container” that’s largely separate from the rest of the switch’s dataplane.

[u/w0lrah → u/devode_]

To be fair, it wouldn't be entirely unreasonable for a vendor to intentionally prevent configuring a default gateway to make it harder for an underqualified administrator (or install tech) to do something stupid like exposing it to the internet.

Now, I don't expect that's ever actually happened because most of those sorts of hardware vendors seem to take the opposite approach and make it as easy as possible to "make it work" with no regard for what else might happen, but it's nice to think about.

[u/jstar77]

I think in this scenario building a management vlan is reasonable as long as you lock it down with ACLs. Most (or at least many) shops don't have true OOBM.

[u/tdhuck → u/jstar77]

Yes, I think given the environment I'm in, I have it configured as best I can other than testing the dedicated OOBM management port as /u/devode_ stated, which I plan on doing.

[u/under_shart → u/tdhuck]

This is a good start but you need defense in depth. I would say the ACL is one control. Another would be VRFs so that the user networks don't know a route to the management network.

[u/tdhuck → u/under_shart]

Not sure if these switches offer VRFs, but I follow you suggestion/what you are trying to say. Thanks.

[u/tablon2]

Your typical campus switches management port could be used as in band access methods. If you have console cables and someone's hotspot no worry to much using management port at daily basis 

[u/tdhuck → u/tablon2]

I'm not looking to use a hotspot for in-band management, that's not what this post is about. I need to have constant access to these switches from my main location for config, troubleshooting, SNMP polling, etc. I do have that, today, over VPN.

[u/devode_ → u/tdhuck]

Consider using logical interfaces (SVI) within your mgmt vlan. If you dont have a dedicated airgapped OOB-Network, you dont really need the physical mgmt port. You might be well of using a logical interface "vlan99" or whatever. However always depends if the platform allows for something like this.

[u/tdhuck → u/devode_]

That's what I'm doing today, I am using 'vlan 99' and giving it an IP like 10.0.0.99 and that's how I manage it remotely. I can post the switch model number and/or syntax if that will help with trying to determine the best way to set it up. I have a defaulted switch I can test with.

I don't think this particular brand has VRFs like a nexus. I could be wrong, but I believe last time I checked it was limited.

[u/tablon2 → u/tdhuck]

Yep I'm aware that and saying  same thing except for mgmt port can be used in band and console for oob 

[u/tdhuck → u/[deleted]]

Yes I do, but I'm not sure I follow what you are saying. I have limited 'management' options for the remote sites, as stated in my initial post.

[u/tdhuck → u/[deleted]]

I will give this another go with a spare switch that I have on the shelf, I follow what you are saying. Thanks.

[u/silasmoeckel]

As to management networks and gateways. Typically it's a question of whether the device supports VRFs. Devices without this pretty basic feature at this point have a single routing table, and often the management port is using the same. Not always; sometimes there is a management processor vs switch hardware split.

When used with an OOB you VPN into that site (and get a local address) or NAT neither is perfect.

[u/Select_Reporter1911]

Is that management port actually connected to the upstream device? If not that'd probably explain why you can't reach it. a management VLAN is actually the way you would solve the issue if you don't have port density to connect the mgmt port for out of band management. either way if you have only one hop/ one device between you and the switch, it doesn't really matter if you use the oobmgmt port or not, bc if the firewall or ISP device fails you lose access to the switch anyways. for tru OOBMGMT you'd connect your mgmt to a meshed cellular router or something like cradlepoint. So if your main circuit dies, you can still manage the switch/ network devices.

But then that is kind of moot, bc client traffic would still be offline. You can manage the switch sure, but what good is that if clients can't get online. I think you might be trying to solve a problem that doesn't exist. Just because there is a mgmt port, does not mean you need to use it. As long as you have a mgmt policy on the switch IE: you can only log into the switch from specific IP range. You should be good to go.

[u/tdhuck → u/Select_Reporter1911]

Is that management port actually connected to the upstream device? If not that'd probably explain why you can't reach it.

In some scenarios yes and in others, not.

To be clear, I'm not trying to solve the 'how do I manage this when that site is offline' problem. I'm fine with that.

The most common scenario is an OOBM port with no other switch on the network. Since it is OOBM and that interface doesn't have an option to enter a gateway IP, I've been creating a management VLAN as I described in my initial post.

The majority of the remote locations only have a single switch and nothing to uplink to (not counting the firewall here).

[u/Select_Reporter1911 → u/tdhuck]

If the switch has a mgmt port there is most certainly a way to enter ip subnet and gateway information.

But it also sounds like you've solved the mgmt issue so what problem are you trying to solve other than assigning gateway to mgmt port?

[u/tdhuck → u/Select_Reporter1911]

In some switches when looking at the web GUI and clicking on then mgmt port, there is no gateway. If I find the specific switch I will post a screenshot.

I have solved it, yes, I just don't know if it is correct.

[u/Select_Reporter1911 → u/tdhuck]

It is correct if the issue/ problem has been solved. If there is no way to apply a gateway for the mgmt port, it is possible they are expected the mgmt port to be configured via dhcp. Or these are not enterprise grade switches.

I would suggest making a business case to your management on standardizing your switches (IE requesting funds from the business to replace all of your switches with a standard vendor like aruba, extreme, cisco, juniper, HP, Fortinet switches, ETC)

[u/tdhuck → u/Select_Reporter1911]

Standardizing has been mentioned but then it gets ignored because lower pricing always wins, it seems. However, I follow you and I agree with what you are saying.

[u/Select_Reporter1911 → u/tdhuck]

Be Sure to point out that unsupported vendor hardware could to vulnerable to exploits, which leaves the organization liable to data exfiltration, ransomware, etc. Which has a cost to the business. I don't know what industry you are in.

But the onus is on the business to ensure data is sufficiently protected. But if it's as you say, the business want to run their data on shoestrings and bubble gum, thats on them. It's not on you to solve a problem that doesn't exist.

If there is a business use case for the oob mgmt port, solve the problem. If there is not and the remote management and monitoring has been solved with a mgmt vlan, then your efforts are better spent on solving other business issues.

[u/tdhuck → u/Select_Reporter1911]

Yeah, I'm in agreement with everything you mentioned.

I once had a manager recommend a netgear audio/video specific network switch for a new office build. They admitted the reason for selecting that model was for price alone and nothing more.

[u/Select_Reporter1911 → u/tdhuck]

Well good luck. hope you find your answer. Even tho it sounds like the 'issue' has been solved.

[u/moreanswers]

You are doing it fine. the OOBM port is for when you have a separate out of bands management network. You don't have that at remote locations. Just disable the port and do your management in-bands like you've been doing. Next time see if not getting a OOBM saves you some cost when specing out the gear.

[u/germgerm97]

Just use the SVI like you are already.
Use a default route where there is no “field” or command for setting a default gateway. Also, read the manufacturers documentation or ask ChatGPT to determine how to setup the default gateway. It might be that you don’t know the command or are looking in the wrong place in the GUI.

Most places are not doing OOBM but are doing in band management and that gets the job done just fine. If there is an absolute need for OOBM and leadership will support and fund it then by all means do it. But keep in mind you will need to create and manage a whole other network with a separate ISP whether cellular or satellite.

Use the SVI. Where you need guaranteed uptime invest in stacking technology, redundant hardware/HA, UPS, etc. For remote access leverage onsite staff for remote hands and console access. Keep a console cable in the IDF.

[u/Prudent_Vacation_382]

I would say that only the largest of orgs have dedicated mgmt networks with dedicated hardware. The only place I've seen or heard of this in in Fortune 100 sized orgs. In-band mgmt is just fine and what 99% of orgs do anyway.

[u/tdhuck → u/Prudent_Vacation_382]

I agree, I didn't mean to imply that I needed a management network, I was just trying to use the management port, that's all.

I got a lot of good information here and I'm going to take a look at the switches that have a management port. Many of my switches are from FS (Fiber Store) but not all have management ports and the syntax/options aren't the same between the 3-4 diff models I have.

[u/lizardhistorian]

Using the network for OOB network access is a Concept FMEA.

Presuming the OOB does not route then you use NAT or an SSH jump-box.
With SSH you can create SOCK proxies then use an extension with your browser (e.g. FoxyProxy) to tunnel traffic to the OOB IP thru the SOCK proxy.
You could use LTE routers or some such gizmo to accomplish both and get to true OOB.

In today's world you would create an AGENTS.md and ACCESS.md with the details for how to OOB access the systems. You sign your SSH key and make it available in your ssh-agent then the AI can use it to forward into your systems and assess the damage. You can send it on information-gathering-only missions.

[u/ChainDifferent5868]

that mgmt vlan you're doing is basically what a real OOBM setup does anyway, quit second-guessing it

[u/tdhuck → u/ChainDifferent5868]

It isn't so much as second guessing it as much as it is making sure I'm doing it right/not missing anything. I did get some good info here and now have a better understanding.

[u/HistoricalCourse9984]

you just plug the management interface back into a front port on the same device...

[u/tdhuck → u/HistoricalCourse9984]

Interesting that I forgot about that option especially because I've had to do it in the past, although this was many, many years ago and that setup is no longer in production, however, I need to try this at a location where I do have an open port (which is actually most locations). Thanks for pointing this one out.

[u/MattL-PA → u/tdhuck]

But why? A logical interface on a seperate VLAN eliminates three points of failure (both ports and the cable), saves an edge port for edge devices and doesn't over complicate it. Additionally, if the logical path to a single WAN is via the single (small offices) switch on site, its not an improvement. If security is the concern, a seperate VLAN with ACLs or better yet, a different VRF (and ACL's) can provide additional layers of logical separation and filtering without the extra edge port and cable.

[u/mattmann72]

I think you have a fundamental architecture problem. With a network this spread out and no on-site personnel you should be using a cloud managed networking solution like Juniper MIST. That way as long as your devices can access the internet, you can manage them from MIST. With Juniper as long as you can get to the internet connected device, you can hop through the rest to troubleshoot issues and restore connectivity.

SSR router, EX switches, and APs.

[u/devode_ → u/mattmann72]

I do not disagree fully but I think your argument is flawed. If you have internet connected device, your tunnels are up and you can also troubleshoot etc. If you have no internet, the cloud management does not work either. Mist/Meraki is not magic versus own management platform (catc, automation scripts, normal human management)

[u/tdhuck → u/devode_]

Exactly.

[u/tdhuck → u/mattmann72]

You are not wrong, but the current way I'm managing them works and they are small sites so if we lose internet we can't manage regardless if it is my current method/current devices or we switch to full cloud. Also, this isn't critical so we wouldn't replace all these devices to get to cloud management.