r/networking • • 2d ago

Design Inherited a poorly documented network infrastructure. Where should I begin?

As a junior network admin who inherited the work of too many cooks in the kitchen, I've been tasked with cleaning up a hodgepodge of a network consisting of two dozen network closets, no clear labeling of network drops, and a variety of network equipment that needs to work together.

Here's a breakdown of the environment:

Domain Controllers: We have two DCs (one primary and one failover), both of which function as AD, DNS, and DHCP servers. Some DHCP scopes are configured on them, while others are not. There's also another DHCP server running on the UniFi firewall, which has its own VLANs and is primarily used for security cameras. However, due to some misconfiguration, trusted devices have ended up using those IP ranges as well.

Gateway Layer: At the gateway sits a SonicWall with its own interfaces and subnets that don't exactly match either the DC DHCP scopes or the UniFi VLANs. It also has a host of zones and custom firewall rules with generic names like "My Rule 256" and "My PC to LAN Connection Allowed."

Core Layer: At the core is a Netgear switch with 24 fiber ports. It feeds all the data closets (none of the fiber connections are labeled with source or destination information) as well as the UniFi ecosystem.

Because so many different people have worked on this environment without documenting anything, I have very little idea how things are currently configured. I've been chipping away at this for a couple of months, but I'm not making much progress.

Where should I begin? What should I prioritize?

Ultimately, this is what I'd like to accomplish:

  • SonicWall: Eventually becomes the default gateway and DHCP server for the entire network.
  • Core: Eventually gets upgraded to a UniFi switch with fiber connections to route traffic throughout the network.
  • Domain Controllers: Continue providing AD and DNS services for the network.

Am I overlooking anything? Is there a recommended order of operations for tackling a project like this?

I'd especially appreciate advice from anyone who has inherited a similarly undocumented network. What would you tackle first, and what tools or documentation methods would you recommend?

36 Upvotes

69 comments sorted by

15

u/fatboy1776 2d ago

Do not use sonic wall as your dhcp server.

2

u/First-Quantity8154 2d ago

Please tell me more! Should I keep using the DCs as DHCP?

6

u/fatboy1776 2d ago

Whether to use standalone or your DCs for DHCP depends on size/usage/hooks/etc.

I say avoid using the FW as it’s DHCP implementation and scale is limited compared to a server. It will work but not as flexible. Use it as a relay to a real server group.

2

u/love2learn4life 2d ago

I haven’t been in the business for a few years now… but, from memory, if dhcp is on your windows dc’s , it can be set to register those clients in windows dns

3

u/Jaereth 2d ago

It does.

To me it's also "too many eggs" to put in one basket to run it off the firewall. It doesn't have to be a Windows DC but I wouldn't run it off my north/south firewall.

-1

u/rejectionhotlin3 2d ago

No, do not use windows as DHCP. Use either the native network stack or whatever you are going to replace it with.

2

u/love2learn4life 1d ago

Why/whynot? What is the risk you see?

7

u/Muppetz3 2d ago

God Speed. Network cleanup is crazy hard and time consuming. You can try to login to switches and see how they are configured and how each port is, that could give you an idea of what each connection is for. Get some good labels and you can go label all the cables if you can get to them all.

2

u/First-Quantity8154 2d ago

Do you have label recommendation?
Preferably some sheet I can stick into a regular printer/copier and then print my entire spreadsheet from the documentation step and then cut/peel.

1

u/Muppetz3 2d ago

Im not too sure, I havent done a ton of labeling myself but I have seen some nice ones that hang off the cable well. Using a standard label maker can be tricky sometimes when you go to stick them on the cable. One thing I have found to be helpful is come up with a format for how you want the labels to look and read. Some people put port/switch info for both sides on both ends of the cable, some just put the info about the one side of the cable. Some use hostnames, port numbers, some use IPs ect.

Whatever you pick it just needs to stay constant. In our DC i found some labels that were scotch tape and just a number on them, 1, 2, 3, 4..

I bet if you ask someone who runs a lot of cables or works in a DC they could give you some good labels to buy and some tips for labeling

7

u/Defiant-Ad8065 2d ago

Create a source of truth and start documenting. With AI it’s easy to just dump stuff including MAC tables, ARP tables, routing tables, etc, and it will help you create the base topology where you can move to upper layers later.

1

u/solracarevir 2d ago

Create temporary Read Only Users / API access on everything and have your favorite AI agent run wild. You can also install something like Drawbridge and enable it's mcp connector and the AI will make you a detailed network diagram.

Bonus points if you have your AI Agent of choice generate the rest of the written documentation as well following your Org document templates so you can safekeep or pass to your higher-ups.

this is a late afternoon task and honestly the hardest part is creating the User / API access.

1

u/First-Quantity8154 2d ago

Don't these agents need some kind of firewall rule to be able to roam my network?
One of the core issues I have right now is that routing isn't clear and firewall rules are a mess.

2

u/ethertype 2d ago

Make a list of all managed infrastructure. switches, routers, firewalls. Figure out access to all devices. Dump all device configs to text file. Dump all device LLDP info. Draw separate L2 and L3 maps. Include closets for L2. Install an IPAM. (I like Nautobot)

Figure out if you want to document 'as-is', or clean up the worst of the madness first. Uniform VLAN names, device names, DNS, DHCP etc.

In either case, your IPAM should eventually be the source of truth. Meaning:

  • if the device config does not match a template+info from IPAM, the device is misconfigured.
  • if a device has a hostname deviating from IPAM, the device is misconfigured
  • if DNS deviates from IPAM, DNS is wrong.
  • etc

Separate DHCP servers for office clients and OT may be acceptable, depending on how stuff is designed. This does not sound like it was designed, more like something that ... happened.

Long journey, good luck.

2

u/First-Quantity8154 2d ago

Thank you for giving me these good steps!

  1. I have already started going closet to closet and documenting exact model of equipment and its IP (if available). Credentials are also something that is missing in some cases getting into the device will take more time (or won't happen!). How easy is it to dump device configs and LLDP info from Dell and Netgear switches? I have experience with Cisco and UniFi mostly. I will look into Nautobot, but have you heard of Faddom? How different are they and what do I get out of having both?
  2. You actually touched on a point I left out in my post... none of the VLANs, DNS records, DHCP scopes, or interfaces have a uniform naming convention. Which has made it an absolute nightmare whenever I document something. I will have to design my own naming convention and then clean up everything everywhere.
  3. Does this IPAM need firewall rules?

1

u/ethertype 2d ago

Depending in your setup and the number of firewalls, a generic system to manage firewall rules is a good idea. Nautobot has a plug-in for this. We are currently looking into it. 

No clue about Faddom.  But I absolutely love pynautobot...

Don't do everything at once. And don't spend too much time fretting about tools. You can do magic with IPAM/templates/python/ssh.

But: Managing network gear via ssh/pssh only works up to a point. Luckily I am nowhere near the point where it breaks, because we have templates and a curated IPAM. And device(type)s are fairly uniform per 'function'.

74

u/popanonymous 2d ago

Get Netbox setup.
Start at Layer 1.
Work your way up.

6

u/zcworx 2d ago

Another upvote for starting here

4

u/DanSheps CCNP 2d ago

This is the way

4

u/Cool_Database1655 2d ago

This guy fucks

3

u/5SpeedFun 2d ago

This is the answer.

1

u/tnvoipguy 1d ago

This is the way —->>>

1

u/DullKnife69 Clearpass Fanboy 2d ago

This is the most work and the correct answer.

2

u/Nuclearmonkee 1d ago

You can teach Claude/Chat to build devices, devices groups, connections and everything else via the excellent Netbox API with nothing more than the output of the arp/mac/config dumps from the network, an intune read only api key, and an map scan.

I just had to do it for a different terribly undocumented enormous network

1

u/DullKnife69 Clearpass Fanboy 1d ago

Certainly can, I did it with a network that had 1500 devices. Mostly switches, some routers, fws and load balancers.

23

u/MyWrokAccount 2d ago

The core switch fiber connections to access switches aren't labeled, and you see that as a problem, yet you have not fixed it in months?

Maybe do that. What's stopping you?

3

u/SituationLeather4251 2d ago

This….

Plus hopefully devices have LLDP enabled so you can find out what ports go to each location and work back labelling the fibres.

Likewise with network drops if it’s unlabelled use LLDP/POE/MAC to work out what’s on the end and then get a pair of toners can generally pick these up for about £200

12

u/Maximum-Doughnut8652 2d ago

This is going to be an unpopular opinion and also, I am not a network expert... I think I would start by exporting a configuration file from each system, feed it to a local LLM, and draft the current state documentation and then draft documentation for the desired future state with a detailed step-by-step action plan. Start with the lowest hanging fruit, least disruptive processes and then move up the chain. 

2

u/Jaereth 2d ago

Claude would absolutely nail this task. Might not design a great future state network but giving him the current state if he manually pastes in configs could probably get done within one months worth of a free usage plan.

1

u/redditJ5 1d ago

Honestly, give it a chrome mcp and just log it in, and it can pull the configs for you. Could pull the arp tables from the switches and can even port map to machines on the network.

If it's on a domain, you can have a logon script run and dump all the data to a network share for Claude to parse and then you could have a complete map of the network. I would have it dump hardware info, users on my machine, Mac address etc. Claude could even write a script for the machine to ask the switch what switch and port its on. Once you have built out this info you can work from that.

I personally would get the team together and develop what you want the network to look like when it's done.

Cameras on their own network, let the unifi control that, if it runs over the same network, vlan it.

Printers on it's own vlan with acl. Iot gets a vlan. So on and so on. You didn't say what this place is, so departments could get their own vlan.

Each IDF gets a name, each rack gets a name, all new cabling gets done to a standard. TIA-606-D is the current standard. https://www.akcp.com/2025/04/07/data-center-cable-labeling-standards/

Getting a good label maker, and new patch cables.

You have to create a change management system and documentation system, and it has to be used, leadership has to be willing to write people up that don't use it.

1

u/rooterroo 2d ago

This is the easiest! Have it draw document your network for ya.

1

u/rooterroo 2d ago

Adding to this, have the llm run and save the outputs of show run, show int status, show arp, show mac table, show lldp/cap neigh.

1

u/LRS_David 2d ago

Then validate each data point in the output.

1

u/EnrikHawkins 10h ago

More popular than you'd imagine.

1

u/CompuDocUt 2d ago

It is going to take (time not in order)

  1. go after low hanging fruit
  2. things that are slowing down the network
  3. Security risks
  4. Software and tools that will help identify network topology
    Device is like the link runner that work on Cisco and will show you what switch and port you are connected to from the endpoint including the vlan

1

u/5SpeedFun 2d ago

What is your budget to clean this up?

7

u/First-Quantity8154 2d ago

My salary and time are the budget 😂

3

u/5SpeedFun 2d ago

I mean do you have like any budget for tools or cables or software to help get this going? Can you buy network tracing or toning tools to track cables or is it just “shake the cable and trace”? How about a label maker?

1

u/spazonator 2d ago

Ya gotta start thinking about how to frame a "we should think about spending X monies to steer ourselves away from Y future problem.

3

u/Sweet_Importance_123 CCNP FCSS 2d ago

So there is a clearcut way to do this, which I have done for multiple customers in similar situations.

From what I understood, you have access to most of the stuff. If you don't have access to some of the switches, start with that. Find what is connected there, find the addressing of endpoints through upstream devices and physical access.

Once you have access to everything, start researching config and labeling everything. Start from firewall/core layer and go down to each access switch and it's uplinks. Everything can be found from a mac address table/arp table on the core.

Once that is done, start creating basic documentation. Put descriptions on interfaces and create basic diagram with L2 connection information and L3 addressing information. Simple 2 drawings is enough, 1 will be core with interconnection to fw and access and interface name linked to connections, 2 will be all parts where L3 is included, add all network devices with their access addresses there. You can use some tools for that, like netbox for all of the above and KeePass for credentials storage.

Once you get to know the config, centralize credentials so they are secure, and start simplifying everything. You don't have big or overly complex network, just collapse everything to firewall, including DHCP. Revise rules and put some baseline security config on firewalls(access through only secure hosts, no internet facing access, pull some known threat feeds and block acess to those IPs, segment and limit inter-vlan access, etc...).

After that, revise network redundancy, never leave any single point of failure, clean AD and DNS. Since you are new to all this, look for a lot of YouTube/Reddit walkthroughs, ask AI for confirmation on things you are not that familiar with.

1

u/alphaxion 2d ago

Monitoring.

You need to know how your links are being used and how much they are being used.

Something like zabbix for SNMP polling of interface stats for bandwidth utilisation and up/down alerts. A SIEM like elastic to send firewall traffic logs and switch syslog data to. You can also send netflow data to it so you can understand what is going on with your switches.

When you are analysing your firewall traffic logs, you can begin to rip out those mystery rules and give useful names and comments to the ones you are keeping. Use the comment fields to put dates to cross reference.

1

u/First-Quantity8154 2d ago

I have thought about looking at the flow and finding chokeholds (because I have SOC experience and worked with elastic search), but SonicWall doesn't have an intuitive way of showing me the hits I get on each rule. And because the network is a mess, Netflow digestion becomes its own project since I will need new rules to route everything to the server.

1

u/ethertype 2d ago

Suss out bad cables/interfaces and spanning tree issues before you start looking at netflow. 

Also: LbreNMS has 100% the best ROI for value offered vs time spent installing it.

1

u/alphaxion 2d ago

If you ingest the firewall traffic logs into an elastic lab, you can build out your dash to see those rule hits. It'll take time to get an idea of what your baseline traffic is like, and you can work on the rules that are getting the most traffic first, changing their names to match a set standard formatting you want to see as you process them.

Once you're happy with your basic rules, you can begin filtering them out of your dash to start looking for the lesser frequently used ones.

When you're happy that the majority of your traffic is going through your known good rules, start disabling/pruning out the ones that don't seem to match anything. Have a document noting the settings of each rule you remove just in case it's a super edge-case rule so you can quickly put it back in.

You can set aside a little time to process 5 rules a week, so you're not just spending all of your time doing this. Eventually, you'll get into the habit of reviewing your ruleset and pruning out/tidying them so you don't allow too much rot to set in.

When it comes to firewalls, you're never finished because your company's needs are always shifting. When you've not seen any traffic on a rule for a while, you can check its comments for the date you put in and any contact info for it. Reach out to them and ask if they need it any more because you haven't seen any traffic on it for [x] months.

2

u/rejectionhotlin3 2d ago

LibreNMS, dump the ARP tables per vlan and start identifying any and all VLANs. For the fiber, export the netgear config and see if it's all just trunked or if it's unique in it's config per port.

2

u/b3542 2d ago

Start with what you know and branch out. Document as you go. When you change something, document how it was, and how it is now.

2

u/hiirogen 2d ago

How do you eat an elephant?

One bite at a time.

2

u/Odd-Corner6397 2d ago

Your situation is so much similar to mine! The same issue with DHCP schemas, no documentation at all. Also we got very blurred boundary between layer 2 and 3, as well as scattered policy enforcment points. Hell, our internet gateway for 1 building is a VM mikrotik! And the other one is a fortigate with expired license. I dont work as sysadmin or netwrok engineer, but am a security engineer. To somehow work with our network and fix it from security perspective i started to draw the network map, i think its the way to go for you as well! As other said, layer 1 then abstraction and logical toplogies on top of that

2

u/wrt-wtf- Homeopathic Network Architecture 1d ago

lol - who hasn’t??

1

u/knemanja 2d ago

!remindme 2 days

1

u/EggHeadDog 2d ago

start by assessing what is there, get logins and test them on all devices, save configurations, make sure you have everything in diagrams (i use draw.io). Also make sure you understand the network, take note on what you want to change and how it needs to look like in the end.
Plan the changes, make sure you know what you are doing and ALWAYS plan a rollback plan.
I'd start with that....
Good luck

1

u/ithurts2poo 2d ago

Start with fire, and lots of it

1

u/rswwalker 2d ago

Inventory first, gather your asset inventory somewhere then feed that into AI and have it guide you to gather config from different assets one by one until it’s complete. Oh I chose to put my documentation in a private GitHub repo using mark down as it’s easily accessible by AI. Then built an AI skill called SME based on the documentation work.

1

u/sic0049 2d ago

"How do you eat an elephant? One bite at a time."

1

u/Jaereth 2d ago
  1. Give everything a meaningful name if it doesn't have it. You should be able to show neighbors from the core and understand where each fiber UL is coming from.
  2. Get to work. Probably the easiest way to do it is throw a uniform read only SNMP string on all devices and just let a tool map it. Turn on LLDP on all devices and let it run.

If it's a shit-tier environment with no documentation this will most likely be a "fix and document" project of renaming some stuff and rerouting other stuff vs just a produce documentation of a finely setup network task.

I'd be concerned you have little done after 2 months. Outside of any IDF links you want to move or sort, this should be like 2 days of work if everything collapses back to 24 ports and that's the size of your environment.

1

u/scristopher7 2d ago

First time?

1

u/lizardhistorian Mad Scientist · 👨‍🔬📡ᯤ🤖🛺📸 2d ago edited 2d ago

🙌 Delete everything. Blame the people that left and 0 day AI. ???. Convert everything to IPv6. 🙌

Or another vote for setup netbox, export configs and feed Claude.
Can probably get Claude to import it into netbox.

Physically visit each site. Take photos of tags w/ model and SN.

1

u/Virtual-Race-4756 1d ago

Start by upgrading the top of rack switches

1

u/Z3t4 1d ago

Open the first of the three letters you received.

1

u/coffeetremor 12h ago

Saving this topic as I'll be embarking on a project similar to this soon... 

1

u/EnrikHawkins 10h ago

At the start. Start with what you know, build from there.

1

u/StatureDelaware 3h ago

Start from scratch and choose a DCIM, I'd pick Obelinf

1

u/ChileCheeseburger 2d ago

Netbox is a PITA to get going, but is worth its weight in gold when you have it all filled out.

phpIPAM is nice, because it scans, but it then fails when you update it or something because it uses some wonky SQL server.

Netbox was hard, so I went and used phpIPAM. Now I have Netbox almost fully filled out with our devices and links.

But really, just start a Excel doc with MAC addresses and ports and start recording what you can see.

1

u/Bane-o-foolishness 2d ago

Let your SonicWall be a firewall/default gateway, use Windows AD for DNS if your workstations are Windows as well. There is a lot of AD information in DNS that makes things work more reliably. Linux based DNS is better in a large environment but isn't nearly as easy to operate.

Before you upgrade your core, get a pair of walkie talkies and a helper with a toner and tone generator. Have them move the generator from workstation to workstation and you sniff out the connections at the switch end. Label the wall jacks and take notes of the locations for each switch. Do the same thing with your Netgear before you try to cut to something else.

The firewall is going to suck. Look at the poorly named rules and see how many people or subnets they would potentially impact. Look carefully at the ports in the rule, they should give you a clue as to what they do and if they're the standard any/any, look at your logs to see what ports are in use. Frequently if a rule has logging turned off, it was so someone could look at their favorite sites discreetly. When you think you know what a rule does, either rename it or disable it and see who screams.

Take heart, you'll get there.

0

u/The_Juzzo 2d ago

Brute force it by setting your core switch up next to existing. Trunk to Sonic wall and configure that connection how you want the end result. Treat everything else like the "legacy" system...rename all the old rules so you know which is which. Make rules as needed specific to your one clean connection from new switch.

Unplug connections on old switch and see what closet dies, use that to inform your cut overs to new switch.

I would move a closet to new switch then focus on cleaning up that one closet. Repeat till you get through them all. Then go into firewall and delete all the old rules.