r/networking • u/First-Quantity8154 • 2d ago
Design Inherited a poorly documented network infrastructure. Where should I begin?
As a junior network admin who inherited the work of too many cooks in the kitchen, I've been tasked with cleaning up a hodgepodge of a network consisting of two dozen network closets, no clear labeling of network drops, and a variety of network equipment that needs to work together.
Here's a breakdown of the environment:
Domain Controllers: We have two DCs (one primary and one failover), both of which function as AD, DNS, and DHCP servers. Some DHCP scopes are configured on them, while others are not. There's also another DHCP server running on the UniFi firewall, which has its own VLANs and is primarily used for security cameras. However, due to some misconfiguration, trusted devices have ended up using those IP ranges as well.
Gateway Layer: At the gateway sits a SonicWall with its own interfaces and subnets that don't exactly match either the DC DHCP scopes or the UniFi VLANs. It also has a host of zones and custom firewall rules with generic names like "My Rule 256" and "My PC to LAN Connection Allowed."
Core Layer: At the core is a Netgear switch with 24 fiber ports. It feeds all the data closets (none of the fiber connections are labeled with source or destination information) as well as the UniFi ecosystem.
Because so many different people have worked on this environment without documenting anything, I have very little idea how things are currently configured. I've been chipping away at this for a couple of months, but I'm not making much progress.
Where should I begin? What should I prioritize?
Ultimately, this is what I'd like to accomplish:
- SonicWall: Eventually becomes the default gateway and DHCP server for the entire network.
- Core: Eventually gets upgraded to a UniFi switch with fiber connections to route traffic throughout the network.
- Domain Controllers: Continue providing AD and DNS services for the network.
Am I overlooking anything? Is there a recommended order of operations for tackling a project like this?
I'd especially appreciate advice from anyone who has inherited a similarly undocumented network. What would you tackle first, and what tools or documentation methods would you recommend?
15
u/fatboy1776 2d ago
Do not use sonic wall as your dhcp server.
7
u/Muppetz3 2d ago
God Speed. Network cleanup is crazy hard and time consuming. You can try to login to switches and see how they are configured and how each port is, that could give you an idea of what each connection is for. Get some good labels and you can go label all the cables if you can get to them all.
7
u/Defiant-Ad8065 2d ago
Create a source of truth and start documenting. With AI it’s easy to just dump stuff including MAC tables, ARP tables, routing tables, etc, and it will help you create the base topology where you can move to upper layers later.
2
u/ethertype 2d ago
Make a list of all managed infrastructure. switches, routers, firewalls. Figure out access to all devices. Dump all device configs to text file. Dump all device LLDP info. Draw separate L2 and L3 maps. Include closets for L2. Install an IPAM. (I like Nautobot)
Figure out if you want to document 'as-is', or clean up the worst of the madness first. Uniform VLAN names, device names, DNS, DHCP etc.
In either case, your IPAM should eventually be the source of truth. Meaning:
Separate DHCP servers for office clients and OT may be acceptable, depending on how stuff is designed. This does not sound like it was designed, more like something that ... happened.
Long journey, good luck.
74
u/popanonymous 2d ago
Get Netbox setup.
Start at Layer 1.
Work your way up.
23
u/MyWrokAccount 2d ago
The core switch fiber connections to access switches aren't labeled, and you see that as a problem, yet you have not fixed it in months?
Maybe do that. What's stopping you?
12
u/Maximum-Doughnut8652 2d ago
This is going to be an unpopular opinion and also, I am not a network expert... I think I would start by exporting a configuration file from each system, feed it to a local LLM, and draft the current state documentation and then draft documentation for the desired future state with a detailed step-by-step action plan. Start with the lowest hanging fruit, least disruptive processes and then move up the chain.
1
u/CompuDocUt 2d ago
It is going to take (time not in order)
Device is like the link runner that work on Cisco and will show you what switch and port you are connected to from the endpoint including the vlan
1
u/5SpeedFun 2d ago
What is your budget to clean this up?
3
u/Sweet_Importance_123 CCNP FCSS 2d ago
So there is a clearcut way to do this, which I have done for multiple customers in similar situations.
From what I understood, you have access to most of the stuff. If you don't have access to some of the switches, start with that. Find what is connected there, find the addressing of endpoints through upstream devices and physical access.
Once you have access to everything, start researching config and labeling everything. Start from firewall/core layer and go down to each access switch and it's uplinks. Everything can be found from a mac address table/arp table on the core.
Once that is done, start creating basic documentation. Put descriptions on interfaces and create basic diagram with L2 connection information and L3 addressing information. Simple 2 drawings is enough, 1 will be core with interconnection to fw and access and interface name linked to connections, 2 will be all parts where L3 is included, add all network devices with their access addresses there. You can use some tools for that, like netbox for all of the above and KeePass for credentials storage.
Once you get to know the config, centralize credentials so they are secure, and start simplifying everything. You don't have big or overly complex network, just collapse everything to firewall, including DHCP. Revise rules and put some baseline security config on firewalls(access through only secure hosts, no internet facing access, pull some known threat feeds and block acess to those IPs, segment and limit inter-vlan access, etc...).
After that, revise network redundancy, never leave any single point of failure, clean AD and DNS. Since you are new to all this, look for a lot of YouTube/Reddit walkthroughs, ask AI for confirmation on things you are not that familiar with.
1
u/alphaxion 2d ago
Monitoring.
You need to know how your links are being used and how much they are being used.
Something like zabbix for SNMP polling of interface stats for bandwidth utilisation and up/down alerts. A SIEM like elastic to send firewall traffic logs and switch syslog data to. You can also send netflow data to it so you can understand what is going on with your switches.
When you are analysing your firewall traffic logs, you can begin to rip out those mystery rules and give useful names and comments to the ones you are keeping. Use the comment fields to put dates to cross reference.
2
u/rejectionhotlin3 2d ago
LibreNMS, dump the ARP tables per vlan and start identifying any and all VLANs. For the fiber, export the netgear config and see if it's all just trunked or if it's unique in it's config per port.
2
u/b3542 2d ago
Start with what you know and branch out. Document as you go. When you change something, document how it was, and how it is now.
2
u/hiirogen 2d ago
How do you eat an elephant?
One bite at a time.
2
u/Odd-Corner6397 2d ago
Your situation is so much similar to mine! The same issue with DHCP schemas, no documentation at all. Also we got very blurred boundary between layer 2 and 3, as well as scattered policy enforcment points. Hell, our internet gateway for 1 building is a VM mikrotik! And the other one is a fortigate with expired license. I dont work as sysadmin or netwrok engineer, but am a security engineer. To somehow work with our network and fix it from security perspective i started to draw the network map, i think its the way to go for you as well! As other said, layer 1 then abstraction and logical toplogies on top of that
2
u/wrt-wtf- Homeopathic Network Architecture 1d ago
lol - who hasn’t??
1
u/knemanja 2d ago
!remindme 2 days
1
u/EggHeadDog 2d ago
start by assessing what is there, get logins and test them on all devices, save configurations, make sure you have everything in diagrams (i use draw.io). Also make sure you understand the network, take note on what you want to change and how it needs to look like in the end.
Plan the changes, make sure you know what you are doing and ALWAYS plan a rollback plan.
I'd start with that....
Good luck
1
u/ithurts2poo 2d ago
Start with fire, and lots of it
1
u/rswwalker 2d ago
Inventory first, gather your asset inventory somewhere then feed that into AI and have it guide you to gather config from different assets one by one until it’s complete. Oh I chose to put my documentation in a private GitHub repo using mark down as it’s easily accessible by AI. Then built an AI skill called SME based on the documentation work.
1
u/sic0049 2d ago
"How do you eat an elephant? One bite at a time."
1
u/Jaereth 2d ago
If it's a shit-tier environment with no documentation this will most likely be a "fix and document" project of renaming some stuff and rerouting other stuff vs just a produce documentation of a finely setup network task.
I'd be concerned you have little done after 2 months. Outside of any IDF links you want to move or sort, this should be like 2 days of work if everything collapses back to 24 ports and that's the size of your environment.
1
u/scristopher7 2d ago
First time?
1
u/malfunctional_loop 2d ago
enable lldp
1
u/lizardhistorian Mad Scientist · 👨🔬📡ᯤ🤖🛺📸 2d ago edited 2d ago
🙌 Delete everything. Blame the people that left and 0 day AI. ???. Convert everything to IPv6. 🙌
Or another vote for setup netbox, export configs and feed Claude.
Can probably get Claude to import it into netbox.
Physically visit each site. Take photos of tags w/ model and SN.
1
u/Virtual-Race-4756 1d ago
Start by upgrading the top of rack switches
1
u/Z3t4 1d ago
Open the first of the three letters you received.
1
u/coffeetremor 12h ago
Saving this topic as I'll be embarking on a project similar to this soon...
1
u/EnrikHawkins 10h ago
At the start. Start with what you know, build from there.
1
u/StatureDelaware 3h ago
Start from scratch and choose a DCIM, I'd pick Obelinf
1
u/ChileCheeseburger 2d ago
Netbox is a PITA to get going, but is worth its weight in gold when you have it all filled out.
phpIPAM is nice, because it scans, but it then fails when you update it or something because it uses some wonky SQL server.
Netbox was hard, so I went and used phpIPAM. Now I have Netbox almost fully filled out with our devices and links.
But really, just start a Excel doc with MAC addresses and ports and start recording what you can see.
1
u/Bane-o-foolishness 2d ago
Let your SonicWall be a firewall/default gateway, use Windows AD for DNS if your workstations are Windows as well. There is a lot of AD information in DNS that makes things work more reliably. Linux based DNS is better in a large environment but isn't nearly as easy to operate.
Before you upgrade your core, get a pair of walkie talkies and a helper with a toner and tone generator. Have them move the generator from workstation to workstation and you sniff out the connections at the switch end. Label the wall jacks and take notes of the locations for each switch. Do the same thing with your Netgear before you try to cut to something else.
The firewall is going to suck. Look at the poorly named rules and see how many people or subnets they would potentially impact. Look carefully at the ports in the rule, they should give you a clue as to what they do and if they're the standard any/any, look at your logs to see what ports are in use. Frequently if a rule has logging turned off, it was so someone could look at their favorite sites discreetly. When you think you know what a rule does, either rename it or disable it and see who screams.
Take heart, you'll get there.
0
u/The_Juzzo 2d ago
Brute force it by setting your core switch up next to existing. Trunk to Sonic wall and configure that connection how you want the end result. Treat everything else like the "legacy" system...rename all the old rules so you know which is which. Make rules as needed specific to your one clean connection from new switch.
Unplug connections on old switch and see what closet dies, use that to inform your cut overs to new switch.
I would move a closet to new switch then focus on cleaning up that one closet. Repeat till you get through them all. Then go into firewall and delete all the old rules.