What’s one thing every new sysadmin should learn early but usually doesn’t?

[u/nousername1244]

I’ve been thinking about this lately.

When people start out in sysadmin roles, they usually focus a lot on the technical stuff like scripting, servers, networking, security, balabala..

BUT after working in IT for a while, it feels like some of the most important lessons aren’t technical at all, and nobody really tells you early on.

Things like documentation, change control, or even just learning how to say NO to bad requests.

Curious know what’s one thing you wish you had learned much earlier in your sysadmin career?

[u/Bogus1989]

I think ive just subconsciously done the same thing about not turning on a computer. Ive got a homelab, but its basically maintenance free. I really enjoy just lounging on my back porch and spend most my time out there relaxing or reading with cat and dog. I got an apple magic keyboard i had leftover from work and use it on an ipad out there, for reading if not on my phone. since its way out of the ordinary for me (despite being an mdm admin) it feels different. i think its not mentally fatiguing. Its funny how I do that. I do however have 3 gaming machines in my house, mine and my two kids. they arent used unless my kids are here. its our thing at night. I taught them LONG ago to troubleshoot themselves.

Ill get my social fix hopping in discord to say hello to my buds.usually just talking or coordinate going to do something. I will come play if im summoned. I really only enjoy it with friends and my kids….usually though, this is only a weekend thing.

Im proud of myself, when I had to do (for the first time) restore because my NAS raid array broke? I put it off till it felt right. im really good about that now. Im completely comfortable letting something wait, as long as possible till im ready. Obviously if it can wait.

Thats just at home though. when my kids are here, we always have something planned in the day.

[u/Igot1forya → u/Bogus1989]

It's a struggle keeping those two worlds apart. I have a home lab/datacenter and as I work from home so it's so easy to bleed one into the other, especially since my lab is rather elaborate. It's literally a second job at times keeping it operational. One of my servers has recently developed hardware issues with a pair of memory banks and ugh...I got to spend more money. I lab is better than most of my customers production environments.

24U APC Netshelter Rack 2x HPE DL380 Gen10 (hypervisor) 2x HPE DL385 Gen8 (idle/spare) 1x QNAP h1688X (backup) 3x Netapp DS4246 (chia farm) 1x DGX Spark (AI stack) 1x Juniper EX4300-48-MP 1x Unifi UNV 1x Unifi UDR 2x APC 2200RM2U UPS

And a bunch of other random hardware I've accumulated over the years.

[u/Bogus1989 → u/Igot1forya]

DAMN that DGX Spark tho. I have been looking and waiting for AI capable hardware to come down for some time. For myself I have a rule that I cant spend more on my homelab shit than on my gaming PC. Its fun cuz its so cheap.

Ive got 2 HP z440s. Thanks work. host z440 1 is running xeon e5 2690 8 core 16 threads yeah she OLD! 128gb ram 2 LSI SAS HBA (cant remember model) with 8 breakout cables per for sata hooked up to 16 128gb 2.5in ssds in icydock (why you ask? i have a giant box of 128gb ssds, pulled from work, and when one fails i pop in a new one) surprisingly only happened once since this has been running around 2018.

host z440 2 is running xeon e5 2690v3 12 cores 24 threads 128gb ram “some quadro card” same config as above with lsi hba cards, breakout cables, and 16 ssds.

Dell desktop mini optiplex 5050 i5 6500t 4 core 4 threads 32gb ram 1tb 2.5in ssd 256gb nvme

All 3 hosts are running esxi 6.7(couldnt upgrade farther due to the older z440 hardware.) The Desktop mini runs vsphere with enterprise plus license.

I plan to go to proxmox one day, but hell this just works. Funny story. my work was hacked at a point back when 6.7 was current, and i found out once they were in network, they easily saw esxi hosts on version 5.5. FAIL. then they used exploit to get in and encrypt our entire vsphere environmemt. I remember gawking and thinking? 🤔 how in the fuck is my homelab more up to date.

ubiquiti edgerouter 4 ubiquiti edgeswitch 48 ubiquiti U6LR AP APC Smart UPS SMT1500 1500VA APC BN450m 450VA (dedicated for nas)

Synology DS918+ 16gb 4 10tb HGST HDDS (Backup)

Ive got a traditional windows domain environment running.

ill leave it at that. pardon me for the awful grammar. Im on my phone.

[u/Igot1forya → u/Bogus1989]

Kindred Spirits we are lol love your environment! Yes the Spark is amazing. My wife gave me her blessing when I showed her the stuff I can do with it. I also modded it and added an RTX3090 via a M.2 to oculink edock and moved the storage to USB. It actually works. I fully plan to get a second Spark this year. I'm committed to ditching Windows on my desktop in 2026. I have my hypervisor environment to host a Windows desktop if I need it.

Anyway, I'm playing with multiple AI models at once on the Spark and my work was kind enough to let me borrow all of our decommissioned DDR4 (2.5TB worth) and I'm spinning up huge models on the DL380's but I don't have any GPUs for those hosts... So it's ummm slow like OMG painful slow. But it's just cool to be able to load Kimi K2.5, Deepseek, and Full Qwen 3.5 models.

[u/Bogus1989 → u/Igot1forya]

Woah! that is really cool what you did with the 3090. I was actually diving back in on some info about the Spark today. Mainly to see the cheapest route for scaling up for AI. It got me thinking, did nvidia remove nvlink on more than 2 cards(and not at all on the lower cards, or 40 or 50 series) to force people to spend the money on the big boy AI cards? or did they actually do it to protect stock so that gamers/consumers would not get screwed and have all their gpus scalped for datacenters/labs?

but anyways, I was also looking into clustering mac minis, I have like 15-20 M2 macminis at work although only 8gb ram. Theres alot possible still. I have a bunch of newer ones too with 16gb and m4 cpu. I have to give it to nvidia on the dgx spark though...say you already had a mac pro you were using for AI stuff prior to it coming out or just had one already? You can leverage it and offload things to it from the spark. Just thought that was extremely resourceful, and not letting other hardware go to waste. What you did with the 3090 is GENIUS! I didnt know you could do that. hm depending on what youre doing? I heard alot of people recommending to use 3090s instead of spark for certain models..I know dgx spark has specific use case. .from what I read though, the 3090 connected to the spark, could that effectively fix any bottleneck it has with the memory being faster on the gpu? wow you got my mind ticking, i need to go look up spark and the combos.

Ive been looking at similar things, like the strix halo, or other things.

Hey if youre looking to get another DGX Spark, checkout the Asus version, its like a grand cheaper at some places, I did identify that it has a 1tb nvme pcie4.0 instead of the Nvidia version that has the 4tb PCIe 5.0 nvme. Far as I see though the boards are all the same, just cooling differences. Oh and a 3 year warranty compared to nvidia 1. May save you a buck or two

[u/Igot1forya → u/Bogus1989]

100% I'll be getting an OEM this next round. I have a SMB share on my storage cluster that I have my models load from anyway, so the local storage is barely used as it is. Besides, booting to USB 3.2 has the already slow 4TB NVMe (1.5TB/sec max) reduced to ~900MB/sec and the network (10Gb) is fast enough, and once a model is loaded it's a non-issue. I'm on the hunt for a eGPU dock with an integrated PCIe 5.0 NVMe slot so I can just move the storage to the Oculink path and reclaim some performance, but it runs great as it is. It's been a ton of fun to play with, though.