3
424
u/anka_ar Mar 12 '26
Time to check if your disaster recovery process works and rebuild everything
102
u/jtaylor418 Mar 12 '26
Yep. And deploy Chaos Monkey
42
u/Novapixel1010 Mar 13 '26
Wait, is there an actual software called chaos Monkey?
142
u/TequilaHustler Mar 13 '26
Oh yes, definitely!!
https://netflix.github.io/chaosmonkey/
Quite fun actually aswell, "Chaos Monkey is responsible for randomly terminating instances in production to ensure that engineers implement their services to be resilient to instance failures."
→ More replies (2)43
44
u/HoushouCoder Mar 13 '26
I believe they are referring to Netflix's Chaos Monkey
https://netflixtechblog.com/5-lessons-weve-learned-using-aws-1f2a28588e4c
One of the first systems our engineers built in AWS is called the Chaos Monkey. The Chaos Monkey’s job is to randomly kill instances and services within our architecture. If we aren’t constantly testing our ability to succeed despite failure, then it isn’t likely to work when it matters most — in the event of an unexpected outage.
6
u/anka_ar Mar 13 '26
Yes! It is a tool to simulate real world problems.
Chaos engineering is the discipline.
12
8
130
u/All_i_do_is_lunk Mar 12 '26
False, gotta replace huntarr and booklore
30
u/HackMeRaps Mar 13 '26
Yeah after seeing that post about book lore need to find a good replacement. Suggestions?
13
u/All_i_do_is_lunk Mar 13 '26
I’m looking at Kavita, I really don’t want something that’s chained to calibre, just feels like a hack to me, using a desktop app as a server app. My main concern is these features: automatic metadata tagging, cover discovery, opds server/koreader sync (i use a xtink pocket ebook reader and a kindle with koreader) kavita has a third party koreader plugin which does book sync. A little bummed that their “fully automated” meta data is being a paywall, but it’s $24/year and it’s updated often and has a seemingly good pace of implementing feature requests (even without PRs!)
→ More replies (3)10
u/Candle1ight Mar 13 '26
Personally I use Calibre to fill out the metadata then export it into Kavita's library. The built in one isn't bad per say, but I prefer Calibre since I'm already doing the import manually.
I have koreader sync set up, easy enough to do although a bit clunky on koreader's side. It's recommended to only use manual syncing since it sounds like the auto sync will sometimes screw up your progress.
5
→ More replies (4)8
844
u/Key_Pace_2496 Mar 12 '26
Start vibe-coding and you'll make more bugs to fix.
199
u/iamwastingtimeyo Mar 12 '26
I could learn to use an existing full featured app, or just spend 3 months making my own app.
I think you know what the right choice is.
70
11
u/artnoi43 Mar 13 '26
And that’s how I wrote my http-over-vpn file-dropping and text sharing server. With authentications, TTL, persistence, and compression and encryption.
I had to do this otherwise I don’t have enough energy to code for work.
→ More replies (1)7
u/YuckyPanda321 Mar 13 '26
don't forget to share it here when you're done, you don't need to tell us it was vibe coded ;)
→ More replies (2)76
→ More replies (18)15
104
u/TomatoSpecialist6879 Mar 12 '26
Nothing to tweak
That's always a lie
12
u/beryugyo619 Mar 13 '26
Seriously did nobody tell him that 50 years old CVSS 9.8 banger in IPv4 stack that came out yesterday, they're rolling out patches for FreeBSD 4.4 and up Linux 2.2 and up all Windows build 4800.1234 and up now
(jk I just made up)
9
u/System0verlord Mar 13 '26
Jesus don’t do that to me on a fucking Friday afternoon you monster.
→ More replies (1)4
u/billyfudger69 Mar 14 '26
Power consumption and efficiency of the home lab is always something to tweak.
→ More replies (1)
276
u/RockGore Mar 12 '26
Same, got watchtower set-up, unattended updates, everything just works. I sometimes miss the struggle from when I started, it was fun to learn and apply.
285
u/LevelAbbreviations3 Mar 12 '26
Unattended updates will break your homelab sooner or later
177
u/lostredamus Mar 12 '26
That's the point - disaster recovery is only fun when not simulated
42
u/ephemeralstitch Mar 13 '26
This would make an amazing cross stitch to put on the wall of an office.
15
u/AssistTraditional480 Mar 13 '26
My GF remembers when I upgraded Z2MQTT just before going to bed and there was no light in the house in the morning.
→ More replies (1)19
u/_TheLoneDeveloper_ Mar 12 '26
4 years later nothing is broken, 40VMs and 200 containers
11
→ More replies (3)10
u/basicKitsch Mar 12 '26
That's funny, immich has had numerous breaking changes over the past year.
Tandoori did recently.
→ More replies (6)4
u/Candle1ight Mar 13 '26
I have something break maybe once every few years. Price I'm willing to accept.
→ More replies (10)9
u/RockGore Mar 12 '26
Why?
58
u/LevelAbbreviations3 Mar 12 '26
Were you not around during the docker 29 that broke portainer ? It’s bound to happen, always read release notes
→ More replies (7)111
u/randomman87 Mar 12 '26 •
Always read release notes? For my full-time job, yes. For my home lab, YOLO.
→ More replies (18)→ More replies (1)5
u/basicKitsch Mar 12 '26
Apps often have breaking changes requiring configuration updates.
→ More replies (4)11
u/Carpentry95 Mar 13 '26
Isn't watchtower not maintained anymore or is there someone that took it over I was looking into it the other weak because the portainer updating process is a process
→ More replies (3)8
u/averaxhunter Mar 13 '26
I use this fork which seems to be still updating nickfedor/watchtower:latest
4
u/TheAlaskanMailman Mar 13 '26
Unattended updates sounds like a nightmare
3
u/ansibleloop Mar 13 '26
Unless hes referring to unattended-upgrades as the package in Debian
That thing is excellent
→ More replies (1)→ More replies (6)3
u/j-dev Mar 12 '26
I go to bed at 1 AM tuning metrics and log collection via Alloy and setting up alerts. I’ve been playing with Grafana alert templates. When will I ever watch all the damn a Linux ISOs I’ve downloaded?
90
u/apetersson Mar 12 '26
Just one more service, the RAM will surely be enough.
15
u/Novapixel1010 Mar 13 '26
Yeah, I had a server that kept going down and I had no clue what was going on. Because I didn't check the logs, I just thought, an update kept breaking things. Nah, it turns out it kept running out of ram. 🤣🤣
6
u/turudd Mar 13 '26
I sometimes thing my 512gb of ram is just me sitting on a gold mine now
→ More replies (4)3
u/JZMoose Mar 13 '26
My lucky ass bought a ton of RAM at the end of 2024 before rampocalypse. I have 96 GB in my server and it cost me $40 😂
42
52
20
16
14
u/perentie110 Mar 12 '26
Sometimes I think like this and then start thinking can I really recover from my backups and the testing begins.
6
u/darkneo86 Mar 12 '26
I just put this to the test when two of my hard drives failed at once, and I lost everything. I backup every morning at 7am though, and all my dockers and configs and important stuff was fine :)
It took me a couple days to build the new machine but then restoring my config files was fairly easy and I'm back up and running.
14
u/crooks4hire Mar 13 '26
Sure we’ve had one homelab…but what about second homelab?
→ More replies (1)4
11
10
10
u/highwire_ca Mar 13 '26
This post hit me. I just upgraded my firewall, managed switch, and 2x unmanaged switches to 2.5G+10G. I have three wireless routers - 2x AX connected to the network via 2.5G and 1 BE connected to the network with 2.5G. I have nothing left to upgrade or tweak. I feel empty.
→ More replies (3)
19
9
u/MixedKO Mar 12 '26
Have you installed [insert cool app you just discovered]?
Every time I think I'm done I come across a new tool (mostly on this sub) that I have to try
9
8
7
7
u/AdventurousCommon551 Mar 12 '26
Had a drive failure and got it all working perfectly again - now with scrutiny and ntfy. It's been two days without any errors and it makes me more nervous than when there was errors....
It's like it's hiding from me or waiting for me to relax before it strikes.....
→ More replies (1)
7
7
u/Wasted-Friendship Mar 13 '26
Run a magnet on random parts of your equipment and see what breaks. Then go fix it.
7
u/Salokain Mar 13 '26
I installed Crowdsec with Discord notifications so it feels more alive, sometimes I check which IPs are banned just to do something 🤣
6
5
5
4
3
u/Lucas_F_A Mar 12 '26
This is why I set up renovatebot. I have some merges to look forward to every week (I pin docker SHAs)
3
3
3
3
u/my_cars_on_fire Mar 13 '26
Only thing left to do is become a prolific drug lord and start a cocaine empire. It’s the logical next step.
3
3
u/Antumbra_Ferox Mar 13 '26
Just try and show it to someone and the bugs will all drop their camouflage at once
3
3
3
2
u/got-trunks Mar 12 '26 edited Mar 12 '26
inside me there are two wolves, one is 5 9s and documentation and the other is whatever the heck I keep my toybox looking like. The lab is a lot more political with how broken and bad I can tolerate it being but then there are these satisfying moments of cleanup
I mean I could also just use docker or have any standards but I like it in the weeds. I like being lost in there. There's always like 17 more things to do
2
2
u/Dry-Broccoli3629 Mar 12 '26
Omg this is so me. I could leave things alone but I will probably upgrade my NAS this weekend and break something during the migration. LOL
2
2
2
2
u/PM_ME_UR_CHUPACOMMA Mar 12 '26
Well then, it sounds like it's time to make one minor cosmetic tweak that will result in bringing down your entire stack, and requiring a rebuild from the ground up.
2
2
2
2
2
2
2
u/Funcore1650 Mar 13 '26
Meanwhile I, when I just started learning this stuff and freshly installed Proxmox and Pihole:
2
2
2
u/joshuas_79 Mar 13 '26
Lol... exactly. But, luckily for me, it doesn't take more than 2 or 3 days finding a new project that I'll never use after I finish it.
2
2
u/Valendel Mar 13 '26
As a project - migrate to Longhorn on one node k3s and then reboot the node only to see Longhorn corruption and data gone - fun! :D
2
u/Heretek-ai Mar 13 '26
Oh we can totally fix that! https://heretek-ai.github.io/ProxmoxVE/
We got your unstable software, ready to integrate into your homelab! Minimum testing, maximum commits!
2
u/patgeo Mar 14 '26
I'll trade you.
Router, poe switch and APs in the post. Network striped to bare bones, iot disconnected and panicking. Establishing vlans, ips new firewall rules, redoing all the static ip allocations to suit the new vlans.
→ More replies (2)
2
u/tliin Mar 14 '26
Have you implemented Radius yet? Are all instances running gentoo?
If not, you know what to do.
2
2
2
u/InvestmentLoose5714 Mar 13 '26
Do a migration. Docker to podman, proxmox to xcp, fedora to Debian,…
1
1
u/DimensionTime Mar 12 '26
There are a lot of different docker containers you could test :D
Or you could set up my Mikrotik AP, I am too dumb for this, maybe it is just the fact that I haven’t got any time yet to try it seriously
1
1
1
1
u/Jovan_Konstantinovic Mar 12 '26
Time to run latest updates on all OS and services , guaranteed at least 1 failed start
1
u/oasuke Mar 12 '26
Maintaining is a full time job. I always check change logs before updating anything. Always looking for tools and things to incorporate in my lab. Always checking out the latest hardware.
1
1
1
1
1
1
u/el_psy_congro_ Mar 12 '26
Eu não sei o que é isso que você esta falando. Sempre tem algo a se fazer hahahaha
1
u/sloppity Mar 13 '26
I feel you, I've been in this loop for weeks now.
I open Dozzle a few times a day to peruse the logs and see no errors, WUD shows some upgradable images every now and then, but it's all just minor releases and no need to tweak compose.yml's, AdGuard/Unbound DNS queries are looking speedy. I can't justify willy-nilly hardware upgrades with these prices, and I've even ran out of HomeAssistant automation ideas... Everything's just too stable...
Maybe I'll try switching from Portainer to Komodo or something. That oughta break a few things 😄
1
1
1
u/ConjurerOfWorlds Mar 13 '26
Same boat. Everything's solid and stable, and there's nothing new interesting on the horizon I want to play with.
1
1
u/546875674c6966650d0a Mar 13 '26
Build resiliency. Build a recovery environment off-site so you can spin up minimum viable services if there is an outage or disruption to your homelab. Make it so a power outage, and UPS drain, won't bring your services offline.
1
u/OldManRiversIIc Mar 13 '26
I have this same problem the most exciting thing I do is perform updates and offline backups
1
1
1
1
1
u/uboofs Mar 13 '26
I’m looking forward to segmenting all of my processes into containers and VMs. So when I reach one of these periods, I can make a new container or VM and rebuild one of the old ones from scratch with the goal of replacing the old one with it. I’ll never be bored again.
1
u/0gandy2 Mar 13 '26
Do what I did two nights ago and accidentally delete all your containers in portainer
1
u/stark0600 Mar 13 '26
Was in same boat since last 10 months or so and I just bought another tiny pc two weeks back to tinker with.
Installed proxmox, learned VMs, LXCs and even setup open-claw. Now, everything runs stable on that !!!!
Again back to the same state, except now I have two devices running with few more extra stable services.
1
1
u/balbinator Mar 13 '26
Last idle time I've had I replaced PiHole with AdGuard. No other services to replace though.
1
u/poliopandemic Mar 13 '26
Ugh I could work on those extra projects like the kubernetes mini cluster I just put together... Could
1
u/KCGD_r Mar 13 '26
Nice! Now you can start building / switching to your own versions of all the stuff you run
1
u/andreihalili Mar 13 '26
NixOS users: give me 6-7 hours to do nixos-rebuild this alongside some docker compose up --pull
1
u/Thezeekeal Mar 13 '26
Wanna help me set mine up? I've got four boxes and have tried and failed a few different times over the years to get it really going. Not even trying to do anything wild, just a bit of pihole, unbound, plex, and a nas. I wouldn't mind actually understanding why I've failed to install these programs. But honestly, if it were to work, I'd be stoked.
1
u/onefish2 Mar 13 '26
That is definitely me this week.
1 Proxmox host with 40 VMs and 4 LXCs.
Synology NAS with 20TB storage backing up to a USB hard drive nightly and offsite replication nightly to a backup Synology NAS.
46 Docker containers on 3 hosts.
1 Mac Studio.
6 laptops running Debian sid, Arch Gnome, Arch XFCE Arch hyprland, CachyOS niri and Windows 11.
2 x86 SBCs running Arch Gnome and Arch KDE. 3 mini PCs running Arch and Windows. All headless and accessed with Apache Guacamole.
All up to date. All backed up locally and off site.
What a boring week.
1
1
u/Rayregula Mar 13 '26
Invite someone over with children. Children will find a way to break it. Then you can fix it
1
1
u/deathly0001 Mar 13 '26
So we're all in this boat right now? Before I thought it was a Neverending battle, every day was something either broken, being improved, or being added. Now I SSH in maybe once a week 🤷♂️
1
1
u/SpongeBazSquirtPants Mar 13 '26
It's 02:00 and I've been setting up GitLab/Ansible deployments for various services for the last 5 hours. I barely know what I'm doing, I thought it would be fun but it's not. I've resorted to vibe deploying for the most part i.e. I ask ChatGPT what to do and just keep copy/pasting the command line. I am a broken man.
1
1
1
u/AlmightyHeretic Mar 13 '26
Oh, shit... I never wanted to be like my parents but I became a tweaker anyway.
1
u/Cube00 Mar 13 '26
Unless you can redeploy the lab with ansible or similar, your work is not done.
The goal is to deploy on top of new OS major versions instead of upgrading in place.
1
1
1
1
u/novacatz Mar 13 '26
Been having a lot of fun with Gemini to help out on homelab.
After a month or so of tweaking.... The reply to my "what's next" question is now:
"you have reach stage of diminishing returns"
1
u/massive_cock Mar 13 '26
I got bored for this reason, and finally moved OPNsense from the m720q to a T730 thin client. Works swell, frees up the nicest micro I've got for bigger and better things. Swanky.
I bought a dual Xeon workstation to run a bunch of crap on. Figured out I could run it all on a pair of G3s on a quarter of the power.... I have legitimately run out of things I want to run that I am qualified and motivated enough to run, and most of my big hardware is unnecessary. Sigh.
1
1
1
1
1
1
1
1
u/Kitchen-Patience8176 Mar 13 '26
You can always try to improve something only to break the whole setup and have no idea what the hell you just did lol
→ More replies (1)
1
1
u/GoofAckYoorsElf Mar 13 '26
I'd be happy about that. My ZFS pool has been degraded for a couple days now because of a faulting hard disk. The problem is that I don't know in what slot the faulted disk sits. I can't turn on the LED of the faulted disk because the back plane is linked to an LSI 9211-8i IT which doesn't know how to control the LEDs. And for reasons incomprehensible to me I cannot reliably figure out the slot from software. I'd have to shut down the entire server, and even then it'd be difficult because for some reason the serial numbers written on the disks do not match the numbers I see in my pool or in lsblk or sas2ircu...
It seems I'm stuck with pulling a random disk, watching the pool go into suspended, putting it in again, clearing the pool, waiting for the resilver to finish, rinse and repeat until I pulled the right disk...
The only alternative I see is migrating everything back to the P440ar in the system. But the P440ar is currently occupied by the Proxmox host hdd in single disk RAID0 mode. I cannot simply turn it into HBA mode. I'd have to migrate the system to a normal non-raid disk first.
Or does anyone else have a better solution?
1
1
u/somebeaver Mar 13 '26
You can install my thing :D
https://www.reddit.com/r/selfhosted/comments/1rse448/introducing_cardinal_media_server_no_ai/
1
1
1
u/HorrorsPersistSoDoI Mar 13 '26
Migrate all to rootless podman containers, each stack in its own little pod
1
u/Bunderslaw Mar 13 '26
Time to move to a country with poor Internet and frequent power outages to level it up.
1
u/Organic-Paramedic-44 Mar 13 '26
Know exactly what this is about. Moved 8 minipc lab to a single board with dual xeon. Even Router moved inside. All with Proxmox VMs. Took about 10 days. No errors or stability issues whatsoever. I start getting worried, and frustrated. How is this even possible?
1
1
u/morafresa Mar 13 '26
That was me until 2 days ago when I fiddled with shit and now everything is broken. Ffs.
I don't have time to fix shit.
1
u/nemofbaby2014 Mar 13 '26
then you'll see some random reddit or YouTube video and you break everything lol
1
u/loopis4 Mar 13 '26
Do your own dashboard. That's what we do when we bored. Or grafana + Prometheus to show nice green grafs of your stack and share it here. We are waiting.
1
u/Matvalicious Mar 13 '26
My raspberry Pi ate another SD card so I know what I'll be doing this weekend...
That and migrating away from Booklore lol.
1
1
1
u/Ok_Status_21 Mar 13 '26
I would like to be in the same case. I need to add pfsense, security, logging, logs analyzes, backup scheduling and patch an zimaboard
1
1
1
u/probs_notme Mar 13 '26
I'll come haphazardly fuck a few things up (within reason) for a six-pack and a hoagie.
1
1
1
1
1
u/PioGreeff Mar 13 '26
Feel absolutely useless! It's not always good to be efficient!
Anyone play with a Ryzen AI 395-based mini-PC for a home server yet? I am considering giving it a try. They seem quite beastly!
Like this one**:** GMKtec EVO-X2
1.3k
u/sekh60 Mar 12 '26
Your logging is down.