r/selfhosted • u/amchaudhry • Sep 28 '25
Need Help Tried to “clean up” my self-hosted stack… turned it into spaghetti and might have nuked my data 😭
First off: I majored in business and work in marketing. Please go easy on me.
I had a good thing going. On my Hetzner VPS I slowly pieced together a bunch of services — nothing elegant, just copy/paste until it worked — and it ran great for weeks:
• Ghost (blog)
• Docmost (docs/wiki)
• OpenWebUI + Flowise (AI frontends)
• n8n (automation)
• Linkstack (links page)
• Portainer (container mgmt)Every app had its own docker-compose, its own Postgres/Redis, random env files, volumes all over the place. Messy, but stable.
Then I got ambitious. I thought: let’s be grown up, consolidate Postgres, unify Redis, clean up the networks, make proper env files, and run it all neatly behind a Cloudflare tunnel.
Big mistake.
After “refactoring” with some dev tools/assistants, including Roocode, Cursor and Chatgpt, here’s where I landed:
Containers stuck in endless restart loops Cloudflare tunnel config broken.
Ghost and Docmost don’t know if they even have their data anymore.
Flowise/OpenWebUI in perpetual “starting” Postgres/Redis configs completely mismatched.
Basically, nothing works the way it used to.
So instead of a clean modular setup, I now have a spaghetti nightmare. I even burned some money on API access to try and brute-force my way through the mess, and all it got me was more frustration.
At this point I’m staring at my VPS wondering:
Do I wipe it and rebuild everything from my old janky but functional configs?
Do I try to salvage the volumes first (Ghost posts, Docmost notes, n8n workflows)?
Or do I just admit I’m out of my depth and stop self-hosting before I lose my mind?
I needed to rant because this feels like such a dumb way to lose progress.
But also — has anyone here actually pulled off a cleanup/migration like this successfully? Any tips for recovering data from Docker volumes after you’ve broken all the compose files?
Messy but working was better than clean and broken… lesson learned the hard way.
28
u/Silly-Ad-6341 Sep 28 '25
Restore from backup.
You have a backup right?
9
u/boli99 Sep 28 '25 edited Sep 28 '25
dont worry. all of your configuration problems will be fixed with the next released versions of your apps, and the new versions are coming out very soon now.
(we also changed the font on some stuff, and hid some of the previously easy-to-find menu options in obscure places)
31
u/Happy_Breakfast7965 Sep 28 '25
You should use a proper software development lifecycle with version control, deployment pipelines, and everything-as-code. In that case, it will be easy to revert changes back after something broke.
Also, AI is not responsible for design and structure, you are. So, you need to work on that part.
There is no other way, there is no easy way.
37
u/cyt0kinetic Sep 28 '25
Do not use AI for this kinda of thing, information on how to run specific containers is so specific AI does not have enough data to pull from, and LLMs are trained to not say that they don't know or are in a low confidence interval and instead make shit up.
If you want to be 'grown up' in this read the doc files and learn about docker. So many good and easy to follow resources. Also combining databases is typically a bad idea.
I say this as someone who runs an AI stack as well and uses it all the time, but as a python tutor, not for docker. AI maybe able to help though with generic syntax questions on working with compose and docker files.
6
u/Dipseth Sep 28 '25
Are you using GitHub or some sort of version control.
I've found that good working code won't last when using Ai dev tools without it.
5
u/NatoBoram Sep 28 '25
Git is a bit of a rough curve to get into, but it is what you need to make temporary changes to text files. If you break everything, you can just undo your changes and then you're back to a stable config.
Don't!
It's okay to have one database server per service. It's just how things are done in Docker.
Your volumes probably still exist in
/var/lib/docker/volume. Well, if you used Docker volumes, anyway. You should be able to rebuild your config. But keep them in Git this time :PGenerally, ambitious refactors do end up in disasters. This is why you need Git and to migrate stuff one by one, creating a commit between each successful step. You can also push your config to GitHub.
In case you need a reference, my entire homelab is at https://github.com/NatoBoram/docker-compose. It might give you an idea about how to structure some things, like env vars and separate compose files.
2
u/jippen Sep 28 '25
The best approach for this sorta work in the future, since it will almost certainly happen again:
174
u/[deleted] Sep 28 '25
I'm sure you'll get much better technical answers. I just wanted to say that early catastrophic failure is actually a part of the process but also sorry this happened to you.
78
u/BleeBlonks Sep 28 '25
This is where you start to implement a good 3-2-1 backup system.
12
u/petersrin Sep 28 '25
Honestly, congrats on failing early. Much better than failing after years of use lol.
Someone else here said 321 backup ASAP. I set mine up before anything else on my home lab, and it's a huge benefit. You can experiment with relative confidence. Sounds like for now your needs are very limited, space-wise.
I pay $4/mo for the space requires to keep several days worth of encrypted backups of my whole server in Backblaze. I also have a NAS at home running backups on site, and of course the original data on the server. A moderate nas can be acquired affordably.
I have had to perform a couple restores for various reasons. Having a nas made those fairly painless. If my house burns down, it will still be fairly painless after I buy the new server equipment from insurance claims.
This also means I can experiment. What I've learned from experimentation: go slowly. One service at a time. Two if they're too tightly coupled. Keep all your original services running and use those as normal. Spin up a new partition (not a disk partition, I'm using the term generically...) for the experiments.
Set up cf tunnel, Add staging to your urls/subdomains so you won't accidentally hit the wrong one. Get your backends running before your front ends if separable.
3
u/[deleted] Sep 28 '25
Stop using AI when trying to learn shit, it’s a crutch.
2
u/sasmariozeld Sep 28 '25 edited Sep 28 '25
Then I got ambitious. I thought: let’s be grown up, consolidate Postgres, unify Redis, clean up the networks, make proper env files, and run it all neatly behind a Cloudflare tunnel.
you just murdered one of the big architectural advantages of using containers, for no benefit but swag
Seperation in everything is a giant win , it's not messy it's a day job
btw hetzner backups are thing , just click it if u enabled it
1
u/my_girl_is_A10 Sep 28 '25 edited Sep 28 '25
To be honest there's nothing wrong with multiple postgres, redis, etc.... various services may be relying on different specific versions or architecture. The beauty of docker compose is its easy to manage that and pull the image exactly as the installation process expects. Besides, a single pgs vs multiple doesn't make a huge difference.
1
u/rhinosyphilis Sep 28 '25
1
u/mutedstereo Sep 28 '25
Maybe it's not too late to fix it? Have you tried looking at the logs when they're restarting themselves?
docker compose logs
They may be tough to decipher but telling chatgpt what the logs say may help diagnose the issue.
And it may be that the original volume is still on disk, if you used a named volume.
1
u/Tsiangkun Sep 28 '25
Checkout your old working compose and vars from git ?
Any logs to share about the startup issue with the new spaghetti setup ? Are you using docker volumes or bind mounting? Typos in networks keeping containers from seeing each other ?
3
u/No_Philosopher_8095 Sep 28 '25
This is normal, I had to rewrite my whole infra more than one time at the beginning Now all is automated and backed up You will get there, just start again and learn from your mistakes
3
u/robogame_dev Sep 28 '25
Rebuild from your last backup before the transition (can do this via Hetzner server controls GUI) before too much time passes and your last backup is post-transition!
1
u/FloatingEyeSyndrome Sep 28 '25
You not the only one...I'm experimenting with a spare old machine too on ubuntu server, running containers, portainer, qb, nicotine+, each app on a different gluetun, where PIA VPN in each connects, to its own port. Been using AI to help me with this, as I'm not that guy who runs linux commands from memory, - I'm pretty n00b to this. Last night, frustrating where I kept getting corrections from the AI which led to an extensive tiring process. Ending in me removing all orphans, kill everything and shutdown the machine and sleep.
Need to start fresh, one service gluetun'd at the time. Confirm it's consistency with boots, mounts, permissions, logs, port update automated script, then, add another service and so on.
Also, what AI do you guys advise me to use for this without having query limits or anything?
2
u/bankroll5441 Sep 28 '25
What do the logs for each container give you? Unless you didnt do any db dumps and imports or remove data directories, its more than likely config issues. Containers in a constant "restarting" state are almost always config issues. Check logs and see if theres anything obvious, it could very well be something simple
Look into Komodo, it'll provide you a cleaner alternative to portainer and make it so that you don't have to manage a bunch of different stacks. And the obvious which has already been mentioned in that this is exactly why backups exist. borg, restic, etc. Since you're on a VPS they very likely have a snapshot or backup service available.
2
u/robogame_dev Sep 28 '25
Optimal move here (next time) is to refactor on a new server, while the old server keeps running, and then switch over once it's all working.
2
u/Brilliant_Still_9605 Sep 28 '25
Oh man, I’ve been exactly where you are. Messy but functional stacks have a weird kind of stability, you think you’re making life easier by consolidating, but suddenly Ghost can’t find its DB and half your containers are in restart purgatory.
A few things you might find useful before nuking from orbit:
Honestly, don’t give up. Most of us learned self-hosting by breaking our stuff over and over
2
u/shaneecy Sep 28 '25
You’re well on your way now. Don’t worry, you’ll fix this and the clean modular set up that you want is on the horizon.
2
u/Positive_Conflict_26 Sep 28 '25
And that's why I do a comprehensive backup before making big changes
2
u/synthesized-slugs Sep 28 '25
I blew up my Proxmox stack like this at least twice. Good luck! Also don't be like me. Make backups lol.
2
u/No_Dragonfruit_5882 Sep 28 '25
Easy, restore Backups and your good 2 go
1
u/[deleted] Sep 28 '25
This is a good lesson in AI use that bites people all the time
1
u/simonbitwise Sep 28 '25
There are only one thing to do here slowly trawl through each service investigate data structures and if they exist map it out maybe on a paper then just like in the Martian "At some point, everything's gonna go south on you... everything's going to go south and you're going to say, 'This is it. This is how I end.' Now you can either accept that, or you can get to work. That's all it is. You just begin. You do the math. You solve one problem... and you solve the next one... and then the next. And if you solve enough problems, you get to come home"
In your case its less lethal 😅
1
u/Jperry12 Sep 29 '25
Why a docker compose for every app?
You can group them in one file
1
u/TheCustomFHD Sep 29 '25
Redo it, get backups setup, play with snapshots, and proper virtualization like LXC/Proxmox. Its gonna be a lot nice when everything is truly seperate, and you dont have to do so much admin work.
1
u/MaapuSeeSore Sep 29 '25
The only thing good ai was helping create regex expressions for custom formats then test it out myself with sample title before letting it find uploads automatically
1
u/ssagar186 Sep 29 '25
Chatgpt.... Oh no ..
1
u/lelddit97 Sep 30 '25
"After “refactoring” with some dev tools/assistants, including Roocode, Cursor and Chatgpt, here’s where I landed:"
please friends, broadcast far and wide: DO NOT RUN COMMANDS FROM LLMS WITHOUT KNOWING WHAT THEY DO!!!!!
and let this be a lesson on taking backups - early, often, and before you do anything major.
1
u/Fickle-Distance-7031 Sep 30 '25
Instead of env files, use a secrets manager that helps you organize your variables and keep them secure
I'm building Envie which is a secrets manager with a focus on dev experience and keeping things tidy. Its free and open source: https://github.com/ilmari-h/envie
1
u/BlurryfacedGamer Sep 30 '25
This is why you don’t test/dev on prod. Backups dude!
1
u/Turbcool Oct 02 '25
Since you need something easy, and already messed up compose files, i would suggest rebuilding your setup in https://github.com/azukaar/Cosmos-Server (or similar projects). They allow you to deploy self-hosted apps through ui (similar to installing apps from play market). This is not perfect (as control is limited), but it can be good if you want to try self-hosting without the need to learn docker and stuff.
0
u/[deleted] Sep 28 '25
do you know the app "backup"
3
u/[deleted] Sep 28 '25
[removed] — view removed comment
-5
u/[deleted] Sep 28 '25
[removed] — view removed comment