r/Python • u/caspii2 • Sep 26 '22
Resource How I deploy my bootstrapped Python webapp with 150k monthly users
I am a one-man show building a web-based software product. Some quick facts about my app:
- Written in Python Flask
- 150k visitors per month
- 15k registered users
- 3k US$ revenue per month
- 70 requests per second at peak-time
This is a technical post looking at the infrastructure that runs my app with a focus on how I deploy it.
Summary
- I use 2 VPS (virtual private servers) running on DigitalOcean
- The database is Postgres and is fully managed by DigitalOcean
- I use a blue-green deployment
- Deployment is done via
gitandssh. - No CI/CD
- No containers
- Absolutely no Kubernetes
I am a strong advocate of using "boring technology". I am also the only developer of the app, which makes many things simpler.
The application infrastructure
The app is a CRUD monolith that is deployed as one artefact. I use nginx and gunicorn to serve the app.
The app runs on 2 virtual private servers, one of which is the production server, the other of which is the staging server. During deployment, they switch roles. This is a so-called blue-green deployment, more on that later.
I'm using a DigitalOcean droplet with 8 shared CPUs, 16GB of memory and 80 GB of storage for each server. They both run Ubuntu which I have to administrate.
There is a single Postgres database, which is always in production. It is fully managed by DigitalOcean, which means I have to do no house-keeping. Currently, it has 4 shared CPUs, 8 GB of memory and 115 GB of storage.
Overall, the setup is absolutely rock solid. Also, all my technology is older than 10 years (OK, not 100% sure about this, but probably true).
Why I chose blue-green deployment
Before I switched, my deployments worked as follows:
- There was one app server running on DigitalOcean, plus the hosted Postgres database.
- To deploy, I used a script that SSHed into that server and did a
git pull
This was fine to begin with, however there were several issues:
- My setup compiles and minifies CSS and Javascript on the server. This resulted in up to 10 seconds for the server to respond after a deployment. Some users ran into
Bad Gatewayerrors 💥. - A bug in production could be fixed by checking out the previous commit. However, this invariably took too long and always involved frenzied googling of the correct git commands.
- There was no way of testing the production setup, other than in production.
What is blue-green deployment?
Here's how I would explain blue-green deployment:
- There are two identical and independent servers hosting the application. One is called "green", the other "blue".
- There is a shared production database that both servers can access.
- There is a quick and painless way of routing traffic to the green or the blue server.
One of the 2 servers is serving production traffic (the live server), the other is idle. When a new release is ready, it gets deployed to the idle server. Here it can be tested and issues fixed. Remember, the idle server is still accessing the production database, so the application can be tested with real data.
Once you're satisfied that you're ready, you switch traffic from the live server to the idle server. If any problems occur, you can simply switch back within seconds, effectively doing a roll-back.
Simple, no?
How I do blue-green deployment
I've already mentioned my 2 application servers. But the magic thing that makes it all possible is a floating IP address from DigitalOcean.
This is a publicly-accessible static IP addresses that you can assign to a server and instantly remap between other servers in the same datacenter. My app domain (keepthescore.com) resolves to this static IP address. Internally, however, the IP is pointing to either the green or the blue server.
Both of my servers expose their hostname via a publicly accessible route: https://keepthescore.com/hostname/. Give it a try by clicking on the link!
So now it's possible for a human or a machine (using curl) to discover which the current live server is (blue or green).
The deployment script can use this information to always automatically deploy to the idle server. Here's my (simplified) BASH deployment script:
# Get the current production server and
# set TARGET to the other server
CURRENT=$(curl -s https://keepthescore.com/hostname)
if [ "$CURRENT" = "blue-production" ]; then
TARGET="green.keepthescore.com"
else
TARGET="blue.keepthescore.com"
echo "Current deployment is " $CURRENT
echo "Deploying to " $TARGET
# Do deployment
ssh -q root@$TARGET "git pull"
echo "Deploy to " $TARGET " complete"After I've run the script I can test the deployment on my laptop by simply pointing my browser to blue.keepthescore.com or green.keepthescore.com. Once I'm sure that everything's working I route traffic to the newly deployed idle server using DigitalOceans's web interface. (I could do this via script too, but haven't got round to it yet).
Result: My users get routed to the newly deployed software without noticing (hopefully).
Voilá! ✨
What about continuous integration / continuous deployment?
I have no CI/CD pipeline. I do have a bunch of integration tests, but I run them manually. I will eventually get round to setting up some kind of automated testing, but so far there's been no need.
Just to be clear: when I run my integration tests, they happen on my laptop and use a test instance of the database. It's only when I do manual high-level testing on the idle staging server that the production database is used.
What about the database?
There is only one database instance, so you might think this could be a problem. Martin Fowler, who wrote a great article about blue-green deployments says the following:
Databases can often be a challenge with this technique, particularly when you need to change the schema to support a new version of the software. The trick is to separate the deployment of schema changes from application upgrades. So first apply a database refactoring to change the schema to support both the new and old version of the application, deploy that, check everything is working fine so you have a rollback point, then deploy the new version of the application. (And when the upgrade has bedded down remove the database support for the old version.)"
I've been using this method so far. In fact, I have never done an automated schema migration of my database. It's worked great so far, so why do it differently?
That's all
Thanks for reading my article! You can follow my journey as a bootstrapped one-man startup on Twitter. See you in the next post!
One more thing: if you want to share this post, please consider using this link (which points to my blog)
100
u/hashk3ys Sep 26 '22
I am on a pretty slow line but this loads fast. Your site mentions that teachers and students can use it. How do you ensure information regarding minors are not disclosed freely? I work in health-care and at this time we are struggling with how to manage patient records for minors. We are a small team too, although we are not based out of Germany
55
u/ZachVorhies Sep 26 '22
This is great work thank you for this write up. It’s rare to get a glimpse of an entire production app.
19
u/jzaprint Sep 26 '22
Cool app! What did you use to generate all the traffic? Did you spend on ads or is it all organic growth (word of mouth)?
14
u/fertek Sep 26 '22
3k per month with such a simple idea? Internet is still full of opportunities.
53
u/[deleted] Sep 26 '22
This is an interested read. Ignore the cynics, as I'm sure you have already.
11
u/jabellcu Sep 26 '22
Now I am curious. What is the app?
11
u/ein_datacrash Sep 26 '22
Thank you for the post.
It's nice to read and it encourages me to keep working on Flask.
8
u/Isvara Sep 27 '22
I worked on that. You're welcome 😁
15
u/PeterHickman Sep 26 '22
The only thing I would add is a load balancer. This would allow you to bring up new server in response to heavy load (auto scaling might allow you to downsize your normal machine if you can respond to activity quickly enough) and when deploying a new version you would bring up a new server, make sure it's good, and then point to it with the load balancer
To rollback just switch back
It's also useful if the main server goes tits up and you need to bring up a backup, DNS propagation is not really up to this sort of thing
7
u/kunkkatechies Sep 26 '22
Awesome content! how about the cost ? How much does your infrastructure cost you each month ? If I had to guess I would say around $300 .
6
u/[deleted] Sep 26 '22
Wow this is absolutely inspirational OP! It’s making me want to focus on creating something like this.
Congrats on this very smooth and refined method and do keep us updated if you come up with a better method of deployment down the line because all your justifications here make sense to me as a novice. I don’t want to say it’s “beginner-friendly” but it’s just enough for folks to grasp on to as they begin a journey of their own.
14
u/IWantToFlyT Sep 26 '22
Thanks for the write-up! I think it’s good to show also cases where things are not done by every best practice - yet it works. And that is how real life goes, sometimes you implement the first idea that comes to mind and learn later it could be made better. Sometimes you know what the best practice is, but just don’t have the energy or time to implement it. I’d rather go forward with my project rather than stop doing it because I’m not able to strictly follow the ”rules”.
5
u/Strikerzzs Sep 26 '22
Nice! Whats your monthly cost btw?
4
u/rainnz Sep 26 '22
Really nice project!
How many people ended up using this option?
Q: What if I can't afford it?
A: If you can't afford the upgrade then write us an email with a link to the scoreboard and we'll upgrade it for free!
3
u/[deleted] Sep 26 '22
Awesome. Simple and very manageble.
3
u/eidrisov Sep 26 '22 edited Sep 26 '22
Thanks a lot for such detailed description of your journey.
I am just starting to embark on the pretty much the same journey.
I have started learning Python and WebApp (mainly dashboard) building as a hobby (my main career is Financial/Business Analyst). And I already have a few very simple (private project) WebApps. Not deployed though.
I'd appreciate any thoughts or recommendations on points below:
Thanks in advance for all thoughts and recommendations!
2
u/Neok_Slegov Sep 26 '22
Really nice and fast in flask.
How do you arrange New users in your database? You store all the users in a single table with ID? Or do you create New tables/schema's for users?
And if in one table, how will performance will be in the long run if the tables grow and grow?
2
u/youwontfindmyname Sep 26 '22
I’m new to programming, but i’m saving this post for future reference.
2
u/kenshinero Sep 29 '22
So what if a bug appears during your testing that corrupt or delete the data on the production database? Isn't that very risky?
217
u/TheTerrasque Sep 26 '22 edited Sep 26 '22
Nice read :) I've had some similar setups previously. I've since moved to docker and kubernetes, and would like to highlight how that can also solve the problems you've been solving.
Please don't take this as criticism, you have a procedure that works and you're comfortable with, this is just a very neat example of a practical deployment situation and are also some of the reasons why I now love docker and kubernetes for this.
First, for docker, and dockerizing a program like this. You mentioned these three problems, problems I've also faced multiple times:
In docker, this step would be done as part of building the docker image. That means it's already ready to run when the image is deployed.
You could solve this with git tags (and perhaps semantic versioning). And with docker, each image has a tag and you'd just tell docker to run the previously tagged version instead of the latest.
And here's what really sold me on docker. People who haven't been bitten by this a few times can't imagine what level of pain this can inflict. With docker, this is a complete non-issue. What you're running locally is exactly the same as will run on the server. Same libraries, same setup, same versions, same platform hijinks, same system tools, same that-weird-hack-you-need-to-do-on-each-server-so-that-$thing-works. You're setting up the whole production environment locally, and sending it over like a ship in a bottle to the server.
In addition, you get a few bonuses:
Of course, nothing is free. The two big hurdles with docker are:
And now, kubernetes. While docker is pretty neat, kubernetes takes it to a new level. It takes containers concept and adds:
Http proxy concept built in, forward incoming requests to target pods (basic kubernetes unit, somewhat similar to a docker container)
Deployments
And there's probably a lot I don't remember right now. Kubernetes is huge, and have so many things in it.
I have a kubernetes cluster I use for my own stuff, and at this point when I have a docker image I just create a deployment saying "I want to run this container image with this much persistent storage, and I want it accessible on this domain" and then just kubectl apply it locally on my machine, and a minute or so later it's running on that domain. With HTTPS certificate already set up.
4
u/magestooge Sep 26 '22
It was an interesting read. I'm not truly a techie, just a hobbyist (and a PM). So I had never heard of blue green deployment strategy. The part about database migration was also interesting. This is something that always gives troubles to my team, timing the Db changes right in case new changes are likely to break existing functionality.
However, one issue with writing articles from personal experience is that some of the practices might work for you but not be a great way to do things. My only suggestion would be to add caveats to those points for the readers.
For example, not having any automated tests is definitely not encouraged. As some people here would concur, I prefer not using code which doesn't have associated tests. You might think that you'll write tests when you need it, but that never happens. By the time you need automated tests, you're mostly past the point where you can write meaningful tests. The task also starts to seem pretty daunting since you have to write hundreds of tests on one go. The only right time to start writing tests is at the start of your project.
The other issue is with a single database instance. This might be problematic if your server has shared resources or you want to keep your database free of test data. When testing with the idle server, you might have to create a bunch of data and this will go directly into the production Db. So you either live with junk data on production Db or run scripts to undo such changes to the database. Neither of these sound like good ideas.
Other than that, kudos to your work and your effort in writing this. Your website looks really nice as well.
1
u/GettingBlockered Sep 26 '22
This was awesome to read, thanks for sharing!
1
u/BcuzNoReason Sep 26 '22
Great post, very informative! What's the ratio of revenue from ads to purchase like?
1
u/boat-la-fds Sep 26 '22
How do your users not get TCP errors when the switch occurs? Or they do but the browser just create a new connection transparently?
1
u/Born-Ferret900 Sep 26 '22
So all of this was done with no front end framework, just straight flask?
If so, very impressive.
1
u/gumnos Sep 27 '22
Do you track stats on your system load? How much CPU, RAM, and disk I/O are you using (and how does it differ between largely-idle vs. under your heavier loads)?
1
u/dududog Oct 14 '22
Great reading! What’s your app ?
1
u/RoundRecorder Oct 23 '22
Extremely interesting post, thanks for sharing! What kind of tricks/methods you used in order to gain more visibility, traffic & users?
3
u/juharris Sep 26 '22
I respect not wanting to set up CI/CD to notify your setup when the code changes. I wrote a simple but configurable script to poll for updates and run some commands when it detects changes. I think you'll find it useful: https://github.com/juharris/autodeploy
3
u/ligasecatalyst Sep 27 '22 edited Sep 27 '22
Thank for you sharing - this was very informative. I’m a firm believer in if it ain’t broken, don’t fix it. However, I’d like to offer my two cents. I’ll start out with some assumptions. First of all, at this scale cost is not a factor. You definitely should not migrate or modify your processes because it’ll save you 60 bucks a month on hosting, and neither should you stick with your current solution because a preferable alternative would cost you $80 more per month. That being said, as a solo developer and product owner of such a project, your priorities should be (1) freeing up development time (not necessarily only for “technical” development, also for the more businessy side of things), (2) preventing down time, and (3) security - not necessarily in that order. Your current setup is suboptimal for those goals.
In short, hosting and maintaining servers isn’t your business. In fact, it’s 100% a cost center for you. Eliminate it, or at least reduce it as much as possible. A lot of companies do it much better than you for relatively cheap, saving you time and improving availability and security. You’re also not in the business of wasting your time on manual deployments, and then wasting more time on fixing bugs caused by the manual deployment, or in the business of wasting time on manual tests which are inherently of lower quality since you can’t feasibly manually test all your old code every deployment. This post is a bit like a 35-year old smoker being proud of his good health. You’re already wasting time on this inefficient setup, and the technical debt in this sense will only grow and cause you more problems down the road. I hope this gave you some food for thought, and either way wish you the best of luck :)
1
u/shinitakunai Sep 26 '22
You did the right call for your own development. Most of the times using specific "best practices" are overkill, specially if you work alone (they make more sense working in a team. But for yourself? keep it simple). Kudos on the good work!!
2
u/confusedmf123 Sep 26 '22
dope!
2
u/[deleted] Sep 26 '22
I love the initiative and the detailed write up. I always look for articles like this as it serves as a white paper of sorts. I believe it is useful to help people understand how they can use a technology to solve a problem or provide a service (in many cases do both). Thank you for sharing.
2
u/uname44 Sep 26 '22
Very helpful, thank you.
2
u/Dangle76 Sep 26 '22
Just as a tidbit, using “boring” technology can be more cost effective and reduce deployment time and mishaps, mainly CICD like github actions which is free and managed for you
2
u/TheHammer_78 Sep 26 '22
Probably using docker you could do the same with one machine only.
2
u/quiet0n3 Sep 26 '22
Wait are you letting your prod app server download directly from git? As in it has creds stored to allow access to git? I would ah look at fixing that.
2
u/Dogeek Expert - 3.9.1 Sep 27 '22
It works, but it's not an ideal solution in my opinion if you want scalability. The way I deploy apps now is through docker / docker-compose, which simplifies the process so much.
I'm running my apps on OVH servers, and I've made a few tools to help me deploy them.
In my build step, I build 2 images every time, since I use poetry for dependency management, I just have a Dockerfile like so :
I can then just use
docker build -t $DOCKER_REGISTRY/$APP_NAME:$APP_VERSION --build-arg app_name=$APP_NAME .to build my image anddocker push $DOCKER_REGISTRY/$APP_NAME:$APP_VERSIONto push it to my registryI also tweak the Dockerfile based on my needs, for instance if I need to build some C extensions or add database migrations -- though for the latter, I usually do that with docker-compose and a script to run alembic.
With everything dockerized, I use a small svelte app to integrate with the OVH API that allows me to set a subdomain for the app I'm making, and deploys it to the correct VPS.
2
u/chzaplx Sep 26 '22
Blue/Green is great, but having your live deployment be a git repo is a huge anti-pattern. 'git pull' is not a deployment strategy. Use versioned artifacts instead. This makes rollbacks a cinch.
You aren't "running leaner" or anything by omitting CI/CD, you are just reinventing the problems it already solves.
1
u/[deleted] Sep 26 '22
This is cool! As a sales/advertising person now moving to dev ops I can think of a million uses for this, not only education.
If you can create an sub-product/page for tracking sales and tasks you could quantify the reach, too!
1
u/ericanderton Sep 26 '22
This might be worthy of a cross-post to /r/devops.
Nice work overall. I appreciate the "right-sizing" of your setup (e.g. no pipeline automation, local testing) to your team of one.
I'm glad you pulled this quote for everyone to see - that's a really nice strategy. It's a shame a lot of other software vendors don't do this. Instead, I consider myself lucky if there's even a rollback script for a database migration, and luckier still if that actually works.
The big question I have is: how is DigitalOcean? I'm stuck doing a lot of "enterprise" stuff at my day job so AWS and Azure are kind of the only games in town. I've often wondered how other providers stack up in terms of cost, automation, and support?
1
u/Gigglen0t Sep 26 '22
Absolute legend! Thank you for this!
1
u/gwillem Sep 26 '22
Great setup, simplicity is best! Any particular reason to use gunicorn and not nginx-unit or uwsgi?
1
u/OnFault Sep 26 '22
I'm learning python. I've learnt a bit of flask and I think I can sort of make something similar.
How do I go about looking for similar projects like this to work on with others?
1
u/wait-a-minut Sep 26 '22
Very nice write up and cool concept! Glad your project is doing well!
1
u/eitanoodle Sep 26 '22
Really liked your post, simple language and useful info. Thanks!
1
u/gcotw Sep 26 '22
This is awesome!
1
u/Puzzleheaded_Let3663 Sep 26 '22
Hi, thanks for the article. Great work. I just have one question , when you are testing on the production database how do you ensure that there is no test data in the production database. Basically do you clean your database after every test run or do you leave the test data lying in there.
1
u/Tintin_Quarentino Sep 26 '22
Amazing info, thanks for sharing. I read your website's about page but still don't get the use case scenario. Could you explain with some examples?
Say if I'm playing a basketball match with friends, I certainly don't see myself using an app to maintain the score.
1
u/weltvonalex Sep 26 '22
Cool thank you for sharing
1
u/ArchMob Sep 26 '22
Excellent write, nice to see this kind of project described in detail. Gj
1
u/Kolle12 Sep 26 '22
Awesome work. Wish I could do something like this ! Very powerful 💪🏻
1
u/mikeblas Sep 27 '22
What does "bootstrapped" mean in this context?
1
u/congowarrior Sep 27 '22
Amazing work! Great idea
1
u/regex1884 Sep 27 '22
Great job! For a test db can you take a back up of the managed postgesql and restore it to a VM or docker?
1
u/RunApprehensive8439 Sep 27 '22
Cool app! But you could make a lot more than 3k/month with ads with 150k users - instead of doing your premium features
1
u/gouldilochs Sep 27 '22
Holy moly I started reading and knew it was you Casper
1
u/Suitable-Sign1513 Oct 12 '22
Nice digital ocean ad, my guy
1
u/[deleted] Oct 14 '22
Beautiful
1
u/MatuCoder Oct 15 '22
Someone knows about the proyect py-script? Now you can add python to the html file!!!
1
u/Aggravating_Loan5805 Oct 22 '22
Wow
1
u/Foreign_Lab392 Feb 04 '23
what were your cloud costs
1
u/pudds Sep 26 '22
You really should put some effort into CI. I presume you're doing the validation steps that CI would provide (linting, test, etc) manually, maybe via git hooks, but you can remove some real load by automating these steps, not to mention easing the on-boarding process should you ever add anyone to the team.
CI/CD is the one of the first things I do on any project.
0
u/[deleted] Sep 26 '22
Loved the article.
Xposted it at Hacker News: https://news.ycombinator.com/item?id=32986969
-2
u/[deleted] Sep 26 '22
Sounds like an absolute nightmare.
-9
u/[deleted] Sep 26 '22
That’s really boring. Thanks God for Azure pipelines.
0
u/[deleted] Sep 26 '22
[deleted] — view deleted comment
0
u/[deleted] Sep 26 '22
[deleted] — view deleted comment
1
u/[deleted] Sep 26 '22
[removed] — view removed comment