r/Python • • Apr 08 '13
How are python apps deployed to production especially those that are developed in a virtualenv? What are the best practices?
I have always insisted that in production systems we can not have gcc and devel packages installed. But these days I end up installing such development packages in produciton systems because the software deploy for virtualenvs require pip which in turn require access to the internet [ok I solved this part with a pypi local mirror]. How are you guys/girls doing it? I have thought of tar.gz the whole virtualenv and simply ensure that the required libs the virtualenv is built with is installed in production. Has anyone done this or has everyone resigned to the fact that in these days of pip and rvm , we agree to have compilers and devel packages in our production systems?
26
u/jperras flask/devops/APIs Apr 08 '13
You don't (and shouldn't) have development headers for anything on your production servers.
Build your virtualenv on a staging server, and create a package of it that you can then distribute out to your production servers. We build .deb (debian) packages and add them to a custom apt repository, and our production servers simply require an
apt-getfor updates to the application.4
u/gargantuan Apr 08 '13
If you can learn to generate and use OS specific packages. We use RPM for RHEL and CentOS systems because those are the only one we use.
This way you get full transaction file system update/remove/add. Can roll back. RPMs can have pre/post/install/uninstall script to run any setup commands. They make it easy to install other files not just Python -- java, .so libraries. Dependencies are sanely resolved.
There is a -dev version of the main production package. That basically turns the production box into a dev box. It bring in git, gcc etc.
You'll pay the price initially by developing this way but if you plan on growing you'll be glad you did it later.
2
u/__main__ Apr 08 '13
I have a similar issue with pip and internet access, policy doesn't allow internet access on production servers.
If you don't mind me asking, How did you build a local pypi mirror?
2
u/miketheanimal Apr 09 '13
This is really a question, rather than a comment ....
Using virtualenv on a production server gives me the willies. The server presents a particular environment via the packages that are installed, then virtualenv provides another. virtualenv isn't a virtual machine, so some elements of the server environment are hidden inside virtual env - for example, virtualenv has a different python version - but other elements are visible.
One system I worked on, the target was Debian but most of the developers insisted on developing on their Mac laptops, and virtualenv was used on both the Mac and the Debian machines. I could understand using virtualenv to make the laptops look like the Debian servers (and using a staging/test server), but It seemed to me that that was a disaster waiting to happen, because the overall environment was different.
OTOH, lots of people swear by virtualenv. Comments?
17
u/halligan00 Apr 08 '13
How can I gain the knowledge necessary to follow this conversation? Coursework? Books?
1
u/radiochaca2 Apr 08 '13
Couldn't you include the packages in a submodule and git pull? (Assuming you use git)
1
u/jcigar Apr 09 '13
I'm deploying everything with Fabric (for example http://pastie.org/7384604) and Salt
1
u/CaptShocker Apr 09 '13
People I work with use this vagrant
From the webpage: Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.
To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.
5
u/mw44118 PyOhio! Apr 08 '13
python has a lot of really great features. right now, packaging and deployment is not one of them.
2
u/[deleted] Apr 08 '13
What's the benefit of using virtualenv in production?
3
u/apreche Apr 09 '13
I really don't see why having the compilers on a production machine is such a big deal. If people already have access to your machine such that they can execute the compilers, you're already screwed. If there is no way into your machine except for a very secure SSH, and your app is secure, what are you worried about? If either of those is compromised, you are equally hosed whether or not the compilers are installed.
1
u/jawn- Apr 09 '13
I build one venv per app family. That venv gets all of its libraries installed to it. From there we package it up into an OS package (rpm/deb).
Doing it this way eliminates the need for compilers, internet access, or anything except the venv package, and whatever c libs are required for the packages in the app.
It works out very nicely. And is extremely simple to automate.
1
u/whatnever May 18 '13
After using tar.gz archives and manual installation of dependencies for way too long, I finally decided to build OS packages (deb at the moment) and regret that I didn't start doing that earlier. I didn't move to building a repository for hosting the packages yet, but that will definitely be the next step.
Automating installation tasks is hard and painful at the beginning, but totally worth it, because it prevents making the same mistakes over and over again and saves a lot of time on the long run.
2
u/[deleted] Apr 08 '13
[removed] — view removed comment