Multistage Django deployments: Part 1

15th April 2009, 2022

At Europython 2008 I gave a presentation of how we were using Capistrano for deploying the Pylons based widgets.opera.com site. When rewriting this blog in Django I decided to look again at more pythonic alternatives. This is the first part of a two part blog post, in this part you will learn the tools I use to create a sane, reproducible deployment. The second part will tie the tools together with version control to perform one click deployment.

Environment

Through a development iteration my website will pass though each of the following stages;

Development

New features and functionality begin life on my own machine. The code runs via the Django development server and sqlite is used as a database to avoid the need for running any extra servers.

Test

Once I'm satisfied that something works fine on my local machine I will then push it to a testing installation on my Dreamhost account. Here the application runs as an fcgi application, with all the caching enabled that would normally exist in production and runs against a backup of the production database so that I can test the functionality still works with the existing data.

Staging

Once I have a version in testing that I'm sure works well and isn't going to destroy my data I then promote my code to a staging installation of the site. This staging version of the application is configured against the real production database so I can perform a final test that everything is going to work fine before I flick the switch to make it live.

Production

Once satisfied that everything looks good in staging the site is then promoted to production in one smooth operation, no users will notice any downtime or broken functionality during the upgrade.

Environment Management

When developing python applications or deploying several versions to the same machine, it is considered bad practice to install the application in the global site-packages. This means that you need to install your python packages elsewhere and modify the PYTHONPATH before running the application. There are two tools that try and automate this process, these are Virtualenv and zc.buildout.

Although both tools produce a similar result, with Virtualenv you need to use setuptools to install your application and its dependencies into the environment, whereas zc.buildout performs everything in a single step along with adding the possibility to perform other tasks, such as generating configuration files.

For my blog I have the following buildout config

[buildout]
parts = django

[django]
recipe = djangorecipe
version = 1.0.2
project = sharebear
settings = settings
eggs =
    docutils
    staticgenerator
    flup
    south
fcgi = true

This file is all I need to get started with my Django project. After running buildout I have all of my dependencies downloaded (defined in the eggs argument) and there is a bin/ folder created which contains two files. The first of these files is an executable script named django that I can use to run all of the normal django management commands (runserver, syncdb, createapp, etc.) with a correctly configured python path. The other file is django.fcgi an fcgi file suitable for running my Django app using an fcgi server.

Database Migrations

In my opinion the most important part of a database oriented project is the database schema and the data within it. Unfortunately it seems that many projects still run without any safe method of testing and performing changes to the database schema. Most projects still seem to rely on a human being running the SQL scripts, that have received limited testing, against the databases as necessary.

The solution to this problem is to make use of a database migration tool. Migration tools allow you to define the steps necessary to upgrade the database to the state it needs to be in and also keeps track of which migrations have already been run against a particular database instance so you don't run the risk of running the same script twice against the same database. Migration tools based on scripting languages can also allow you to do advanced manipulation of data in the database so that you fix corrupt data caused by software bugs as well as doing simple schema changes such as adding columns and indexes.

There are several database migration tools targeted at Django projects but for my blog I have chosen to try out South, which seems to have the feature-set I desire and is under active development. I'm not going to go into detail about how to use South but here is an example migration from my project.

from south.db import db
from django.db import models

class Migration:
    def forwards(self):
        # Adding field 'Entry.content_html'
        db.add_column('blog_entry', 'content_html',
                      models.TextField(editable=False, blank=True))

    def backwards(self):
        # Deleting field 'Entry.content_html'
        db.delete_column('blog_entry', 'content_html')

One additional feature of migration tools as you can see here is that, not only can you migrate a database forwards but you can also rollback a migration if you need to back out of a software deployment. This is a situation that many manual migration strategies fail to consider.

Wrapping up

Now you've had a brief introduction to the tools that help me organise my website deployment. In my next post I'll share with you details of how I use Fabric to tie everything together into a single action for deployment.

This Blog represents my personal views and experiences which are not necessarily the same as those of my employer.