Multistage Django deployments: Part 2

What is Fabric?

Fabric is a python tool for interacting with one or many remote machines over the SSH protocol. These interactions can be grouped into tasks as regular python functions.

When you are deploying a website you generally scp the projects files to the remote server(s) and then execute some commands to prepare or install the files as necessary. These tasks should be grouped into atomic operations so that the person performing the deployment has little margin for error when performing these interactions.

My fabfile

Fabric is configured by defining your desired actions as python functions in a file called fabfile.py and here are the contents of my fabfile.

import os
import tempfile
import shutil
from datetime import datetime

config(fab_hosts=["*******.dreamhost.com"])

prod_db = {
    'host': '********',
    'name': '********',
    'user': '********',
    'password': '********'
}
test_db = {
    'host': '********',
    'name': '********',
    'user': '********',
    'password': '********'
}

def export_package():
    """
    Creates a package of the source suitable for deployment.

    Exports the parent of the current mercurial repository as a gzipped tarball
    for transferring to the server.

    """
    # Create a destination file name with instantly understandable timestamp
    tempdir = tempfile.mkdtemp()
    timestamp = datetime.now().strftime("%Y-%m-%d-%H-%M")
    archive_file = os.path.join(tempdir,
                                'sharebearcouk-%s-%%h.tar.gz' % timestamp)

    # Perform the actual export to our know file
    local('hg archive --type tgz ' + archive_file)

    # Parse the filename to get the name including revision hash
    generated_file = os.listdir(tempdir)[0]
    prefix = generated_file.split('.')[0]

    # Copy file to PWD. So we don't lose it when removing temp dir
    shutil.copy(os.path.join(tempdir, generated_file), os.getcwd())

    # Store some file names for later use
    config.archive_prefix = prefix
    config.archive_filename = generated_file

    # Cleanup everything we don't need anymore
    os.remove(os.path.join(tempdir, generated_file))
    os.rmdir(tempdir)

def backup_db():
    """Makes a backup of the production db to a folder on the server."""
    run("""
        cd db_backup && \
        mysqldump \
            --host=%(host)s \
            --user=%(user)s \
            --password=%(password)s \
            %(name)s \
        > $(date +%%F-%%H%%M%%S).sql
        """ % prod_db)

def restore_db_to_test():
    """
    Restores the latest backup to the test db.

    The definition of latest is based upon the alphabetic sorting of the
    contents of the backup directory so you need to ensure that backups are made
    with an appropriate filename.

    """
    run("""
        cd db_backup && \
        mysql \
            --host=%(host)s \
            --user=%(user)s \
            --password=%(password)s \
            %(name)s \
        < $(ls -1 | tail -n 1)
        """ % test_db)

def deploy():
    """Deploy's the site to the testing environment."""
    invoke(export_package)
    put(config.archive_filename, 'releases/file.tar.gz')
    run('cd releases && tar zxf file.tar.gz')
    run('rm releases/file.tar.gz')

    # Buildout with test settings module
    run("""
        cd releases/$(archive_prefix) && \
        python2.4 bootstrap.py && \
        bin/buildout \
            django:settings=test_settings \
            django:download-cache=/home/sharebearcouk/downloads
        """)

    # Install a clean DB from most recent backup (so we can test migrations)
    invoke(restore_db_to_test)

    # Perform DB migrations
    run('cd releases/$(archive_prefix) && bin/django migrate blog')

    # Move the symlink
    run('ln -snf ../releases/$(archive_prefix) sites/test')

def promote_test():
    """Promotes the current test site to staging."""
    # Clear static content caches
    run('cd sites/test && rm -rf public/blog public/about')

    # Buildout with prod settings module
    run("""
        cd sites/test && \
        bin/buildout \
            django:settings=prod_settings \
            django:download-cache=/home/sharebearcouk/downloads
        """)

    # Perform DB migrations
    run('cd sites/test && bin/django migrate blog')
    invoke(backup_db) # Backup immediately after migrate so we can always have a
                      # backup with the current schema

    # Move the symlink
    run('rm sites/staging && cp --no-dereference sites/test sites/staging')

def promote_staging():
    """Promotes the current staging site to production."""
    # Only symlink change needed
    run('rm sites/current && cp --no-dereference sites/staging sites/current')

This fabfile defines three commands that are intended for direct use when performing a deployment they are;

• deploy
• promote_test
• promote_staging

The remaining three commands are mainly helpers for the above three commands but may be useful to run on their own at some point.

deploy

When all code has been checked in and tested locally ready for testing on the server it is time to run the deploy command. This is the most complex stage in the deployment, running the command will;

• Export the contents of the mercurial repository into a tarball for transporting to the server
• Send the tarball to the server
• Unpack the tarball
• Perform a buildout with test environment settings to install the application
• Restore the most recent database backup from production into the testing database
• Run the South migrations
• Set the testing site symlink to point to this new installation of the application

The most notable thing here is the database restore and running migrations. This means that every time I push code to the server my migrations for this release get tested against an up to date copy of the production database. This ensures that when the time comes to run the migrations for real I know they will work.

promote_test

Once the site has been tested and found to be working in the test environment, it's time to promote the test environment to staging. For this you use the promote_test command which will;

• Clean up any files generated in the test environment
• Run buildout with the production settings (this configures the database connection settings)
• Run the South migrations
• Performs a backup of the database
• Swtiches the staging symlink to point the staging site at what was the test site

The important point here is that we take a backup of the database after we have run the migrations. This means that we always have an up to date backup of the database schema ready for testing migrations for a following iteration of the project, without having to wait until the next nightly backup.

It would also be wise to perform a backup just before performing the migrations so that in the event of a complete disaster you have a recent backup in a known good state. I just noticed this was missing when writing this post and will add it soon.

promote_staging

After you've checked the site in the staging environment to be sure there are no last minute surprises it's time to use the promote_staging command. This, rather predictably, promotes the staging site to production. As the database migrations have already been run against the production database from the staging environment there is very little left to do other than move a symlink so that apache starts to serve the staging site as the main site.

Further work

This script isn't yet complete, but it's getting there. The two main missing features are rollback on failure and a rollback command.

Comparison to Capistrano

Capistrano and its deploy recipes are designed to be generic enough to apply to many different situations, different version control systems, running from different operating systems, etc. This results in lots of abstractions and a reasonable amount of magic/uncertainty when trying to bend it to work the way you want.

In contrast the only assumption that Fabric makes is that you wish to use SSH to interact with the remote server(s). Everything else about your deployment process is completely under your control and although my fabfile might not be portable to everyone's scenario it is only 130 lines of python (including comments) and any python developer can read it and customise it to their own needs without needing to understand the internals of Fabric.

Final notes

If you're developing a Django based site and use the same server for testing and production installations of the application then feel free to grab my fabfile and customise it to your own project. If however you have a larger project with separate servers for testing and production environments you will probably have to rethink most of the work.

When working in a multi-server environment just switching a symlink will not be good enough (unless your servers all share the same storage over nfs or similar). If working in a multi-server environment, I would probably place the tarball in a commonly accessible area and download that tarball to the appropriate server as necessary. This would ensure the same "build" of files ends up on each server.