Hi django fridayers !

Today on python friday, we will discuss the model and data migration of django.

Many of you are probably developing the web app in python and are absolutely amazed about how easy it to create a database schema from your django models. Just run an manage.py syncdb and …voila ! Your brand new application has all the support needed in the backend database.

But let’s face it, this is not really practical when it comes to modify an application that is ALREADY online and have data (registered users, stored personal information, …).

The problem

How can you migrate from one model to another without loosing any data ?

The solution.

South

This python package will take care of most of the trouble caused by your model modifications.
With South, you can :

• Modify the schema of your database according to your model
• Migrate your data to make sure that you wont lose any data across the migration

Installing South

• download
• setup.py install (as root)
• edit your settings.py to activate “south” in the installed apps

A simple example

You can now type :

$> ./manage syncdb

to create the good tables in your database. South keep track of every migration in the project database.

Now, let’s consider that you have a the following model:

class Customer(models.Model):
    name = models.CharField(max_length= 60)

    def __unicode__(self):
        return self.name

The first thing you need to do is to initialize south so it can keep track of your
modification of models:

$> manage.py startmigration customer initial --initial

This will create your first migration (initial). The migration will basically creates the tables in your database.
Now, you can apply the migration.

$> manage.py migrate

Now, let’s assume that you want to change it into:

class Customer(models.Model):
    name = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text password', max_length=60)

You have to prepare the migration with the following command:

$> ./manage.py startmigration customer add_password --auto

This creates a migration file that contains your modifications. I let you try to figure out what forward and backward methods are for. Once again, when you think that your migration file is good, you can apply the changes :

$> ./manage.py migrate

At this point, your database is totally synced with your model, so let’s create some data into it:

$> ./manage.py shell
> from customer.models import Customer
> c = Customer()
> c.name="sebastien requiem"
> c.password = "secret"
> c.save()
> Customer.objects.all()
[\]

A more complex example with DATA migration

You now decide to split the field name into firstname and lastname. The natural way would be to delete the field name and create two fields named firstname and lastname. And this is Wrong for many reasons:

1. South will not understand that you want to split the name into two distinct fields
2. South won’t be able to migrate your data forward AND backward if you want to roll back

The good (and ONLY) way to do it is to so it in three steps:

1. modify your model so you add firstname and lastname fields and migrate the schema
2. create a migration that migrate your DATA (while name, name, firstname and lastname are all accessible)
3. delete the name field

Now, let’s assume that you want to change you previous model into the following :

class Customer(models.Model):
    name = models.CharField(max_length = 60)
    firstname = models.CharField(max_length = 60)
    lastname = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text pasword', length=60)

    def __unicode__(self):
        return self.firstname + ", " + self.lastname

Migrating the schema

Nothing fancy here:

$> ./manage.py startmigration customer add_first_last_name --auto
$> ./manage.py migrate

Migrate the data

$> ./manage.py startmigration customer first_last_data
now edit the migration file newly created and add the following lines :

    def forwards(self, orm):
        for customer in orm.Customer.objects.all():
            try:
                customer.first_name, customer.last_name = adopter.name.split(" ", 1)
            except ValueError:
                customer.first_name, customer.last_name = customer.name, ""
            customer.save()

    def backwards(self, orm):
        for customer in orm.Customer.objects.all():
            customer.name = customer.firstname + " " + customer.lastname
            customer.save()

As you can see, the variable “orm” is in fact the real django ORM. you can use it to write data migration in python with the usual django syntax, which is VERY convenient.

run now you data migration:

$> ./manage.py migrate

A quick django shell will ensure that we have the data present in the database where we expect it to be :

$>./manage shell
> from customer.models import Customer
> Customer.objects.all()
[]

We now just need to delete the remaining an unused field.

Change your model :

class Customer(models.Model):
    firstname = models.CharField(max_length = 60)
    lastname = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text password', length=60)

and run a simple migration with –auto

$> ./manage.py startmigration customer remove_name --auto

and apply

$> ./manage.py migrate

Voila.

References:

• The South migration tools