Hi django fridayers !

Today on python friday, we will discuss the model and data migration of django.

Many of you are probably developing the web app in python and are absolutely amazed about how easy it to create a database schema from your django models. Just run an manage.py syncdb and …voila ! Your brand new application has all the support needed in the backend database.

But let’s face it, this is not really practical when it comes to modify an application that is ALREADY online and have data (registered users, stored personal information, …).

The problem

How can you migrate from one model to another without loosing any data ?

The solution.

South

This python package will take care of most of the trouble caused by your model modifications.
With South, you can :

• Modify the schema of your database according to your model
• Migrate your data to make sure that you wont lose any data across the migration

Installing South

• download
• setup.py install (as root)
• edit your settings.py to activate “south” in the installed apps

A simple example

You can now type :

$> ./manage syncdb

to create the good tables in your database. South keep track of every migration in the project database.

Now, let’s consider that you have a the following model:

class Customer(models.Model):
    name = models.CharField(max_length= 60)

    def __unicode__(self):
        return self.name

The first thing you need to do is to initialize south so it can keep track of your
modification of models:

$> manage.py startmigration customer initial --initial

This will create your first migration (initial). The migration will basically creates the tables in your database.
Now, you can apply the migration.

$> manage.py migrate

Now, let’s assume that you want to change it into:

class Customer(models.Model):
    name = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text password', max_length=60)

You have to prepare the migration with the following command:

$> ./manage.py startmigration customer add_password --auto

This creates a migration file that contains your modifications. I let you try to figure out what forward and backward methods are for. Once again, when you think that your migration file is good, you can apply the changes :

$> ./manage.py migrate

At this point, your database is totally synced with your model, so let’s create some data into it:

$> ./manage.py shell
> from customer.models import Customer
> c = Customer()
> c.name="sebastien requiem"
> c.password = "secret"
> c.save()
> Customer.objects.all()
[\]

A more complex example with DATA migration

You now decide to split the field name into firstname and lastname. The natural way would be to delete the field name and create two fields named firstname and lastname. And this is Wrong for many reasons:

1. South will not understand that you want to split the name into two distinct fields
2. South won’t be able to migrate your data forward AND backward if you want to roll back

The good (and ONLY) way to do it is to so it in three steps:

1. modify your model so you add firstname and lastname fields and migrate the schema
2. create a migration that migrate your DATA (while name, name, firstname and lastname are all accessible)
3. delete the name field

Now, let’s assume that you want to change you previous model into the following :

class Customer(models.Model):
    name = models.CharField(max_length = 60)
    firstname = models.CharField(max_length = 60)
    lastname = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text pasword', length=60)

    def __unicode__(self):
        return self.firstname + ", " + self.lastname

Migrating the schema

Nothing fancy here:

$> ./manage.py startmigration customer add_first_last_name --auto
$> ./manage.py migrate

Migrate the data

$> ./manage.py startmigration customer first_last_data
now edit the migration file newly created and add the following lines :

    def forwards(self, orm):
        for customer in orm.Customer.objects.all():
            try:
                customer.first_name, customer.last_name = adopter.name.split(" ", 1)
            except ValueError:
                customer.first_name, customer.last_name = customer.name, ""
            customer.save()

    def backwards(self, orm):
        for customer in orm.Customer.objects.all():
            customer.name = customer.firstname + " " + customer.lastname
            customer.save()

As you can see, the variable “orm” is in fact the real django ORM. you can use it to write data migration in python with the usual django syntax, which is VERY convenient.

run now you data migration:

$> ./manage.py migrate

A quick django shell will ensure that we have the data present in the database where we expect it to be :

$>./manage shell
> from customer.models import Customer
> Customer.objects.all()
[]

We now just need to delete the remaining an unused field.

Change your model :

class Customer(models.Model):
    firstname = models.CharField(max_length = 60)
    lastname = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text password', length=60)

and run a simple migration with –auto

$> ./manage.py startmigration customer remove_name --auto

and apply

$> ./manage.py migrate

Voila.

References:

• The South migration tools

As a new freelancer, I decided to create my own business cards using moo services. This website allow you to create collectors business cards. The principle is simple, you can upload one image per card so the Verso of the card will be different each time.

My problem : How to create a business card that  match my website layout and make it uniq at the same time ?

This is when Python and Inkscape came to rescue me. Inkscape is a well know (GPL’ed) SVG graphic editor. If you have never tried it, consider doing it. It is amazing what this soft can produce within a couple of minutes. Beside Inkscape, the website commandlinfu.com is a (cool) website providing Unix Commands and explanations of what they do (be careful some commands may be dangerous or invalid though).

So I had an idea. Why not using the website commandlinefu, extract commands from the RSS feed and generate images for my Collector Business cards ?

How ?

Inkscape support plugins and offer a nice python module to help you to create SVG objects.

Let’s go !

Inkscape expect a couple of things from the module structure.

#!/usr/bin/env python

# These two lines are only needed if you don't put the script directly into
# the installation directory
import sys
from xml.dom import minidom
import urllib
import re

# This import is for MAc os X only. In GNU systems it should
# be somewhere in /usr/share/inkscape/extensions

sys.path.append('/Applications/Inkscape.app/Contents/Resources/extensions')

# We will use the inkex module with the predefined Effect base class.
import inkex
# The simplestyle module provides functions for style parsing.
from simplestyle import *

class BusinessCard(inkex.Effect)
    """
    Example Inkscape effect extension. It reads an RSS feed and create objects according to the content.
    """
    def __init__(self):
        """
        Constructor.
        Defines the "--what" option of a script.
        """
        # Call the base class constructor.
        inkex.Effect.__init__(self)

        # Define string option "--what" with "-w" shortcut and default value "World".
        self.OptionParser.add_option('-w', '--what', action = 'store',
        type = 'integer', dest = 'cards_number', default = '10',
        help = 'How many images do you want to create?')

The optionparser is required by Inkscape as it will pass some arguments to the script. We could, for example use this argument to know how many images we want to generate. In our case, the argument will be stored in the variable call cards_number. I let you have a closer look to this Wiki Page for any further explanation.

Now that we have the basis for creating an Inkscape plugin, we should define the main actions that this plugin is gonna do:

1. Read the RSS feed
2. Extract the usefull information (and filter)
3. Create the SVG objects
4. Add them to the SVG Layer
5. Loop to 1

Let’s use Python wonderful XML parser to read our RSS feed

    # Open the XML ressource and read it
    socket = urllib.urlopen("http://feeds2.feedburner.com/Command-line-fu?format=xml");
    # Create an object representing the XML structure
    self.xmldoc = minidom.parse(socket).documentElement
    socket.close();

Create the function that will do the loop

    def effect(self):
        """
        Effect behaviour.
        Overrides base class' method and inserts "Hello World" text into SVG document.
        """
        # Get script's "--what" option value.
        cards_number = self.options.cards_number

        # Get access to main SVG document element and get its dimensions.
        svg = self.document.getroot()

        # Create a new layer.
        layer = inkex.etree.SubElement(svg, 'g')
        layer.set(inkex.addNS('label', 'inkscape'), 'Hello %s Layer' % (what))
        layer.set(inkex.addNS('groupmode', 'inkscape'), 'layer')

        # Create text element from the RSS feed
        itemlist = self.xmldoc.getElementsByTagName('item')

        #we will loop so better initialize variables
        item_x = 0
        item_y = 0

        # in the RSS feed, the iformation we want in contained in between
        # the <code></code> markup. Let's use a regexp to get it.
        regexp = re.compile('<code>.*</code>');

        # Note that here, we should use the cards_number instead of reading
        # the complete rss feed.
        for node in itemlist:
            title = node.getElementsByTagName('title')[0].firstChild.data
            command = regexp.search(node.getElementsByTagName('description')[0].firstChild.data)

            # Get ride of the markup
            command = command.group()
            command = command.replace("<code>", "");
            command = command.replace("</code>", "");

            # filter in case the comman is too long...
            if len(command) > 40:
                continue

Ok so now we have to start creating the objects. For this, Inkscape offer us a nice way to do it. Use the inkex extension. the function inkex.addNS is usefull to create everything you need in SVG.

        #Design the background grey business card
        background = inkex.etree.Element(inkex.addNS('rect', 'svg'))
        background.set('x', str(item_x))
        background.set('y', str(item_y))
        background.set('width', str(520))
        background.set('height', str(348))
        background.set('fill', 'rgb(32, 32, 32)')

        #Our lovely ornge square will fit here
        square = inkex.etree.Element(inkex.addNS('rect', 'svg'))
        square.set('x', str(item_x + 30))
        square.set('y', str(item_y + 30))
        square.set('width', str(84))
        square.set('height', str(84))
        square.set('rx', str(20))
        square.set('ry', str(20))
        square.set('fill', 'rgb(224, 121, 26)')

        #Insert the "K" letter. K stands for Kolios
        K_logo = inkex.etree.Element(inkex.addNS('text','svg'))
        K_logo.text = ("K")
        # Set text position to center of document.
        K_logo.set('x', str(item_x + 61))
        K_logo.set('y', str(item_y + 95))
        K_logo.set('fill', "white")

        # Center text horizontally with CSS style.
        style = {'text-align' : 'left',
            'font-family': 'MammaGamma',
            'font-size': '56'
        }
        K_logo.set('style', formatStyle(style))

        #Insert the Unix Tip
        tip = inkex.etree.Element(inkex.addNS('text','svg'))
        tip.text = (title)
        # Set text position to center of document.
        tip.set('x', str(item_x + 40))
        tip.set('y', str(item_y + 200))
        tip.set('fill', 'rgb(170, 170, 170)')

        # Center text horizontally with CSS style.
        style = {'text-align' : 'left',
            'font-family': 'terminal',
            'font-style': 'italic'
        }
        tip.set('style', formatStyle(style))

        #Insert the Unix Command
        unix = inkex.etree.Element(inkex.addNS('text','svg'))
        unix.text = (command)
        # Set text position to center of document.
        unix.set('x', str(item_x + 30))
        unix.set('y', str(item_y + 180))
        unix.set('fill', 'rgb(224, 121, 26)')

        # Center text horizontally with CSS style.
        style = {'text-align' : 'left',
            'font-family': 'Monaco',
            'font-size': '20'
        }
        unix.set('style', formatStyle(style))

        # Connect elements together.
        layer.append(background)
        layer.append(square)
        layer.append(K_logo)
        layer.append(kolios)
        layer.append(slogan)
        layer.append(unix)
        layer.append(tip)
        item_y = item_y + 360

    # Create effect instance and apply it.
    effect = BusinessCard()
    effect.affect()

You can notice that we use css-style for the text. This is possible with the use of the module simplestyle.

Basically, you create an object and THEN add it to the layer. Pay attention that the order you add objects MATTERS for the rendering depth. In this case the background will be rendered first, then the square… and so on. All type of objects can be created and the entire list is available at the W3c webpage.

But how do I add my plugin to Inkscape ?

Well, last thing but not the least. You python script should be placed in .inkscape/extensions with a description file

<inkscape-extension>
    <_name>Business Card</_name>
    <id>org.ekips.filter.business_card</id>
    <dependency type="executable" location="extensions">card.py</dependency>
    <dependency type="executable" location="extensions">inkex.py</dependency>
    <param name="what" type="integer" _gui-text="How many business cards ?">10</param>
    <effect>
    <object-type>all</object-type>
    <effects-menu>
        <submenu _name="Examples"/>
    </effects-menu>
    </effect>
    <script>
        <command reldir="extensions" interpreter="python">card.py</command>
    </script>
</inkscape-extension>

The file should be named with a .inx extension. I won’t explain this as this is pretty obvious.

At last, you can take a llok at the final output. The first picture is NOT generated but … it gives a good example.