For those who didn’t know, we have a cow at work.

Hi django fridayers !

Today on python friday, we will discuss the model and data migration of django.

Many of you are probably developing the web app in python and are absolutely amazed about how easy it to create a database schema from your django models. Just run an manage.py syncdb and …voila ! Your brand new application has all the support needed in the backend database.

But let’s face it, this is not really practical when it comes to modify an application that is ALREADY online and have data (registered users, stored personal information, …).

The problem

How can you migrate from one model to another without loosing any data ?

The solution.

South

This python package will take care of most of the trouble caused by your model modifications.
With South, you can :

• Modify the schema of your database according to your model
• Migrate your data to make sure that you wont lose any data across the migration

Installing South

• download
• setup.py install (as root)
• edit your settings.py to activate “south” in the installed apps

A simple example

You can now type :

$> ./manage syncdb

to create the good tables in your database. South keep track of every migration in the project database.

Now, let’s consider that you have a the following model:

class Customer(models.Model):
    name = models.CharField(max_length= 60)

    def __unicode__(self):
        return self.name

The first thing you need to do is to initialize south so it can keep track of your
modification of models:

$> manage.py startmigration customer initial --initial

This will create your first migration (initial). The migration will basically creates the tables in your database.
Now, you can apply the migration.

$> manage.py migrate

Now, let’s assume that you want to change it into:

class Customer(models.Model):
    name = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text password', max_length=60)

You have to prepare the migration with the following command:

$> ./manage.py startmigration customer add_password --auto

This creates a migration file that contains your modifications. I let you try to figure out what forward and backward methods are for. Once again, when you think that your migration file is good, you can apply the changes :

$> ./manage.py migrate

At this point, your database is totally synced with your model, so let’s create some data into it:

$> ./manage.py shell
> from customer.models import Customer
> c = Customer()
> c.name="sebastien requiem"
> c.password = "secret"
> c.save()
> Customer.objects.all()
[\]

A more complex example with DATA migration

You now decide to split the field name into firstname and lastname. The natural way would be to delete the field name and create two fields named firstname and lastname. And this is Wrong for many reasons:

1. South will not understand that you want to split the name into two distinct fields
2. South won’t be able to migrate your data forward AND backward if you want to roll back

The good (and ONLY) way to do it is to so it in three steps:

1. modify your model so you add firstname and lastname fields and migrate the schema
2. create a migration that migrate your DATA (while name, name, firstname and lastname are all accessible)
3. delete the name field

Now, let’s assume that you want to change you previous model into the following :

class Customer(models.Model):
    name = models.CharField(max_length = 60)
    firstname = models.CharField(max_length = 60)
    lastname = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text pasword', length=60)

    def __unicode__(self):
        return self.firstname + ", " + self.lastname

Migrating the schema

Nothing fancy here:

$> ./manage.py startmigration customer add_first_last_name --auto
$> ./manage.py migrate

Migrate the data

$> ./manage.py startmigration customer first_last_data
now edit the migration file newly created and add the following lines :

    def forwards(self, orm):
        for customer in orm.Customer.objects.all():
            try:
                customer.first_name, customer.last_name = adopter.name.split(" ", 1)
            except ValueError:
                customer.first_name, customer.last_name = customer.name, ""
            customer.save()

    def backwards(self, orm):
        for customer in orm.Customer.objects.all():
            customer.name = customer.firstname + " " + customer.lastname
            customer.save()

As you can see, the variable “orm” is in fact the real django ORM. you can use it to write data migration in python with the usual django syntax, which is VERY convenient.

run now you data migration:

$> ./manage.py migrate

A quick django shell will ensure that we have the data present in the database where we expect it to be :

$>./manage shell
> from customer.models import Customer
> Customer.objects.all()
[]

We now just need to delete the remaining an unused field.

Change your model :

class Customer(models.Model):
    firstname = models.CharField(max_length = 60)
    lastname = models.CharField(max_length = 60)
    password = models.CharField('this is a clear text password', length=60)

and run a simple migration with –auto

$> ./manage.py startmigration customer remove_name --auto

and apply

$> ./manage.py migrate

Voila.

References:

• The South migration tools

Hi everyone,

Welcome for the first edition for python-friday ! This python friday will be dedicated to the so popular django web framework because I really have my hands in it for the time being.

Django offers a very nice User class to manage all sort of users, set rights, contact information and so on. But sometimes, you want to extend that class to add some extra information.

The problem:
You want to use your – let’s call it – Customer class as the central user class and authenticate using django’s user modules.

The solution:
Extend the user classs to add your data and configure django framework to use your new class as a middleware and authenticate on it.

Extending the class.

from django.contrib.auth.models import User, UserManager

class Customer(User):
	"""
	Model for customer.Hold administrative information
	"""
	company_name = models.CharField('company name', unique=True, max_length = 50)
	tax_number = models.CharField('tax number', max_length = 12)
	address = models.CharField('address line 1', max_length = 200)
	zip_code = models.IntegerField('zip code', max_length = 10)
	city_name = models.CharField('city name', max_length = 20)
	country = models.CharField('country', max_length = 20)
	edited_on = models.DateTimeField(auto_now=True, auto_now_add=True, editable=False)

	# We setup a manager here.. not really sure this is useful
	objects = UserManager()

	def __unicode__(self):
		return self.company_name

	# In the save function, we implement our own password
	# management. If the password is already hashed in the form
	# we just dont change anything otherwise we call the set_password()
	def save(self):
		password = ""
		r = re.compile('sha1\$.*')
		if not r.match(self.password):
			password = self.password
			self.set_password(self.password)
		User.save(self)

Now that we have extended the class, we can play a bit with it to see how things goes :

> c = Customer()
> c.username = "mms"
> c.lastname = "sauvage"
> c.firstname = "manumanu"
> c.company_name = "the Corp"
#and so on ...
and finally
> c.save()

But wait …. Haven’t we said that django has its own authentication methods ? Well, how do you want Django framework to know that it has to authenticate against the customer table and NOT against django users’ table ?

Well, django can let us redefine the authentication backend. Doing so will then change the authentication behavior the way we want to be.

Redefining djjango’s authentication backend.

First we have to declare our new authenticate backend in the settings.py. While we are in that file, we will also tell django which class to use to authenticate :

AUTHENTICATION_BACKENDS = (
    'project.auth_backend.CustomerModelBackend',
)

CUSTOM_USER_MODEL = 'accounts.Customer'

We just say here that the backend will be in the file auth_backen.py and the class will be named CustomerModelBackend.

Then create a file called auth-backend.py in your project directory and add it the following content:

from django.conf import settings
from django.contrib.auth.backends import ModelBackend
from django.core.exceptions import ImproperlyConfigured
from django.db.models import get_model

class CustomerModelBackend(ModelBackend):
	def authenticate(self, username=None, password=None):
		try:
			user = self.user_class.objects.get(username=username)
			if user.check_password(password):
				return user
		except self.user_class.DoesNotExist:
			return None

	def get_user(self, user_id):
		try:
			return self.user_class.objects.get(pk=user_id)
		except self.user_class.DoesNotExist:
			return None

	@property
	def user_class(self):
		if not hasattr(self, '_user_class'):
			self._user_class = get_model(*settings.CUSTOM_USER_MODEL.split('.', 2))
		if not self._user_class:
			raise ImproperlyConfigured('Could not get custom user model')
		return self._user_class

The authentication backend is a class with 2 methods (get_user and authenticate). This is well explained in the documentation of django framework.

Now we have a Class for our Extended users, an authentication backend that is configured through our settings.py.

Yes ! This is what we want. The most lazy of you can stop here and start coding his favorite user management webapp.

Syncdb / admin problem.

But .. wait a minute. Do you remember when you do your first syncdb ? manage.py ask you if you want to fill in an administrator. If you say yes at that point, manage.py will create a django user and now that we authenticate on Custome’s table, there is a lot of chances that the admin module will not authenticate properly (for the admin only though …).

The solution is to modify the authentication backend to authenticate on django users’ table in case of a failed authentication on customer’s table.

       def authenticate(self, username=None, password=None):
               try:
                        user = self.user_class.objects.get(username=username)
                except self.user_class.DoesNotExist:
                        try:
                                user = User.objects.get(username=username)
                        except User.DoesNotExist:
                                return None

As you can see, the code is pretty straight forward and self explanatory. The only tricky part is the “@property” but I let you dig into that yourself as you will learn a lot of things if you want to understand it completely.

Et voila !

via kings and queens