While considering a reorganisation of your Django programme, you may find yourself in need of relocating a Django model. Although while Django migrations make it possible to copy a model from one app to another, the process is complex and error-prone.
Copying data, modifying constraints, and renaming objects when transferring models across Django projects is often a time-consuming and laborious process. Django’s object-relational mapper (ORM) does not provide automated detection and migration procedures because of these complexities. Alternatively, Django’s developers may use the ORM’s supplied set of low-level migration operations to implement the process themselves in the migration framework. What you’ll study here in this guide is:
-
How to
move a Django model
from one app to another -
How to use
advanced features
of the Django migration command line interface (CLI), such as
sqlmigrate
,
showmigrations
, and
sqlsequencereset
-
How to produce and inspect a
migration plan
-
How to make a migration reversible and how to
reverse migrations
-
What
introspection
is and how Django uses it in migrations
This lesson will provide you with the knowledge to decide which method of migrating a Django model across applications is most suitable for your needs.
Here’s an Application Case Study: Porting a Django Model to Another Framework
In this guide, you will develop a retail app from scratch. You’ll have two Django
apps:
-
catalog
: This app is for storing data on products and product categories. -
sale
: This app is for recording and tracking product sales.
When you’ve built up the aforementioned applications, you’ll go over to the product app, where you’ll transfer the Django model named Product. These are some of the challenges you’ll encounter:
challenges:
- The model being moved has foreign key relationships with other models.
- Other models have foreign key relationships with the model being moved.
- The model being moved has an index on one of the fields (besides the primary key).
These tests were motivated by actual refactoring activities. After you’ve conquered those obstacles, you’ll be able to begin formulating a migration strategy tailored to your unique scenario.
Assemble: Set the Stage
You need to establish the starting point of your project before you can begin making any changes. Nevertheless, although Django 3 and Python 3.8 are used in this lesson, the principles presented are universal and may be applied to other frameworks and versions of the programming language.
Prepare a sandbox for Python
Start by creating a new file that will serve as your virtual
directory:
$ mkdir django-move-model-experiment
$ cd django-move-model-experiment
$ python -m venv venv
Python Virtual Environments: A Primer provides detailed instructions for setting up your own sandbox.
Start a Django Project
A virtual machine may be activated and software installed by entering the appropriate commands into the terminal.
Django:
$ source venv/bin/activate
$ pip install django
Collecting django
Collecting pytz (from django)
Collecting asgiref~=3.2 (from django)
Collecting sqlparse>=0.2.2 (from django)
Installing collected packages: pytz, asgiref, sqlparse, django
Successfully installed asgiref-3.2.3 django-3.0.4 pytz-2019.3 sqlparse-0.3.1
You may begin developing your Django application immediately. Create an experiment named “django-move-model-experiment” with the use of the django-admin startproject command.
:
$ django-admin startproject django-move-model-experiment
$ cd django-move-model-experiment
Django has generated new files and folders once this command has been executed. Check visit Beginning a Django Project if you want to learn more about kickstarting a new Django project.
Django app development
Build a shop-specific app using the new Django project you just created.
catalogue:
$ python manage.py startapp catalog
The following pieces of equipment need to be included in the new catalogue.
app:
# catalog/models.py
from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
Your catalogue app now has working Category and Product models. You’re ready to put your catalogue to use and start making sales. Developing another another app for
sales:
$ python manage.py startapp sale
To the new sale, please include the following Sale model.
app:
# sale/models.py
from django.db import models
from catalog.models import Product
class Sale(models.Model):
created = models.DateTimeField()
product = models.ForeignKey(Product, on_delete=models.PROTECT)
Take note that a ForeignKey is used to link the Sale model to the Product model.
Develop and implement preliminary migrations
Just make some migrations, and then apply them, and you’ll be all set.
them:
$ python manage.py makemigrations catalog sale
Migrations for 'catalog':
catalog/migrations/0001_initial.py
- Create model Category
- Create model Product
Migrations for 'sale':
sale/migrations/0001_initial.py
- Create model Sale
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, sale, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying catalog.0001_initial... OK
Applying sale.0001_initial... OK
Applying sessions.0001_initial... OK
If you want to learn more about Django migrations, I recommend reading Django Migrations: A Primer. Now that your migrations are complete, you may begin making test data.
Produce Some Sample Data
Django shell must be activated from the terminal to make the migration situation as realistic as feasible.
window:
$ python manage.py shell
It follows that you must develop the following
objects:
>>>
>>> from catalog.models import Category, Product
>>> clothes = Category.objects.create(name='Clothes')
>>> shoes = Category.objects.create(name='Shoes')
>>> Product.objects.create(name='Pants', category=clothes)
>>> Product.objects.create(name='Shirt', category=clothes)
>>> Product.objects.create(name='Boots', category=shoes)
You split the content into the “Clothes” and “Shoes” sections. Next you created three new products: pants, a shirt, and boots for the Shoes category.
Congratulations! Initiating your project to its starting point is now complete. Planning the refactoring begins at this point in a real-world situation. From here, we’ll go on to the first of three different methods discussed in this course.
Data Transfer to a Fresh Django Installation
You’re going to begin with a lengthy
road:
- Create a new model
- Copy the data to it
- Drop the old table
You should be aware of the potential drawbacks of this strategy. In the next chapters, you’ll learn everything about them.
Make a Brand-New Template
Make a fresh app for your product first. Type enter the following commands into your terminal:
command:
$ python manage.py startapp product
When you execute this command, you’ll see that a new folder named “product” has been created to the project.
Adding the new app to the list of INSTALLED APPS in Django’s settings.py will register it with your current Django project.
:
--- a/store/store/settings.py
+++ b/store/store/settings.py
@@ -40,6 +40,7 @@ INSTALLED_APPS = [
'catalog',
'sale',
+ 'product',
]
MIDDLEWARE = [
Django has been updated to include your latest product app. The next step is to build a product model in the fresh app. The catalogue has the necessary code, which may be copied.
app:
# product/models.py
from django.db import models
from catalog.models import Category
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
So, now that the model has been created, the next step is to attempt to build migrations for it.
it:
$ python manage.py makemigrations product
SystemCheckError: System check identified some issues:
ERRORS:
catalog.Product.category: (fields.E304) Reverse accessor for 'Product.category' clashes with reverse accessor for 'Product.category'.
HINT: Add or change a related_name argument to the definition for 'Product.category' or 'Product.category'.
product.Product.category: (fields.E304) Reverse accessor for 'Product.category' clashes with reverse accessor for 'Product.category'.
HINT: Add or change a related_name argument to the definition for 'Product.category' or 'Product.category'.
Django encountered a problem because it discovered many models that shared the same category field’s reverse accessor. This is due to a collision between two models titled “Product,” both of which make reference to the Category model.
Django automatically generates a reverse accessor in the associated model when you include foreign keys in your model. Here, goods serves as the backwards accessor. The category.products example shows how the reverse accessor may be used to go to a related object.
To avoid losing the new model in the process of merging the two, edit catalog/models.py and delete the reverse accessor from the older model.
:
--- a/store/catalog/models.py
+++ b/store/catalog/models.py
@@ -7,4 +7,4 @@ class Category(models.Model):
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
- category = models.ForeignKey(Category, on_delete=models.CASCADE)
+ category = models.ForeignKey(Category, on_delete=models.CASCADE, related_name='+')
Specifying a related name for a reverse accessor is possible through the related name property. In this case, the value + is used to prevent Django from generating a reverse accessor.
Create a catalogue migration right now.
app:
Do not initiate this migration at this time,
$ python manage.py makemigrations catalog
Migrations for 'catalog':
catalog/migrations/0002_auto_20200124_1250.py
- Alter field category on product
. It’s possible that after this change is implemented, code that now makes use of the inverted accessor will no longer function properly.
Try to produce the migrations for the new product now that there is no longer a collision between the reverse accessors.
app:
$ python manage.py makemigrations product
Migrations for 'product':
product/migrations/0001_initial.py
- Create model Product
Great! The next phase may begin now that you’re ready.
Transfer Information to the Latest Model
You’ve already taken the first step in migrating a product model by creating a new product app with a Product model that’s a carbon copy of the target model. The data from the previous model must now be imported into the new one.
Execute the following command from your
terminal:
$ python manage.py makemigrations product --empty
Migrations for 'product':
product/migrations/0002_auto_20200124_1300.py
Replace the old data copying procedure with one in the new migration file.
table:
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('product', '0001_initial'),
]
operations = [
migrations.RunSQL("""
INSERT INTO product_product (
id,
name,
category_id
)
SELECT
id,
name,
category_id
FROM
catalog_product;
""", reverse_sql="""
INSERT INTO catalog_product (
id,
name,
category_id
)
SELECT
id,
name,
category_id
FROM
product_product;
""")
]
It is necessary to utilise the specialised RunSQL migration command in order to carry out SQL in a migration. As the first parameter, the SQL to execute will be evaluated. In addition, the reverse sql option specifies a course of action to execute in order to undo the migration.
Finding a way to undo a migration is useful if a mistake is made and you need to undo the change. All standard migration operations may be undone. When you add a new field to a form, you can delete it just as easily. Dropping an existing table is the antithesis of adding a new one. If anything goes wrong during a run of RunSQL, it’s recommended that you provide reverse SQL as an additional argument.
Data is transferred from product product to catalog product during this forward migration procedure. The inverse procedure will move data from catalog product into product product. If anything goes wrong during the migration, you may undo the process by giving Django access to the corresponding reverse action.
You haven’t even completed half of the migration yet. Yet, a lesson may be drawn from this, so feel free to put it into practise.
migrations:
$ python manage.py migrate product
Operations to perform:
Apply all migrations: product
Running migrations:
Applying product.0001_initial... OK
Applying product.0002_auto_20200124_1300... OK
Make an effort to generate original ideas before moving on to the
product:
>>>
>>> from product.models import Product
>>> Product.objects.create(name='Fancy Boots', category_id=2)
Traceback (most recent call last):
File "/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "product_product_pkey"
DETAIL: Key (id)=(1) already exists.
Django generates a sequence in the database to generate unique IDs for new objects when an auto-incrementing primary key is used. Take the fact that you failed to include the new item’s identifier as an example. In most cases, you should let the database assign unique primary keys in a predetermined sequence on its own without providing any identifying information. Even though ID 1 already existed in the new database, it was assigned to the new product.
What happened, then? You forgot to synchronise the sequence when you transferred the information to the new table. Another tool available to Django admins, sqlsequencereset may be used to synchronise the sequence. This command generates a script that utilises the current data in the table to determine the next value in the series. This is a common command for using existing data to populate new models.
In order to generate a script to synchronise the
sequence:
$ python manage.py sqlsequencereset product
BEGIN;
SELECT setval(pg_get_serial_sequence('"product_product"','id'), coalesce(max("id"), 1), max("id") IS NOT null)
FROM "product_product";
COMMIT;
The command creates database-specific script. PostgreSQL is being used as the database system here. The current value of the sequence is changed by the script to the next expected value of the sequence, which is the maximum ID in the database plus one.
Finalize by include the fragment in the information
migration:
--- a/store/product/migrations/0002_auto_20200124_1300.py
+++ b/store/product/migrations/0002_auto_20200124_1300.py
@@ -22,6 +22,8 @@ class Migration(migrations.Migration):
category_id
FROM
catalog_product;
+
+ SELECT setval(pg_get_serial_sequence('"product_product"','id'), coalesce(max("id"), 1), max("id") IS NOT null) FROM "product_product";
""", reverse_sql="""
INSERT INTO catalog_product (
id,
After you apply the migration, the snippet will synchronise the sequence, so resolving the previous sequence problem.
There is some confusion in your code as a result of your detour to study synchronising sequences. Clean up the Django database by erasing the information from the updated model.
shell:
>>>
>>> from product.models import Product
>>> Product.objects.all().delete()
(3, {'product.Product': 3})
Because the copied data has been removed, the transfer may be undone. Reverting a migration requires a second migration to an earlier
migration:
$ python manage.py showmigrations product
product
[X] 0001_initial
[X] 0002_auto_20200124_1300
$ python manage.py migrate product 0001_initial
Operations to perform:
Target specific migration: 0001_initial, from product
Running migrations:
Rendering model states... DONE
Unapplying product.0002_auto_20200124_1300... OK
To begin, you ran showmigrations to see all of the app product migrations. Seeing the results, it’s clear that both migrations were carried out. You rolled back to migration 0001 initial after performing migration 0002 auto 20200124 1300.
Second migration is no longer reported as cancelled if you run showmigrations again.
applied:
$ python manage.py showmigrations product
product
[X] 0001_initial
[ ] 0002_auto_20200124_1300
As the box is now empty, we know that the second migration was unsuccessful. With a blank slate in front of you, it’s time to do the migrations using the
code:
$ python manage.py migrate product
Operations to perform:
Apply all migrations: product
Running migrations:
Applying product.0002_auto_20200124_1300... OK
In this case, the migration was effectively implemented. Verify that a new Product can be created in the Django
shell:
>>>
>>> from product.models import Product
>>> Product.objects.create(name='Fancy Boots', category_id=2)
<Product: Product object (4)>
Amazing! You’ve earned this next level because of all your effort.
Bring Over the New Model’s Foreign Keys
At the present, the old table serves as a foreign key for other tables. In order to safely delete the obsolete model, you must first update any dependent models to use the successor.
Sale, a model used in the sale app, is one such example. Make that the Sell model’s foreign key refers to the updated Product.
model:
--- a/store/sale/models.py
+++ b/store/sale/models.py
@@ -1,6 +1,6 @@
from django.db import models
-from catalog.models import Product
+from product.models import Product
class Sale(models.Model):
created = models.DateTimeField()
Iterate the migration and deploy
it:
$ python manage.py makemigrations sale
Migrations for 'sale':
sale/migrations/0002_auto_20200124_1343.py
- Alter field product on sale
$ python manage.py migrate sale
Operations to perform:
Apply all migrations: sale
Running migrations:
Applying sale.0002_auto_20200124_1343... OK
The updated Product model in the product app is now being referred to by the Sale model. The new model has no constraints being violated as a result of your diligent data transfer efforts.
Eliminate the Previous Template
All traces of the prior Product model have been eradicated. The discontinued product may finally be taken out of circulation.
app:
--- a/store/catalog/models.py
+++ b/store/catalog/models.py
@@ -3,8 +3,3 @@ from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
-
-
-class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
- category = models.ForeignKey(Category, on_delete=models.CASCADE, related_name='+')
You can create a migration, but you shouldn’t use it just yet.
:
$ python manage.py makemigrations
Migrations for 'catalog':
catalog/migrations/0003_delete_product.py
- Delete model Product
The following should be added to the process to delay the deletion of the previous model until after the data has been replicated.
dependency:
--- a/store/catalog/migrations/0003_delete_product.py
+++ b/store/catalog/migrations/0003_delete_product.py
@@ -7,6 +7,7 @@ class Migration(migrations.Migration):
dependencies = [
('catalog', '0002_auto_20200124_1250'),
+ ('sale', '0002_auto_20200124_1343'),
]
operations = [
It is crucial to include this reliance. If you skip this step, you may have disastrous results, such as data loss. Check out Diving Further Into Django Migrations to learn more about migration files and inter-migration dependencies. A time and date stamp of its creation may be seen in the migration’s name. These identifiers will be changed if you’re using your own code to follow along.
Now that the dependence has been inserted, the
migration:
$ python manage.py migrate catalog
Operations to perform:
Apply all migrations: catalog
Running migrations:
Applying catalog.0003_delete_product... OK
At of this moment, the transfer is finalised. You have successfully created a new model and copied the data from the Product model in the catalogue app to the new product app.
Additional benefit: Turn Back the Migrations
Django migrations may be undone, which is one of its many advantages. When talking about migration, what does it imply if it can be undone? In case of an error during migration, the database may be restored to its previous condition.
You may recall passing the reverse sql parameter to RunSQL in the past. Finally, the benefits begin to show.
Use a clean installation for all migrations.
database:
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, product, sale, sessions
Running migrations:
Applying product.0001_initial... OK
Applying product.0002_auto_20200124_1300... OK
Applying sale.0002_auto_20200124_1343... OK
Applying catalog.0003_delete_product... OK
Now, using the unique zero term, invert them all.
:
$ python manage.py migrate product zero
Operations to perform:
Unapply all migrations: product
Running migrations:
Rendering model states... DONE
Unapplying catalog.0003_delete_product... OK
Unapplying sale.0002_auto_20200124_1343... OK
Unapplying product.0002_auto_20200124_1300... OK
Unapplying product.0001_initial... OK
The database has been reset to its original configuration. If an error is found after this version has been deployed, it may be rolled back.
Take Care of Exceptions
Certain Django features could need extra care when you’re transferring models across apps. Particularly, attention must be used while using generic relations and when adding or altering database constraints.
Adjusting Limits
Adding limitations to already-populated tables is risky business in a production environment. Before a database constraint can be implemented, it must pass verification. The database may be unable to do further actions on the table while the verification is in progress because it will have acquired a lock on the table.
In order to ensure that the new data is genuine, certain constraints, such NOT NULL and CHECK, may need a complete scan of the database. Validation of additional restrictions, such FOREIGN KEY, might be time-consuming if the table being referenced is large.
Processing Relationships Based on Generics
Whenever you use
If you need to work with generic relations, then that’s one more thing to do. When referencing a record in a database, generic relations employ both the main key and the content type ID of the model.
Any sample desk will do. Since the content type ID is different between the old and new models, generic connections may fail. As the database does not always ensure the integrity of generic foreign keys, this may go unreported.
For general foreign phrases, you may either
keys:
- Update the content type ID of the new model to that of the old model.
- Update the content type ID of any referencing tables to that of the new model.
Test thoroughly before releasing to the live environment, whatever path you choose.
Implications of Data Duplication: Benefits and Disadvantages
There are benefits and drawbacks to replicating a Django model and using it in another project. Only a few of the benefits of this are listed below:
approach:
-
It’s supported by the ORM
: Performing this transition using built-in migration operations guarantees proper database support. -
It’s reversible
: Reversing this migration is possible if necessary.
Several potential drawbacks of this approach are outlined below.
approach:
-
It’s slow
: Copying large amounts of data can take time. -
It requires downtime
: Changing the data in the old table while it’s being copied to the new table will cause data loss during the transition. To prevent this from happening, downtime is necessary. -
It requires manual work to sync the database
: Loading data to existing tables requires syncing sequences and generic foreign keys.
This method of transferring Django models to other applications is quite time-consuming, as you’ll discover in the coming sections.
To Sum It Up Quickly: Use the New Django Model as an Example of the Old Table
The prior method included simply copying all the information to the new table. The migration required a period of downtime, and its completion time would be proportional to the volume of data being copied.
What if, rather than recreating the data, you referenced the original table in the new model?
Make a Brand-New Template
The models will be updated in bulk this time, and Django will automatically construct the necessary migrations.
As a first step, please delete the Product template from the database.
app:
--- a/store/catalog/models.py
+++ b/store/catalog/models.py
@@ -3,8 +3,3 @@ from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
-
-
-class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
- category = models.ForeignKey(Category, on_delete=models.CASCADE)
It seems that you have deleted the Product model from the cataloguing app. Include the new product into the Product model now.
app:
# store/product/models.py
from django.db import models
from catalog.models import Category
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
Because the Product model is now part of the product app, you may update any links that previously pointed to the obsolete Product model. The foreign key in sales must be updated to refer to product.Product in this scenario.
:
--- a/store/sale/models.py
+++ b/store/sale/models.py
@@ -1,6 +1,6 @@
from django.db import models
-from catalog.models import Product
+from product.models import Product
class Sale(models.Model):
created = models.DateTimeField()
There’s one more tweak to the new Product that must be made before you can go on to creating the migrations.
model:
--- a/store/product/models.py
+++ b/store/product/models.py
@@ -5,3 +5,6 @@ from catalog.models import Category
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
+
+ class Meta:
+ db_table = 'catalog_product'
The db table Meta option is available on Django model types. While using this feature, you may override Django’s default table name with one of your own choosing. When implementing the ORM on an already-existing database schema whose table names don’t conform to Django’s naming convention, this is the most often used option.
In this situation, you’ll need to edit the table’s name in the product app so that it points to the catalogue app’s preexisting table.
In order to finalise the framework, produce
migrations:
$ python manage.py makemigrations sale product catalog
Migrations for 'catalog':
catalog/migrations/0002_remove_product_category.py
- Remove field category from product
catalog/migrations/0003_delete_product.py
- Delete model Product
Migrations for 'product':
product/migrations/0001_initial.py
- Create model Product
Migrations for 'sale':
sale/migrations/0002_auto_20200104_0724.py
- Alter field product on sale
Create a strategy for the upcoming migration with the help of the —plan option.
:
$ python manage.py migrate --plan
Planned operations:
catalog.0002_remove_product_category
Remove field category from product
product.0001_initial
Create model Product
sale.0002_auto_20200104_0724
Alter field product on sale
catalog.0003_delete_product
Delete model Product
The command’s output specifies the sequence in which Django will apply the migrations.
Remove Database Updates
The primary advantage of this method is that it requires no modifications to the database itself, simply the code. The unique migration procedure SeparateDatabaseAndState may be used to avoid any modifications to the database.
During a migration, you may alter the steps Django takes by using the SeparateDatabaseAndState setting. To learn more about using SeparateDatabaseAndState, check out the guide on How to Build an Index in Django Without Downtime.
Examining the resulting migrations from Django will reveal that the framework replaces the old model with a new one. These migrations will result in the table being created empty and the data being lost. To prevent this, ensure that no database modifications are made by Django during the conversion.
Wrapping each migration activity with a SeparateDatabaseAndState operation will prevent any database modifications from occurring. Setting db operations to an empty list will prevent Django from making any modifications to the database.
Since you want to use the old table again, you must stop Django from deleting it. Django will first remove any fields that need the model before removing the model itself. First, make sure Django doesn’t accidentally delete the sale product foreign key.
:
--- a/store/catalog/migrations/0002_remove_product_category.py
+++ b/store/catalog/migrations/0002_remove_product_category.py
@@ -10,8 +10,14 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.RemoveField(
- model_name='product',
- name='category',
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.RemoveField(
+ model_name='product',
+ name='category',
+ ),
+ ],
+ # You're reusing the table, so don't drop it
+ database_operations=[],
),
]
Django is free to remove the model now that it has dealt with all of the associated objects. Keep Django from erasing the Product table if you really want to maintain it.
it:
--- a/store/catalog/migrations/0003_delete_product.py
+++ b/store/catalog/migrations/0003_delete_product.py
@@ -11,7 +11,13 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.DeleteModel(
- name='Product',
- ),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.DeleteModel(
+ name='Product',
+ ),
+ ],
+ # You want to reuse the table, so don't drop it
+ database_operations=[],
+ )
]
To stop Django from deleting the table, you set database operations=[]. Then, stop Django from making the new
table:
--- a/store/product/migrations/0001_initial.py
+++ b/store/product/migrations/0001_initial.py
@@ -13,15 +13,21 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.CreateModel(
- name='Product',
- fields=[
- ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
- ('name', models.CharField(db_index=True, max_length=100)),
- ('category', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='catalog.Category')),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.CreateModel(
+ name='Product',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('name', models.CharField(db_index=True, max_length=100)),
+ ('category', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='catalog.Category')),
+ ],
+ options={
+ 'db_table': 'catalog_product',
+ },
+ ),
],
- options={
- 'db_table': 'catalog_product',
- },
- ),
+ # You reference an existing table
+ database_operations=[],
+ )
]
To stop Django from creating a new table, simply set database operations=[]. To conclude, you must stop Django from duplicating the Sell foreign key constraint in the new Product model. The restriction still applies since you’re utilising the same table.
place:
--- a/store/sale/migrations/0002_auto_20200104_0724.py
+++ b/store/sale/migrations/0002_auto_20200104_0724.py
@@ -12,9 +12,14 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.AlterField(
- model_name='sale',
- name='product',
- field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, to='product.Product'),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.AlterField(
+ model_name='sale',
+ name='product',
+ field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, to='product.Product'),
+ ),
+ ],
+ database_operations=[],
),
]
After making the necessary changes to the migration files, run the
migrations:
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, product, sale, sessions
Running migrations:
Applying catalog.0002_remove_product_category... OK
Applying product.0001_initial... OK
Applying sale.0002_auto_20200104_0724... OK
Applying catalog.0003_delete_product... OK
Your updated model is now referencing the previous database. In this case, Django’s model state was modified in the code, but the database was left untouched. The new model’s state should agree with the database before you declare victory and proceed.
An Extra: Tailor the New Model to Your Needs
Verify that Django successfully detects a change to the new model to guarantee that the model state matches the database state.
The name field is indexed in the Product model. Take that away
index:
--- a/store/product/models.py
+++ b/store/product/models.py
@@ -3,7 +3,7 @@ from django.db import models
from catalog.models import Category
class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
+ name = models.CharField(max_length=100)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
class Meta:
When you got rid of db index=True, you got rid of the index. In the next step, a
migration:
$ python manage.py makemigrations
Migrations for 'product':
product/migrations/0002_auto_20200104_0856.py
- Alter field name on product
Look at the SQL that Django has produced for this first.
migration:
$ python manage.py sqlmigrate product 0002
BEGIN;
--
-- Alter field name on product
--
DROP INDEX IF EXISTS "catalog_product_name_924af5bc";
DROP INDEX IF EXISTS "catalog_product_name_924af5bc_like";
COMMIT;
Great! Using the “catalogue_*” prefix, Django was able to identify the legacy index. You may now carry out the
migration:
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, product, sale, sessions
Running migrations:
Applying product.0002_auto_20200104_0856... OK
Be certain that the outcome of the
database:
django_migration_test=# \d catalog_product
Table "public.catalog_product"
Column | Type | Nullable | Default
-------------+------------------------+----------+---------------------------------------------
id | integer | not null | nextval('catalog_product_id_seq'::regclass)
name | character varying(100) | not null |
category_id | integer | not null |
Indexes:
"catalog_product_pkey" PRIMARY KEY, btree (id)
"catalog_product_category_id_35bf920b" btree (category_id)
Foreign-key constraints:
"catalog_product_category_id_35bf920b_fk_catalog_category_id"
FOREIGN KEY (category_id) REFERENCES catalog_category(id)
DEFERRABLE INITIALLY DEFERRED
Referenced by:
TABLE "sale_sale" CONSTRAINT "sale_sale_product_id_18508f6f_fk_catalog_product_id"
FOREIGN KEY (product_id) REFERENCES catalog_product(id)
DEFERRABLE INITIALLY DEFERRED
Success! The name column’s index was removed.
Changes to the Model Reference: Pros and Disadvantages
There are benefits and drawbacks associated with switching the model’s reference. Only a few of the benefits of this are listed below:
approach:
-
It’s fast
: This approach doesn’t make any changes to the database, so it’s very fast. -
It doesn’t require downtime
: This approach doesn’t require copying data, so it can be performed on a live system without downtime. -
It’s reversible
: It’s possible to reverse this migration if necessary. -
It’s supported by the ORM
: Performing this transition using built-in migration operations guarantees proper database support. -
It doesn’t require any sync to the database
: With this approach, related objects, such as indices and sequences, remain unchanged.
The most significant drawback of this method is that it deviates from the established naming pattern. Utilizing the preexisting table will result in the table retaining the former app’s naming conventions.
Compared to just copying the data, this method is more simpler.
Django Reinvents the Dining Room Table
Before, you used a database table reference to link the two models. Hence, you went against the Django naming standard. This method works in reverse, with the old table pointing to the new one.
In particular, you will be responsible for developing both the new model and its corresponding migration. Next, using the AlterModelTable migration method, you rename the existing table to the new model’s name, which is the name of the table Django produced for you.
Make a Brand-New Template
Create a new product app to implement all of the updates at once, as was done before. As a first step, please delete the Product template from the database.
app:
--- a/store/catalog/models.py
+++ b/store/catalog/models.py
@@ -3,8 +3,3 @@ from django.db import models
class Category(models.Model):
name = models.CharField(max_length=100)
-
-
-class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
- category = models.ForeignKey(Category, on_delete=models.CASCADE)
You have removed Product from our current offering. To proceed, transfer the Product model to a different product.
app:
# store/product/models.py
from django.db import models
from catalog.models import Category
class Product(models.Model):
name = models.CharField(max_length=100, db_index=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
Your Product app now has a Product model. Alter the Sale table’s foreign key to refer to product.Product.
:
--- a/store/sale/models.py
+++ b/store/sale/models.py
@@ -1,6 +1,6 @@
from django.db import models
-from catalog.models import Product
+from product.models import Product
class Sale(models.Model):
created = models.DateTimeField()
--- a/store/store/settings.py
+++ b/store/store/settings.py
@@ -40,6 +40,7 @@ INSTALLED_APPS = [
'catalog',
'sale',
+ 'product',
]
Finally, have Django create migrations for
you:
$ python manage.py makemigrations sale catalog product
Migrations for 'catalog':
catalog/migrations/0002_remove_product_category.py
- Remove field category from product
catalog/migrations/0003_delete_product.py
- Delete model Product
Migrations for 'product':
product/migrations/0001_initial.py
- Create model Product
Migrations for 'sale':
sale/migrations/0002_auto_20200110_1304.py
- Alter field product on sale
Because you want to rename the table, you must stop Django from removing it.
Producing the SQL for the migration that makes a Product model in the product app will give you the name of the Product model in the app.
:
$ python manage.py sqlmigrate product 0001
BEGIN;
--
-- Create model Product
--
CREATE TABLE "product_product" ("id" serial NOT NULL PRIMARY KEY, "name" varchar(100) NOT NULL, "category_id" integer NOT NULL);
ALTER TABLE "product_product" ADD CONSTRAINT "product_product_category_id_0c725779_fk_catalog_category_id" FOREIGN KEY ("category_id") REFERENCES "catalog_category" ("id") DEFERRABLE INITIALLY DEFERRED;
CREATE INDEX "product_product_name_04ac86ce" ON "product_product" ("name");
CREATE INDEX "product_product_name_04ac86ce_like" ON "product_product" ("name" varchar_pattern_ops);
CREATE INDEX "product_product_category_id_0c725779" ON "product_product" ("category_id");
COMMIT;
The Django-generated table for the Product model in the product app is called product product.
Please Rename the Current Dining Room Set
You may now rename the legacy table to the name Django produced for the model. Django created two new models to replace Product in the catalogue app.
migrations:
-
catalog/migrations/0002_remove_product_category
removes the foreign key from the table. -
catalog/migrations/0003_delete_product
drops the model.
Stopping Django from removing the foreign key to Category is necessary before renaming the table.
:
--- a/store/catalog/migrations/0002_remove_product_category.py
+++ b/store/catalog/migrations/0002_remove_product_category.py
@@ -10,8 +10,13 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.RemoveField(
- model_name='product',
- name='category',
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.RemoveField(
+ model_name='product',
+ name='category',
+ ),
+ ],
+ database_operations=[],
),
]
Django will not remove the column if you use SeparateDatabaseAndState and set database operations to an empty list.
If you need to change the name of a model table, Django’s migration operation AlterModelTable has you covered. To avoid data loss, change the migration that deletes the previous table to product product.
:
--- a/store/catalog/migrations/0003_delete_product.py
+++ b/store/catalog/migrations/0003_delete_product.py
@@ -11,7 +11,17 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.DeleteModel(
- name='Product',
- ),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.DeleteModel(
+ name='Product',
+ ),
+ ],
+ database_operations=[
+ migrations.AlterModelTable(
+ name='Product',
+ table='product_product',
+ ),
+ ],
+ )
]
You gave Django a new migration action to perform in the database by using SeparateDatabaseAndState in conjunction with AlterModelTable.
The next step is to prohibit Django from creating a table for the new Product model. You’d rather it utilise the renamed table instead. Adjust the product’s first migration as follows:
app:
--- a/store/product/migrations/0001_initial.py
+++ b/store/product/migrations/0001_initial.py
@@ -13,12 +13,18 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.CreateModel(
- name='Product',
- fields=[
- ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
- ('name', models.CharField(db_index=True, max_length=100)),
- ('category', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='catalog.Category')),
- ],
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.CreateModel(
+ name='Product',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('name', models.CharField(db_index=True, max_length=100)),
+ ('category', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='catalog.Category')),
+ ],
+ ),
+ ],
+ # Table already exists. See catalog/migrations/0003_delete_product.py
+ database_operations=[],
),
]
The line database operations=[] prevents the migration from really creating the table in the database, even if it does construct the model in Django’s state. Do you recall renaming the original table product product? Django can be tricked into sticking with the old table by using the old table’s data by renaming the table to the name Django would have given it for the new model.
Last but not least, you need to stop Django from recreating the foreign key constraint in the Sale table.
model:
--- a/store/sale/migrations/0002_auto_20200110_1304.py
+++ b/store/sale/migrations/0002_auto_20200110_1304.py
@@ -12,9 +12,15 @@ class Migration(migrations.Migration):
]
operations = [
- migrations.AlterField(
- model_name='sale',
- name='product',
- field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, to='product.Product'),
- ),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.AlterField(
+ model_name='sale',
+ name='product',
+ field=models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, to='product.Product'),
+ ),
+ ],
+ # You're reusing an existing table, so do nothing
+ database_operations=[],
+ )
]
Getting set to run the
migration:
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, catalog, contenttypes, product, sale, sessions
Running migrations:
Applying catalog.0002_remove_product_category... OK
Applying product.0001_initial... OK
Applying sale.0002_auto_20200110_1304... OK
Applying catalog.0003_delete_product... OK
Great! The relocation went off without a hitch. Nevertheless, before continuing, check to see whether it is possible.
reversed:
$ python manage.py migrate catalog 0001
Operations to perform:
Target specific migration: 0001_initial, from catalog
Running migrations:
Rendering model states... DONE
Unapplying catalog.0003_delete_product... OK
Unapplying sale.0002_auto_20200110_1304... OK
Unapplying product.0001_initial... OK
Unapplying catalog.0002_remove_product_category... OK
Amazing! The move may be undone if necessary. To be clear, AlterModelTable has many advantages over RunSQL.
As a first step, AlterModelTable
to manage fields with many-to-many relationships using names derived from the model’s name. It might take some extra effort to rename the tables using RunSQL.
Database independence is another advantage of built-in migration processes like AlterModelTable over RunSQL. If your app has to support numerous database engines, for instance, you may struggle to write SQL that works for all of them.
Bonus: Practice Self-Reflection
Django’s object-relational mapping (ORM) layer acts as a wrapper over Python data types, allowing for more direct access to and manipulation of database tables. Django will generate a table named product product if you build a model called Product in the product app. Other than tables, the ORM generates database objects like indexes, constraints, sequences, and more. All of these objects follow a standard naming scheme in Django that is derived from the app and model names.
Explore the catalog category table in the
database:
django_migration_test=# \d catalog_category
Table "public.catalog_category"
Column | Type | Nullable | Default
--------+------------------------+----------+----------------------------------------------
id | integer | not null | nextval('catalog_category_id_seq'::regclass)
name | character varying(100) | not null |
Indexes:
"catalog_category_pkey" PRIMARY KEY, btree (id)
Catalog category is the name given to the table that Django created for the Category model in the app catalogue. The other database follows the same naming pattern.
objects.
-
catalog_category_pkey
refers to a primary key index. -
catalog_category_id_seq
refers to a sequence to generate values for the primary key field
id
.
After that, have a look at the data table of the Product model you converted from catalogue to product.
:
django_migration_test=# \d product_product
Table "public.product_product"
Column | Type | Nullable | Default
-------------+------------------------+----------+---------------------------------------------
id | integer | not null | nextval('catalog_product_id_seq'::regclass)
name | character varying(100) | not null |
category_id | integer | not null |
Indexes:
"catalog_product_pkey" PRIMARY KEY, btree (id)
"catalog_product_category_id_35bf920b" btree (category_id)
"catalog_product_name_924af5bc" btree (name)
"catalog_product_name_924af5bc_like" btree (name varchar_pattern_ops)
Foreign-key constraints:
"catalog_product_category_id_35bf920b_fk_catalog_category_id"
FOREIGN KEY (category_id)
REFERENCES catalog_category(id)
DEFERRABLE INITIALLY DEFERRED
The number of seemingly connected things is larger than it really is. But, a deeper inspection shows that the names of the associated objects do not match the name of the table. For instance, whereas product product describes the database itself, catalog product pkey describes the primary key constraint. You took the model from the catalogue app, which suggests that the migration procedure AlterModelTable does not alter the names of all associated database objects.
You may examine the resulting SQL for this migration to get a feel for AlterModelTable’s inner workings.
operation:
$ python manage.py sqlmigrate catalog 0003
BEGIN;
--
-- Custom state/database change combination
--
ALTER TABLE "catalog_product" RENAME TO "product_product";
COMMIT;
These results demonstrate that AlterModelTable can only rename tables. What would happen if you attempted to modify a database object that references the tables of these objects, assuming that this is the case? Could these modifications be handled by Django?
You may try unindexing the relevant field in the Product table to see if it helps.
model:
--- a/store/product/models.py
+++ b/store/product/models.py
@@ -3,5 +3,5 @@ from django.db import models
from catalog.models import Category
class Product(models.Model):
- name = models.CharField(max_length=100, db_index=True)
+ name = models.CharField(max_length=100, db_index=False)
category = models.ForeignKey(Category, on_delete=models.CASCADE)
In the next step, a
migration:
$ python manage.py makemigrations
Migrations for 'product':
product/migrations/0002_auto_20200110_1426.py
- Alter field name on product
This is a positive indicator, since it means the order was carried out successfully. Verify the outputs that were created.
SQL:
$ python manage.py sqlmigrate product 0002
BEGIN;
--
-- Alter field name on product
--
DROP INDEX IF EXISTS "catalog_product_name_924af5bc";
DROP INDEX IF EXISTS "catalog_product_name_924af5bc_like";
COMMIT;
The index catalog product name 924af5bc is removed from the database by the produced SQL statements. Django found the existing index despite the inconsistency between the index name and the table name. You’re engaging in what’s called “introspection” here.
You won’t find much information regarding introspection since it is utilised internally by the ORM. An introspection module is built into each database backend, allowing database objects to be recognised by their attributes. Typically, the metadata tables supplied by the database would be used by the introspection module. The ORM may perform manipulations on objects independently of the naming convention by using introspection. This is how Django learned what index name to remove.
Renaming the Table: Benefits and Disadvantages
There are benefits and drawbacks to changing the name of the table. Only a few of the benefits of this are listed below:
approach:
-
It’s fast
: This approach only renames database objects, so it’s very fast. -
It doesn’t require downtime
: With this approach, database objects are locked only for a short time while they are renamed, so it can be performed on a live system without downtime. -
It’s reversible
: Reversing this migration is possible if necessary. -
It’s supported by the ORM
: Performing this transition using built-in migration operations guarantees proper database support.
The only possible downside to this strategy is that it deviates from the name norm. If you merely change the table names, your database won’t be compliant with Django’s naming standards. When interacting with the database directly, this might lead to some misunderstandings. Django can still utilise introspection to detect and handle such objects, thus this is not a huge issue.
Advice: Choose the Most Effective Method
This lesson showed you three distinct approaches to copying a Django model across applications. The methods discussed in this section are compared below.
tutorial:
| Metric | Copy Data | Change Table | Rename Table |
|---|---|---|---|
| Fast | ✗ | ✔️ | ✔️ |
| No downtime | ✗ | ✔️ | ✔️ |
| Sync related objects | ✗ | ✔️ | ✔️ |
| Preserve naming convention | ✔️ | ✗ | ✔️ |
| Built-in ORM support | ✔️ | ✔️ | ✔️ |
| Reversible | ✔️ | ✔️ | ✔️ |
Please take into consideration that the aforementioned table seems to imply that changing the table keeps Django’s naming standard. While this isn’t quite accurate, you already know that Django can use introspection to fix the naming problems that arise with this method.
The aforementioned methods each come with their own set of benefits and drawbacks. Which method should you use, then?
When dealing with a small table and can afford some downtime, data copying is a good practise. A better option would be to rename the table and include a reference to the updated model.
The reality, however, is that every project has its own set of specific demands. If one strategy makes more sense than another for you and your group, go with that one.