Converting to Django - Webucator’s Story
Why Switch Stacks?
I had a couple of big redesign projects in mind, and I felt it would be easier for our experienced ColdFusion developer, who had deep knowledge of our code base, to learn Python, Django, and Vue.js then it would be for me and anyone else we brought on to learn how to work effectively in our existing code base. I was a little worried that he wouldn’t be as excited about the switch as I was, but he had had his own frustrations with the code base and was excited about doing something new.
Our Pre-Plan Plan
The first thing we did was figure out what we needed to do. Here was our pre-plan plan:
- Freeze our current code.
- Make a plan to learn Python, Django, and Vue.js.
- Take an inventory of all of our sites and tools.
- Interview staff on how tools are used.
- Make a plan for tackling development.
From this, we were able to come up with a development plan and divide up the work.
The Scariest Piece: The Data
The piece I was most concerned about was importing the data from our SQL Server database to our new PostgreSQL database. Our database was not trivial. It included 216 tables containing millions of rows. Some of those tables were no longer used; others were awkwardly designed or no longer optimal for how our business had evolved. We had to figure out how to move all this data into our new Django models.
Although it is possible to integrate Django with a legacy database, it’s hard and I suspect you would lose a lot of the benefits of working with Django if you went that route. Your models would be designed based on your database tables rather than the other way around.
So, we decided to recreate all of our apps and then import our existing data into the tables generated by our Django models.
How to Maintain Relationships?
Of course, those 216 tables were related to each other in all sorts of different ways. We had to figure out how to maintain those relationships.
For example, in our legacy database, we had
Companies tables, which were connected via the
People.CompanyID column. We would need to retain this relationship between our
Company models in Django. To do that, we included
old_id fields in all the models. For example:
class Company(models.Model): old_id = models.CharField(max_length=50, null=True, unique=True) class CustomUser(AbstractUser): old_id = models.CharField(max_length=50, null=True, unique=True) old_company_id = models.CharField(max_length=50, null=True) company = models.ForeignKey( Company, on_delete=models.PROTECT, null=True, blank=True, related_name="users" )
Our importing code looked something like this:
def create_users(): # Get all the users from SQL Server query_file = settings.BASE_DIR / "importer/management/queries/Users.sql" q = query_file.read_text() cursor = execute_query(q) rows = get_rows_as_dicts(cursor) # Loop through the users, adding companies as new ones are found. for row in rows: # Add or get company company = None if row["old_company_id"]: try: # Use existing company if it exists company = Company.objects.get(old_id=row["old_company_id"]) except: # Create new company company = Company( name=row["company_name"], old_id=row["old_company_id"], … ) company.save() # Add user user = User( date_joined=tz.localize(row["DateEntered"]) company=company, old_id=row["old_id"], … ) user.save() cursor.close()
The code was significantly more complicated, with data cleanup (e.g. trimming, identifying and removing junk data, etc.), error checking, and reporting, but the above gives the general idea.
In some cases we needed to take a second pass through a table. For example, our
Company model includes a
salesperson field, which is a foreign key referencing the
CustomUser table. We couldn’t add that value until we were sure the user existed, so we needed to take a second pass through the company data after we had imported all the users to add salespeople to the company records.
Verifying the Data Import
It was extremely difficult to verify that we were importing all of our data correctly. We had two primary methods for doing this:
- Reporting. By recreating all of our old reports, we were able to verify that the numbers matched up. For example, we could check our monthly revenue by line of business to make sure the new reports matched the old reports.
- About three months before making the switch we made the new site available to staff and we asked them to try to do everything they do in both places. This was of course painful and time consuming, but it helped ensure that the data on the new site matched that on the old site. It also had the added benefits of:
- Training staff on Django admin before they’d have to use it in production.
- Identifying features staff needed that we had failed to bring over.
Unexpected Challenges: Docker and Heroku
We knew most of the challenges we’d face:
- Learning the new technologies turned out to be easier than we expected, and it was a ton of fun.
- Porting the data was a lot of work, even more than I expected, but I felt early on that it was doable.
- Working with staff on recreating the apps was both fun and helpful in getting a better understanding of how our own business works.
But we didn’t know about the challenges we’d face with Docker and Heroku. After more than a year of working with those technologies, I feel the same way about them that I feel about AWS:
You can always get things to work, but you often don’t quite know how you did it.
Docker and Heroku both caused us some headaches, and I still don’t feel like I’m an expert at either of them, but we have them working well.
How long did it take and how did it go?
With two full-time and two part-time developers (two computer science majors who had taken a Covid-inspired gap year from Brown University), it took 12 months to complete the conversion. This included the learning process: most of the team was relatively new to Python and Django, and also to Vue.js, which we used heavily for our self-paced course application.
We had a few minor hiccups the first week or so after launching, but we were able to resolve them quickly.
Our staff is rapidly getting used to using Django admin. We have kept our old site up (with the old data) in case people need to verify something or get access to some feature we forgot, but that’s rarely necessary. We’re planning to shut it down soon.
The part-time developers are back at college, so it’s just two of us in development now. We both love the new stack. Our new code base looks something like this:
So, overall, it has been a great success. We’re happy.