How to Parallel Change
How to change you database structure with no downtime and test your code with production data
At some point, you’ll need to deeply change the structure of your database, which is not ideal.
Maybe you need to restructure your platform, or maybe you want to add new features that were not possible with the current structure, or maybe both.
You’ll need to answer these questions in any case:
How can you ensure that new data is correct?
How can you achieve zero downtime when making the change?
Can you roll back if there are issues?
Parallel Change achieves this by creating a different “store” living together with the current store and making changes in parallel to both until you can deprecate the current one, leaving only the new store.
The Parallel Change
The entire process follows these rules and is split into 4 phases:
Each phase has a dedicated Feature Flag.
It must be possible to rollback to the previous phase during every phase by turning off its Feature Flag (**no code changes, no new deploy**).
Do not touch the current store structure.
Write-Both
This is when the new store is created.
You will start writing to the new store during this phase. **The new functions you create must be completely decoupled from the current store.** There is an optional migration to initialize the structure of the new store and a final one to backfill data in the new store.
By the end of this phase, you should have the current store untouched and the new one with partial data.
Note: When I say “store” in this context I don’t mean a different database, it could be another table in the same database or even a different field on the same table.
If your database requires it, you need to start with a migration to create the structure for the new schema.
Change all instances of writing to the current store to **also** write to the new store
In the code above, the change is to port Person to a new table called People and also to have a single field (“fullName") instead of two fields ("firstName" and "lastName”).
Make sure **the new functions receive input from the client, without coupling with the current store.**
Wait until the Feature Flag reaches 100% rollout.
Remove the Feature Flag
Do not remove the code writing to the current store with the Feature Flag!
Removing that code will be a very big issue because we are still reading from the current store. Keep that code living together with the new one.
That’s why we call it "Parallel Change”.
Backfill migration
After step 4 of the Write-Both phase, the new store will have some data, but not all the data.
Before you start reading from the new store, make sure it is complete with all the data existing in the current store. This is necessary because it is possible that some data didn’t get updated since you released the Write-Both changes, so the functions never wrote them to the new store.
We need to migrate all the data from the current store to the new one.
Read-Both (and compare)
Now it’s time to start reading data from the new store.
Your API will read data from the new store, but it won’t expose that data to the client yet. The API will only read that data to compare it against the data in the current store. Only data in the current store will be returned to the client.
The steps are:
Create a function to compare the current structure and the new structure.
Change all the places where you read the data to also read data from the new store, and use the `compare` function to check the validity of the data.
Wait until the Feature Flag reaches 100% rollout.
Remove the Feature Flag and the compare code.
Do not remove the code that reads from the current store!
We still need that code because we are still reading and returning data from the current store.
This is the most delicate phase.
During this phase, you’ll validate new data with production data. This is the moment where you can find bugs in the code writing to the new store. All the code added after the end of this phase is not validated.
Add all the new code before the end of this phase.
Read-New
This phase should produce very small changes.
Here are the steps:
When the Feature Flag is enabled, return data from the new store instead of the current store.
Wait until the Feature Flag reaches 100% rollout.
Remove the Feature Flag and also remove the code reading from the current store.
The API is still perfectly backward-compatible before this phase.
Write-Both
Like the Read-New phase, this phase should be very easy if the first 2 phases were done correctly.
At this point, you should have code reading only from the new store, so it is very unlikely that the code created during the Write-Both phase to write to the new store has bugs. You should be safe enough to simply remove the code writing to the current store. When this phase is finished you can also consider doing another migration to clean-up your database from data that is no more needed.
Most of the time, this phase results in removing code and making very few changes to the same places you changed during the Write-Both.