Over the course of my career I’ve had the privilege to work on some very talented teams. As a part of these teams, I’ve been a part of countless cutovers. Some have gone really well, some have been disastrous. Here are some of the lessons I’ve had to learn the hard way:
Know the risks. Identify the worst case scenario and use this to guide the rest of your planning.
Have a plan. Not just a general plan, but a detailed, minute by minute plan of how things should go. Write down every single command that needs to be issued, a list of tests to confirm the cutover, and the commands to roll back if things go sideways. They should be written in such detail that someone without prior knowledge of the cutover could complete it successfully.
Include the right people. Ensure that all people needed for a worst case scenario are present. Include all technical resources and someone with the power to make big decisions. Ensure support is aware of the outage window and any potential risks. Include someone with the sole responsibility of communicating with stakeholders in case of emergency.
Minimize user impact. Always plan for the worst case scenario and choose a period with the lowest traffic.
Don’t be afraid to roll back. If things go sideways, it’s almost always better to roll back and evaluate why things went wrong and live to fight another day. Trying to power through and fix things on the fly almost always compounds the issues.
Celebrate. When a cutover finishes successfully, celebration is in order. Have plenty of beer and high 5’s handy for when things go right.