OpenShift 4 Migration: A sample path

by Josh and Laine | Processes, Technology

November 5, 2019

The Problem

Moving stuff between Kubernetes clusters can be a pain in the butt. You have to do this when:

Migrating between container platforms, such as EKS -> AKS
Upgrading clusters when you want to upgrade in parallel (move everything from old to new) as opposed to upgrading in-place (update one cluster from old to new). This would be something like what we’ll talk about in a minute, going from OpenShift 3.x to OpenShift 4.x
Moving work/applications between clusters, e.g. two clusters in different datacenters

Migrating work between clusters requires some thought and planning, and good solid processes. Specifically, the migration from OpenShift 3.x to OpenShift 4.x requires a parallel upgrade, because there is no in-place upgrade available for all of the new goodies in RHEL CoreOS (the underlying infrastructure of the cluster). OpenShift 4.2 released recently, so we thought it would be good timing to put our migration thoughts down here. However, the advice below is generally good for any Kubernetes cluster parallel upgrade or other migration.

The Plan

1. Understand what’s new and fundamentally different for OpenShift 4.

This video, talking about the upstream OKD, is pretty good, and these slides cover a lot of important stuff. If you’re a Red Hat customer, contact your account team – they can talk you through the platform updates and discuss your situation in detail. Be sure to talk about what you’re currently using OpenShift for, and what you plan to use it for, and also any pain points you have with what you’re doing today.

The big OpenShift changes come in three big chunks: security (including the CoreOS changes), operations (specifically operators and finding ways to lessen the need for esoteric infrastructure knowledge), and enablement of developers.

2. Make the Plan!

Get your infrastructure/operations people, your architects, and your DevOps people in a room (ideally with a whiteboard or two), and talk through the plan. Talk about if you want help migrating, and if so, how much. OpenShift 4.2 comes with a migration tool that will do at least some of the work for you. The migration tool can automate your migration, project-by-project, which is pretty slick. Check out this video:

If you’re migrating between various versions of OpenShift, this tool will work nicely for you. Project-by-project is a fine place to start with an OpenShift migration, because projects typically represent logical buckets of applications or teams. If you prefer to write your own migration script, you can also do that via any scripting tool that has access to your clusters – Jenkins, Ansible, etc, all of these will work fine. Tools like Skopeo (for image migration) and Valero (for application component migration) can also be helpful.

3. The Migration Script in Concept

First, verify all incoming networking traffic goes through a front-end load balancer or obeys DNS rules that you can control.
Spin up the new OpenShift cluster – don’t route live traffic to it yet
Install the components on the new cluster necessary to support the migration’s automation – the migration tool if doing OpenShift -> OpenShift, or the things necessary for Jenkins/Ansible/Skopeo/etc to work.
Migrate a tester project or two, with all included components, between clusters using your automation of choice. Make sure the tester project is functional and that the applications within it start and run. Pick stateless apps and be careful where they’re writing their data to.
Once you’ve verified the cluster is up and the migration automation works, work through a few more projects to test the automation out. Don’t change routing over to the new cluster just yet, this is to test out your process and make sure it will be smooth at go time.
Now that your confidence in your process is more solid, pick a date to change the cluster routing – to officially point to the new cluster instead of the old one.
On the cluster routing change date: block deploys to the old cluster
Run your full migration automation again, to pick up any changes since the last time.
1. For stateless apps: just migrate them! One of the beauties of stateless apps.
2. For stateful apps (there will be a longer outage for these, sadly, if you don’t want the database/other “state” to get very confused):
  1. Shut down access to the old cluster’s stateful apps
  2. run the migration
  3. Start those apps on the new cluster
Do final testing of apps on your new cluster (ideally, this would be covered with health checks so it isn’t by-hand testing).
Officially switch the routing so that work goes to the new cluster.
Verify that the new routing is working well.
Done! High five each other, and enjoy your new features and fun new Kubernetes toys!

Conclusion

We intend this as a high-level instruction regarding Kubernetes cluster migration. Migration can be stressful and painful, or it can be relatively smooth – and usually it’s planning, and having time to practice and automate that makes all the difference. Red Hat tries to make this as easy as possible, and we’ve been pushing Kubernetes upgrades since 1.0, but it helps to talk to people who have done it before and have seen the common pitfalls.

If you have questions, please ask here or hit us up on Twitter.