Migrated workload fails on 1.20 and later with Ubuntu nodes

AppArmor lets a system administrator restrict capabilities of a deployed container. To deploy a migrated container workload, Migrate to Containers requires the AppArmor profile provided by the Linux LXC package.

However, as part of the update to Google Kubernetes Engine (GKE) 1.20, Ubuntu node images no longer include the LXC package by default. That means new deployment clusters that use Ubuntu nodes, or Ubuntu nodes for existing deployment clusters that are upgraded to GKE 1.20 and later, cannot run your migrated workloads.

When deploying container workloads on Ubuntu nodes for GKE versions 1.20 and later, the workload can enter a crash-loop and the logs contain error messages in the form:

D0806 01:59:15.000000 8 hcutil.py:136] SHELL CMD: aa-exec -p lxc-container-default echo 123
D0806 01:59:15.000000 8 hcutil.py:168] SHELL STDERR: b"aa-exec: ERROR: profile 'lxc-container-default' does not exist\n"
D0806 01:59:15.000000 8 hcutil.py:168] SHELL STDERR: b'\n'
D0806 01:59:15.000000 8 hcutil.py:210] SHELL COMPLETED: 1 (aa-exec -p lxc-container-default echo 123)
required AppArmor profile 'lxc-container-default' does not exist, if this pod runs on a
GKE node of type 'Container Optimized OS' (COS) please run
`migctl setup install --cos-runtime` to install it
E0806 01:59:15.000000    8 hcmain.py:22] Traceback (most recent call last):
  File "./hcmain.py", line 19, in safe_action
  File "./hcmain.py", line 74, in run_action
  File "./hcrunner.py", line 392, in validateSystem
Exception: Invalid system or AppArmor profile

E0806 01:59:15.000000 8 hcmain.py:23] Invalid system or AppArmor profile
I0806 01:59:15.000000 8 termination_log.py:4] writing error to termination log at /dev/termination-log

To resolve this issue:

  1. Connect to the deployment cluster using a command in the form:

    gcloud container clusters get-credentials CLUSTER --zone ZONE --project PROJECT

  2. Manually install the Linux LXC package on the deployment cluster by using the command:

    migctl setup install --cos-runtime