Debugging node issues using toolbox

You might need to install additional packages or tools on Container-Optimized OS for certain tasks, such as debugging. For example, debug node connectivity issues by installing tcpdump in toolbox. Although Container-Optimized OS does not include a package manager, you can use the pre-installed toolbox utility to install any additional packages or tools you require. Using /usr/bin/toolbox is the preferred method for installing and running one-off debugging tools.

Note: toolbox is a debugging utility and not intended for production use.

/usr/bin/toolbox essentially provides you a shell in a Debian chroot-like environment. When you invoke /usr/bin/toolbox, it runs following commands:

1. docker pull and docker create to set up the environment. These are only run the first time you invoke /usr/bin/toolbox.
2. systemd-nspawn to run the given command or (in absence of any command) provides you a shell

toolbox has some other properties to keep in mind:

• Invoking toolbox after the first invocation does not require a working Docker daemon, nor does it incur any network/disk overhead.
• The toolbox environment is set up once for each user invoking it. Running sudo toolbox sets it up for root user.
• The toolbox environment is created under /var/lib/toolbox and is persistent across reboots.
• You can access sections of the root filesystem, such as user home directories, from inside the toolbox environment.

Customizing toolbox for your deployment

In the following examples, we assume that your user name is USER and your Container-Optimized OS instance name is cos-dev.

You can customize the Docker image that toolbox uses, as well as the paths available to toolbox in the root filesystem. These settings are located in the file /etc/default/toolbox. The default /etc/default/toolbox file typically resembles the following:

USER@cos-dev ~ $ cat /etc/default/toolbox
# Copyright 2016 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

TOOLBOX_DOCKER_IMAGE="gcr.io/cos-cloud/toolbox"
TOOLBOX_DOCKER_TAG="v20220722"
TOOLBOX_BIND="--bind=/:/media/root/ --bind=/mnt/disks/:/media/root/mnt/disks/ --bind=/var/:/media/root/var/ --bind=/home:/media/root/home/ --bind=/etc/resolv.conf:/etc/resolv.conf"
: ${USER:=root}

• The TOOLBOX_DOCKER_IMAGE and TOOLBOX_DOCKER_TAG variables specify the Docker image to be used. The default gcr.io/cos-cloud/toolbox comes with some of the common tools like the Google Cloud CLI pre-installed.
• The TOOLBOX_BIND variable specifies the paths from rootfs to be made available inside the toolbox environment.

To change the default settings, modify the /etc/default/toolbox file, or specify new values for the variables in ${HOME}/.toolboxrc for the appropriate user. For example, assuming that you want toolbox to use fedora:latest as its container, you can run the following commands:

USER@cos-dev ~ $ echo "TOOLBOX_DOCKER_IMAGE=docker.io/library/fedora" > "${HOME}/.toolboxrc"
USER@cos-dev ~ $ echo "TOOLBOX_DOCKER_TAG=latest" >> "${HOME}/.toolboxrc"
USER@cos-dev ~ $ toolbox

In Milestone 93, toolbox will fail to pull custom toolbox image. To workaround this problem, build custom toolbox docker image tar ball and export it via either ${HOME}/.toolboxrc file or /etc/default/toolbox file.

USER@cos-dev ~ $ echo "TOOLBOX_DOCKER_IMAGE_TARBALL=[docker image tar ball]" > "${HOME}/.toolboxrc"
USER@cos-dev ~ $ toolbox

The `/etc/default/toolbox` file can be modified as shown below.

USER@cos-dev ~ $ cat /etc/default/toolbox
    # Copyright 2016 The Chromium OS Authors. All rights reserved.
    # Use of this source code is governed by a BSD-style license that can be
    # found in the LICENSE file.

    TOOLBOX_DOCKER_IMAGE="gcr.io/cos-cloud/toolbox"
    TOOLBOX_DOCKER_TAG="v20220722"
    TOOLBOX_DOCKER_IMAGE_TARBALL=[location/to/tarball]
    TOOLBOX_BIND="--bind=/:/media/root/ --bind=/mnt/disks/:/media/root/mnt/disks/ --bind=/var/:/media/root/var/ --bind=/home:/media/root/home/"
    : ${USER:=root}
USER@cos-dev ~ $ toolbox

Installing and running tools from toolbox

Once you've invoked the toolbox utility to start the shell, you can use apt-get inside the resulting container to install packages. For example:

# Inside the toolbox shell
USER@cos-dev ~ $ toolbox
root@cos-dev:~# apt-get update && apt-get install -y htop psmisc
root@cos-dev:~# htop
root@cos-dev:~# pstree -p
root@cos-dev:~# exit

You do not always have to enter toolbox to run commands in it. You can simply prefix your commands with toolbox. For example, to install and run the strace utility to trace the execution of the Docker daemon, you can do the following:

USER@cos-dev ~ $ toolbox apt-get install -y strace
USER@cos-dev ~ $ toolbox strace -p `pidof dockerd`

The Google Cloud CLI is pre-installed in toolbox:

USER@cos-dev ~ $ toolbox
root@cos-dev:~# which gcloud
/google-cloud-sdk/bin/gcloud
# View installed components
root@cos-dev:~# gcloud components list

Your current gcloud CLI version is: 287.0.0
The latest available version is: 295.0.0
...

Keep in mind that the gcloud CLI can only access resources that the VM instance is authorized to access. Read the Compute Engine documentation to learn more about provisioning VM instances with access to other resources.

Read the Google Cloud CLI documentation to learn more about using the gcloud CLI.

Getting files into and out of toolbox

The root filesystem of the host is accessible inside toolbox through the /media/root path. On the host, the root directory of toolbox is accessible through the following path: /var/lib/toolbox/USER-gcr.io_cos-cloud_toolbox-VERSION/root where USER is your user name and VERSION is the toolbox version number (for example, v20220722).

The following examples show how to get files into and out of toolbox:

# Access the host filesystem inside the toolbox
USER@cos-dev ~ $ toolbox
root@cos-dev:~# ls /media/root
bin  boot  dev  etc  home  lib  lib64  ...
root@cos-dev:~# cp /media/root/home/USER/some-file .

# Access toolbox directory from the host
USER@cos-dev ~ $ sudo cp some-file /var/lib/toolbox/USER-gcr.io_cos-cloud_toolbox-v20220722/root

# Run a command inside toolbox and save its output in your home directory
USER@cos-dev ~ $ toolbox strace -o /media/root/$HOME/ls.strace ls
USER@cos-dev ~ $ more $HOME/ls.strace

Cleaning up disk space used by toolbox

The toolbox installation consumes approximately 2-3 GB of disk space in /var (disk usage varies with the Container-Optimized OS version). To free disk space used by toolbox, perform the following tasks depending on your Container-Optimized OS version:

Milestone 89 and earlier

# remove files in /var/lib/toolbox
sudo rm -rf /var/lib/toolbox/*

# remove container artifacts
docker image rm $IMAGE_PATH:$TAG

You can run docker image ls to determine these values. For example, IMAGE_PATH=gcr.io/cos-cloud/toolbox and TAG=v20220722.

Milestone 93 and later

# remove files in /var/lib/toolbox
sudo rm -rf /var/lib/toolbox/*

# remove container artifacts
sudo ctr images rm $IMAGE_NAME

You can run sudo ctr images ls -q to determine this value. For example, IMAGE_NAME=gcr.io/cos-cloud/toolbox:v20220722.