Version 1.9. This version is no longer supported. For information about how to upgrade to version 1.10, see Upgrading Anthos on bare metal in the 1.10 documentation. For more information about supported and unsupported versions, see the Version history page in the latest documentation.
Sometimes, to remove a broken node for repair or replacement, you may have to
force its removal from the cluster. Force removal only removes the broken node
from the cluster management's perspective. Force removal bypasses clean up jobs
for the installed components on the node itself. Upon recovery of the node, you
run bmctl reset nodes to clean up the installed components on the node so that
it can be reused.
Force-removing nodes
The following methods apply to both control plane nodes and worker nodes. For
control plane nodes, controllers in Google Distributed Cloud also take care of the
bookkeeping of etcd memberships.
Using bmctl
You can use bmctl to remove the node from the cluster. Normally, bmctl reset
triggers a reset job to try to clean up installed components on the node. To
remove the node from the cluster without being blocked on cleaning up installed
packages, you can run the bmctl command with the --force flag:
NODE_IP: the IP address of the node to reset, such as
10.200.0.8.
ADMIN_KUBECONFIG: the path to the admin cluster
kubeconfig file.
CLUSTER_NAME: the name of the target cluster that contains the nodes.
Using kubectl
In Google Distributed Cloud, you can add an annotation to mark a node for
force removal.
After removing the node from the parent nodepool, run the following command
to annotate the corresponding failing machine with the
baremetal.cluster.gke.io/force-remove annotation. The value of the annotation itself
does not matter:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-03-05 UTC."],[[["Force removal of a node is used when a node is broken and needs to be removed from the cluster for repair or replacement, bypassing the cleanup of installed components on the node."],["The `bmctl reset nodes` command with the `--force` flag can be used to remove a node from the cluster without waiting for the cleanup of its installed packages."],["After the node is removed using the force removal method, you must run `bmctl reset nodes` on the recovered node to clean up any remaining installed components so it can be reused."],["Nodes can also be force-removed using `kubectl` by annotating the failing machine with `baremetal.cluster.gke.io/force-remove`."]]],[]]
