This document describes how to troubleshoot vCPU soft lockups. A soft lockup occurs when a virtual machine (VM) instance's vCPU is unable to run a new task for more than 20 seconds. Most soft lockups are caused by bugs in application software.
Soft lockups can cause VMs to become unresponsive for short periods of time, disrupt SSH access to VMs, and trigger application timeouts or failover. VMs that are experiencing a soft lockup might also have unusually high or unusually low CPU utilization, depending on the exact cause of the soft lockup.
Identify soft lockups
To identify whether your VM is experiencing a soft lockup, do one of the following:
- If you previously enabled serial port output logging for your VM, review serial port output for a soft lockup stack trace.
- Review your VM's operating system logs (
/var/log/messages) for a soft lockup stack trace.
Example soft lockup stack trace
watchdog: BUG: soft lockup - CPU#3 stuck for 22s!
To detect future soft lockups, you can do the following:
Create a log-based alerting policy for the following log:
resource.type="gce_instance" log_id("serialconsole.googleapis.com/serial_port_1_output") textPayload=~"watchdog.*lockup"
Troubleshoot soft lockups
After you've identified that a soft lockup is occurring, try the following troubleshooting steps to resolve the issue:
- Check your OS vendor's site for known errors with your OS version. Sometimes you might find reference to specific kernel modules in the stack trace that suggests a particular function or operation that is involved.
- Identify whether the soft lockup repeats with any frequency, such as coinciding with high load or certain activities. If the soft lockups correlate with high load, you might need to reconfigure your workload, for example by using a larger VM or splitting the load across more VMs.
- Check if the soft lockups correlate with any changes to your runtime environment such as new software deployments or OS image updates.
- Evaluate whether any maintenance events have taken place around the time of the soft lockup, by reviewing audit logs for system event audit logs.
If the proceeding troubleshooting steps didn't resolve the issue, file a support case and include all of the information you gathered from troubleshooting.
Best practices to avoid soft lockups
To help prevent your VMs from experiencing soft lockups, we recommend implementing the following best practices:
- Ensure that you have appropriate redundant components configured for your system, such as high availability clusters, to provide a failover capability if a particular VM experiences a prolonged soft lockup. For more information, see Designing resilient systems.
- For compute-intensive workloads, consider using compute-optimized machine families.
- Test your workload with simulated maintenance events to learn how your workload performs during live migration (if enabled), particularly under load testing.
- If you're running a custom Linux Kernel or custom modules in your VM, test new changes under load before deploying them to your production environment. Confirm that your custom changes don't disqualify you from receiving support from your OS vendor.
- Keep your operating system up to date. For more information, see Operating system details.