Remediating Security Command Center errors

This page provides a list of reference guides and techniques for remediating SCC errors.

Before you begin

You need adequate Identity and Access Management (IAM) roles to view or edit findings, and to access or modify Google Cloud resources. If you encounter permission errors when accessing Security Command Center in the Google Cloud console, ask your administrator for assistance. To learn about roles, see Access control. To resolve resource errors, read documentation for affected products.

Review findings in the Google Cloud console

SCC errors are configuration errors that prevent Security Command Center from working as expected. The Security Command Center source generates these findings.

As long as Security Command Center is set up for your organization or project, it generates error findings as it detects them. You can view SCC errors in the Google Cloud console.

Use the following procedure to review findings in the Google Cloud console:

Google Cloud console

1. In the Google Cloud console, go to the Findings page of Security Command Center.

   Go to Findings

2. Select your Google Cloud project or organization.
3. In the Quick filters section, in the Source display name subsection, select Security Command Center. The findings query results are updated to show only the findings from this source.
4. To view the details of a specific finding, click the finding name under Category. The details panel for the finding opens and displays the Summary tab.
5. On the Summary tab, review the details of the finding, including information about what was detected, the affected resource, and—if available—steps that you can take to remediate the finding.
6. Optional: To view the full JSON definition of the finding, click the JSON tab.

Security Operations console

Enterprise tier only: This feature is available only with the Security Command Center Enterprise tier.

1. In the Security Operations console, go to the Findings page.

   https://CUSTOMER_SUBDOMAIN.backstory.chronicle.security/posture/findings
   

   Replace CUSTOMER_SUBDOMAIN with your customer-specific identifier.

2. In the Aggregations section, click to expand the Source Display Name subsection.
3. Select Security Command Center. The findings query results are updated to show only the findings from this source.
4. To view the details of a specific finding, click the finding name under Category. The details panel for the finding opens and displays the Summary tab.
5. On the Summary tab, review the details of the finding, including information about what was detected, the affected resource, and—if available—steps that you can take to remediate the finding.
6. Optional: To view the full JSON definition of the finding, click the JSON tab.

Deactivation of SCC errors after remediation

After you remediate an SCC error finding, Security Command Center automatically sets the state of the finding to INACTIVE during the next scan. How long it takes for Security Command Center to set the state of a remediated finding to INACTIVE depends on when you fix the finding and the schedule of the scan that detects the error.

For information about the scan frequency for an SCC error finding, see the summary of the finding in Error detectors.

Remediate SCC errors

This section includes remediation instructions for all SCC errors.

Caution: If done improperly, modifying the configuration of your resources can negatively affect your production environment. Also, enabling some suggested Google Cloud services can increase your cloud costs. Understand the impact of all remediation steps before making changes.

API disabled

Category name in the API: API_DISABLED

One of the following services is disabled for the project:

• Container Threat Detection
• Web Security Scanner

The disabled service can't generate findings.

To remediate this finding, follow these steps:

1. Review the finding to determine which API is disabled.

2. Enable the API:

   • Enable the Container Threat Detection API.

     Enable the API

   • Enable the Web Security Scanner API.

     Enable the API

Learn about this finding type's supported assets and scan settings.

APS no resource value configs match any resources

Category name in the API: APS_NO_RESOURCE_VALUE_CONFIGS_MATCH_ANY_RESOURCES

Resource value configurations are defined for attack path simulations, but they don't match any resource instances in your environment. The simulations are using the default high-value resource set instead.

Resource value configurations might not match any resources for the following reasons, which are identified in the finding description in Google Cloud console:

• None of the resource value configurations match any resource instances.
• One or more resource value configurations that specify NONE override every other valid configuration.
• All the defined resource value configurations specify a value of NONE.

To remediate this finding, follow these steps:

1. Go to the Attack path simulation page in Security Command Center Settings:

   Go to Settings

2. Select your organization. The Attack path simulation page opens with the existing configurations displayed.

3. In the Resource value column of the Resource value configurations list, check for values of None.

4. For any configuration that specified None, do the following:

   1. Click the name of any resource value configuration to display the configuration specifications.
   2. If necessary, edit the resource attribute specifications to reduce the number of resource instances that match the configuration.

5. If the problem is not caused by an overly broad None specification, do the following:

   1. Click the names of each configuration that specifies a value of HIGH, MEDIUM, or LOW to display the resource attribute specifications.
   2. Review and, necessary, edit the configuration to correct the scope, resource type, tag, or label specification to match the intended resources.

6. If necessary, create a new resource value configuration.

Your changes are applied to the next attack path simulation.

Learn about this finding type's supported assets and scan settings.

APS resource value assignment limit exceeded

Category name in the API: APS_RESOURCE_VALUE_ASSIGNMENT_LIMIT_EXCEEDED

In the last attack path simulation, the number of high-value resource instances, as identified by the Resource value configurations, exceeded the limit of 1,000 resource instances in a high-value resource set. As a result, Security Command Center excluded the excess number of instances from the high-value resource set.

To remediate this finding, you can try the following actions:

• Use tags or labels to reduce the number of matches for a given resource type or within a specified scope. The tags or labels have to be applied to the resources instances before they can be matched by a resource value configuration.

• Create a resource value configuration that assigns a resource value of NONE to a subset of the resources that are specified in another configuration.

  Specifying a value of NONE overrides any other configurations and excludes the resource instances from your high-value resource set.

• Reduce the scope resource attribute specification in the resource value configuration.

• Delete resource value configurations that assign a value of LOW.

For instructions on creating, editing, or deleting a resource value configuration, see Define and manage your high-value resource set.

Learn about this finding type's supported assets and scan settings.

CIEM service account missing permissions

Enterprise tier only: This feature is available only with the Security Command Center Enterprise tier.

Category name in the API: CIEM_SERVICE_ACCOUNT_MISSING_PERMISSIONS

The service account that is used by the CIEM service is missing permissions. CIEM cannot generate one or more finding categories.

To remediate this finding, restore the required IAM roles on the CIEM service account:

1. In the Google Cloud console, go to the IAM page.

   Go to IAM

2. Select your organization's CIEM service account. The service account's identifier is an email address with the following format:

   [e-mail]
   

   Replace ORGANIZATION_ID with your organization's numerical ID.

   If you don't see the service account listed, click GRANT ACCESS at the top of the page and enter the service account as a new principal.

3. Grant the CIEM Service Agent role (roles/ciem.serviceAgent) to the service account. If you use custom roles, make sure they include the following permissions:

   • cloudasset.assets.exportResource
   • cloudasset.assets.exportIamPolicy

4. Click Save.

CIEM AWS CloudTrail configuration error

Enterprise tier only: This feature is available only with the Security Command Center Enterprise tier.

Category name in the API: AWS_CLOUDTRAIL_CONFIGURATION_ERROR

Either all or some CIEM AWS findings aren't being sent to Security Command Center. The AWS CloudTrail feed failed and is unable to successfully fetch data due to a configuration error.

There are three possible causes for this finding:

• Missing AWS CloudTrail feed

  To fix this issue, create and configure a feed in Security Operations console to ingest AWS CloudTrail logs. Set the Ingestion label key-value pair to CIEM and TRUE.

  For instructions on creating a feed, see Create the feed in the Google SecOps documentation.

• Errors in feed configuration

  Make sure you have configured the feed correctly.

  To configure a feed, see Configure feed in Google Security Operations to ingest AWS logs in the Google SecOps documentation.

• Incomplete AWS CloudTrail configuration

  To fix this issue, set up the S3 bucket in your AWS CloudTrail configuration to log both data events and management events from all AWS accounts where you intend to use CIEM.

  To set up CloudTrail, see Configure AWS CloudTrail (or other service) in the Google SecOps documentation.

GKE service account missing permissions

Category name in the API: GKE_SERVICE_ACCOUNT_MISSING_PERMISSIONS

Container Threat Detection can't generate findings for a Google Kubernetes Engine cluster, because the GKE default service account on the cluster is missing permissions. This prevents Container Threat Detection from being successfully enabled on the cluster.

To remediate this finding, restore the GKE default service account, and confirm that the service account has the Kubernetes Engine Service Agent (roles/container.serviceAgent) role.

Learn about this finding type's supported assets and scan settings.

KTD blocked by admission controller

Category name in the API: KTD_BLOCKED_BY_ADMISSION_CONTROLLER

Container Threat Detection can't be enabled on a cluster because a third-party admission controller is preventing the deployment of the required Kubernetes DaemonSet object.

To remediate this finding, make sure that the admission controllers that are running on the cluster allow Container Threat Detection to create the required Kubernetes objects.

Check the admission controller

Check to see if the admission controller in your cluster is denying the deployment of the Container Threat Detection DaemonSet object.

1. In the finding description in finding details in the Google Cloud console, review the included error message from Kubernetes. The Kubernetes error message should be similar to the following message:

   generic::failed_precondition: incompatible admission webhook:
   admission webhook "example.webhook.sh" denied the request:
   [example-constraint] you must provide labels: {"example-required-label"}.
   

2. In the Admin Activity Cloud Audit Logs for the project that contains your cluster, look for the error message shown in the Description field of the finding details.

3. If your admission controller is working, but is denying the deployment of the Container Threat Detection DaemonSet object, configure your admission controller to allow the service agent for Container Threat Detection to manage objects in the kube-system namespace.

   The service agent for Container Threat Detection must be able to manage specific Kubernetes objects.

For more information about using admission controllers with Container Threat Detection, see PodSecurityPolicy and Admission Controllers.

Confirm the fix

After you fix the error, Security Command Center automatically attempts to enable Container Threat Detection. After waiting for enablement to complete, you can check if Container Threat Detection is active by using the following steps:

1. Go to Kubernetes Engine Workloads page in the console.

   Go to Kubernetes workloads

2. If necessary, select Show system workloads.

3. On the Workloads page, filter the workloads first by the cluster name.

4. Look for the container-watcher workload. If container-watcher is present and its status shows OK, Container Threat Detection is active.

KTD image pull failure

Category name in the API: KTD_IMAGE_PULL_FAILURE

Container Threat Detection can't be enabled on the cluster because a required container image can't be pulled (downloaded) from gcr.io, the Container Registry image host.

The pulling or downloading of a container image can fail for any of multiple possible reasons.

Check the following:

• Make sure that your VPC network, DNS, or firewall settings are not blocking network access from the cluster to the gcr.io image host.
• If the cluster is private, make sure that Private Google Access is enabled to allow access to the gcr.io image host.
• If the network settings and Private Google Access are not the cause of the failure, see the GKE troubleshooting documentation for ImagePullBackOff and ErrImagePull errors.

Learn about this finding type's supported assets and scan settings.

KTD service account missing permissions

Category name in the API: KTD_SERVICE_ACCOUNT_MISSING_PERMISSIONS

The Container Threat Detection service account that is identified in the finding details in the Google Cloud console is missing required permissions. Either all or some Container Threat Detection findings are not being sent to Security Command Center.

To remediate this finding, follow these steps:

1. Grant the Container Threat Detection Service Agent role (roles/containerthreatdetection.serviceAgent) to the service account. For more information, see Grant a single role.

   Alternatively, if you want to use a custom role, make sure it has the permissions in the Container Threat Detection Service Agent role.

2. Make sure that there are no IAM deny policies preventing the service account from using any of the permissions in the Container Threat Detection Service Agent role. If there is a deny policy blocking access, add the service account as an exception principal in the deny policy.

For more information about the Container Threat Detection service account and the role and permissions it requires, see Required IAM permissions

Learn about this finding type's supported assets and scan settings.

Misconfigured Cloud Logging Export

Category name in the API: MISCONFIGURED_CLOUD_LOGGING_EXPORT

The project configured for continuous export to Cloud Logging is unavailable. As a result, Security Command Center can't send findings to Logging.

To remediate this finding, do one of the following:

• If the project recovery period has not elapsed, restore the missing project.

  Note: You can verify the number of days left in the recovery period by using the gcloud projects describe Google Cloud CLI method.

• If the project has been permanently deleted, configure a new or existing project for Logging exports.

Learn about this finding type's supported assets and scan settings.

VPC Service Controls Restriction

Category name in the API: VPC_SC_RESTRICTION

Security Health Analytics can't produce certain findings for a project, because the project is protected by a service perimeter. You must grant the Security Command Center service account inbound access to the service perimeter.

The service account's identifier is an email address with the following format:

[e-mail]

Replace the following:

• RESOURCE_KEYWORD: the keyword org or project, depending on what resource owns the service account

• RESOURCE_ID: one of the following:

  • The organization ID if the service account is owned by the organization
  • The project number if the service account is owned by a project

If you have both organization-level and project-level service accounts, apply the remediation to both of them.

To remediate this finding, follow these steps.

Step 1: Determine which service perimeter is blocking Security Health Analytics

1. Get the VPC Service Controls unique ID and the project ID associated with the finding:

   1. To view the finding's details, click its category name.
   2. In the Description field, copy the VPC Service Controls unique ID—for example, 5e4GI409D6BTWfOp_6C-uSwmTpOQWcmW82sfZW9VIdRhGO5pXyCJPQ.
   3. In the Resource path field, copy the project's ID.

2. Obtain the access policy ID and the service perimeter's name:

   1. In the Google Cloud console, go to the Logs Explorer page.

      Go to Logs Explorer

   2. On the toolbar, select the project associated with the finding.

      

   3. In the search box, enter the error's unique ID.

      

      If the error doesn't appear in the query results, extend the timeline in the Histogram, and then rerun the query.

   4. Click the error that appears.

   5. Click Expand nested fields.

   6. Copy the value of the servicePerimeterName field. The value has the following format:

      accessPolicies/ACCESS_POLICY/servicePerimeters/SERVICE_PERIMETER
      

      In this example, the service perimeter's full resource name is accessPolicies/540107806624/servicePerimeters/vpc_sc_misconfigured.

      • ACCESS_POLICY is the access policy ID—for example, 540107806624.

      • SERVICE_PERIMETER is the service perimeter's name—for example, vpc_sc_misconfigured.

        

   7. To get the display name that corresponds to the access policy ID, use the gcloud CLI.

      If you can't make organization-level queries, ask your administrator to perform this step.

      gcloud access-context-manager policies list \
          --organization ORGANIZATION_ID
      

      Replace ORGANIZATION_ID with your organization's numerical ID.

      You get an output similar to the following:

      NAME          ORGANIZATION  SCOPES                 TITLE           ETAG
      540107806624  549441802605                         default policy  2a9a7e30cbc14371
      352948212018  549441802605  projects/393598488212  another_policy  d7b47a9ecebd4659
      

      The display name is the title that corresponds to the access policy ID. Take note of the access policy's display name and the service perimeter's name. You need them in the next section.

Step 2: Create an ingress rule that grants access to the project

This section requires you to have organization-level access to VPC Service Controls. If you don't have organization-level access, ask your administrator to perform these steps.

In the following steps, you create an ingress rule on the service perimeter that you identified in step 1.

Note: If you were unable to identify the restricted service perimeter in step 1, we recommend performing the following steps for each service perimeter in your organization that restricts access to the project and to the services that Security Health Analytics calls. You can check each service perimeter's settings through the Google Cloud console or through the Google Cloud CLI.

To grant a service account inbound access to a service perimeter, follow these steps.

1. Go to VPC Service Controls.

   Go to VPC Service Controls

2. On the toolbar, select your Google Cloud organization.

   

3. In the drop-down list, select the access policy that contains the service perimeter you want to grant access to.

   

   The service perimeters associated with the access policy appear in the list.

4. Click the name of the service perimeter.

5. Click Edit perimeter

6. In the navigation menu, click Ingress Policy.

7. Click Add rule.

8. Configure the rule as follows:

   FROM attributes of the API client

   1. For Source, select All sources.
   2. For Identity, select Selected identities.
   3. In the Add User/Service Account field, click Select.
   4. Enter the service account email address. If you have both organization-level and project-level service accounts, add both of them.
   5. Click Save.

   TO attributes of GCP services/resources

   1. For Project, select All projects, or select the project specified in the finding.

   2. For Services, select All services or select each of the following individual services that Security Health Analytics requires:

      • BigQuery API
      • Binary Authorization API
      • Cloud Logging API
      • Cloud Monitoring API
      • Compute Engine API
      • Kubernetes Engine API

   If a service perimeter restricts access to a required service, Security Health Analytics can't produce findings for that service.

9. In the navigation menu, click Save.

For more information, see Configuring ingress and egress policies.

Learn about this finding type's supported assets and scan settings.

Security Command Center service account missing permissions

Category name in the API: SCC_SERVICE_ACCOUNT_MISSING_PERMISSIONS

Security Command Center's service agent is missing the permissions needed to function properly.

The service account's identifier is an email address with the following format:

[e-mail]

Replace the following:

• RESOURCE_KEYWORD: the keyword org or project, depending on what resource owns the service account

• RESOURCE_ID: one of the following:

  • The organization ID if the service account is owned by the organization
  • The project number if the service account is owned by a project

If you have both organization-level and project-level service accounts, apply the remediation to both of them.

To remediate this finding, follow these steps:

1. Grant the Security Center Service Agent (roles/securitycenter.serviceAgent) role to the service account.

   For more information, see Grant a single role.

   Alternatively, if you want to use a custom role, make sure it has the permissions in the Security Center Service Agent role.

2. Make sure that there are no IAM deny policies preventing the service account from using any of the permissions in the required roles. If there is a deny policy blocking access, add the service account as an exception principal in the deny policy.

Learn about this finding type's supported assets and scan settings.

What's next

Learn about Security Command Center errors.