Troubleshoot permission issues with a Google service account

This page shows you how to resolve permission issues that occur when using Config Sync with a Google service account.

Missing reader access

When using a Google service account (spec.git.gcpServiceAccountEmail, or spec.oci.gcpServiceAccountEmail, or spec.helm.gcpServiceAccountEmail) to authenticate to Cloud Source Repositories or Artifact Registry, the Google service account requires the following reader access:

  • Cloud Source Repositories: roles/source.reader
  • Artifact Registry: roles/artifactregistry.reader

If the service account is missing these roles, git-sync or oci-sync or helm-sync fails with errors similar to the following:

failed to pull image us-docker.pkg.dev/...: GET https://us-docker.pkg.dev/v2/token?scope=repository...: DENIED: Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource \"projects/.../locations/us/repositories/...\" (or it may not exist)

"Err":"failed to render the helm chart: exit status 1, stdout: Error: failed to download ...

To fix this issue, grant the service account the correct reader access.

Missing IAM policy binding with Workload Identity Federation for GKE

When using a Google service account for authentication, an IAM policy binding is required between the Google service account and the Kubernetes service account. If the IAM policy binding is missing, you get the following error:

KNV2004: unable to sync repo Error in the git-sync container: ERROR: failed to call ASKPASS callback URL: auth URL returned status 500, body: "failed to get token from credentials: oauth2/google: status code 403: {\n \"error\": {\n \"code\": 403,\n \"message\": \"The caller does not have permission\",\n \"status\": \"PERMISSION_DENIED\"\n }\n}\n"

To fix the issue, create the following IAM policy binding:

gcloud iam service-accounts add-iam-policy-binding \
   --role roles/iam.workloadIdentityUser \
   --member "serviceAccount:PROJECT_ID.svc.id.goog[config-management-system/KSA_NAME]" \
   GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Replace the following:

  • PROJECT_ID: If you're using GKE Workload Identity Federation for GKE, add your organization's project ID. If you're using fleet Workload Identity Federation for GKE, you can use two different project IDs. In serviceAccount:PROJECT_ID, add the project ID of the fleet that your cluster is registered to. In GSA_NAME@PROJECT_ID, add a project ID for any project that has read access to the repository in Cloud Source Repositories.

  • KSA_NAME: the Kubernetes service account for the reconciler. For root repositories, if the RootSync name is root-sync, KSA_NAME is root-reconciler. Otherwise, it is root-reconciler-ROOT_SYNC_NAME.

    For namespace repositories, if the RepoSync name is repo-sync, KSA_NAME is ns-reconciler-NAMESPACE. Otherwise, it is ns-reconciler-NAMESPACE-REPO_SYNC_NAME.

  • GSA_NAME: the custom Google service account that you want to use to connect to Cloud Source Repositories. Make sure that the Google service account that you select has the source.reader role.

Creating this policy binding requires the iam.serviceAccounts.setIamPolicy permission.

Make sure that fleet Workload Identity Federation for GKE is enabled on your registered clusters. For more information, see Register a cluster. If your cluster is in a project different from the fleet host project, you need to bind the Google service account with the Kubernetes service account in the fleet host project.

Missing cloud-platform scope to access Cloud Source Repositories

When granting a Google service account access to your Git repository in Cloud Source Repositories, the read-only scope must be included in access scopes for the nodes in the cluster.

The read-only scope can be added by including cloud-source-repos-ro in the --scopes list specified at cluster creation time, or by using the cloud-platform scope at cluster creation time. For example:

gcloud container clusters create CLUSTER_NAME --scopes=cloud-platform

If the read-only scope is missing, you see an error similar to the following:

Error in the git-sync container: {"Msg":"unexpected error syncing repo, will retry","Err":"Run(git clone -v --no-checkout -b main --depth 1 https://source.developers.google.com/p/PROJECT_ID/r/csr-auth-test /repo/source): exit status 128: { stdout: \"\", stderr: \"Cloning into '/repo/source'...\\nremote: INVALID_ARGUMENT: Request contains an invalid argument\\nremote: [type.googleapis.com/google.rpc.LocalizedMessage]\\nremote: locale: \\\"en-US\\\"\\nremote: message: \\\"Invalid authentication credentials. Please generate a new identifier: https://source.developers.google.com/new-password\\\"\\nremote: \\nremote: [type.googleapis.com/google.rpc.RequestInfo]\\nremote: request_id: \\\"fee01d10ba494552922d42a9b6c4ecf3\\\"\\nfatal: unable to access 'https://source.developers.google.com/p/PROJECT_ID/r/csr-auth-test/': The requested URL returned error: 400\\n\" }","Args":{}}

Missing storage-ro scope to access Artifact Registry

When you're using Artifact Registry, image layers are kept in Cloud Storage buckets. When granting a Google service account access to your OCI image or Helm Chart in Artifact Registry, the read-only scope must be included in access scopes for the nodes in the cluster.

The read-only scope can be added by including storage-ro in the --scopes list specified at cluster creation time, or by using the cloud-platform scope at cluster creation time. For example:

gcloud container clusters create CLUSTER_NAME --scopes=cloud-platform

If the read-only scope is missing, you see an error similar to the following:

Error in the oci-sync container: {"Msg":"unexpected error fetching package, will retry","Err":"failed to pull image us-docker.pkg.dev/...: GET https://us-docker.pkg.dev/v2/token?scope=repository%3A...%3Apull\u0026service=us-docker.pkg.dev: UNAUTHORIZED: failed authentication","Args":{}}

The error message might not include the full details for what caused the error, but it does provide a command which prints the logs from the git-sync container which might have more information.

If you've enabled the RootSync or RepoSync APIs, run the following command:

kubectl logs -n config-management-system -l app=reconciler -c git-sync

If you haven't enabled the RootSync or RepoSync APIs, run the following command:

kubectl logs -n config-management-system -l app=git-importer -c git-sync

RootSync and RepoSync APIs are enabled by default if you used the Google Cloud console or the Google Cloud CLI to install Config Sync and you cannot disable it.

What's next

  • If you're still experiencing issues, check to see if your problem is a known issue.