Troubleshooting environment creation

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page provides troubleshooting information for problems that you might encounter while creating Cloud Composer environments.

For troubleshooting information related to updating and upgrading environments, see Troubleshooting environment updates and upgrades.

When Cloud Composer environments are created, the majority of issues happen because of the following reasons:

• Service account permission problems.

• Incorrect Firewall, DNS or routing information.

• Network-related issues. For example, invalid VPC configuration, IP address conflicts, or network IP ranges that are too narrow.

• Quota-related issues.

• Incompatible Organization Policies.

Insufficient permissions to create an environment

Note: You can validate that you configured permissions are correct using the IAM permission validation script. The script validates the permissions for both Shared VPC and non-Shared VPC configuration.

If Cloud Composer cannot create an environment because your account has insufficient permissions, it outputs the following error messages:

ERROR: (gcloud.composer.environments.create) PERMISSION_DENIED: The caller
does not have permission

or

ERROR: (gcloud.composer.environments.create) PERMISSION_DENIED: User not
authorized to act as service account <service-account-name>.
The user must be granted iam.serviceAccounts.actAs permission, included in
Owner, Editor, Service Account User role. See https://cloud.google.com/iam/docs
/understanding-service-accounts for additional details.

Solution: Assign roles to both to your account and to the service account of your environment as described in Access control.

• In Cloud Composer 2, make sure that Cloud Composer Service Agent service account ([e-mail]) has the Cloud Composer v2 API Service Agent Extension role assigned.

• Make sure that Google APIs Service Agent ([e-mail]) has the Editor role assigned.

• In the Shared VPC configuration, follow Configure Shared VPC instructions.

The service account of the environment has insufficient permissions

When creating a Cloud Composer environment, you specify a service account that runs the environment's GKE cluster nodes. If this service account does not have enough permissions for the requested operation, Cloud Composer outputs the following error:

Errors in: [Web server]; Error messages:
  Creation of airflow web server version failed. This may be an intermittent
  issue of the App Engine service. You may retry the operation later.
{"ResourceType":"appengine.v1.version","ResourceErrorCode":"504","ResourceError
Message":"Your deployment has failed to become healthy in the allotted time
and therefore was rolled back. If you believe this was an error, try adjusting
the 'app_start_timeout_sec' setting in the 'readiness_check' section."}

Solution: Assign roles to both to your account and to the service account of your environment as described in Access control.

Warnings about missing IAM roles in service accounts

When an environment creation fails, Cloud Composer generates the following warning message after an error occurred: The issue may be caused by missing IAM roles in the following Service Accounts ....

This warning message highlights possible causes for the error. Cloud Composer checks for required roles on the service accounts in your project, and if these roles are not present, it generates this warning message.

Note: This check applies only to the project where you create the environment. In case of Shared VPC configuration, Cloud Composer does not perform this check in the host project. To enable this check in Shared VPC, the Cloud Composer Service Agent account from the service project must have permission to view roles in the host project. To allow it, grant the Role Viewer (roles/iam.roleViewer) role to the Cloud Composer Service Agent account from the service project at the host project level.

Solution: Check that service accounts mentioned in the warning message have the required roles. For more information about roles and permissions in Cloud Composer, see Access control.

In some cases, you can ignore this warning. Cloud Composer does not check individual permissions assigned to roles. For example, If you use custom IAM roles, it is possible that the service account mentioned in the warning message already has all required permissions. In this case, you can ignore this warning.

Incompatible organization policies

The following policies must be configured appropriately so that Cloud Composer environments can be created successfully.

Organization Policy	Cloud Composer 3	Cloud Composer 2	Cloud Composer 1
compute.disableSerialPortLogging	Any value is allowed	Must be disabled	Disabled for versions earlier than 1.13.0; otherwise any value
compute.requireOsLogin	Any value is allowed	Any value is allowed	Must be disabled
compute.vmCanIpForward	Any value is allowed	Any value is allowed	Must be allowed (required for Cloud Composer-owned GKE clusters) when VPC-native mode (using alias IP) is not configured
compute.vmExternalIpAccess	Any value is allowed	Must be allowed for Public IP environments	Must be allowed for Public IP environments
compute.restrictVpcPeering	Can be enforced	Cannot be enforced	Cannot be enforced
compute.disablePrivateServiceConnectCreationForConsumers	Any value is allowed	Cannot disallow SERVICE_PRODUCERS for Private and Public IP environments. Does not affect existing environments, they can operate when this policy is enabled.	Cannot disallow SERVICE_PRODUCERS for Private IP environments. Does not affect existing environments, they can operate when this policy is enabled.
compute.restrictPrivateServiceConnectProducer	When active, allowlist the google.com organization	When active, allowlist the google.com organization	Any value is allowed

For more information, see the Known issues page and Organization policy constraints.

Restricting services used within organization or project

Organization or project administrators can restrict what Google services can be used in their projects using the gcp.restrictServiceUsage organization policy constraint.

When using this organization policy, it's important to allow all the services required by Cloud Composer.

What's next

• Creating environments
• Troubleshooting environment updates and upgrades
• Access control