Troubleshoot serving issues

The following sections cover app initialization and serving errors.

Permission error when creating an app with the default service account

When you create an app after enabling the App Engine API for the first time, it fails with the following errors:

gcloud CLI

An internal error occurred while calling service consumer manager for service account.
Creating  App Engine application in project [PROJECT__ID] and [REGION]....failed. DEBUG: (gcloud.app.create) Error Response: [13] an internal error has occurred

Request logs

Service account creation is not allowed on this project.

Console

Error while initialising App Engine.

This error can occur due to the enforcement of the organization policy constraint constraints/iam.disableServiceAccountCreation when creating your app. This policy prevents the provisioning of the App Engine default service account PROJECT_ID@appspot.gserviceaccount.com.

To resolve this issue, you must temporarily remove the organization policy constraint constraints/iam.disableServiceAccountCreation to allow for the creation and deployment of the App Engine default service account. The default service account is necessary for app creation and can't be skipped. This is also applicable when you use a per-version service account. The App Engine default service account can then be deleted or replaced with a service account that you create after successful deployment.

If you are using a service account that you created, review the Overview of role recommendations to understand how to enforce restricting permissions, such as providing a token creator role on the service account you create for the service agent.

Nginx fails to connect or contact the app container

The following error only occurs in the App Engine flexible environment and typically returns with 502 errors immediately after the error:

recv() failed (104: Connection reset by peer) while reading response header from upstream

This error indicates that nginx reverse proxy (nginx sidecar) is unable to reach the app container. In the logs, you can compare the close timing of the 502 error in the nginx log with the timing of the nginx.error log. A nginx.error followed immediately by a 502 nginx error is likely the cause of the nginx 502 error.

This error often occurs when the connection keepalive timeout of the application is smaller than the keepalive timeout of nginx. As nginx in the App Engine flexible environment has keepalive_timeout of 650 seconds, applications need to keep connections alive for at least this long. By default, Node.js applications have keepAliveTimeout of 5000 milliseconds. In this case, you can set server.keepAliveTimeout to 700000 milliseconds.

To troubleshoot, check the logs written by the code running in your app container by connecting to the VM instance, and add more logging, if necessary, to find the root cause.

Insufficient memory

The following error occurs in the App Engine flexible environment and typically returns with 502 errors:

kernel: [  133.706951] Out of memory: Kill process 4490 (java) score 878 or sacrifice child
kernel: [  133.714468] Killed process 4306 (java) total-vm:5332376kB, anon-rss:2712108kB, file-rss:0kB

This error indicates that App Engine has terminated the application.

This error occurs when the instance has insufficient memory. By default App Engine flexible environment has 1GB of memory, with only 600MB available for the application container.

To troubleshoot, check the logs and if you see an Out of Memory entry. Update your memory_gb configuration and redeploy.

Insufficient open connections to handle incoming requests

Apps might encounter a 502 error if the maximum number of waiting connections is equal to or greater than 75% of the number of active connections.

To resolve the issue, check the Cloud Monitoring metrics for the maximum number of active and waiting connections, and decrease the number of waiting connections to ensure that the maximum number of waiting connections is less than or equal to 75% of the number of active connections.