Migrate from Python 2.7 to the latest Python 3 runtime

This page covers instructions for migrating from the first-generation to the second-generation Python runtimes. To upgrade your second-generation app to use the latest supported version of Python, see Upgrade an existing application.

Python 2.7 has reached end of support on January 31, 2024. Your existing Python 2.7 applications will continue to run and receive traffic. However, App Engine might block re-deployment of applications that use runtimes after their end of support date. We recommend that you migrate to the latest supported version of Python by using the guidelines in this page.

Migrating to the Python 3 runtime lets you use up-to-date language features and build apps that are more portable, with idiomatic code. The Python 3 runtime uses the latest version of the open source Python interpreter provided by the Python Software Foundation. Apps built-in the Python 3 runtime can use Python's rich ecosystem of packages and frameworks in your app, including those that use C code, by declaring dependencies in a requirements.txt file.

Overview of the runtime migration process

We recommend the following incremental approach to the runtime migration, in which you maintain a functioning and testable application throughout the process:

  1. Upgrade your app to be compatible with Python 3.

    Several solutions are available to help with this upgrade. For example, use Six, Python-Future, or Python-Modernize.

    For more information about this step of the runtime migration process, see Porting Python 2 Code to Python 3 on the Python Software Foundation documentation site.

  2. Pick one of these implementation strategies for any App Engine bundled service your app uses:

    1. Migrate the legacy bundled services in your Python 2 app to unbundled Google Cloud services, third-party services, or other recommended replacements.

    2. Continue using legacy bundled services in your Python 3 apps. This approach gives you flexibility to move to unbundled services later in the migration cycle.

    Make sure to test your app after migrating each service.

  3. Prepare App Engine configuration files for the Python 3 runtime. Several important changes affect the configuration settings in app.yaml, including but not limited to:

    • Apps are now assumed to be threadsafe. If your application isn't threadsafe, you should set max_concurrent_requests in your app.yaml to 1. This setting may result in more instances being created than you need for a threadsafe app, and lead to unnecessary cost.

    • The app.yaml file no longer routes requests to your scripts. Instead, you are required to use a web framework with in-app routing, and update or remove all script handlers in app.yaml. For an example of how to do this with the Flask framework, see the App Engine migration guide code sample in GitHub.

      To learn more about changing this and other configuration files, see the Configuration files section.

  4. In the second-generation runtimes, app logs are no longer nested within the request logs. Additional steps are necessary to display the nested view of request and app logs in the Logs Explorer. For more information, see Migrate to Cloud Logging.

  5. Test and deploy your upgraded app in a Python 3 environment.

    After all of your tests pass, deploy the upgraded app to App Engine but prevent traffic from automatically routing to the new version. Use traffic splitting to slowly migrate traffic from your app in the Python 2 runtime to the app in the Python 3 runtime. If you run into problems, you can route all traffic to a stable version until the problem is fixed.

For examples of how to convert your Python 2 apps to Python 3, you can refer to these additional resources.

Key differences between the Python 2 and Python 3 runtimes

Most of the changes you need to make during the runtime migration come from the following differences between the Python 2 and Python 3 runtimes:

Memory usage differences

Second-generation runtimes see a higher baseline of memory usage compared to first-generation runtimes. This is due to multiple factors, such as different base image versions, and differences in how the two generations calculate memory usage.

Second-generation runtimes calculate instance memory usage as the sum of what an application process uses, and the number of application files dynamically cached in memory. To avoid memory-intensive applications from experiencing instance shutdowns due to exceeding memory limits, upgrade to a larger instance class with more memory.

CPU usage differences

Second-generation runtimes can see a higher baseline of CPU usage upon instance cold-start. Depending on an application's scaling configuration, this might have unintended side effects, such as, a higher instance count than anticipated if an application is configured to scale based on CPU utilization. To avoid this issue, review and test application scaling configurations to ensure the number of instances are acceptable.

Request header differences

First-generation runtimes allow request headers with underscores (e.g. X-Test-Foo_bar) to be forwarded to the application. Second-generation runtimes introduces Nginx into the host architecture. As a result of this change, second-generation runtimes are configured to automatically remove headers with underscores (_). To prevent application issues, avoid using underscores in application request headers.

Gunicorn worker differences

For Python 3+ runtimes, the number of Gunicorn workers have a direct impact on memory usage. The increase in memory usage is directly proportional to the increase in worker count. To reduce memory consumption, consider reducing the number of Gunicorn workers. See Entrypoint best practices for instructions on configuring Gunicorn worker count

Compatibility issues between Python 2 and Python 3

When Python 3 was first released in 2008, several backward incompatible changes were introduced to the language. Some of these changes require only minor updates to your code, such as changing the print statement to a print() function. Other changes may require significant updates to your code, such as updating the way that you handle binary data, text, and strings.

Many popular open source libraries, including the Python standard libraries, also changed when they moved from Python 2 to Python 3.

App Engine bundled services in the Python 3 runtime

To reduce migration effort and complexity, App Engine standard environment allows you to access many of legacy bundled services and APIs in the Python 3 runtime, such as Memcache. Your Python 3 app can call the bundled services APIs through language idiomatic libraries, and access the same functionality as on the Python 2 runtime.

You also have the option to use Google Cloud products that offer similar functionality as the legacy bundled services. We recommend that you consider migrating to the unbundled Google Cloud products, as doing so lets you take advantage of ongoing improvements and new features.

For the bundled services that are not available as separate products in Google Cloud, such as image processing, search, and messaging, you can use our suggested third-party providers or other workarounds.

Configuration files

Before you can run your app in the Python 3 runtime of the App Engine standard environment, you may need to change some of the configuration files that App Engine uses:

Web framework required to route requests for dynamic content

In the Python 2 runtime, you can create URL handlers in the app.yaml file to specify which app to run when a specific URL or URL pattern is requested.

In the Python 3 runtime, your app needs to use a web framework such as Flask or Django to route requests for dynamic content instead of using URL handlers in app.yaml. For static content, you can continue to create URL handlers in your app's app.yaml file.

Apps with only static content

When hosting a static webapp on App Engine, you specify handlers in your app.yaml file to map URLs to your static files.

In Python 2, if a request does not match any of the handlers specified in the app.yaml file, App Engine returns a 404 error code.

In Python 3, if a request does not match any of the handlers, App Engine looks for a main.py file, and returns a 5xx error if a main.py file is not found. Since App Engine apps with only static content do not require a main.py file, most users see this error, in addition to seeing instance startup errors in app logs.

To keep the same behavior of returning a 404 error when none of the static handlers match, and to avoid errors in logs, you can either:

  • Add a catch-all static handler pointing to an empty directory in the app.yaml file
  • Add a simple dynamic app in the main.py file to return a 404 error

Examples of using either option:

app.yaml

Create an empty directory in the root app directory, such as empty/. In the app.yaml file's handler section, create a new handler at the very end to catch all other URL patterns, and specify the empty directory in the static_files and upload elements:

  handlers:
  - url:
    .
    .
    .
  - url: /(.*)$
    static_files: empty/\1
    upload: empty/.*$

main.py

Create a main.py file and add the following code to return a 404 error:

  def app(env, start_response):
    start_response('404 Not Found', [('Content-Type','text/html')])
    return [b"Not Found"]

Testing

We recommend that you use a testing approach that is idiomatic to Python rather than being dependent on dev_appserver. For example, you might use venv to create an isolated local Python 3 environment. Any standard Python testing framework can be used to write your unit, integration, and system tests. You might also consider setting up development versions of your services or use the local emulators that are available for many Google Cloud products.

Optionally, you can use the preview version of dev_appserver which supports Python 3. To learn more about this testing feature, see Using the Local Development Server.

Deploying

Deployments via appcfg.py is not not supported for Python 3. Instead, use the gcloud command line tool to deploy your app.

Logging

Logging in the Python 3 runtime follows the logging standard in Cloud Logging. In the Python 3 runtime, app logs are no longer bundled with the request logs but are separated in different records. To learn more about reading and writing logs in the Python 3 runtime, see the logging guide.

Additional migration resources

For additional information on how to migrate your App Engine apps to standalone Cloud services or the Python 3 runtime, you can refer to these App Engine resources: