Download Python packages using direct repository access

After you integrate Assured OSS with Security Command Center, Assured Open Source Software packages are hosted in an Artifact Registry repository that is created in a project that you control.

This page explains how you can connect to the Artifact Registry repository for Assured OSS to directly access and download the Python packages.

This document applies to the Assured OSS premium tier only. For the free tier, see Download Python packages using direct repository access for the free tier.

Before you begin

Integrate Assured OSS with Security Command Center.
Validate connectivity to Assured OSS for the requested service accounts.
Install the latest version of the Google Cloud CLI.
If you have installed the Google Cloud CLI previously, make sure you have the latest version by running the command:
```
gcloud components update
```

Set up authentication

You can authenticate to the Assured OSS Python package repository using one of the following methods:

Authenticate with a keyring
Authenticate with a service account key

The following sections describe how to set up these authentication methods.

Authenticate with keyring

To use the Python keyring to authenticate to the Artifact Registry, see authenticating with keyring and the information on the credential search order. We recommend that you use the Python keyring for authentication.

To set up the keyring for authentication, do the following:

Install the keyring library:
```
pip install keyring
```

Install the Artifact Registry backend:

pip install keyrings.google-artifactregistry-auth

List backends to confirm the installation:
```
keyring --list-backends
```
The list should include the following:
- ChainerBackend(priority:10)
- GooglePythonAuth(priority:9)
For information about setting up Application Default Credentials, see Set up authentication.

This step ensures that the Assured OSS credential helper obtains your key when connecting with the repositories.

Authenticate with a service account key

Authenticate with a service account key when the application requires authentication with a username and a password.

Replace the package repository URL https://us-python.pkg.dev/PROJECT_ID/assuredoss-python/simple with the URL https://_json_key_base64:BASE64_KEY@us-python.pkg.dev/PROJECT_ID/assuredoss-python/simple

Replace the following:

Replace PROJECT_ID with the ID of the project that you selected when you set up Assured Open Source Software.
Replace BASE64_KEY with the base64-encoded service account JSON key file. To convert the entire service account JSON key file to base64 encoding, use the following command:
```
BASE64_KEY=$(cat KEY_FILE_LOCATION | base64 -w 0)
```
Replace KEY_FILE_LOCATION with the location of the service account JSON key file.

Install packages

The following instructions assume that you're using PyPI as the repository to download your dependencies. If you're using another repository, you must use different steps to download dependencies.

Install packages existing in Assured OSS

To specify the Assured OSS Python packages that you want to download, create two requirements.txt files. The sample files are the following:

requirements-google.txt

# Packages present in Google's Artifact Registry

urllib3==1.26.11 --hash=sha256:1cffe1aa066363a75c856f261c8fce62d87f7c40ce0f46453ea12bf652b12a13
jsonschema==4.13.0 --hash=sha256:29895bfe55b93b75552fbdd1e09aa0c82b7c1c9395d4f267e10c7d43cd31a74e

requirements-pypi.txt

# Packages present in Google's Artifact Registry are mentioned here so that pip
# downloads their dependencies from PyPI.

urllib3==1.26.11
jsonschema==4.13.0

# Below this comment, add any package version which you need but is NOT
# present in Google's Artifact Registry and therefore needs to be downloaded from
# PyPI.

To download the packages, run the following commands:

To download the required packages which are present in the Artifact Registry for Assured OSS, run the following:
```
 pip install --require-hashes --requirement=requirements-google.txt \
     --index-url https://_json_key_base64:BASE64_KEY@us-python.pkg.dev/PROJECT_ID/assuredoss-python repo/simple -v \
     --no-deps
```
Consider the following:
- --require-hashes is optional. If included, hashes are specified for all packages and for all package versions in the requirements.txt file.
- Replace BASE64_KEY with the base64-encoded service account JSON file.
- -v is optional. If specified, the command gives more output.
To download the required packages which aren't present in the Artifact Registry for Assured OSS, run the following:
```
 pip install --requirement=requirements-pypi.txt --index-url https://pypi.org/simple -v
```
This command also downloads the missing dependencies of the packages that you have downloaded using the previous command.

List all Python packages available in Assured OSS

To use an API to get a list of all the Python packages in the Artifact Registry repository, see List all Python packages available in Assured OSS.

Generating the `requirements-google.txt` file

This section provides additional information about generating the requirements-google.txt file that you need to download the Python packages. To download the hash and generate the requirements-google.txt file for your environment, you can use one of the following two options:

Use a script to generate a single requirements file for all artifacts.
Download a separate requirements file for each artifact.

The following sections provide more information about these options.

Option 1: Use a script to generate a single requirements file for all artifacts

Use the generator.sh script to generate a single requirements-google.txt file for all the Python packages (along with their hashes) available with Assured OSS for linux operating system. The packages must satisfy constraints such as Python version, machine architecture, and operating system. You can then delete the package versions which you don't require and use the resulting file.

The generator.sh script helps in the following two ways:

It generates the most recent list of Python package versions available with Assured OSS that can be successfully installed on your system.
It generates the requirements-google.txt file along with all the hashes.

The required script and its README.md file are available in a Cloud Storage bucket (gs://cloud-aoss/utils/python-requirements-txt/v1.0) which can be downloaded using the Google Cloud CLI.

To download the script and the README.md file, follow these steps:

Authenticate with the service account to access the Cloud Storage bucket using the following command:
```
gcloud auth activate-service-account --key-file KEY_FILE
```
Replace KEY_FILE with the path to the file containing the credentials of the service account provided during the set up.
Download the generator.sh to your machine using the following command:
```
gcloud storage cp gs://cloud-aoss/utils/python-requirements-txt/v1.0/generator.sh PATH_TO_LOCAL_STORE --recursive
```
Replace PATH_TO_LOCAL_STORE with the local path where you want to save the downloaded file.
Download the README.md file using the following command:
```
gcloud storage cp gs://cloud-aoss/utils/python-requirements-txt/v1.0/README.md PATH_TO_LOCAL_STORE --recursive
```
Replace PATH_TO_LOCAL_STORE with the local path where you want to save the downloaded file. The README.md file contains instructions on how to use the script.

To run the script, use the following commands:

To generate the requirements-google.txt file, run the following command:
```
 chmod +x generator.sh
 ./generator.sh
```
To get the information of the package in a CSV file, run the following command:
```
chmod +x generator.sh
./generator.sh -i
```

Option 2: Download `requirements.txt` for each required artifact

You can also download a separate requirements.txt (containing the hash) file for each Python artifact and then combine them into a single requirements.txt file.

Artifact hashes are available in a Cloud Storage bucket which can be downloaded using the gcloud CLI. Hashes for each package and a version are located at the gs://cloud-aoss/python/PACKAGE_NAME/VERSION Cloud Storage bucket location.

To download the requirements.txt file, follow these steps:

Authenticate with the service account to access the Cloud Storage bucket using the following command:
```
gcloud auth activate-service-account --key-file KEY_FILE
```
Replace KEY_FILE with the path to the file containing the service account credentials.
Download the requirements.txt of a specific package and version to your local machine using the following command:
```
gcloud storage cp gs://cloud-aoss/python/PACKAGE_NAME/VERSION PATH_TO_LOCAL_STORE --recursive
```
Replace the following:
- PACKAGE_NAME: the package name
- VERSION: the version of the package
- PATH_TO_LOCAL_STORE: the local path where you want to download the file
Sample command:

gcloud storage cp gs://cloud-aoss/python/bleach/5.0.0 /tmp/bleach --recursive

Sample requirements.txt file:
```
bleach==5.0.0 \
 --hash=sha256:6d286e765bfd3e309209cfa1d063e4d46afa966dea8cb97431c02b1e3067d812
```
The contents of each such requirements.txt file can be combined into a single requirements-google.txt file.