You're viewing Apigee and Apigee hybrid documentation.
View
Apigee Edge documentation.
This document describes how Google manages security vulnerabilities and patches for Apigee hybrid. Except where otherwise noted, Apigee includes both the management plane and the data plane.
Shared patching responsibility
Patching is a shared responsibility between Google and the customer. Apigee X and Apigee hybrid have different shared responsibilities given the data plane for Apigee hybrid is entirely managed by the customer. For information on the Apigee hybrid shared responsibilities, see Apigee hybrid shared responsibility model.
How vulnerabilities are discovered
Google takes a proactive approach to the security of software systems, using Secure by Default design principles and implementing various security hardening practices.
For example, containerized applications power the various Apigee API Management Platform features. The containerized applications are deployed on Kubernetes. The container images are built on top of minimal base images (for example, distroless base images) for maximum security and improved performance.
However, even the best designed software systems can have vulnerabilities. To find those vulnerabilities and patch them before they can be exploited, Google has made significant investments on multiple fronts.
Security scanners
Google proactively identifies and fixes vulnerabilities on different types of container images:
- First party containers: Container images built and distributed by Google as part of the Apigee platform. These are Google proprietary applications that power the Apigee API Management platform including core functionality such as traffic routing, quota management, and key management.
- Third party containers: Container images built by the open source community but distributed by Google as part of the Apigee platform. These are mostly open source components that are used by the platform for common operational tasks such as logging, monitoring, and certificate management.
Google scans containers using Container Registry Container Analysis to discover vulnerabilities and missing patches in first party and third party containers. If fixes are available, Google begins the patching and release process. Such scans are run on a regular basis (when new images are published) as well as on-demand (before a release) to maximize the chances of detecting new vulnerabilities and early remediation or mitigation planning.
Security research
In addition to automated scanning, Google discovers and patches vulnerabilities unknown to scanners in these ways:
- Google performs its own security and compliance audits, application and network
penetration testing, segmentation testing, and security vulnerability discovery across all Apigee
components.
Specialized teams inside Google and trusted third-party security vendors conduct their own attack research. - Google collaborates with other industry and open source software partners who share
vulnerabilities, security research, and patches before the public release of the vulnerability.
The goal of this collaboration is to patch large pieces of Internet infrastructure before the vulnerability is announced to the public. In some cases, Google contributes vulnerabilities found to this community. For example, Google's Project Zero discovered and publicized the Spectre and Meltdown vulnerabilities. The Google Cloud Security team also regularly finds and fixes vulnerabilities in the Kernel-based Virtual Machine (KVM).
Vulnerability Reward Programs
Google actively engages with the security research community through multiple vulnerability reward programs. A dedicated Google Cloud vulnerability rewards program provides significant bounties, including $133,337 for the best cloud vulnerability found each year. The program covers all Apigee software dependencies.
OSS
Google's security collaboration happens on many levels. Sometimes it occurs formally through programs where organizations sign up to receive pre-release notifications about software vulnerabilities for products such as Kubernetes and Envoy. Collaboration also happens informally due to our engagement with many open source projects such as the Linux kernel, container runtimes, virtualization technology, and others.
While less severe vulnerabilities are discovered and patched outside of these processes, most critical security vulnerabilities are reported privately through one of the channels previously listed. Early reporting gives Google time before the vulnerability becomes public to research how it affects Apigee, develop patches or mitigations, and prepare advice and communications for customers. When possible, Google patches all clusters (for Apigee X) before the public release of the vulnerability. For Apigee hybrid, patch releases are made available on a regular basis for addressing security vulnerabilities in the container images and customers are encouraged to stay up-to-date with the latest patch versions.
How vulnerabilities are classified
Apigee makes large investments in security hardening on the entire stack, including the base image, the third party libraries and the first party application software, in addition to setting good defaults, security-hardened configurations, and managed components. Combined, these efforts help to reduce the impact and likelihood of vulnerabilities.
The Apigee security team classifies vulnerabilities according to the Common Vulnerability Scoring System (CVSS).
The following table describes vulnerability severity categories:
| Severity | Description |
| Critical | A vulnerability easily exploitable in all clusters by an unauthenticated remote attacker that leads to full system compromise. |
| High | A vulnerability easily exploitable for many clusters that leads to loss of confidentiality, integrity, or availability. |
| Medium | A vulnerability exploitable for some clusters where loss of confidentiality, integrity, or availability is limited by common configurations, difficulty of the exploit itself, required access, or user interaction. |
| Low | All other vulnerabilities. Exploitation is unlikely or consequences of exploitation are limited. |
How vulnerabilities and patches are communicated
Google's goal is to mitigate detected vulnerabilities within a time period appropriate for the risks they represent. Apigee is included within Google Cloud's FedRAMP provisional ATO which requires that known vulnerabilities be remediated within specific timeframes according to their severity level as specified in FedRAMP RA-5d. Google Cloud's FedRAMP provisional ATO doesn't include Apigee hybrid data plane components (customer-managed) but we aim for the same remediation timeframes on those products.
Vulnerabilities detected by proactive scanning
Google detects security vulnerabilities through proactive scanning of the binaries and the underlying infrastructure that hosts the Apigee platform. Apigee releases regular patch updates to address these vulnerabilities in a timely manner depending on the severity of underlying CVE(s). Patching a vulnerability involves upgrading to a new Apigee hybrid version-‐either a minor version upgrade or a patch version upgrade, depending on the nature of the vulnerability. Such vulnerabilities are mostly addressed as part of monthly patch releases for Apigee hybrid and included in the regular software updates to our managed fleet in the case of Apigee X. Security patches are communicated through release notes for both Apigee X and Apigee hybrid.
Fixing some vulnerabilities requires only a control plane upgrade, performed automatically by Google on Apigee, while others require new binaries to be rolled out to the data plane. In case of Apigee X, Google takes care of rolling out the new binaries to the entire fleet. Customers running Apigee Apigee hybrid must apply the patch release to their Apigee hybrid clusters in order to roll out the updated binaries.
To keep clusters patched and hardened against vulnerabilities of all severities, we recommend applying the latest patch release for any given minor version of Apigee hybrid. For those running Apigee hybrid on Anthos, Google recommends upgrading your Anthos components at least monthly.
Customer-reported vulnerabilities
With Apigee hybrid, customers are given Apigee binaries to run in their data centers or their preferred cloud platforms. As part of a customer's security standards for launching software into production in their own data centers, they may run a series of security tests. These tests could include penetration testing, container scanning, static code scanning, etc. These tests might report possible vulnerabilities in the Apigee software. Prior to enabling these packages in their data centers, customers need to determine if these reported items are exploitable and therefore a security risk.
Vulnerability with an exploit Proof of Concept
If the customer identifies an exploitable vulnerability and has a proof of concept (POC) on how to exploit the vulnerability, they should report this issue through the Google Vulnerabilities Reward Program found at goo.gle/vulnz. This reports the issue to the Google security team who will validate the proof of concept and determine its severity and potential impact. The issue will then be escalated to Apigee. The customer may be eligible for a reward through the VRP program.
Vulnerability identified by an automated tool
If the customer has generated a report of possible vulnerabilities in the Apigee product based on static scanning or another tool or technique, but does not have any proof of concepts on how to exploit the issue(s), these items can be reported to support through Google Cloud support portal. By reporting the issue to support, the customer has a ticket number for tracking and can see updates and responses to the report. The support team will then escalate the reported issue(s) internally as appropriate.
CVE identifiers
Customers should include as much information about the vulnerability as possible, and specifically include the CVE identifier, library/package name, name of the container image etc. for each item. CVEs allow us to know we are looking at the same vulnerability. Providing only a description of the issue or other proprietary tracking number does not allow correlation of the issue across detection tools or reporting processes. Without a CVE, Google may be unable to respond to the specific item.
Response
For items reported to support which have a severity score of critical or high, customers can expect a response through the support ticketing system. For items reported to the VRP, please see the rules and documentation provided by the program.