Create and manage feeds using the feed management UI

Supported in:

Google SecOps SIEM

This page provides information about how to create, manage, and troubleshoot feeds using the feed management UI. Managing the feeds includes modifying, enabling, and deleting the feeds.

Before you begin

Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Google Security Operations. For information about prerequisites specific to a feed type, see Configuration by source type. Search for the data feed type you need to setup and follow the instructions provided.

Supported compression formats

Supported compression formats for feed ingestion include: .gz, .tar.gz, .tar, .targz, and solr.gz.

Add a feed

To add a feed to your Google Security Operations account, complete the following steps. You can add up to five feeds for each log type.

1. From the Google Security Operations menu, select Settings, and then click Feeds. The data feeds listed on this page include all the feeds that Google has configured for your account in addition to the feeds that you have configured.

2. Click Add New. The Add feed window is displayed.

3. Add a feed name.

Note: New feeds require a feed name. Existing feeds will show [Not Configured] on the Feeds page.

1. In the Source type list, select the source type through which you intend to bring data into Google Security Operations. You can select from the following feed source types:

   • Amazon Data Firehose
   • Amazon S3
   • Amazon SQS
   • Google Cloud Pub/Sub
   • Google Cloud Storage
   • HTTP(S) Files (non-API)
   • Microsoft Azure Blob Storage
   • Third party API
   • Webhook
   Note: For Amazon SQS, make sure you enable the permissions for Google Security Operations to be able to delete the Amazon SQS queue messages.

2. In the Log type list, select the log type corresponding to the logs that you want to ingest. The logs available vary depending on which source type you selected previously. Click Next.

   If you select Google Cloud Storage as the source type, use the Get service account option to get a unique service account. In this document, see Google Cloud Storage feed setup example.

3. Specify the parameters needed from the Input Parameters tab. The options presented here vary depending on the source and log type selected on the Set Properties tab. Hold the pointer over the question icon for each field to get additional information on what you need to provide.

4. (Optional) You can specify a namespace here. For more information about namespaces, see the asset namespace documentation.

5. Click Next.

6. Review your new feed configuration from the Finalize tab. Click Submit when you are ready. Google Security Operations completes a validation check of the new feed. If the feed passes the check, a name is generated for the feed, it is submitted to Google Security Operations, and Google Security Operations begins to attempt to fetch data.

   

Delete Source Files

For several feed types, including Cloud Storage, there is a field in the Add new or Edit feed workflow labeled SOURCE DELETION OPTION. This menu has three options:

1. Never delete files
2. Delete transferred files and empty directories
3. Delete transferred files

Options 2 and 3 involve deletions: one for files and one for files and any empty directories. If you select either of those options, you need to add the permissions specific to your feed type. For information about permissions specific to a feed type, see Configuration by source type.

This option lets you delete an object out of the storage system after you have transferred it. Feeds always remember which objects (or files) they have transferred and never transfer the same file twice (unless it has been updated), but you have to set this option if you want the system to delete the source object after it has been (successfully) transferred.

Microsoft Azure Blob Storage doesn't support deletion of source files. The following source deletion options mustn't be used with Microsoft Azure Blob Storage source type:

• Delete transferred files and empty directories
• Delete transferred files

When you create a feed with Microsoft Azure Blob Storage source, select only the Never delete files option.

Set up a Pub/Sub push feed

To set up a Pub/Sub push feed, do the following:

1. Create a Pub/Sub push feed.
2. Specify the endpoint URL in a Pub/Sub subscription.

Create a Pub/Sub push feed

1. From the Google Security Operations menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Google Cloud Pub/Sub Push.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need this endpoint URL to create a push subscription in Pub/Sub.
11. To disable the feed, click the Feed Enabled toggle. The feed is enabled by default.
12. Click Done.

Specify the endpoint URL

After you create a Pub/Sub push feed, in Pub/Sub, create a push subscription, specify the HTTPS endpoint, and enable authentication.

1. Create a push subscription in Pub/Sub. For more information about how to create a push subscription, see Create push subscriptions.
2. Specify the endpoint URL, which is available in the Google Cloud Pub/Sub push feed.
3. Select Enable authentication, and select a service account.

Note: If you need to push logs from an another Google Cloud project that isn't bound to Google Security Operations, such as from a user-managed service account, you must set up a log sink. For more information about log sinks, see Route logs to supported destinations.

Set up an Amazon Data Firehose feed

To set up an Amazon Data Firehose feed, do the following:

1. Create an Amazon Data Firehose feed and copy the endpoint URL and secret key.
2. Create an API key to authenticate to Google Security Operations. You can also reuse your existing API key to authenticate to Google Security Operations.
3. Specify the endpoint URL in Amazon Data Firehose.

Create an Amazon Data Firehose feed

1. From the Google Security Operations menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Amazon Data Firehose.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key as you cannot view this secret again. You can generate a new secret key again, but regeneration of the secret key makes the previous secret key obsolete.
12. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need this endpoint URL when you specify the destination settings for your delivery stream in Amazon Data Firehose.
13. To disable the feed, click the Feed Enabled toggle. The feed is enabled by default.
14. Click Done.

Create an API key for the Amazon Data Firehose feed

1. Go to the Google Cloud console Credentials page.
2. Click Create credentials, and then select API key.
3. Restrict the API key access to the Google Security Operations API.

Specify the endpoint URL

In Amazon Data Firehose, specify the HTTPS endpoint and access key.

1. Append the API key to the feed endpoint URL and specify this URL as the HTTP endpoint URL in the following format:

     ENDPOINT_URL?key=API_KEY
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google Security Operations.

2. For the access key, specify the secret key that you obtained when you created the Amazon Data Firehose feed.

Set up an HTTPS webhook feed

To set up an HTTPS webhook feed, do the following:

1. Create an HTTPS webhook feed and copy the endpoint URL and secret key.
2. Create an API key that is specified with the endpoint URL. You can also reuse your existing API key to authenticate to Google Security Operations.
3. Specify the endpoint URL in your application.

Prerequisites

• Ensure that a Google Cloud project for Google Security Operations is configured and the Chronicle API is enabled for the project.

• Link a Google Security Operations instance to Google Cloud services.

  Note: Batches of data sent through Webhook feeds may experience ingestion delays if the request size or QPS limits are set too low. When calling the HTTPS push endpoint, the maximum request size is 4MB, and the maximum QPS is 15K.

Create an HTTPS webhook feed

1. From the Google Security Operations menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Webhook.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key as you cannot view this secret again. You can generate a new secret key again, but regeneration of the secret key makes the previous secret key obsolete.
12. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need to specify this endpoint URL in your client application.
13. To disable the feed, click the Feed Enabled toggle. The feed is enabled by default.
14. Click Done.

Create an API key for the webhook feed

1. Go to the Google Cloud console Credentials page.
2. Click Create credentials, and then select API key.
3. Restrict the API key access to the Google Security Operations API.

Specify the endpoint URL

1. In your client application, specify the HTTPS endpoint, which is available in the webhook feed.

2. Enable authentication by specifying the API key and secret key as part of the custom header in the following format:

   X-goog-api-key = API_KEY

   X-Webhook-Access-Key = SECRET

   We recommend that you specify the API key as a header instead of specifying it in the URL. If your webhook client doesn't support custom headers, you can specify the API key and secret key by using query parameters in the following format:

     ENDPOINT_URL?key=API_KEY&secret=SECRET
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google Security Operations.
   • SECRET: the secret key that you generated to authenticate the feed.

Google Cloud Storage feed setup example

1. From the Google Security Operations menu, select Settings, and then click Feeds.
2. Click Add New.
3. Select Google Cloud Storage for Source Type.
4. Select the Log type. For example, to create a feed for Google Kubernetes Engine audit logs, select Google Kubernetes Engine audit logs as the Log Type.
5. Click Get service account. Google Security Operations provides a unique service account that Google Security Operations uses to ingest data.
6. Configure access for the service account to access the Cloud Storage objects. In this document, see Grant access to the Google Security Operations service account.
7. Click Next.
8. Based on the Cloud Storage configuration that you created, specify values for the following fields:
   • Storage bucket URI
   • URI is a
   • Source deletion option
9. Click Next and then click Submit.

Grant access to the Google Security Operations service account

1. In the Google Cloud console, go to the Cloud Storage Buckets page.
   

   Go to Buckets

2. Grant access to the service account to the relevant Cloud Storage objects.

   • To grant read permission to a specific file, complete the following steps:

     1. Select the file and click Edit access.
     2. Click Add principal.
     3. In the New principals field, enter the name of the Google Security Operations service account.
     4. Assign a role that contains the read permission to the Google Security Operations service account. For example, Storage Object Viewer (roles/storage.objectViewer). This can only be done if you have not enabled uniform bucket-level access.
     5. Click Save.

   • To grant read permission to multiple files, you must grant access at the bucket level. You must add the Google Security Operations service account as a principal to your storage bucket and grant it the IAM Storage Object Viewer (roles/storage.objectViewer) role.

     If you configure the feed to delete source files, you must add the Google Security Operations service account as a principal on your bucket and grant it the IAM Storage Object Admin (roles/storage.objectAdmin) role.

Configure VPC Service Controls

If VPC Service Controls is enabled, an ingress rule is required to provide access to the Cloud Storage bucket.

The following Cloud Storage methods must be allowed in the ingress rule:

• google.storage.objects.list. Required for a single file feed.
• google.storage.objects.get. Required for feeds that require directory or subdirectory access.
• google.storage.objects.delete. Required for feeds that require deletion of the source file.

Sample ingress rule

- ingressFrom:
  identities:
    - serviceAccount:[e-mail]
  sources:
  - accessLevel: "*"
  ingressTo:
  operations:
  - serviceName: storage.googleapis.com
    methodSelectors:
    - method: google.storage.objects.list
    - method: google.storage.objects.get
    - method: google.storage.objects.delete
  resources:
  - projects/PROJECT_ID

Feed status

You can monitor the status of the feed from the initial Feeds page. Feeds can have the following statuses:

• Active—Feed is configured and ready to ingest data into your Google Security Operations account.
• InProgress—Google Security Operations is now attempting to pull data from the configured third party.
• Completed—Data successfully retrieved by this feed.
• Archived—Disabled feed.

• Failed—Feed is failing to successfully fetch data. This is likely due to a configuration issue. Click the question to display the configuration error. Once you have corrected the error and resubmitted the feed, return to the Feeds page to determine whether or not the feed is now working.

  Note: The Last succeeded on column on the Feeds page shows the last time data was fetched successfully by that feed. For a new feed, this field is blank. It might take approximately 5-10 minutes for a new feed to populate as Google Security Operations takes some time to process the data. Some feeds, such as Amazon SQS, could take longer than that. For feeds that never succeeded in the past, this field stays blank.

Edit feeds

From the Feeds page, you can edit an existing feed:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click Edit Feed. You can now alter the input parameters for the feed and resubmit it to Google Security Operations. Google Security Operations will attempt to use the edited feed.

   Note: Existing feeds show [Not Configured] as a feed name on the Feeds page. Edit the feed to add a feed name.

Enable and disable feeds

In the Status column, enabled feeds are labeled as Active, InProgress, Completed, or Failed. Disabled fields are labeled as Archived. For a description, see the feed status.

From the Feeds page, you can enable or disable any of the existing feeds:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. To enable a feed, click the Enable Feed toggle.

3. To disable a feed, click the Disable Feed toggle. The feed is now labeled as Archived.

   Note: When you disable a feed, it prevents new data from being added to the ingestion queue. Existing transfers (active or throttled) continue until finished. To stop data ingestion immediately, delete the feed.

Delete feeds

From the Feeds page, you can also delete an existing feed:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click Delete Feed. The DELETE FEED window opens. To permanently delete the feed, click Yes, delete it.

   Note: When you delete a feed, no new data will be ingested. There may be data already in the queue that will still be processed.

Control the rate of ingestion

When the data ingestion rate for a tenant reaches a certain threshold, Google Security Operations restricts the rate of ingestion for new data feeds to prevent a source with a high ingestion rate from affecting the ingestion rate of another data source. In this case, there is a delay but no data is lost. The ingestion volume and tenant's usage history determine the threshold.

You can request a rate limit increase by contacting Cloud Customer Care.

Troubleshooting

From the Feeds page, you can view details such as source type, log type, feed ID, and status of the existing feeds:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click View Feed. A dialog appears showing the feed details. For a failed feed, you can find error details under Details > Status.

For a failed feed, the details include the cause of the error and steps to fix it. The following table describes the error messages that you might encounter when working with data feeds.

Error Code	Cause	Troubleshooting
ACCESS_DENIED	The authentication account provided in the feed configuration lacks required permissions.	Verify the authentication account provided in the feed configuration has required permissions. Refer to the feeds documentation for the necessary permissions.
ACCESS_TOO_FREQUENT	The feed failed because there were too many attempts to reach the source.	Contact Google Security Operations support.
CONNECTION_DROPPED	A connection to the source was established, but the connection closed before the feed was complete.	This error is transient and application will retry the request. If the issue persists, contact support.
CONNECTION_FAILED	The application can't connect to the source IP address and port.	Check the following: The source is available. A firewall isn't blocking the connection. The IP address associated with the server is correct. If the problem continues, contact Google Security Operations support.
DNS_ERROR	The source hostname can't be resolved.	The server hostname may be spelled incorrectly. Check the URL and verify the spelling.
FILE_FAILED	A connection to the source was established, but there was a problem with the file or resource.	Check the following: The file isn't corrupt. The file-level permissions are correct. If the problem continues, contact Google Security Operations support.
FILE_NOT_FOUND	A connection to the source was established, but the file or resource can't be found.	Check the following: The file exists on the source. Appropriate users have access to the file. If the problem continues, contact Google Security Operations support.
GATEWAY_ERROR	API returned a gateway error to the call made by Google Security Operations.	Verify the source details of the feed. The application will retry the request.
INTERNAL_ERROR	Unable to ingest data due to an internal error.	If the problem continues, contact Google Security Operations support.
INVALID_ARGUMENT	A connection to the source was established, but the feed failed because of invalid arguments.	Check the feed configuration. Refer to the feeds documentation to learn more about setting up feeds. If the problem continues, contact Google Security Operations support.
INVALID_FEED_CONFIG	The feed configuration contains invalid values.	Review the feed configuration for incorrect settings. Refer to the feeds documentation for correct syntax.
INVALID_REMOTE_RESPONSE	A connection to the source was established, but the response was incorrect.	Check the feed configuration. Learn more about setting up feeds. If the problem continues, contact Google Security Operations support.
LOGIN_FAILED	A connection to the source was established, but credentials were incorrect or missing.	Re-enter the credentials for the source to confirm they're correct.
NO_RESPONSE	A connection to the source was established, but the source didn't respond.	Make sure the source can support requests from Google Security Operations. If the problem continues, contact Google Security Operations support.
PERMISSION_DENIED	A connection to the source was established, but there was a problem with authorization.	Verify required accesses and permissions are added.
REMOTE_SERVER_ERROR	A connection to the source was established, but the source didn't respond with data.	Make sure the source is available and is responding with data. If the problem continues, contact Google Security Operations support.
REMOTE_SERVER_REPORTED_BAD_REQUEST	A connection to the source was established, but the source rejected the request.	Check the feed configuration. Refer to the feeds documentation for more details. If the problem continues, contact Google Security Operations support.
SOCKET_READ_TIMEOUT	A connection to the source was established, but the connection timed out before the data transfer was complete.	This error is transient and application will retry the request. If the issue persists, contact Google Security Operations support.
TOO_MANY_ERRORS	The feed timed out because because it encountered multiple errors from the source.	Contact Google Security Operations support.
TRANSIENT_INTERNAL_ERROR	Feed encountered temporary internal error.	This error is transient and application will retry the request. If the issue persists, contact Google Security Operations support.
UNSAFE_CONNECTION	The application failed to make a connection because the IP address was restricted.	This error is transient and Google Security Operations will retry the request. If the issue persists, contact Google Security Operations support.
HTTP_400	The feed failed because of an invalid request.	Check the feed configuration. Learn more about setting up feeds. If the problem continues, contact Google Security Operations support.
HTTP_403	A connection to the source was established, but there was a problem with authorization.	Verify required accesses and permissions are added.
HTTP_404	A connection to the source was established, but the file or resource can't be found.	Check the following: The file exists on the source. Appropriate users have access to the file. If the problem continues, contact Google Security Operations support.
HTTP_429	The feed timed out because there were too many attempts to reach the source.	Contact Google Security Operations support.
HTTP_500	A connection to the source was established, but the source didn't respond with data.	Make sure the source is available and is responding with data. If the problem continues, contact Google Security Operations support.
HTTP_502	Feed encountered a gateway error.	This error is transient and application will retry the request. If the issue persists, contact Google Security Operations support.
HTTP_504	Google Security Operations can't connect to the source IP address and port.	This error is transient and application will retry the request. Check the following: The source is available. A firewall isn't blocking the connection. The IP address associated with the server is correct. If the problem continues, contact Google Security Operations support.