Add `awsfirehose` integration package #7146

kaiyan-sheng · 2023-07-26T02:50:27Z

What does this PR do?

WIP: Im working on using routing-rules.yml instead.

This PR is to add awsfirehose integration package as its own package. This integration supports reroute processors to provide routing configurations for logs. For example, if the field aws.cloudwatch.log_stream value contains CloudTrail, then route this log entry from the default logs-awsfirehose.logs-default to logs-aws.cloudtrail-default data stream.

The original plan is to add awsfirehose as an input package. But package spec does not support predefined pipelines in input packages nor routing rules. Please see elastic/package-spec#566 for more details.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

reroute processor for cloudtrail logs
reroute processor for firewall logs
reroute processor for route53 public logs
reroute processor for route53 resolver logs
reroute processor for vpcflow logs
reroute processor for waf logs

How to test this PR locally

Install awsfirehose integration and aws integration assets

You can check in Kibana for ingest pipelines to see if the reroute processors are there in logs-awsfirehose.logs-0.1.0

In the dev tool, send ES a test document, and ensure it's stored in the default data stream logs-awsfirehose.logs-default. For example:

 POST /logs-awsfirehose.logs-default/_doc { "cloud.region": "us-east-2", "aws.firehose.arn": "arn:aws:firehose:us-east-2:123456789:deliverystream/firehose-cloudtrail-logs-to-elastic", "data_stream.namespace": "default", "aws.firehose.subscription_filters": "[cloudtrail-to-firehose]", "message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AWSService\",\"invokedBy\":\"cloudtrail.amazonaws.com\"},\"eventTime\":\"2023-07-17T21:02:26Z\",\"eventSource\":\"sts.amazonaws.com\",\"eventName\":\"AssumeRole\",\"awsRegion\":\"sa-east-1\",\"sourceIPAddress\":\"cloudtrail.amazonaws.com\",\"userAgent\":\"cloudtrail.amazonaws.com\",\"requestParameters\":{\"roleArn\":\"arn:aws:iam::123456789:role/service-role/aws-cloudtrail-logs-123456789-b888baff_Role\",\"roleSessionName\":\"CLOUDWATCH_LOGS_DELIVERY_SESSION\"},\"responseElements\":{\"credentials\":{\"accessKeyId\":\"TEST\",\"sessionToken\":\"TEST\",\"expiration\":\"Jul 12, 2023, 10:02:26 PM\"},\"assumedRoleUser\":{\"assumedRoleId\":\"AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION\",\"arn\":\"arn:aws:sts::123456789:assumed-role/aws-cloudtrail-logs-123456789-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION\"}},\"requestID\":\"041c9e5f-a031-47d2-a4a0-011bc8d5352c\",\"eventID\":\"3096b662-7aa9-43e6-8bee-541a45686745\",\"readOnly\":true,\"resources\":[{\"accountId\":\"123456789\",\"type\":\"AWS::IAM::Role\",\"ARN\":\"arn:aws:iam::123456789:role/service-role/aws-cloudtrail-logs-123456789-b888baff_Role\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"123456789\",\"sharedEventID\":\"a1c94275-884f-4c1f-b8dc-2e1bf4c94d29\",\"eventCategory\":\"Management\"}", "aws.kinesis.type": "deliverystream", "data_stream.type": "logs", "aws.firehose.request_id": "971ae05f-a128-4a7f-b623-30f9bc513e55", "aws.cloudwatch.log_stream": "123456789_CloudTrail_us-east-2_3", "cloud.provider": "aws", "@timestamp": "2023-07-17T21:04:35Z", "cloud.account.id": "123456789", "data_stream.dataset": "awsfirehose.logs", "aws.kinesis.name": "firehose-cloudtrail-logs-to-elastic", "event.id": "37670326805251200781477669690942747782212394134076063744", "aws.cloudwatch.log_group": "aws-cloudtrail-logs-123456789-1c167310" }

Check in Discover to see what's the document looks like. It should be under logs-aws.cloudtrail-default instead and with message field all parsed by the cloudtrail ingest pipelines.

{ "_index": ".ds-logs-aws.cloudtrail-default-2023.08.31-000001", "_id": "byqgTYoBEaFhK0d36mzB", "_version": 1, "_score": 0, "_source": { "cloud.region": "us-east-2", "aws.firehose.arn": "arn:aws:firehose:us-east-2:123456789:deliverystream/firehose-cloudtrail-logs-to-elastic", "data_stream.namespace": "default", "aws.firehose.subscription_filters": "[cloudtrail-to-firehose]", "source": { "address": "cloudtrail.amazonaws.com" }, "aws.kinesis.type": "deliverystream", "data_stream.type": "logs", "aws.firehose.request_id": "971ae05f-a128-4a7f-b623-30f9bc513e55", "aws.cloudwatch.log_stream": "123456789_CloudTrail_us-east-2_3", "cloud": { "provider": "aws", "region": "sa-east-1" }, "cloud.provider": "aws", "@timestamp": "2023-07-17T21:02:26.000Z", "ecs": { "version": "8.0.0" }, "cloud.account.id": "123456789", "data_stream": { "dataset": "aws.cloudtrail" }, "aws.kinesis.name": "firehose-cloudtrail-logs-to-elastic", "event.id": "37670326805251200781477669690942747782212394134076063744", "event": { "agent_id_status": "missing", "ingested": "2023-08-31T22:05:06Z", "provider": "sts.amazonaws.com", "created": "2023-08-31T21:46:22Z", "kind": "event", "action": "AssumeRole", "id": "3096b662-7aa9-43e6-8bee-541a45686745", "type": [ "info" ], "category": [ "authentication" ], "outcome": "success" }, "aws": { "cloudtrail": { "event_version": "1.08", "flattened": { "request_parameters": { "roleArn": "arn:aws:iam::123456789:role/service-role/aws-cloudtrail-logs-123456789-b888baff_Role", "roleSessionName": "CLOUDWATCH_LOGS_DELIVERY_SESSION" }, "response_elements": { "assumedRoleUser": { "assumedRoleId": "AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION", "arn": "arn:aws:sts::123456789:assumed-role/aws-cloudtrail-logs-123456789-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION" }, "credentials": { "accessKeyId": "TEST", "sessionToken": "TEST", "expiration": "Jul 12, 2023, 10:02:26 PM" } } }, "event_type": "AwsApiCall", "read_only": true, "user_identity": { "invoked_by": "cloudtrail.amazonaws.com", "type": "AWSService" }, "recipient_account_id": "123456789", "event_category": "Management", "request_parameters": "{roleArn=arn:aws:iam::123456789:role/service-role/aws-cloudtrail-logs-123456789-b888baff_Role, roleSessionName=CLOUDWATCH_LOGS_DELIVERY_SESSION}", "request_id": "041c9e5f-a031-47d2-a4a0-011bc8d5352c", "response_elements": "{assumedRoleUser={assumedRoleId=AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION, arn=arn:aws:sts::123456789:assumed-role/aws-cloudtrail-logs-123456789-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION}, credentials={accessKeyId=TEST, sessionToken=TEST, expiration=Jul 12, 2023, 10:02:26 PM}}", "management_event": true } }, "user_agent": { "original": "cloudtrail.amazonaws.com", "name": "Other", "device": { "name": "Other" } }, "aws.cloudwatch.log_group": "aws-cloudtrail-logs-123456789-1c167310" } }

Screenshots

Related issues

Closes https://github.com/elastic/obs-infraobs-team/issues/1085

elasticmachine · 2023-07-26T02:56:29Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-09-01T20:55:12.175+0000
Duration: 14 min 21 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8
Skipped	0
Total	8

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-07-26T05:10:20Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚
Classes	100.0% (`1/1`)	💚
Methods	60.0% (`3/5`)	👍 26.667
Lines	100.0% (`71/71`)	💚
Conditionals	100.0% (`0/0`)	💚

packages/awsfirehose/data_stream/log/routing_rules.yml

…n_package

felixbarny · 2023-08-10T07:02:15Z

packages/awsfirehose/data_stream/log/routing_rules.yml

I think you can remove the whole namespace block.

That's because you don't need to add the explicit fall back for default. Even if the documents don't contain the data_stream.namespace field, it will use the namespace from the current data stream name. If the logs are sent to logs-awsfirehose.log-default, the reroute processor will parse out default from that data stream name in case it can't find a data_stream.namespace field. See the options table in https://www.elastic.co/guide/en/elasticsearch/reference/current/reroute-processor.html for more details.

As {{data_stream.namespace}} is the default value for namespace, you can simply omit it.

Ha good point thanks!! I realized elastic-package doesn't accept empty namespace, target_dataset or if so I'm adding it in elastic/elastic-package#1391. Will remove them afterwards.

packages/awsfirehose/_dev/build/docs/README.md

tommyers-elastic

great work so far kaiyan - thanks !

some small comments, but overall looking great.

packages/awsfirehose/data_stream/log/fields/base-fields.yml

tommyers-elastic · 2023-09-01T12:30:42Z

packages/awsfirehose/data_stream/log/fields/fields.yml

i'm trying to figure out if these (cloudwatch) fields live here or we should delegate all service-specific mappings to the downstream integrations. i think having the mapping here too is probably right, since then any generic logs shipped via cloudwatch at least have these fields. so after all, ignore this comment... :)

I think for this integration in particular, it would be useful to use the dynamic ECS mappings instead of explicit mappings.

packages/awsfirehose/data_stream/log/fields/fields.yml

packages/awsfirehose/data_stream/log/manifest.yml

packages/awsfirehose/_dev/build/docs/README.md

kaiyan-sheng · 2023-09-05T19:45:25Z

This PR is getting too big to review. I will merge it first. @tommyers-elastic Let's keep the conversation ongoing and I will create separate PRs to address them.

elasticmachine · 2023-09-05T20:08:46Z

Package awsfirehose - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=awsfirehose

lucabelluccini · 2023-09-07T08:12:04Z

Hello @tommyers-elastic

Does the documentation need to reflect this?
Is there anything to do for users now that the Firehose integration exists? Do they need to migrate? Is this documented?

tommyers-elastic · 2023-09-07T10:57:00Z

@lucabelluccini yes - but not yet, we still have work to do to figure out exactly what the upgrade process looks like for existing users, and how that works with fresh installs. for now the existing documentation is still accurate.

the docs in the package contain more details on how to configure things right now and enable the routing. we will combine everything when it's time to GA the integration.

lucabelluccini · 2023-09-07T12:40:47Z

Thanks @tommyers-elastic

move awsfirehose to its own integration package

1bf0f76

kaiyan-sheng self-assigned this Jul 26, 2023

add changelog

247c985

kaiyan-sheng added 2 commits July 25, 2023 22:14

add code owner

cee122b

Use routing_rules.yml

0c4204e

felixbarny reviewed Jul 26, 2023

View reviewed changes

packages/awsfirehose/data_stream/log/routing_rules.yml Outdated Show resolved Hide resolved

packages/awsfirehose/data_stream/log/routing_rules.yml Outdated Show resolved Hide resolved

packages/awsfirehose/data_stream/log/routing_rules.yml Outdated Show resolved Hide resolved

kaiyan-sheng mentioned this pull request Jul 26, 2023

Support routing_rules.yml and ingest pipeline test for reroute processor elastic/elastic-package#1372

Merged

adding reroute processor back into ingest pipeline for testing

2ed809a

andrewkroh added Integration:awsfirehose Amazon Data Firehose New Integration Issue or pull request for creating a new integration package. labels Aug 1, 2023

kaiyan-sheng added 2 commits August 9, 2023 16:17

Merge remote-tracking branch 'upstream/main' into firehose_integratio…

14ab712

…n_package

combine routing rules

44a4c0c

felixbarny reviewed Aug 10, 2023

View reviewed changes

kaiyan-sheng added 4 commits August 11, 2023 15:30

add reroute public log and firewall log for testing

0a5707f

add routing rules for route53 vpcflow and waf log

5031492

add documentation

bfca6eb

add es_datastream_name parameter into documentation

d307142

kaiyan-sheng requested a review from tommyers-elastic August 29, 2023 13:39

constanca-m reviewed Aug 31, 2023

View reviewed changes

packages/awsfirehose/_dev/build/docs/README.md Outdated Show resolved Hide resolved

Fix link to screenshots

c139ee9

kaiyan-sheng requested a review from constanca-m August 31, 2023 22:11

tommyers-elastic approved these changes Sep 1, 2023

View reviewed changes

kaiyan-sheng added 2 commits September 1, 2023 13:41

change data stream name from log to logs

06b7f0c

add test case for logs sent from waf to cloudwatch

e101d2f

kaiyan-sheng merged commit a391b43 into elastic:main Sep 5, 2023

kaiyan-sheng deleted the firehose_integration_package branch September 5, 2023 19:45

felixbarny mentioned this pull request Sep 13, 2023

Make reroute processor GA elastic/elasticsearch#99531

Merged

ruflin mentioned this pull request Oct 31, 2023

Rally benchmark elastic/elastic-package#1522

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `awsfirehose` integration package #7146

Add `awsfirehose` integration package #7146

Uh oh!

kaiyan-sheng commented Jul 26, 2023 •

edited

Loading

elasticmachine commented Jul 26, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 26, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

felixbarny Aug 10, 2023

kaiyan-sheng Aug 17, 2023

Uh oh!

tommyers-elastic left a comment

Uh oh!

tommyers-elastic Sep 1, 2023

felixbarny Sep 1, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaiyan-sheng commented Sep 5, 2023

elasticmachine commented Sep 5, 2023

lucabelluccini commented Sep 7, 2023

tommyers-elastic commented Sep 7, 2023 •

edited

Loading

lucabelluccini commented Sep 7, 2023

Labels

7 participants

Add awsfirehose integration package #7146

Add awsfirehose integration package #7146

Uh oh!

Conversation

kaiyan-sheng commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Screenshots

Related issues

elasticmachine commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Uh oh!

Uh oh!

felixbarny Aug 10, 2023

Choose a reason for hiding this comment

kaiyan-sheng Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

tommyers-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

tommyers-elastic Sep 1, 2023

Choose a reason for hiding this comment

felixbarny Sep 1, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kaiyan-sheng commented Sep 5, 2023

elasticmachine commented Sep 5, 2023

lucabelluccini commented Sep 7, 2023

tommyers-elastic commented Sep 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

lucabelluccini commented Sep 7, 2023

Labels

7 participants

Add `awsfirehose` integration package #7146

Add `awsfirehose` integration package #7146

kaiyan-sheng commented Jul 26, 2023 •

edited

Loading

elasticmachine commented Jul 26, 2023 •

edited

Loading

elasticmachine commented Jul 26, 2023 •

edited

Loading

tommyers-elastic commented Sep 7, 2023 •

edited

Loading