- Notifications
You must be signed in to change notification settings - Fork 519
Add awsfirehose integration package #7146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
kaiyan-sheng merged 14 commits into elastic:main from kaiyan-sheng:firehose_integration_package Sep 5, 2023
Merged
Changes from all commits
Commits
Show all changes
14 commits Select commit Hold shift + click to select a range
1bf0f76 move awsfirehose to its own integration package
kaiyan-sheng 247c985 add changelog
kaiyan-sheng cee122b add code owner
kaiyan-sheng 0c4204e Use routing_rules.yml
kaiyan-sheng 2ed809a adding reroute processor back into ingest pipeline for testing
kaiyan-sheng 14ab712 Merge remote-tracking branch 'upstream/main' into firehose_integratio…
kaiyan-sheng 44a4c0c combine routing rules
kaiyan-sheng 0a5707f add reroute public log and firewall log for testing
kaiyan-sheng 5031492 add routing rules for route53 vpcflow and waf log
kaiyan-sheng bfca6eb add documentation
kaiyan-sheng d307142 add es_datastream_name parameter into documentation
kaiyan-sheng c139ee9 Fix link to screenshots
kaiyan-sheng 06b7f0c change data stream name from log to logs
kaiyan-sheng e101d2f add test case for logs sent from waf to cloudwatch
kaiyan-sheng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| dependencies: | ||
| ecs: | ||
| reference: git@v8.0.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # Amazon Kinesis Data Firehose | ||
| Amazon Kinesis Data Firehose integration offers users a way to stream logs from Firehose to Elastic Cloud. | ||
| This integration includes predefined rules that automatically route AWS service logs to the respective integrations, which | ||
| include field mappings, ingest pipelines, predefined dashboards and ect. Here is a list of log types that are supported | ||
| by this integration: | ||
| | ||
| | AWS service log | Log destination | | ||
| |--------------------|---------------------------| | ||
| | CloudTrail | CloudWatch | | ||
| | Network Firewall | Firehose, CloudWatch, S3 | | ||
| | Route53 Public DNS | CloudWatch | | ||
| | Route53 Resolver | Firehose, CloudWatch, S3 | | ||
| | VPC Flow | Firehose, CloudWatch, S3 | | ||
| | WAF | Firehose, CloudWatch | | ||
| | ||
| ## Limitation | ||
| It is not possible to configure a delivery stream to send data to Elastic Cloud via PrivateLink (VPC endpoint). | ||
| This is a current limitation in Firehose, which we are working with AWS to resolve. | ||
| | ||
| ## Instructions | ||
| 1. Install the relevant integrations in Kibana | ||
| | ||
| In order to make the most of your data, install AWS integrations to load index templates, ingest pipelines, and | ||
| dashboards into Kibana. In Kibana, navigate to **Management** > **Integrations** in the sidebar. | ||
| Find the **AWS** integration by searching or browsing the catalog. | ||
| | ||
|  | ||
| | ||
| Navigate to the **Settings** tab and click **Install AWS assets**. Confirm by clicking **Install AWS** in the popup. | ||
| | ||
|  | ||
| | ||
| 2. Create a delivery stream in Amazon Kinesis Data Firehose | ||
| | ||
| Sign into the AWS console and navigate to Amazon Kinesis. Click **Create delivery stream**. | ||
| Configure the delivery stream using the following settings: | ||
| | ||
|  | ||
| | ||
| **Choose source and destination** | ||
| | ||
| Unless you are streaming data from Kinesis Data Streams, set source to Direct PUT (see Setup guide for more details on data sources). | ||
| | ||
| Set destination to **Elastic**. | ||
| | ||
| **Delivery stream name** | ||
| | ||
| Provide a meaningful name that will allow you to identify this delivery stream later. | ||
| | ||
|  | ||
| | ||
| **Destination settings** | ||
| | ||
| 1. Set **Elastic endpoint URL** to point to your Elasticsearch cluster running in Elastic Cloud. | ||
| This endpoint can be found in the Elastic Cloud console. An example is https://my-deployment-28u274.es.eu-west-1.aws.found.io. | ||
| | ||
| 2. **API key** should be a Base64 encoded Elastic API key, which can be created in Kibana by following the | ||
| instructions under API Keys. If you are using an API key with “Restrict privileges”, be sure to review the Indices | ||
| privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this | ||
| delivery stream. | ||
| | ||
| 3. We recommend leaving **Content encoding** set to **GZIP** for improved network efficiency. | ||
| | ||
| 4. **Retry duration** determines how long Firehose continues retrying the request in the event of an error. | ||
| A duration of 60-300s should be suitable for most use cases. | ||
| | ||
| 5. Elastic requires a **Buffer size** of `1MiB` to avoid exceeding the Elasticsearch `http.max_content_length` | ||
| setting (typically 100MB) when the buffer is uncompressed. | ||
| | ||
| 6. The default **Buffer interval** of `60s` is recommended to ensure data freshness in Elastic. | ||
| | ||
| 7. **Parameters** | ||
| | ||
| 1. Elastic recommends setting the `es_datastream_name` parameter to `logs-awsfirehose.logs-default` in order to | ||
| leverage the routing rules defined in this integration. If this parameter is not specified, data is sent to the | ||
| `logs-generic-default` data stream by default. | ||
|  | ||
| | ||
| 2. The **include_cw_extracted_fields** parameter is optional and can be set when using a CloudWatch logs subscription | ||
| filter as the Firehose data source. When set to true, extracted fields generated by the filter pattern in the | ||
| subscription filter will be collected. Setting this parameter can add many fields into each record and may significantly | ||
| increase data volume in Elasticsearch. As such, use of this parameter should be carefully considered and used only when | ||
| the extracted fields are required for specific filtering and/or aggregation. | ||
| | ||
| 3. The **include_event_original** field is optional and should only be used for debugging purposes. When set to `true`, each | ||
| log record will contain an additional field named `event.original`, which contains the raw (unprocessed) log message. | ||
| This parameter will increase the data volume in Elasticsearch and should be used with care. | ||
| | ||
| 3. Send data to the Firehose delivery stream | ||
| | ||
| Consult the [AWS documentation](https://docs.aws.amazon.com/firehose/latest/dev/basic-write.html) for details on how to | ||
| configure a variety of log sources to send data to Firehose delivery streams. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # newer versions go on top | ||
| - version: 0.1.0 | ||
| changes: | ||
| - description: initial release | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/7146 |
22 changes: 22 additions & 0 deletions 22 packages/awsfirehose/data_stream/logs/_dev/test/pipeline/test-cloudtrail-log.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "events": [ | ||
| { | ||
| "cloud.region": "us-east-2", | ||
| "aws.firehose.arn": "arn:aws:firehose:us-east-2:123456:deliverystream/firehose-cloudtrail-logs-to-elastic", | ||
| "data_stream.namespace": "default", | ||
| "aws.firehose.subscription_filters": "[cloudtrail-to-firehose]", | ||
| "message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AWSService\",\"invokedBy\":\"cloudtrail.amazonaws.com\"},\"eventTime\":\"2023-07-17T21:02:26Z\",\"eventSource\":\"sts.amazonaws.com\",\"eventName\":\"AssumeRole\",\"awsRegion\":\"sa-east-1\",\"sourceIPAddress\":\"cloudtrail.amazonaws.com\",\"userAgent\":\"cloudtrail.amazonaws.com\",\"requestParameters\":{\"roleArn\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\",\"roleSessionName\":\"CLOUDWATCH_LOGS_DELIVERY_SESSION\"},\"responseElements\":{\"credentials\":{\"accessKeyId\":\"ASIAZEDJODE3A5LVGLFB\",\"sessionToken\":\"IQoJb3JpZ2luX2VjEGUaCXNhLWVhc3QtMSJHMEUCIHgHmtcrhwDhosJlQVky+C2zsYDKuR99qVlNjGIp8FLWAiEAsJtTDQ3Arq8iXEOHwv0ImEQdGb5tbgc+fLpoK58Enb4q9AII3v//////////ARAEGgw2MjcyODYzNTAxMzQiDN5gNdfO4ZdSqDmmwSrIAicTBYZg+ZXjwiJTN/Bz2YsMWYU6psw5znG3/Gh3EJ1P3RCmB7d79X6XZzFVi2u2xdrnaY/sTKDfp1jdl8OoAsSKYwJiGbzjoQlv59bB6JqPbKfAKUPAmz6JEMWNFgWTtaQL9rNkdPz23u/1msoUSzxCcxR9f3A2dD4yqnVpNJe8ipuhxpBMzQ61vcGL4G5hQEDM/o8sORP2PXbK4O7QAuWOyuryYkHAPwY9RrL0WHfflGBEBQV6XlidGpsRCtIppZVn025n3DQOypDEaL3fKp0gUsMkDH+frFjxop4o4wRYC3CxXe3XRJ5/Te886rQry7RUfXlQtiCfojZO5ohcLB+z6Y/uCK0IHp3zrfl5shKsQIAFt7p0B8W7PK5yHE4W9HHRiktJ9wTtq1YCTaWECpnjW0bISNgumRmDOAJvVHAjSjfkr4yAlJkw4qm8pQY6vwGbBiuf98AfRFrXMy01hVdE3GNTBrIS68zxUJaOjBLgw8l0nEC00L+LPuqaASFWz65Dnq5JAjXaDD9E3iCi4klp4gZFAcj7uGgeBIPkP7Bpr4SvBfnnqCgE2oyFrWke3NnYtqkL5iHLJeGlOTrvI5ND2H4jurQv0KbiqwHt6DmGF3poZOrtf8R3piNcuCCDLU8RvhRVLHy5rKPzsWgNokBc9XXmgltwvB6rIgdZhBJzupzmy/NSoWZcOeH2ooEELw==\",\"expiration\":\"Jul 12, 2023, 10:02:26 PM\"},\"assumedRoleUser\":{\"assumedRoleId\":\"AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION\",\"arn\":\"arn:aws:sts::123456:assumed-role/aws-cloudtrail-logs-123456-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION\"}},\"requestID\":\"041c9e5f-a031-47d2-a4a0-011bc8d5352c\",\"eventID\":\"3096b662-7aa9-43e6-8bee-541a45686745\",\"readOnly\":true,\"resources\":[{\"accountId\":\"123456\",\"type\":\"AWS::IAM::Role\",\"ARN\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"123456\",\"sharedEventID\":\"a1c94275-884f-4c1f-b8dc-2e1bf4c94d29\",\"eventCategory\":\"Management\"}", | ||
| "aws.kinesis.type": "deliverystream", | ||
| "data_stream.type": "logs", | ||
| "aws.firehose.request_id": "971ae05f-a128-4a7f-b623-30f9bc513e55", | ||
| "aws.cloudwatch.log_stream": "123456_CloudTrail_us-east-2_3", | ||
| "cloud.provider": "aws", | ||
| "@timestamp": "2023-07-25T21:04:35Z", | ||
| "cloud.account.id": "123456", | ||
| "data_stream.dataset": "awsfirehose.logs", | ||
| "aws.kinesis.name": "firehose-cloudtrail-logs-to-elastic", | ||
| "event.id": "37670326805251200781477669690942747782212394134076063744", | ||
| "aws.cloudwatch.log_group": "aws-cloudtrail-logs-123456-1c167310" | ||
| } | ||
| ] | ||
| } |
28 changes: 28 additions & 0 deletions 28 ...es/awsfirehose/data_stream/logs/_dev/test/pipeline/test-cloudtrail-log.json-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2023-07-25T21:04:35Z", | ||
| "aws.cloudwatch.log_group": "aws-cloudtrail-logs-123456-1c167310", | ||
| "aws.cloudwatch.log_stream": "123456_CloudTrail_us-east-2_3", | ||
| "aws.firehose.arn": "arn:aws:firehose:us-east-2:123456:deliverystream/firehose-cloudtrail-logs-to-elastic", | ||
| "aws.firehose.request_id": "971ae05f-a128-4a7f-b623-30f9bc513e55", | ||
| "aws.firehose.subscription_filters": "[cloudtrail-to-firehose]", | ||
| "aws.kinesis.name": "firehose-cloudtrail-logs-to-elastic", | ||
| "aws.kinesis.type": "deliverystream", | ||
| "cloud": { | ||
| "provider": "aws" | ||
| }, | ||
| "cloud.account.id": "123456", | ||
| "cloud.provider": "aws", | ||
| "cloud.region": "us-east-2", | ||
| "data_stream.dataset": "aws.cloudtrail", | ||
| "data_stream.namespace": "default", | ||
| "data_stream.type": "logs", | ||
| "ecs": { | ||
| "version": "8.0.0" | ||
| }, | ||
| "event.id": "37670326805251200781477669690942747782212394134076063744", | ||
| "message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AWSService\",\"invokedBy\":\"cloudtrail.amazonaws.com\"},\"eventTime\":\"2023-07-17T21:02:26Z\",\"eventSource\":\"sts.amazonaws.com\",\"eventName\":\"AssumeRole\",\"awsRegion\":\"sa-east-1\",\"sourceIPAddress\":\"cloudtrail.amazonaws.com\",\"userAgent\":\"cloudtrail.amazonaws.com\",\"requestParameters\":{\"roleArn\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\",\"roleSessionName\":\"CLOUDWATCH_LOGS_DELIVERY_SESSION\"},\"responseElements\":{\"credentials\":{\"accessKeyId\":\"ASIAZEDJODE3A5LVGLFB\",\"sessionToken\":\"IQoJb3JpZ2luX2VjEGUaCXNhLWVhc3QtMSJHMEUCIHgHmtcrhwDhosJlQVky+C2zsYDKuR99qVlNjGIp8FLWAiEAsJtTDQ3Arq8iXEOHwv0ImEQdGb5tbgc+fLpoK58Enb4q9AII3v//////////ARAEGgw2MjcyODYzNTAxMzQiDN5gNdfO4ZdSqDmmwSrIAicTBYZg+ZXjwiJTN/Bz2YsMWYU6psw5znG3/Gh3EJ1P3RCmB7d79X6XZzFVi2u2xdrnaY/sTKDfp1jdl8OoAsSKYwJiGbzjoQlv59bB6JqPbKfAKUPAmz6JEMWNFgWTtaQL9rNkdPz23u/1msoUSzxCcxR9f3A2dD4yqnVpNJe8ipuhxpBMzQ61vcGL4G5hQEDM/o8sORP2PXbK4O7QAuWOyuryYkHAPwY9RrL0WHfflGBEBQV6XlidGpsRCtIppZVn025n3DQOypDEaL3fKp0gUsMkDH+frFjxop4o4wRYC3CxXe3XRJ5/Te886rQry7RUfXlQtiCfojZO5ohcLB+z6Y/uCK0IHp3zrfl5shKsQIAFt7p0B8W7PK5yHE4W9HHRiktJ9wTtq1YCTaWECpnjW0bISNgumRmDOAJvVHAjSjfkr4yAlJkw4qm8pQY6vwGbBiuf98AfRFrXMy01hVdE3GNTBrIS68zxUJaOjBLgw8l0nEC00L+LPuqaASFWz65Dnq5JAjXaDD9E3iCi4klp4gZFAcj7uGgeBIPkP7Bpr4SvBfnnqCgE2oyFrWke3NnYtqkL5iHLJeGlOTrvI5ND2H4jurQv0KbiqwHt6DmGF3poZOrtf8R3piNcuCCDLU8RvhRVLHy5rKPzsWgNokBc9XXmgltwvB6rIgdZhBJzupzmy/NSoWZcOeH2ooEELw==\",\"expiration\":\"Jul 12, 2023, 10:02:26 PM\"},\"assumedRoleUser\":{\"assumedRoleId\":\"AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION\",\"arn\":\"arn:aws:sts::123456:assumed-role/aws-cloudtrail-logs-123456-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION\"}},\"requestID\":\"041c9e5f-a031-47d2-a4a0-011bc8d5352c\",\"eventID\":\"3096b662-7aa9-43e6-8bee-541a45686745\",\"readOnly\":true,\"resources\":[{\"accountId\":\"123456\",\"type\":\"AWS::IAM::Role\",\"ARN\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"123456\",\"sharedEventID\":\"a1c94275-884f-4c1f-b8dc-2e1bf4c94d29\",\"eventCategory\":\"Management\"}" | ||
| } | ||
| ] | ||
| } |
20 changes: 20 additions & 0 deletions 20 packages/awsfirehose/data_stream/logs/_dev/test/pipeline/test-firewall-log.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| { | ||
| "events": [ | ||
| { | ||
| "cloud.region": "us-east-1", | ||
| "aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-firewall-logs", | ||
| "data_stream.namespace": "default", | ||
| "aws.firehose.subscription_filters": "[test-firewall-logs-to-firehose]", | ||
| "message": "{\"firewall_name\":\"AWSNetworkFirewall\",\"availability_zone\":\"us-east-2a\",\"event_timestamp\":\"1636381332\",\"event\":{\"timestamp\":\"2021-11-08T14:22:12.637611+0000\",\"flow_id\":706471429191862,\"event_type\":\"alert\",\"src_ip\":\"81.2.69.143\",\"src_port\":51254,\"dest_ip\":\"216.160.83.57\",\"dest_port\":80,\"proto\":\"TCP\",\"alert\":{\"action\":\"blocked\",\"signature_id\":1000003,\"rev\":1,\"signature\":\"Deny all other TCP traffic\",\"category\":\"\",\"severity\":3},\"http\":{\"hostname\":\"216.160.83.57\",\"url\":\"/\",\"http_user_agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"length\":0},\"app_proto\":\"http\"}}", | ||
| "aws.kinesis.type": "deliverystream", | ||
| "data_stream.type": "logs", | ||
| "aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e", | ||
| "cloud.provider": "aws", | ||
| "@timestamp": "2023-08-11T20:01:37Z", | ||
| "cloud.account.id": "123456789", | ||
| "data_stream.dataset": "awsfirehose.logs", | ||
| "aws.kinesis.name": "test-firewall-logs", | ||
| "event.id": "37728046078123216000395549868459931814660237705210691585" | ||
| } | ||
| ] | ||
| } |
26 changes: 26 additions & 0 deletions 26 ...ages/awsfirehose/data_stream/logs/_dev/test/pipeline/test-firewall-log.json-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2023-08-11T20:01:37Z", | ||
| "aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-firewall-logs", | ||
| "aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e", | ||
| "aws.firehose.subscription_filters": "[test-firewall-logs-to-firehose]", | ||
| "aws.kinesis.name": "test-firewall-logs", | ||
| "aws.kinesis.type": "deliverystream", | ||
| "cloud": { | ||
| "provider": "aws" | ||
| }, | ||
| "cloud.account.id": "123456789", | ||
| "cloud.provider": "aws", | ||
| "cloud.region": "us-east-1", | ||
| "data_stream.dataset": "aws.firewall_logs", | ||
| "data_stream.namespace": "default", | ||
| "data_stream.type": "logs", | ||
| "ecs": { | ||
| "version": "8.0.0" | ||
| }, | ||
| "event.id": "37728046078123216000395549868459931814660237705210691585", | ||
| "message": "{\"firewall_name\":\"AWSNetworkFirewall\",\"availability_zone\":\"us-east-2a\",\"event_timestamp\":\"1636381332\",\"event\":{\"timestamp\":\"2021-11-08T14:22:12.637611+0000\",\"flow_id\":706471429191862,\"event_type\":\"alert\",\"src_ip\":\"81.2.69.143\",\"src_port\":51254,\"dest_ip\":\"216.160.83.57\",\"dest_port\":80,\"proto\":\"TCP\",\"alert\":{\"action\":\"blocked\",\"signature_id\":1000003,\"rev\":1,\"signature\":\"Deny all other TCP traffic\",\"category\":\"\",\"severity\":3},\"http\":{\"hostname\":\"216.160.83.57\",\"url\":\"/\",\"http_user_agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"length\":0},\"app_proto\":\"http\"}}" | ||
| } | ||
| ] | ||
| } |
22 changes: 22 additions & 0 deletions 22 packages/awsfirehose/data_stream/logs/_dev/test/pipeline/test-route53-public-log.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "events": [ | ||
| { | ||
| "cloud.region": "us-east-1", | ||
| "aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-route53-public-logs", | ||
| "data_stream.namespace": "default", | ||
| "aws.firehose.subscription_filters": "[test-route53-public-logs-to-firehose]", | ||
| "message": "1.0 2023-08-11T20:01:37Z Z0786514BU8K9GJ587CT filebeat-firehose.com NAPTR NOERROR UDP EWR52-C2 44.199.191.178 -", | ||
| "aws.kinesis.type": "deliverystream", | ||
| "data_stream.type": "logs", | ||
| "aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e", | ||
| "aws.cloudwatch.log_stream": "Z0786514BU8K9GJ587CT/EWR52-C2", | ||
| "cloud.provider": "aws", | ||
| "@timestamp": "2023-08-11T20:01:37Z", | ||
| "cloud.account.id": "123456789", | ||
| "data_stream.dataset": "awsfirehose.logs", | ||
| "aws.kinesis.name": "test-route53-public-logs", | ||
| "event.id": "37728046078123216000395549868459931814660237705210691585", | ||
| "aws.cloudwatch.log_group": "/aws/route53/filebeat-firehose.com" | ||
| } | ||
| ] | ||
| } |
28 changes: 28 additions & 0 deletions 28 ...wsfirehose/data_stream/logs/_dev/test/pipeline/test-route53-public-log.json-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2023-08-11T20:01:37Z", | ||
| "aws.cloudwatch.log_group": "/aws/route53/filebeat-firehose.com", | ||
| "aws.cloudwatch.log_stream": "Z0786514BU8K9GJ587CT/EWR52-C2", | ||
| "aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-route53-public-logs", | ||
| "aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e", | ||
| "aws.firehose.subscription_filters": "[test-route53-public-logs-to-firehose]", | ||
| "aws.kinesis.name": "test-route53-public-logs", | ||
| "aws.kinesis.type": "deliverystream", | ||
| "cloud": { | ||
| "provider": "aws" | ||
| }, | ||
| "cloud.account.id": "123456789", | ||
| "cloud.provider": "aws", | ||
| "cloud.region": "us-east-1", | ||
| "data_stream.dataset": "aws.route53_public_logs", | ||
| "data_stream.namespace": "default", | ||
| "data_stream.type": "logs", | ||
| "ecs": { | ||
| "version": "8.0.0" | ||
| }, | ||
| "event.id": "37728046078123216000395549868459931814660237705210691585", | ||
| "message": "1.0 2023-08-11T20:01:37Z Z0786514BU8K9GJ587CT filebeat-firehose.com NAPTR NOERROR UDP EWR52-C2 44.199.191.178 -" | ||
| } | ||
| ] | ||
| } |
22 changes: 22 additions & 0 deletions 22 packages/awsfirehose/data_stream/logs/_dev/test/pipeline/test-route53-resolver-log.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "events": [ | ||
| { | ||
| "cloud.region": "us-east-1", | ||
| "aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-route53-resolver-logs", | ||
| "data_stream.namespace": "default", | ||
| "aws.firehose.subscription_filters": "[test-route53-resolver-logs-to-firehose]", | ||
| "message": "{\"version\":\"1.100000\",\"account_id\":\"626345947581\",\"region\":\"us-east-1\",\"vpc_id\":\"vpc-01e31a7c\",\"query_timestamp\":\"2021-12-11T22:46:23Z\",\"query_name\":\"does-not-exist.abc.com.\",\"query_type\":\"A\",\"query_class\":\"IN\",\"rcode\":\"NXDOMAIN\",\"answers\":[],\"srcaddr\":\"172.31.86.159\",\"srcport\":\"48701\",\"transport\":\"UDP\",\"srcids\":{}}", | ||
| "aws.kinesis.type": "deliverystream", | ||
| "data_stream.type": "logs", | ||
| "aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e", | ||
| "aws.cloudwatch.log_stream": "Z0786514BU8K9GJ587CT/EWR52-C2", | ||
| "cloud.provider": "aws", | ||
| "@timestamp": "2023-08-11T20:01:37Z", | ||
| "cloud.account.id": "123456789", | ||
| "data_stream.dataset": "awsfirehose.logs", | ||
| "aws.kinesis.name": "test-route53-resolver-logs", | ||
| "event.id": "37728046078123216000395549868459931814660237705210691585", | ||
| "aws.cloudwatch.log_group": "/aws/route53/filebeat-firehose.com" | ||
| } | ||
| ] | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.