Skip to content
Merged
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
/packages/aws/kibana @elastic/obs-cloud-monitoring @elastic/kibana-visualizations
/packages/aws_logs @elastic/obs-cloud-monitoring
/packages/awsfargate @elastic/obs-cloud-monitoring
/packages/awsfirehose @elastic/obs-cloud-monitoring
/packages/azure @elastic/obs-cloud-monitoring
/packages/azure_app_service @elastic/obs-infraobs-integrations
/packages/azure_application_insights @elastic/obs-cloud-monitoring
Expand Down
3 changes: 3 additions & 0 deletions packages/awsfirehose/_dev/build/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
dependencies:
ecs:
reference: git@v8.0.0
92 changes: 92 additions & 0 deletions packages/awsfirehose/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose integration offers users a way to stream logs from Firehose to Elastic Cloud.
This integration includes predefined rules that automatically route AWS service logs to the respective integrations, which
include field mappings, ingest pipelines, predefined dashboards and ect. Here is a list of log types that are supported
by this integration:

| AWS service log | Log destination |
|--------------------|---------------------------|
| CloudTrail | CloudWatch |
| Network Firewall | Firehose, CloudWatch, S3 |
| Route53 Public DNS | CloudWatch |
| Route53 Resolver | Firehose, CloudWatch, S3 |
| VPC Flow | Firehose, CloudWatch, S3 |
| WAF | Firehose, CloudWatch |

## Limitation
It is not possible to configure a delivery stream to send data to Elastic Cloud via PrivateLink (VPC endpoint).
This is a current limitation in Firehose, which we are working with AWS to resolve.

## Instructions
1. Install the relevant integrations in Kibana

In order to make the most of your data, install AWS integrations to load index templates, ingest pipelines, and
dashboards into Kibana. In Kibana, navigate to **Management** > **Integrations** in the sidebar.
Find the **AWS** integration by searching or browsing the catalog.

![AWS integration](../img/aws.png)

Navigate to the **Settings** tab and click **Install AWS assets**. Confirm by clicking **Install AWS** in the popup.

![Install AWS assets](../img/install-assets.png)

2. Create a delivery stream in Amazon Kinesis Data Firehose

Sign into the AWS console and navigate to Amazon Kinesis. Click **Create delivery stream**.
Configure the delivery stream using the following settings:

![Amazon Kinesis Data Firehose](../img/aws-firehose.png)

**Choose source and destination**

Unless you are streaming data from Kinesis Data Streams, set source to Direct PUT (see Setup guide for more details on data sources).

Set destination to **Elastic**.

**Delivery stream name**

Provide a meaningful name that will allow you to identify this delivery stream later.

![Choose Firehose Source and Destination](../img/source-destination.png)

**Destination settings**

1. Set **Elastic endpoint URL** to point to your Elasticsearch cluster running in Elastic Cloud.
This endpoint can be found in the Elastic Cloud console. An example is https://my-deployment-28u274.es.eu-west-1.aws.found.io.

2. **API key** should be a Base64 encoded Elastic API key, which can be created in Kibana by following the
instructions under API Keys. If you are using an API key with “Restrict privileges”, be sure to review the Indices
privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this
delivery stream.

3. We recommend leaving **Content encoding** set to **GZIP** for improved network efficiency.

4. **Retry duration** determines how long Firehose continues retrying the request in the event of an error.
A duration of 60-300s should be suitable for most use cases.

5. Elastic requires a **Buffer size** of `1MiB` to avoid exceeding the Elasticsearch `http.max_content_length`
setting (typically 100MB) when the buffer is uncompressed.

6. The default **Buffer interval** of `60s` is recommended to ensure data freshness in Elastic.

7. **Parameters**

1. Elastic recommends setting the `es_datastream_name` parameter to `logs-awsfirehose.logs-default` in order to
leverage the routing rules defined in this integration. If this parameter is not specified, data is sent to the
`logs-generic-default` data stream by default.
![Firehose Destination Settings](../img/destination-settings.png)

2. The **include_cw_extracted_fields** parameter is optional and can be set when using a CloudWatch logs subscription
filter as the Firehose data source. When set to true, extracted fields generated by the filter pattern in the
subscription filter will be collected. Setting this parameter can add many fields into each record and may significantly
increase data volume in Elasticsearch. As such, use of this parameter should be carefully considered and used only when
the extracted fields are required for specific filtering and/or aggregation.

3. The **include_event_original** field is optional and should only be used for debugging purposes. When set to `true`, each
log record will contain an additional field named `event.original`, which contains the raw (unprocessed) log message.
This parameter will increase the data volume in Elasticsearch and should be used with care.

3. Send data to the Firehose delivery stream

Consult the [AWS documentation](https://docs.aws.amazon.com/firehose/latest/dev/basic-write.html) for details on how to
configure a variety of log sources to send data to Firehose delivery streams.
6 changes: 6 additions & 0 deletions packages/awsfirehose/changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# newer versions go on top
- version: 0.1.0
changes:
- description: initial release
type: enhancement
link: https://github.com/elastic/integrations/pull/7146
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"events": [
{
"cloud.region": "us-east-2",
"aws.firehose.arn": "arn:aws:firehose:us-east-2:123456:deliverystream/firehose-cloudtrail-logs-to-elastic",
"data_stream.namespace": "default",
"aws.firehose.subscription_filters": "[cloudtrail-to-firehose]",
"message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AWSService\",\"invokedBy\":\"cloudtrail.amazonaws.com\"},\"eventTime\":\"2023-07-17T21:02:26Z\",\"eventSource\":\"sts.amazonaws.com\",\"eventName\":\"AssumeRole\",\"awsRegion\":\"sa-east-1\",\"sourceIPAddress\":\"cloudtrail.amazonaws.com\",\"userAgent\":\"cloudtrail.amazonaws.com\",\"requestParameters\":{\"roleArn\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\",\"roleSessionName\":\"CLOUDWATCH_LOGS_DELIVERY_SESSION\"},\"responseElements\":{\"credentials\":{\"accessKeyId\":\"ASIAZEDJODE3A5LVGLFB\",\"sessionToken\":\"IQoJb3JpZ2luX2VjEGUaCXNhLWVhc3QtMSJHMEUCIHgHmtcrhwDhosJlQVky+C2zsYDKuR99qVlNjGIp8FLWAiEAsJtTDQ3Arq8iXEOHwv0ImEQdGb5tbgc+fLpoK58Enb4q9AII3v//////////ARAEGgw2MjcyODYzNTAxMzQiDN5gNdfO4ZdSqDmmwSrIAicTBYZg+ZXjwiJTN/Bz2YsMWYU6psw5znG3/Gh3EJ1P3RCmB7d79X6XZzFVi2u2xdrnaY/sTKDfp1jdl8OoAsSKYwJiGbzjoQlv59bB6JqPbKfAKUPAmz6JEMWNFgWTtaQL9rNkdPz23u/1msoUSzxCcxR9f3A2dD4yqnVpNJe8ipuhxpBMzQ61vcGL4G5hQEDM/o8sORP2PXbK4O7QAuWOyuryYkHAPwY9RrL0WHfflGBEBQV6XlidGpsRCtIppZVn025n3DQOypDEaL3fKp0gUsMkDH+frFjxop4o4wRYC3CxXe3XRJ5/Te886rQry7RUfXlQtiCfojZO5ohcLB+z6Y/uCK0IHp3zrfl5shKsQIAFt7p0B8W7PK5yHE4W9HHRiktJ9wTtq1YCTaWECpnjW0bISNgumRmDOAJvVHAjSjfkr4yAlJkw4qm8pQY6vwGbBiuf98AfRFrXMy01hVdE3GNTBrIS68zxUJaOjBLgw8l0nEC00L+LPuqaASFWz65Dnq5JAjXaDD9E3iCi4klp4gZFAcj7uGgeBIPkP7Bpr4SvBfnnqCgE2oyFrWke3NnYtqkL5iHLJeGlOTrvI5ND2H4jurQv0KbiqwHt6DmGF3poZOrtf8R3piNcuCCDLU8RvhRVLHy5rKPzsWgNokBc9XXmgltwvB6rIgdZhBJzupzmy/NSoWZcOeH2ooEELw==\",\"expiration\":\"Jul 12, 2023, 10:02:26 PM\"},\"assumedRoleUser\":{\"assumedRoleId\":\"AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION\",\"arn\":\"arn:aws:sts::123456:assumed-role/aws-cloudtrail-logs-123456-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION\"}},\"requestID\":\"041c9e5f-a031-47d2-a4a0-011bc8d5352c\",\"eventID\":\"3096b662-7aa9-43e6-8bee-541a45686745\",\"readOnly\":true,\"resources\":[{\"accountId\":\"123456\",\"type\":\"AWS::IAM::Role\",\"ARN\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"123456\",\"sharedEventID\":\"a1c94275-884f-4c1f-b8dc-2e1bf4c94d29\",\"eventCategory\":\"Management\"}",
"aws.kinesis.type": "deliverystream",
"data_stream.type": "logs",
"aws.firehose.request_id": "971ae05f-a128-4a7f-b623-30f9bc513e55",
"aws.cloudwatch.log_stream": "123456_CloudTrail_us-east-2_3",
"cloud.provider": "aws",
"@timestamp": "2023-07-25T21:04:35Z",
"cloud.account.id": "123456",
"data_stream.dataset": "awsfirehose.logs",
"aws.kinesis.name": "firehose-cloudtrail-logs-to-elastic",
"event.id": "37670326805251200781477669690942747782212394134076063744",
"aws.cloudwatch.log_group": "aws-cloudtrail-logs-123456-1c167310"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"expected": [
{
"@timestamp": "2023-07-25T21:04:35Z",
"aws.cloudwatch.log_group": "aws-cloudtrail-logs-123456-1c167310",
"aws.cloudwatch.log_stream": "123456_CloudTrail_us-east-2_3",
"aws.firehose.arn": "arn:aws:firehose:us-east-2:123456:deliverystream/firehose-cloudtrail-logs-to-elastic",
"aws.firehose.request_id": "971ae05f-a128-4a7f-b623-30f9bc513e55",
"aws.firehose.subscription_filters": "[cloudtrail-to-firehose]",
"aws.kinesis.name": "firehose-cloudtrail-logs-to-elastic",
"aws.kinesis.type": "deliverystream",
"cloud": {
"provider": "aws"
},
"cloud.account.id": "123456",
"cloud.provider": "aws",
"cloud.region": "us-east-2",
"data_stream.dataset": "aws.cloudtrail",
"data_stream.namespace": "default",
"data_stream.type": "logs",
"ecs": {
"version": "8.0.0"
},
"event.id": "37670326805251200781477669690942747782212394134076063744",
"message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"AWSService\",\"invokedBy\":\"cloudtrail.amazonaws.com\"},\"eventTime\":\"2023-07-17T21:02:26Z\",\"eventSource\":\"sts.amazonaws.com\",\"eventName\":\"AssumeRole\",\"awsRegion\":\"sa-east-1\",\"sourceIPAddress\":\"cloudtrail.amazonaws.com\",\"userAgent\":\"cloudtrail.amazonaws.com\",\"requestParameters\":{\"roleArn\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\",\"roleSessionName\":\"CLOUDWATCH_LOGS_DELIVERY_SESSION\"},\"responseElements\":{\"credentials\":{\"accessKeyId\":\"ASIAZEDJODE3A5LVGLFB\",\"sessionToken\":\"IQoJb3JpZ2luX2VjEGUaCXNhLWVhc3QtMSJHMEUCIHgHmtcrhwDhosJlQVky+C2zsYDKuR99qVlNjGIp8FLWAiEAsJtTDQ3Arq8iXEOHwv0ImEQdGb5tbgc+fLpoK58Enb4q9AII3v//////////ARAEGgw2MjcyODYzNTAxMzQiDN5gNdfO4ZdSqDmmwSrIAicTBYZg+ZXjwiJTN/Bz2YsMWYU6psw5znG3/Gh3EJ1P3RCmB7d79X6XZzFVi2u2xdrnaY/sTKDfp1jdl8OoAsSKYwJiGbzjoQlv59bB6JqPbKfAKUPAmz6JEMWNFgWTtaQL9rNkdPz23u/1msoUSzxCcxR9f3A2dD4yqnVpNJe8ipuhxpBMzQ61vcGL4G5hQEDM/o8sORP2PXbK4O7QAuWOyuryYkHAPwY9RrL0WHfflGBEBQV6XlidGpsRCtIppZVn025n3DQOypDEaL3fKp0gUsMkDH+frFjxop4o4wRYC3CxXe3XRJ5/Te886rQry7RUfXlQtiCfojZO5ohcLB+z6Y/uCK0IHp3zrfl5shKsQIAFt7p0B8W7PK5yHE4W9HHRiktJ9wTtq1YCTaWECpnjW0bISNgumRmDOAJvVHAjSjfkr4yAlJkw4qm8pQY6vwGbBiuf98AfRFrXMy01hVdE3GNTBrIS68zxUJaOjBLgw8l0nEC00L+LPuqaASFWz65Dnq5JAjXaDD9E3iCi4klp4gZFAcj7uGgeBIPkP7Bpr4SvBfnnqCgE2oyFrWke3NnYtqkL5iHLJeGlOTrvI5ND2H4jurQv0KbiqwHt6DmGF3poZOrtf8R3piNcuCCDLU8RvhRVLHy5rKPzsWgNokBc9XXmgltwvB6rIgdZhBJzupzmy/NSoWZcOeH2ooEELw==\",\"expiration\":\"Jul 12, 2023, 10:02:26 PM\"},\"assumedRoleUser\":{\"assumedRoleId\":\"AROAZEDJODE3NLJAH2FZC:CLOUDWATCH_LOGS_DELIVERY_SESSION\",\"arn\":\"arn:aws:sts::123456:assumed-role/aws-cloudtrail-logs-123456-b888baff_Role/CLOUDWATCH_LOGS_DELIVERY_SESSION\"}},\"requestID\":\"041c9e5f-a031-47d2-a4a0-011bc8d5352c\",\"eventID\":\"3096b662-7aa9-43e6-8bee-541a45686745\",\"readOnly\":true,\"resources\":[{\"accountId\":\"123456\",\"type\":\"AWS::IAM::Role\",\"ARN\":\"arn:aws:iam::123456:role/service-role/aws-cloudtrail-logs-123456-b888baff_Role\"}],\"eventType\":\"AwsApiCall\",\"managementEvent\":true,\"recipientAccountId\":\"123456\",\"sharedEventID\":\"a1c94275-884f-4c1f-b8dc-2e1bf4c94d29\",\"eventCategory\":\"Management\"}"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"events": [
{
"cloud.region": "us-east-1",
"aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-firewall-logs",
"data_stream.namespace": "default",
"aws.firehose.subscription_filters": "[test-firewall-logs-to-firehose]",
"message": "{\"firewall_name\":\"AWSNetworkFirewall\",\"availability_zone\":\"us-east-2a\",\"event_timestamp\":\"1636381332\",\"event\":{\"timestamp\":\"2021-11-08T14:22:12.637611+0000\",\"flow_id\":706471429191862,\"event_type\":\"alert\",\"src_ip\":\"81.2.69.143\",\"src_port\":51254,\"dest_ip\":\"216.160.83.57\",\"dest_port\":80,\"proto\":\"TCP\",\"alert\":{\"action\":\"blocked\",\"signature_id\":1000003,\"rev\":1,\"signature\":\"Deny all other TCP traffic\",\"category\":\"\",\"severity\":3},\"http\":{\"hostname\":\"216.160.83.57\",\"url\":\"/\",\"http_user_agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"length\":0},\"app_proto\":\"http\"}}",
"aws.kinesis.type": "deliverystream",
"data_stream.type": "logs",
"aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e",
"cloud.provider": "aws",
"@timestamp": "2023-08-11T20:01:37Z",
"cloud.account.id": "123456789",
"data_stream.dataset": "awsfirehose.logs",
"aws.kinesis.name": "test-firewall-logs",
"event.id": "37728046078123216000395549868459931814660237705210691585"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"expected": [
{
"@timestamp": "2023-08-11T20:01:37Z",
"aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-firewall-logs",
"aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e",
"aws.firehose.subscription_filters": "[test-firewall-logs-to-firehose]",
"aws.kinesis.name": "test-firewall-logs",
"aws.kinesis.type": "deliverystream",
"cloud": {
"provider": "aws"
},
"cloud.account.id": "123456789",
"cloud.provider": "aws",
"cloud.region": "us-east-1",
"data_stream.dataset": "aws.firewall_logs",
"data_stream.namespace": "default",
"data_stream.type": "logs",
"ecs": {
"version": "8.0.0"
},
"event.id": "37728046078123216000395549868459931814660237705210691585",
"message": "{\"firewall_name\":\"AWSNetworkFirewall\",\"availability_zone\":\"us-east-2a\",\"event_timestamp\":\"1636381332\",\"event\":{\"timestamp\":\"2021-11-08T14:22:12.637611+0000\",\"flow_id\":706471429191862,\"event_type\":\"alert\",\"src_ip\":\"81.2.69.143\",\"src_port\":51254,\"dest_ip\":\"216.160.83.57\",\"dest_port\":80,\"proto\":\"TCP\",\"alert\":{\"action\":\"blocked\",\"signature_id\":1000003,\"rev\":1,\"signature\":\"Deny all other TCP traffic\",\"category\":\"\",\"severity\":3},\"http\":{\"hostname\":\"216.160.83.57\",\"url\":\"/\",\"http_user_agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36\",\"http_method\":\"GET\",\"protocol\":\"HTTP/1.1\",\"length\":0},\"app_proto\":\"http\"}}"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"events": [
{
"cloud.region": "us-east-1",
"aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-route53-public-logs",
"data_stream.namespace": "default",
"aws.firehose.subscription_filters": "[test-route53-public-logs-to-firehose]",
"message": "1.0 2023-08-11T20:01:37Z Z0786514BU8K9GJ587CT filebeat-firehose.com NAPTR NOERROR UDP EWR52-C2 44.199.191.178 -",
"aws.kinesis.type": "deliverystream",
"data_stream.type": "logs",
"aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e",
"aws.cloudwatch.log_stream": "Z0786514BU8K9GJ587CT/EWR52-C2",
"cloud.provider": "aws",
"@timestamp": "2023-08-11T20:01:37Z",
"cloud.account.id": "123456789",
"data_stream.dataset": "awsfirehose.logs",
"aws.kinesis.name": "test-route53-public-logs",
"event.id": "37728046078123216000395549868459931814660237705210691585",
"aws.cloudwatch.log_group": "/aws/route53/filebeat-firehose.com"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"expected": [
{
"@timestamp": "2023-08-11T20:01:37Z",
"aws.cloudwatch.log_group": "/aws/route53/filebeat-firehose.com",
"aws.cloudwatch.log_stream": "Z0786514BU8K9GJ587CT/EWR52-C2",
"aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-route53-public-logs",
"aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e",
"aws.firehose.subscription_filters": "[test-route53-public-logs-to-firehose]",
"aws.kinesis.name": "test-route53-public-logs",
"aws.kinesis.type": "deliverystream",
"cloud": {
"provider": "aws"
},
"cloud.account.id": "123456789",
"cloud.provider": "aws",
"cloud.region": "us-east-1",
"data_stream.dataset": "aws.route53_public_logs",
"data_stream.namespace": "default",
"data_stream.type": "logs",
"ecs": {
"version": "8.0.0"
},
"event.id": "37728046078123216000395549868459931814660237705210691585",
"message": "1.0 2023-08-11T20:01:37Z Z0786514BU8K9GJ587CT filebeat-firehose.com NAPTR NOERROR UDP EWR52-C2 44.199.191.178 -"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"events": [
{
"cloud.region": "us-east-1",
"aws.firehose.arn": "arn:aws:firehose:us-east-1:123456789:deliverystream/test-route53-resolver-logs",
"data_stream.namespace": "default",
"aws.firehose.subscription_filters": "[test-route53-resolver-logs-to-firehose]",
"message": "{\"version\":\"1.100000\",\"account_id\":\"626345947581\",\"region\":\"us-east-1\",\"vpc_id\":\"vpc-01e31a7c\",\"query_timestamp\":\"2021-12-11T22:46:23Z\",\"query_name\":\"does-not-exist.abc.com.\",\"query_type\":\"A\",\"query_class\":\"IN\",\"rcode\":\"NXDOMAIN\",\"answers\":[],\"srcaddr\":\"172.31.86.159\",\"srcport\":\"48701\",\"transport\":\"UDP\",\"srcids\":{}}",
"aws.kinesis.type": "deliverystream",
"data_stream.type": "logs",
"aws.firehose.request_id": "afa49f89-ad73-4f39-807f-7f4d666b038e",
"aws.cloudwatch.log_stream": "Z0786514BU8K9GJ587CT/EWR52-C2",
"cloud.provider": "aws",
"@timestamp": "2023-08-11T20:01:37Z",
"cloud.account.id": "123456789",
"data_stream.dataset": "awsfirehose.logs",
"aws.kinesis.name": "test-route53-resolver-logs",
"event.id": "37728046078123216000395549868459931814660237705210691585",
"aws.cloudwatch.log_group": "/aws/route53/filebeat-firehose.com"
}
]
}
Loading