Increase the ZT event coverage for Cloudflare Logpush

packages/cloudflare_logpush/_dev/build/docs/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-The [Cloudflare Logpush](https://www.cloudflare.com/) integration allows you to monitor Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics and Spectrum Event Logs. Cloudflare is a content delivery network and DDoS mitigation company. Cloudflare provides a network designed to make everything you connect to the Internet secure, private, fast, and reliable; secure your websites, APIs, and Internet applications; protect corporate networks, employees, and devices; and write and deploy code that runs on the network edge.
+The [Cloudflare Logpush](https://www.cloudflare.com/) integration allows you to monitor Access Request, Audit, CASB, Device Posture, DNS, Firewall Event, Gateway DNS, Gateway HTTP, Gateway Network, HTTP Request, NEL Report, Network Analytics, Spectrum Event and Network Session logs. Cloudflare is a content delivery network and DDoS mitigation company. Cloudflare provides a network designed to make everything you connect to the Internet secure, private, fast, and reliable; secure your websites, APIs, and Internet applications; protect corporate networks, employees, and devices; and write and deploy code that runs on the network edge.
 
 The Cloudflare Logpush integration can be used in three different modes to collect data:
 - HTTP Endpoint mode - Cloudflare pushes logs directly to an HTTP endpoint hosted by your Elastic Agent.
@@ -13,10 +13,28 @@ For example, you could use the data from this integration to know which websites
 
 ## Data streams
 
-The Cloudflare Logpush integration collects logs for seven types of events: Audit, DNS, Firewall Event, HTTP Request, NEL Report, Network Analytics, and Spectrum Event.
+The Cloudflare Logpush integration collects logs for the following types of events.
+
+### Zero Trust events
+
+**Access Request**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/access_requests/).
 
 **Audit**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/audit_logs/).
 
+**CASB findings**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/casb_findings/).
+
+**Device Posture Results**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/device_posture_results/).
+
+**Gateway DNS**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/gateway_dns/).
+
+**Gateway HTTP**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/gateway_http/).
+
+**Gateway Network**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/gateway_network/).
+
+**Zero Trust Network Session**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/account/zero_trust_network_sessions/).
+
+### Non Zero Trust events
+
 **DNS**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/zone/dns_logs/).
 
 **Firewall Event**: See Example Schema [here](https://developers.cloudflare.com/logs/reference/log-fields/zone/firewall_events/).
@@ -43,15 +61,22 @@ This module has been tested against **Cloudflare version v4**.
 - Configure the [Data Forwarder](https://developers.cloudflare.com/logs/get-started/enable-destinations/aws-s3/) to ingest data into an AWS S3 bucket.
 - The default value of the "Bucket List Prefix" is listed below. However, the user can set the parameter "Bucket List Prefix" according to the requirement.
 
- | Data Stream Name | Bucket List Prefix |
- | ----------------- | ---------------------- |
- | Audit Logs | audit_logs |
- | DNS | dns |
- | Firewall Event | firewall_event |
- | HTTP Request | http_request |
- | NEL Report | nel_report |
- | Network Analytics | network_analytics_logs |
- | Spectrum Event | spectrum_event |
+ | Data Stream Name | Bucket List Prefix |
+ | -------------------------- | ---------------------- |
+ | Access Request | access_request |
+ | Audit Logs | audit_logs |
+ | CASB findings | casb |
+ | Device Posture Results | device_posture |
+ | DNS | dns |
+ | Firewall Event | firewall_event |
+ | Gateway DNS | gateway_dns |
+ | Gateway HTTP | gateway_http |
+ | Gateway Network | gateway_network |
+ | HTTP Request | http_request |
+ | NEL Report | nel_report |
+ | Network Analytics | network_analytics_logs |
+ | Zero Trust Network Session | network_session |
+ | Spectrum Event | spectrum_event |
 
 ### To collect data from AWS SQS, follow the below steps:
 1. If data forwarding to an AWS S3 Bucket hasn't been configured, then first setup an AWS S3 Bucket as mentioned in the above documentation.
@@ -107,6 +132,17 @@ curl --location --request POST 'https://api.cloudflare.com/client/v4/zones/<ZONE
 
 ## Logs reference
 
+### access_request
+
+This is the `access_request` dataset.
+Default port for HTTP Endpoint: _9572_
+
+#### Example
+
+{{event "access_request"}}
+
+{{fields "access_request"}}
+
 ### audit
 
 This is the `audit` dataset.
@@ -118,6 +154,28 @@ Default port for HTTP Endpoint: _9560_
 
 {{fields "audit"}}
 
+### casb
+
+This is the `casb` dataset.
+Default port for HTTP Endpoint: _9571_
+
+#### Example
+
+{{event "casb"}}
+
+{{fields "casb"}}
+
+### device_posture
+
+This is the `device_posture` dataset.
+Default port for HTTP Endpoint: _9573_
+
+#### Example
+
+{{event "device_posture"}}
+
+{{fields "device_posture"}}
+
 ### dns
 
 This is the `dns` dataset.
@@ -140,6 +198,39 @@ Default port for HTTP Endpoint: _9562_
 
 {{fields "firewall_event"}}
 
+### gateway_dns
+
+This is the `gateway_dns` dataset.
+Default port for HTTP Endpoint: _9567_
+
+#### Example
+
+{{event "gateway_dns"}}
+
+{{fields "gateway_dns"}}
+
+### gateway_http
+
+This is the `gateway_http` dataset.
+Default port for HTTP Endpoint: _9568_
+
+#### Example
+
+{{event "gateway_http"}}
+
+{{fields "gateway_http"}}
+
+### gateway_network
+
+This is the `gateway_network` dataset.
+Default port for HTTP Endpoint: _9569_
+
+#### Example
+
+{{event "gateway_network"}}
+
+{{fields "gateway_network"}}
+
 ### http_request
 
 This is the `http_request` dataset.
@@ -173,6 +264,17 @@ Default port for HTTP Endpoint: _9565_
 
 {{fields "network_analytics"}}
 
+### network_session
+
+This is the `network_session` dataset.
+Default port for HTTP Endpoint: _9570_
+
+#### Example
+
+{{event "network_session"}}
+
+{{fields "network_session"}}
+
 ### spectrum_event
 
 This is the `spectrum_event` dataset.

packages/cloudflare_logpush/_dev/deploy/docker/docker-compose.yml

@@ -56,3 +56,59 @@ services:
  - STREAM_PROTOCOL=webhook
  - STREAM_ADDR=http://elastic-agent:9566/
  command: log --start-signal=SIGHUP --delay=5s /sample_logs/spectrum_event.log
+ cloudflare-logpush-gateway-dns-http-endpoint:
+ image: docker.elastic.co/observability/stream:v0.7.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=http://elastic-agent:9567/
+ command: log --start-signal=SIGHUP --delay=5s /sample_logs/gateway_dns.log
+ cloudflare-logpush-gateway-http-http-endpoint:
+ image: docker.elastic.co/observability/stream:v0.7.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=http://elastic-agent:9568/
+ command: log --start-signal=SIGHUP --delay=5s /sample_logs/gateway_http.log
+ cloudflare-logpush-gateway-network-http-endpoint:
+ image: docker.elastic.co/observability/stream:v0.7.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=http://elastic-agent:9569/
+ command: log --start-signal=SIGHUP --delay=5s /sample_logs/gateway_network.log
+ cloudflare-logpush-network-session-http-endpoint:
+ image: docker.elastic.co/observability/stream:v0.7.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=http://elastic-agent:9570/
+ command: log --start-signal=SIGHUP --delay=5s /sample_logs/network_session.log
+ cloudflare-logpush-casb-http-endpoint:
+ image: docker.elastic.co/observability/stream:v0.7.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=http://elastic-agent:9571/
+ command: log --start-signal=SIGHUP --delay=5s /sample_logs/casb.log
+ cloudflare-logpush-access-request-http-endpoint:
+ image: docker.elastic.co/observability/stream:v0.7.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=http://elastic-agent:9572/
+ command: log --start-signal=SIGHUP --delay=5s /sample_logs/access_request.log
+ cloudflare-logpush-device-posture-http-endpoint:
+ image: docker.elastic.co/observability/stream:v0.7.0
+ volumes:
+ - ./sample_logs:/sample_logs:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=http://elastic-agent:9573/
+ command: log --start-signal=SIGHUP --delay=5s /sample_logs/device_posture.log

packages/cloudflare_logpush/_dev/deploy/docker/sample_logs/access_request.log

@@ -0,0 +1 @@
+{"Action":"login","Allowed":true,"AppDomain":"partner-zt-logs.cloudflareaccess.com/warp","AppUUID":"123e4567-e89b-12d3-a456-426614174000","Connection":"onetimepin","Country":"us","CreatedAt":1684862313000000000,"Email":"user@example.com","IPAddress":"67.43.156.93","PurposeJustificationPrompt":"Please provide your reason for accessing the application.","PurposeJustificationResponse":"I need to access the application for work purposes.","RayID":"00c0ffeeabc12345","TemporaryAccessApprovers":["approver1@example.com","approver2@example.com"],"TemporaryAccessDuration":7200,"UserUID":"166befbb-00e3-5e20-bd6e-27245333949f"}

packages/cloudflare_logpush/_dev/deploy/docker/sample_logs/casb.log

@@ -0,0 +1 @@
+{"AssetDisplayName":"John Doe","AssetExternalID":"0051N000004mG2LAAA","AssetLink":"https://example.com/resource","AssetMetadata":{"Id":"0051N000004mG2LAAA","Fax":null,"Name":"John Doe","Alias":"JDoe","Email":"user@example.com","Phone":"+3460000000","Title":"Customer Solutions Engineer","Address":{"city":"Singapore","state":null,"street":null,"country":"Singapore","latitude":null,"longitude":null,"stateCode":null,"postalCode":null,"countryCode":"SG","geocodeAccuracy":null},"Division":null,"IsActive":false,"LastName":"Doe","UserType":"Standard","AccountId":null,"BadgeText":"","ContactId":null,"Extension":null,"FirstName":"John","Signature":null,"Department":"521","SenderName":null,"UserRoleId":"00E2G000001E","attributes":{"url":"/services/data/userID","type":"User"},"CompanyName":"MyCompany","MobilePhone":null,"SenderEmail":"sender@example.com","CallCenterId":null,"FullPhotoUrl":"https://photos.com/profilephoto/001","LocaleSidKey":"en_SG","LastLoginDate":"2021-10-06T06:32:09.000+0000","SmallPhotoUrl":"https://photos.com/photo/001","BannerPhotoUrl":"/profilephoto/001","EmployeeNumber":"18124","LastViewedDate":null,"TimeZoneSidKey":"Asia/Singapore","DigestFrequency":"D","ForecastEnabled":false,"EmailEncodingKey":"UTF-8","CommunityNickname":"Doe.John","LanguageLocaleKey":"en_US","LastReferencedDate":null,"ReceivesInfoEmails":true,"SmallBannerPhotoUrl":"/profilephoto/001/D","FederationIdentifier":null,"IsProfilePhotoActive":false,"MediumBannerPhotoUrl":"/profilephoto/001/E","EmailPreferencesAutoBcc":true,"ReceivesAdminInfoEmails":true,"OfflineTrialExpirationDate":null,"UserPermissionsOfflineUser":false,"UserPermissionsSupportUser":false,"UserPermissionsMarketingUser":false,"UserPermissionsInteractionUser":true,"DefaultGroupNotificationFrequency":"N","UserPermissionsCallCenterAutoLogin":false},"DetectedTimestamp":"2023-05-16T10:00:00Z","FindingTypeDisplayName":"Salesforce User Sending Email with Different Email Address","FindingTypeID":"a2790c4f-03f5-449f-b209-5f4447f417aa","FindingTypeSeverity":"Medium","InstanceID":"6b187be4-2dd5-42c5-a37b-111111111111","IntegrationDisplayName":"Salesforce Testing","IntegrationID":"c772678d-5cf1-4c73-bf3f-111111111111","IntegrationPolicyVendor":"Salesforce Connection"}

packages/cloudflare_logpush/_dev/deploy/docker/sample_logs/device_posture.log

@@ -0,0 +1 @@
+{"ClientVersion":"2023.3.258","DeviceID":"083a8354-d56c-11ed-9771-111111111","DeviceManufacturer":"Google Compute Engine","DeviceModel":"Google Compute Engine","DeviceName":"zt-test-vm1","DeviceSerialNumber":"GoogleCloud-ABCD1234567890","DeviceType":"linux","Email":"user@example.com","OSVersion":"5.15.0","PolicyID":"policy-abcdefgh","PostureCheckName":"Ubuntu","PostureCheckType":"os_version","PostureEvaluatedResult":true,"PostureExpectedJSON":{"version":"5.15.0-1025-gcp","operator":"==","os_distro_name":"ubuntu","os_distro_revision":"20.04"},"PostureReceivedJSON":{"version":"5.15.0-1025-gcp","operator":"==","os_distro_name":"ubuntu","os_distro_revision":"20.04"},"Timestamp":"2023-05-17T12:00:00Z","UserUID":"user-abcdefgh"}

packages/cloudflare_logpush/_dev/deploy/docker/sample_logs/gateway_dns.log

@@ -0,0 +1 @@
+{"ApplicationID":0,"ColoCode":"ORD","ColoID":14,"Datetime":"2023-05-02T22:49:53Z","DeviceID":"083a8354-d56c-11ed-9771-6a842b111aaa","DeviceName":"zt-test-vm1","DstIP":"89.160.20.129","DstPort":443,"Email":"user@test.com","Location":"GCP default","LocationID":"f233bd67-78c7-4050-9aff-ad63cce25732","MatchedCategoryIDs":[7,163],"MatchedCategoryNames":["Photography","Weather"],"Policy":"7bdc7a9c-81d3-4816-8e56-de1acad3dec5","PolicyID":"1412","Protocol":"https","QueryCategoryIDs":[26,155],"QueryCategoryNames":["Technology","Technology"],"QueryName":"security.ubuntu.com","QueryNameReversed":"com.ubuntu.security","QuerySize":48,"QueryType":1,"QueryTypeName":"A","RCode":0,"RData":[{"type":"1","data":"CHNlY3VyaXR5BnVidW50dQMjb20AAAEAAQAAAAgABLl9vic="},{"type":"1","data":"CHNlY3VyaXR5BnVidW50dQNjb20AAAEAABAAAAgABLl9viQ="},{"type":"1","data":"CHNlT3VyaXR5BnVidW50dQNjb20AAAEAAQAAAAgABFu9Wyc="}],"ResolvedIPs":["67.43.156.1","67.43.156.2","67.43.156.3"],"ResolverDecision":"allowedOnNoPolicyMatch","SrcIP":"67.43.156.2","SrcPort":0,"TimeZone":"UTC","TimeZoneInferredMethod":"fromLocalTime","UserID":"166befbb-00e3-5e20-bd6e-27245000000"}

packages/cloudflare_logpush/_dev/deploy/docker/sample_logs/gateway_http.log

@@ -0,0 +1 @@
+{"AccountID":"e1836771179f98aabb828da5ea69a348","Action":"block","BlockedFileHash":"91dc1db739a705105e1c763bfdbdaa84c0de8","BlockedFileName":"downloaded_test","BlockedFileReason":"malware","BlockedFileSize":43,"BlockedFileType":"bin","Datetime":"2023-05-03T20:55:05Z","DestinationIP":"89.160.20.129","DestinationPort":443,"DeviceID":"083a8354-d56c-11ed-9771-6a842b100cff","DeviceName":"zt-test-vm1","DownloadedFileNames":["downloaded_file","downloaded_test"],"Email":"user@example.com","FileInfo":{"files":[{"name":"downloaded_file","size":43},{"name":"downloaded_test","size":341}]},"HTTPHost":"guce.yahoo.com","HTTPMethod":"GET","HTTPStatusCode":302,"HTTPVersion":"HTTP/2","IsIsolated":false,"PolicyID":"85063bec-74cb-4546-85a3-e0cde2cdfda2","PolicyName":"Block Yahoo","Referer":"https://www.example.com/","RequestID":"1884fec9b600007fb06a299400000001","SourceInternalIP":"192.168.1.123","SourceIP":"67.43.156.2","SourcePort":47924,"UntrustedCertificateAction":"none","UploadedFileNames":["uploaded_file","uploaded_test"],"URL":"https://test.com","UserAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64) Firefox/112.0","UserID":"166befbb-00e3-5e20-bd6e-27245723949f"}

packages/cloudflare_logpush/_dev/deploy/docker/sample_logs/gateway_network.log

@@ -0,0 +1 @@
+{"AccountID":"e1836771179f98aabb828da5ea69a111","Action":"allowedOnNoRuleMatch","Datetime":1684444377058000000,"DestinationIP":"89.160.20.129","DestinationPort":443,"DeviceID":"083a8354-d56c-11ed-9771-6a842b100cff","DeviceName":"zt-test-vm1","Email":"user@test.com","OverrideIP":"175.16.199.4","OverridePort":8080,"PolicyID":"85063bec-74cb-4546-85a3-e0cde2cdfda2","PolicyName":"My policy","SNI":"www.elastic.co","SessionID":"5f2d04be-3512-11e8-b467-0ed5f89f718b","SourceIP":"67.43.156.2","SourceInternalIP":"192.168.1.3","SourcePort":47924,"Transport":"tcp","UserID":"166befbb-00e3-5e20-bd6e-27245723949f"}

packages/cloudflare_logpush/_dev/deploy/docker/sample_logs/network_session.log

@@ -0,0 +1 @@
+{"AccountID":"e1836771179f98aabb828da5ea69a111","BytesReceived":679,"BytesSent":2333,"ClientTCPHandshakeDurationMs":12,"ClientTLSCipher":"TLS_AES_128_GCM_SHA256","ClientTLSHandshakeDurationMs":125,"ClientTLSVersion":"TLS 1.3","ConnectionCloseReason":"CLIENT_CLOSED","ConnectionReuse":false,"DestinationTunnelID":"00000000-0000-0000-0000-000000000000","DeviceID":"083a8354-d56c-11ed-9771-6a842b100cff","DeviceName":"zt-test-vm1","EgressColoName":"ORD","EgressIP":"2a02:cf40::23","EgressPort":41052,"EgressRuleID":"00000000-0000-0000-0000-000000000000","EgressRuleName":"Egress Rule 1","Email":"user@test.com","IngressColoName":"ORD","Offramp":"INTERNET","OriginIP":"89.160.20.129","OriginPort":80,"OriginTLSCertificateIssuer":"DigiCert Inc","OriginTLSCertificateValidationResult":"VALID","OriginTLSCipher":"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","OriginTLSHandshakeDurationMs":130,"OriginTLSVersion":"TLS 1.2","Protocol":"TCP","RuleEvaluationDurationMs":10,"SessionEndTime":"2023-05-04T11:29:14Z","SessionID":"18881f179300007fb0d06d6400000001","SessionStartTime":"2023-05-04T11:29:14Z","SourceInternalIP":"1.128.0.1","SourceIP":"67.43.156.2","SourcePort":52994,"UserID":"166befbb-00e3-5e20-bd6e-27245723949f","VirtualNetworkID":"0ce99869-63d3-4d5d-bdaf-d4f33df964aa"}

packages/cloudflare_logpush/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.5.0"
+ changes:
+ - description: Add new data streams to cover all Zero Trust events.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/6132
 - version: "1.4.0"
  changes:
  - description: Update package to ECS 8.8.0.

packages/cloudflare_logpush/data_stream/access_request/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,4 @@
+fields:
+ tags:
+ - preserve_original_event
+ - preserve_duplicate_custom_fields

packages/cloudflare_logpush/data_stream/access_request/_dev/test/pipeline/test-pipeline-access-request.log

@@ -0,0 +1 @@
+{"Action":"login","Allowed":true,"AppDomain":"partner-zt-logs.cloudflareaccess.com/warp","AppUUID":"123e4567-e89b-12d3-a456-426614174000","Connection":"onetimepin","Country":"us","CreatedAt":1684862313000000000,"Email":"user@example.com","IPAddress":"67.43.156.93","PurposeJustificationPrompt":"Please provide your reason for accessing the application.","PurposeJustificationResponse":"I need to access the application for work purposes.","RayID":"00c0ffeeabc12345","TemporaryAccessApprovers":["approver1@example.com","approver2@example.com"],"TemporaryAccessDuration":7200,"UserUID":"166befbb-00e3-5e20-bd6e-27245333949f"}