[google_scc] Initial release for the Google SCC (#6645)

* Initial release for the Google SCC * Update the changelog entry * Resolve review comments * Resolve review comments and updated dashboard * Update fields desciption

Author: janvi-elastic

.github/CODEOWNERS

@@ -101,6 +101,7 @@
 /packages/github @elastic/security-external-integrations
 /packages/golang @elastic/obs-infraobs-integrations
 /packages/google_cloud_storage @elastic/security-external-integrations
+/packages/google_scc @elastic/security-external-integrations
 /packages/google_workspace @elastic/security-external-integrations
 /packages/hadoop @elastic/obs-infraobs-integrations
 /packages/haproxy @elastic/obs-infraobs-integrations

packages/google_scc/_dev/build/build.yml

@@ -0,0 +1,4 @@
+dependencies:
+ ecs:
+ reference: git@v8.8.0
+ import_mappings: true

packages/google_scc/_dev/build/docs/README.md

@@ -0,0 +1,146 @@
+# Google Security Command Center
+
+## Overview
+
+The [Google Security Command Center](https://cloud.google.com/security-command-center) integration allows users to monitor finding, audit, asset, and source. Security Command Center Premium provides comprehensive threat detection for Google Cloud that includes Event Threat Detection, Container Threat Detection, and Virtual Machine Threat Detection as built-in services.
+
+Use the Google SCC integration to collect and parse data from the Google SCC REST API (finding, asset, and source) or GCP Pub/Sub (finding, asset, and audit). Then visualize that data through search, correlation, and visualization within Elastic Security.
+
+## Data streams
+
+The Google SCC integration collects four types of data: finding, audit, asset, and source.
+
+**Finding** is a record of assessment data like security, risk, health, or privacy, that is ingested into Security Command Center for presentation, notification, analysis, policy testing, and enforcement. For example, a cross-site scripting (XSS) vulnerability in an App Engine application is a finding.
+
+**Audit** logs created by Security Command Center as part of Cloud Audit Logs.
+
+**Asset** lists assets with time and resource types and returns paged results in response.
+
+**Source** is an entity or a mechanism that can produce a finding. A source is like a container of findings that come from the same scanner, logger, monitor, and other tools.
+
+## Compatibility
+
+This module has been tested against the latest Google SCC API version **v1**.
+
+## Requirements
+
+- Elastic Agent must be installed.
+- You can install only one Elastic Agent per host.
+- Elastic Agent is required to stream data from the GCP Pub/Sub or REST API and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.
+
+### Installing and managing an Elastic Agent:
+
+You have a few options for installing and managing an Elastic Agent:
+
+### Install a Fleet-managed Elastic Agent (recommended):
+
+With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier.
+
+### Install Elastic Agent in standalone mode (advanced users):
+
+With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only.
+
+### Install Elastic Agent in a containerized environment:
+
+You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry and we provide deployment manifests for running on Kubernetes.
+
+There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+
+The minimum **kibana.version** required is **8.8.0**.
+
+## Prerequisites
+
+ - Create Google SCC service account [Steps to create](https://developers.google.com/identity/protocols/oauth2/service-account#creatinganaccount).
+ - Permissions required for Service Account: 
+ - Cloud Asset Viewer at Organization Level
+ - Pub/Sub Subscriber at Project Level
+ - Security Center Admin Editor at Organization Level
+ - **Security Command Center API** and **Cloud Asset API** must be enabled.
+
+This integration will make use of the following *oauth2 scope*:
+
+- `https://www.googleapis.com/auth/cloud-platform`
+
+Once Service Account credentials are downloaded as a JSON file, then the integration can be setup to collect data.
+
+If installing in GCP-Cloud Environment, No need to provide any credentials and make sure the account linked with the VM has all the required IAM permissions. Steps to [Set up Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).
+
+## Setup
+
+### To create GCP Pub/Sub, follow the below steps:
+
+- [Create Topic for Pub/sub](https://cloud.google.com/pubsub/docs/create-topic#create_a_topic).
+- [Create Subscription for topic](https://cloud.google.com/pubsub/docs/create-subscription#create_subscriptions)
+
+### To collect data from GCP Pub/Sub, follow the below steps:
+
+- [Configure to export finding to GCP Pub/Sub](https://cloud.google.com/security-command-center/docs/how-to-notifications).
+- [Configure to export asset to GCP Pub/Sub](https://cloud.google.com/asset-inventory/docs/monitoring-asset-changes).
+- [Configure to export audit to GCP Pub/Sub](https://cloud.google.com/logging/docs/export/configure_export_v2?_ga=2.110932226.-66737431.1679995682#overview).
+
+**NOTE**:
+ - **Sink destination** must be **Pub/Sub topic** while exporting audit logs to GCP Pub/Sub.
+ - Create unique Pub/Sub topic per data-stream.
+
+### Enabling the integration in Elastic:
+1. In Kibana go to **Management > Integrations**.
+2. In "Search for integrations" search bar, type **Google Security Command Center**.
+3. Click on the **Google Security Command Center** integration from the search results.
+4. Click on the **Add Google Security Command Center** Integration button to add the integration.
+5. While adding the integration, if you want to **collect logs via Rest API**, turn on the toggle and then put the following details:
+ - credentials type
+ - credentials JSON/file
+ - parent type
+ - id
+ - To collect **asset logs**, put the following details:
+ - content type
+
+ or if you want to **collect logs via GCP Pub/Sub**, turn on the toggle and then put the following details:
+ - credentials type
+ - credentials JSON/file
+ - project id
+ - To collect **asset, audit, or finding logs**, put the following details:
+ - topic
+ - subscription name 
+
+## Logs reference
+
+### Asset
+
+This is the `Asset` dataset.
+
+#### Example
+
+{{event "asset"}}
+
+{{fields "asset"}}
+
+### Finding
+
+This is the `Finding` dataset.
+
+#### Example
+
+{{event "finding"}}
+
+{{fields "finding"}}
+
+### Source
+
+This is the `Source` dataset.
+
+#### Example
+
+{{event "source"}}
+
+{{fields "source"}}
+
+### Audit
+
+This is the `Audit` dataset.
+
+#### Example
+
+{{event "audit"}}
+
+{{fields "audit"}}

packages/google_scc/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,62 @@
+version: '2.3'
+services:
+ google_scc:
+ image: docker.elastic.co/observability/stream:v0.10.0
+ hostname: google_scc
+ ports:
+ - 8090
+ volumes:
+ - ./files:/files:ro
+ environment:
+ PORT: '8090'
+ command:
+ - http-server
+ - --addr=:8090
+ - --config=/files/config.yml
+ gcppubsub-emulator:
+ image: google/cloud-sdk:emulators
+ command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8681
+ ports:
+ - "8681/tcp"
+ gcppubsub-audit:
+ image: docker.elastic.co/observability/stream:v0.10.0
+ volumes:
+ - ./files:/files:ro
+ command:
+ - log
+ - --retry=30
+ - --addr=gcppubsub-emulator:8681
+ - -p=gcppubsub
+ - --gcppubsub-clear=true
+ - --gcppubsub-project=audit
+ - /files/audit.log
+ depends_on:
+ - gcppubsub-emulator
+ gcppubsub-asset:
+ image: docker.elastic.co/observability/stream:v0.10.0
+ volumes:
+ - ./files:/files:ro
+ command:
+ - log
+ - --retry=30
+ - --addr=gcppubsub-emulator:8681
+ - -p=gcppubsub
+ - --gcppubsub-clear=true
+ - --gcppubsub-project=asset
+ - /files/asset.log
+ depends_on:
+ - gcppubsub-emulator
+ gcppubsub-finding:
+ image: docker.elastic.co/observability/stream:v0.10.0
+ volumes:
+ - ./files:/files:ro
+ command:
+ - log
+ - --retry=30
+ - --addr=gcppubsub-emulator:8681
+ - -p=gcppubsub
+ - --gcppubsub-clear=true
+ - --gcppubsub-project=finding
+ - /files/finding.log
+ depends_on:
+ - gcppubsub-emulator

packages/google_scc/_dev/deploy/docker/files/asset.log

@@ -0,0 +1 @@
+{"asset":{"ancestors":["projects/123456987522","folders/123456987520","organizations/523456987520"],"assetType":"logging.googleapis.com/LogBucket","name":"//logging.googleapis.com/projects/123456987522/locations/global/buckets/_Default","resource":{"data":{"description":"Default bucket","lifecycleState":"ACTIVE","name":"projects/123456987522/locations/global/buckets/_Default","retentionDays":30},"discoveryDocumentUri":"https://logging.googleapis.com/$discovery/rest","discoveryName":"LogBucket","location":"global","parent":"//cloudresourcemanager.googleapis.com/projects/123456987522","version":"v2"},"updateTime":"2023-05-28T06:59:48.052491Z"},"priorAsset":{"ancestors":["projects/123456987522","folders/123456987520","organizations/523456987520"],"assetType":"logging.googleapis.com/LogBucket","name":"//logging.googleapis.com/projects/123456987522/locations/global/buckets/_Default","resource":{"data":{"analyticsEnabled":true,"description":"Default bucket","lifecycleState":"ACTIVE","name":"projects/123456987522/locations/global/buckets/_Default","retentionDays":30},"discoveryDocumentUri":"https://logging.googleapis.com/$discovery/rest","discoveryName":"LogBucket","location":"global","parent":"//cloudresourcemanager.googleapis.com/projects/123456987522","version":"v2"},"updateTime":"2023-05-27T18:53:48.843904Z"},"priorAssetState":"PRESENT","window":{"startTime":"2023-05-28T06:59:48.052491Z"}}

packages/google_scc/_dev/deploy/docker/files/audit.log

@@ -0,0 +1 @@
+{"protoPayload":{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"test-user@example.net"},"requestMetadata":{"callerIp":"175.16.199.1","requestAttributes":{},"destinationAttributes":{}},"serviceName":"login.googleapis.com","methodName":"google.login.LoginService.loginFailure","resourceName":"organizations/123"},"insertId":"-nahbepd4l1x","resource":{"type":"audited_resource"},"httpRequest":{"remoteIp":"FE80::0202:B3FF:FE1E:1010"},"timestamp":"2021-09-24T16:16:57.183212Z","severity":"NOTICE","logName":"organizations/123/logs/cloudaudit.googleapis.com%2Fdata_access","receiveTimestamp":"2021-09-24T17:51:25.034361197Z"}

packages/google_scc/_dev/deploy/docker/files/config.yml

@@ -0,0 +1,59 @@
+rules:
+ - path: /token
+ methods: [POST]
+ request_headers:
+ Content-Type:
+ - "application/x-www-form-urlencoded"
+ responses:
+ - status_code: 200
+ headers:
+ Content-Type:
+ - "application/json"
+ body: >
+ {"access_token": "1/fFAGRNJru1FTz70BzhT3Zg","expires_in": 3920,"token_type": "Bearer", "scope": "https://www.googleapis.com/auth/admin.reports.audit.readonly","refresh_token": "1//xEoDL4iW3cxlI7yDbSRFYNG01kVKM2C-259HOF2aQbI"}
+
+ - path: /v1/organizations/xxxx/sources/-/findings
+ methods: [GET]
+ request_headers:
+ Accept:
+ - "application/json"
+ Authorization:
+ - "Bearer 1/fFAGRNJru1FTz70BzhT3Zg"
+ responses:
+ - status_code: 200
+ headers:
+ Content-Type:
+ - "application/json"
+ body: >
+ {"listFindingsResults":[{"finding":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545","parent":"organizations/515665165161/sources/98481484454154454545","resourceName":"//cloudresourcemanager.googleapis.com/projects/45455445554","state":"ACTIVE","category":"application","externalUri":"http://www.adwait.com","securityMarks":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545/securityMarks"},"eventTime":"2023-06-02T05:17:41.936Z","createTime":"2020-02-19T13:37:43.858Z","severity":"CRITICAL","canonicalName":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545","mute":"UNMUTED","muteUpdateTime":"2022-03-23T05:50:21.804Z","externalSystems":{"test":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545/externalSystems/test","assignees":["primary"],"externalUid":"test_scc_finding_2","status":"updated1","externalSystemUpdateTime":"2022-01-05T05:00:35.674Z"}},"muteInitiator":"Unmuted by john@gmail.com"},"resource":{"name":"//cloudresourcemanager.googleapis.com/projects/45455445554"}}]}
+
+ - path: /v1/organizations/xxxx/sources
+ methods: [GET]
+ request_headers:
+ Accept:
+ - "application/json"
+ Authorization:
+ - "Bearer 1/fFAGRNJru1FTz70BzhT3Zg"
+ responses:
+ - status_code: 200
+ headers:
+ Content-Type:
+ - "application/json"
+ body: >
+ {"sources":[{"name":"organizations/595779152576/sources/10134421585261057824","displayName":"Cloudflare Security Events","description":"Extend your security view from the edge.","canonicalName":"organizations/595779152576/sources/10134421585261057824"}]}
+
+ - path: /v1/organizations/xxxx/assets
+ methods: [GET]
+ request_headers:
+ Accept:
+ - "application/json"
+ Authorization:
+ - "Bearer 1/fFAGRNJru1FTz70BzhT3Zg"
+ responses:
+ - status_code: 200
+ headers:
+ Content-Type:
+ - "application/json"
+ body: >
+ {"assets":[{"name":"//cloudbilling.googleapis.com/billingAccounts/012345-A08098-1Ab2CD","assetType":"cloudbilling.googleapis.com/BillingAccount","resource":{"version":"v1","discoveryDocumentUri":"https://cloudbilling.googleapis.com/$discovery/rest","discoveryName":"BillingAccount","data":{"displayName":"New","name":"billingAccounts/012345-A08098-1Ab2CD"},"location":"global"},"ancestors":["organizations/523456987520"],"updateTime":"2022-11-17T12:20:17.601902Z"}]}
+

packages/google_scc/_dev/deploy/docker/files/finding.log

@@ -0,0 +1 @@
+{"finding":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545","parent":"organizations/515665165161/sources/98481484454154454545","resourceName":"//cloudresourcemanager.googleapis.com/projects/45455445554","state":"ACTIVE","category":"application","externalUri":"http://www.adwait.com","sourceProperties":{},"securityMarks":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545/securityMarks"},"eventTime":"2023-06-02T05:17:41.936Z","createTime":"2020-02-19T13:37:43.858Z","severity":"CRITICAL","canonicalName":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545","mute":"UNMUTED","muteUpdateTime":"2022-03-23T05:50:21.804Z","externalSystems":{"test":{"name":"organizations/515665165161/sources/98481484454154454545/findings/414rfrhjebhrbhjbr444454hv54545/externalSystems/test","assignees":["primary"],"externalUid":"test_scc_finding_2","status":"updated1","externalSystemUpdateTime":"2022-01-05T05:00:35.674Z"}},"muteInitiator":"Unmuted by john@gmail.com"},"resource":{"name":"//cloudresourcemanager.googleapis.com/projects/45455445554"}}

packages/google_scc/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.1.0"
+ changes:
+ - description: Initial release.
+ type: enhancement
+ link: https://github.com/elastic/integrations/pull/6645