Skip to content

Conversation

@brijesh-elastic
Copy link
Collaborator

What does this PR do?

  • Generated the skeleton of the Entity Analytics Okta integration package.
  • Added a data stream.
  • Added a data collection logic for the data stream.
  • Added the ingest pipeline for the data stream.
  • Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
  • Added dashboard and visualizations.
  • Added a pipeline test for the data stream.
  • Added a system test cases for the data stream.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

  • Change follows the contributing guidelines
  • Supported versions of the monitoring target is documented
  • Supported operating systems are documented (if applicable)
  • Integration or System tests exist
  • Documentation exists
  • Fields follow ECS and naming conventions
  • At least a manual test with ES / Kibana / Agent has been performed.
  • Required Kibana version set to: ^8.9.0

New Package

  • Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

  • Dashboards exists
  • Screenshots added or updated
  • Datastream filters added to visualizations

Log dataset changes

  • Pipeline tests exist (if applicable)
  • Generated output for at least 1 log file exists
  • Sample event (sample_event.json) exists

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/entityanalytics_okta directory.
  • Run the following command to run tests.

elastic-package test

Screenshots

Integration Page
Overview page

Automated Test

2023/07/07 16:49:35 DEBUG Enable verbose logging Run pipeline tests for the package --- Test results for package: entityanalytics_okta - START --- ╭──────────────────────┬─────────────┬───────────┬────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├──────────────────────┼─────────────┼───────────┼────────────────┼────────┼──────────────┤ │ entityanalytics_okta │ user │ pipeline │ test-user.json │ PASS │ 12.462813ms │ ╰──────────────────────┴─────────────┴───────────┴────────────────┴────────┴──────────────╯ --- Test results for package: entityanalytics_okta - END --- Done 2023/07/07 16:48:02 DEBUG Enable verbose logging Run system tests for the package 2023/07/07 16:48:02 DEBUG Running system tests for data stream 2023/07/07 16:48:02 DEBUG running test with configuration 'default' 2023/07/07 16:48:02 DEBUG setting up service... 2023/07/07 16:48:02 DEBUG setting up service using Docker Compose service deployer 2023/07/07 16:48:02 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:03 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:03 DEBUG output command: /usr/bin/docker network inspect elastic-package-stack_default 2023/07/07 16:48:03 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service up --build -d Creating network "elastic-package-service_default" with the default driver Creating elastic-package-service_entityanalytics_okta_1 ... done 2023/07/07 16:48:04 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service ps -q 2023/07/07 16:48:05 DEBUG Wait for healthy containers: 401e476f2726b3c1e686ad2b47622f4c811187c216ee182bb0048336fb801aed 2023/07/07 16:48:05 DEBUG output command: /usr/bin/docker inspect 401e476f2726b3c1e686ad2b47622f4c811187c216ee182bb0048336fb801aed 2023/07/07 16:48:05 DEBUG Container status: {"Config":{"Image":"docker.elastic.co/observability/stream:v0.10.0","Labels":{"BRANCH_NAME":"v0.10.0","GIT_SHA":"2a076c9b1acdf1c35b5f5c2f8c23904c7c2c441a","GO_VERSION":"1.19.5","TIMESTAMP":"2023-01-30_11:29","com.docker.compose.config-hash":"cdd887d3c90592da5f89631af4604b333c1035986647afaa25239c160a8311be","com.docker.compose.container-number":"1","com.docker.compose.oneoff":"False","com.docker.compose.project":"elastic-package-service","com.docker.compose.project.config_files":"/root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml","com.docker.compose.project.working_dir":"/root/integrations/packages/entityanalytics_okta/_dev/deploy/docker","com.docker.compose.service":"entityanalytics_okta","com.docker.compose.version":"1.29.2"}},"ID":"401e476f2726b3c1e686ad2b47622f4c811187c216ee182bb0048336fb801aed","State":{"Status":"running","ExitCode":0,"Health":null}} 2023/07/07 16:48:05 DEBUG run command: /usr/bin/docker network connect elastic-package-stack_default elastic-package-service_entityanalytics_okta_1 2023/07/07 16:48:05 DEBUG adding service container elastic-package-service_entityanalytics_okta_1 internal ports to context 2023/07/07 16:48:05 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service config 2023/07/07 16:48:06 DEBUG Installing package... 2023/07/07 16:48:06 DEBUG GET https://127.0.0.1:5601/api/status 2023/07/07 16:48:06 DEBUG Build directory: /root/integrations/build/packages/entityanalytics_okta/0.1.0 2023/07/07 16:48:06 DEBUG Clear target directory (path: /root/integrations/build/packages/entityanalytics_okta/0.1.0) 2023/07/07 16:48:06 DEBUG Copy package content (source: /root/integrations/packages/entityanalytics_okta) 2023/07/07 16:48:06 DEBUG Copy license file if needed 2023/07/07 16:48:06 INFO License text found in "/root/integrations/LICENSE.txt" will be included in package 2023/07/07 16:48:06 DEBUG Encode dashboards 2023/07/07 16:48:06 DEBUG Resolve external fields 2023/07/07 16:48:06 DEBUG Package has external dependencies defined 2023/07/07 16:48:06 DEBUG data_stream/user/fields/base-fields.yml: source file hasn't been changed 2023/07/07 16:48:06 DEBUG data_stream/user/fields/beats.yml: source file hasn't been changed 2023/07/07 16:48:06 DEBUG data_stream/user/fields/fields.yml: source file hasn't been changed 2023/07/07 16:48:06 INFO Import ECS mappings into the built package (technical preview) 2023/07/07 16:48:06 DEBUG Build zipped package 2023/07/07 16:48:06 DEBUG Compress using archiver.Zip (destination: /root/integrations/build/packages/entityanalytics_okta-0.1.0.zip) 2023/07/07 16:48:06 DEBUG Create work directory for archiving: /tmp/elastic-package-1001619349/entityanalytics_okta-0.1.0 2023/07/07 16:48:06 DEBUG Skip validation of the built .zip package 2023/07/07 16:48:06 DEBUG POST https://127.0.0.1:5601/api/fleet/epm/packages 2023/07/07 16:48:08 DEBUG creating test policy... 2023/07/07 16:48:08 DEBUG POST https://127.0.0.1:5601/api/fleet/agent_policies 2023/07/07 16:48:12 DEBUG adding package data stream to test policy... 2023/07/07 16:48:12 DEBUG POST https://127.0.0.1:5601/api/fleet/package_policies 2023/07/07 16:48:15 DEBUG deleting old data in data stream... 2023/07/07 16:48:15 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream: index_not_found_exception: no such index [logs-entityanalytics_okta.user-ep] Status=404 2023/07/07 16:48:15 DEBUG GET https://127.0.0.1:5601/api/fleet/agents 2023/07/07 16:48:15 DEBUG filter agents using criteria: NamePrefix=docker-fleet-agent 2023/07/07 16:48:15 DEBUG found 1 enrolled agent(s) 2023/07/07 16:48:15 DEBUG GET https://127.0.0.1:5601/api/fleet/agent_policies/f38003b0-1cb7-11ee-9f7e-77b04ebefa97 2023/07/07 16:48:15 DEBUG assigning package data stream to agent... 2023/07/07 16:48:15 DEBUG PUT https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4/reassign 2023/07/07 16:48:17 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4 2023/07/07 16:48:17 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"f38003b0-1cb7-11ee-9f7e-77b04ebefa97","local_metadata":{"host":{"name":"docker-fleet-agent"}}} 2023/07/07 16:48:17 DEBUG Wait until the policy (ID: f38003b0-1cb7-11ee-9f7e-77b04ebefa97, revision: 2) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)... 2023/07/07 16:48:19 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4 2023/07/07 16:48:19 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"f38003b0-1cb7-11ee-9f7e-77b04ebefa97","local_metadata":{"host":{"name":"docker-fleet-agent"}}} 2023/07/07 16:48:19 DEBUG Wait until the policy (ID: f38003b0-1cb7-11ee-9f7e-77b04ebefa97, revision: 2) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)... 2023/07/07 16:48:21 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4 2023/07/07 16:48:21 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"f38003b0-1cb7-11ee-9f7e-77b04ebefa97","local_metadata":{"host":{"name":"docker-fleet-agent"}}} 2023/07/07 16:48:21 DEBUG Wait until the policy (ID: f38003b0-1cb7-11ee-9f7e-77b04ebefa97, revision: 2) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)... 2023/07/07 16:48:23 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4 2023/07/07 16:48:23 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"f38003b0-1cb7-11ee-9f7e-77b04ebefa97","policy_revision":2,"local_metadata":{"host":{"name":"docker-fleet-agent"}}} 2023/07/07 16:48:23 DEBUG Policy revision assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)... 2023/07/07 16:48:23 DEBUG checking for expected data in data stream... 2023/07/07 16:48:23 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream: index_not_found_exception: no such index [logs-entityanalytics_okta.user-ep] Status=404 2023/07/07 16:48:24 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream: index_not_found_exception: no such index [logs-entityanalytics_okta.user-ep] Status=404 2023/07/07 16:48:25 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream: index_not_found_exception: no such index [logs-entityanalytics_okta.user-ep] Status=404 2023/07/07 16:48:26 DEBUG found 0 hits in logs-entityanalytics_okta.user-ep data stream 2023/07/07 16:48:27 DEBUG found 3 hits in logs-entityanalytics_okta.user-ep data stream 2023/07/07 16:48:27 DEBUG check whether or not synthetics is enabled (component template logs-entityanalytics_okta.user@package)... 2023/07/07 16:48:27 DEBUG data stream logs-entityanalytics_okta.user-ep has synthetics enabled: false 2023/07/07 16:48:28 DEBUG reassigning original policy back to agent... 2023/07/07 16:48:28 DEBUG PUT https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4/reassign 2023/07/07 16:48:28 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4 2023/07/07 16:48:28 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}} 2023/07/07 16:48:28 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 3) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)... 2023/07/07 16:48:30 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4 2023/07/07 16:48:30 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}} 2023/07/07 16:48:30 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 3) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)... 2023/07/07 16:48:32 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4 2023/07/07 16:48:32 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"elastic-agent-managed-ep","local_metadata":{"host":{"name":"docker-fleet-agent"}}} 2023/07/07 16:48:32 DEBUG Wait until the policy (ID: elastic-agent-managed-ep, revision: 3) is assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)... 2023/07/07 16:48:34 DEBUG GET https://127.0.0.1:5601/api/fleet/agents/8e33d03e-a022-4cf9-babb-1a8b586334d4 2023/07/07 16:48:34 DEBUG Agent data: {"id":"8e33d03e-a022-4cf9-babb-1a8b586334d4","policy_id":"elastic-agent-managed-ep","policy_revision":3,"local_metadata":{"host":{"name":"docker-fleet-agent"}}} 2023/07/07 16:48:34 DEBUG Policy revision assigned to the agent (ID: 8e33d03e-a022-4cf9-babb-1a8b586334d4)... 2023/07/07 16:48:34 DEBUG deleting test policy... 2023/07/07 16:48:34 DEBUG POST https://127.0.0.1:5601/api/fleet/agent_policies/delete 2023/07/07 16:48:37 DEBUG DELETE https://127.0.0.1:5601/api/fleet/epm/packages/entityanalytics_okta-0.1.0 2023/07/07 16:48:39 DEBUG tearing down service... 2023/07/07 16:48:39 DEBUG tearing down service using Docker Compose runner 2023/07/07 16:48:39 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:40 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:40 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service logs 2023/07/07 16:48:40 INFO Write container logs to file: /root/integrations/build/container-logs/entityanalytics_okta-1688728720906242006.log 2023/07/07 16:48:40 DEBUG running command: /usr/local/bin/docker-compose -f /root/integrations/packages/entityanalytics_okta/_dev/deploy/docker/docker-compose.yml -p elastic-package-service down --volumes Stopping elastic-package-service_entityanalytics_okta_1 ... done Removing elastic-package-service_entityanalytics_okta_1 ... done Removing network elastic-package-service_default 2023/07/07 16:48:42 DEBUG deleting data in data stream... 2023/07/07 16:48:42 DEBUG Dump Elastic stack data 2023/07/07 16:48:42 DEBUG Dump stack logs (location: /tmp/test-system-3018837560) 2023/07/07 16:48:42 DEBUG Dump stack logs for elasticsearch 2023/07/07 16:48:42 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:42 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:42 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elasticsearch 2023/07/07 16:48:43 DEBUG Dump stack logs for elastic-agent 2023/07/07 16:48:43 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:44 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:44 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs elastic-agent 2023/07/07 16:48:44 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:45 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:45 DEBUG run command: /usr/bin/docker cp elastic-package-stack_elastic-agent_1:/usr/share/elastic-agent/state/data/logs/ /tmp/test-system-3018837560/logs/elastic-agent-internal 2023/07/07 16:48:45 DEBUG Dump stack logs for fleet-server 2023/07/07 16:48:45 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:46 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:46 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs fleet-server 2023/07/07 16:48:47 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:47 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:47 DEBUG run command: /usr/bin/docker cp elastic-package-stack_fleet-server_1:/usr/share/elastic-agent/state/data/logs/ /tmp/test-system-3018837560/logs/fleet-server-internal 2023/07/07 16:48:47 DEBUG Dump stack logs for kibana 2023/07/07 16:48:47 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:48 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:48 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs kibana 2023/07/07 16:48:49 DEBUG Dump stack logs for package-registry 2023/07/07 16:48:49 DEBUG running command: /usr/local/bin/docker-compose version --short 2023/07/07 16:48:49 DEBUG Determined Docker Compose version: 1.29.2, the tool will use Compose V1 2023/07/07 16:48:49 DEBUG running command: /usr/local/bin/docker-compose -f /root/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack logs package-registry 2023/07/07 16:48:50 DEBUG skipped malformed docker-compose log line: Attaching to elastic-package-stack_elastic-agent_1 --- Test results for package: entityanalytics_okta - START --- ╭──────────────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├──────────────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤ │ entityanalytics_okta │ user │ system │ default │ PASS │ 25.068587576s │ ╰──────────────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯ --- Test results for package: entityanalytics_okta - END --- Done 2023/07/07 16:49:57 DEBUG Enable verbose logging Run static tests for the package --- Test results for package: entityanalytics_okta - START --- ╭──────────────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├──────────────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤ │ entityanalytics_okta │ user │ static │ Verify sample_event.json │ PASS │ 137.744969ms │ ╰──────────────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯ --- Test results for package: entityanalytics_okta - END --- Done 2023/07/07 16:50:23 DEBUG Enable verbose logging Run asset tests for the package 2023/07/07 16:50:23 DEBUG installing package... 2023/07/07 16:50:23 DEBUG GET https://127.0.0.1:5601/api/status 2023/07/07 16:50:24 DEBUG Build directory: /root/integrations/build/packages/entityanalytics_okta/0.1.0 2023/07/07 16:50:24 DEBUG Clear target directory (path: /root/integrations/build/packages/entityanalytics_okta/0.1.0) 2023/07/07 16:50:24 DEBUG Copy package content (source: /root/integrations/packages/entityanalytics_okta) 2023/07/07 16:50:24 DEBUG Copy license file if needed 2023/07/07 16:50:24 INFO License text found in "/root/integrations/LICENSE.txt" will be included in package 2023/07/07 16:50:24 DEBUG Encode dashboards 2023/07/07 16:50:24 DEBUG Resolve external fields 2023/07/07 16:50:24 DEBUG Package has external dependencies defined 2023/07/07 16:50:24 DEBUG data_stream/user/fields/base-fields.yml: source file hasn't been changed 2023/07/07 16:50:24 DEBUG data_stream/user/fields/beats.yml: source file hasn't been changed 2023/07/07 16:50:24 DEBUG data_stream/user/fields/fields.yml: source file hasn't been changed 2023/07/07 16:50:24 INFO Import ECS mappings into the built package (technical preview) 2023/07/07 16:50:24 DEBUG Build zipped package 2023/07/07 16:50:24 DEBUG Compress using archiver.Zip (destination: /root/integrations/build/packages/entityanalytics_okta-0.1.0.zip) 2023/07/07 16:50:24 DEBUG Create work directory for archiving: /tmp/elastic-package-2748993626/entityanalytics_okta-0.1.0 2023/07/07 16:50:24 DEBUG Skip validation of the built .zip package 2023/07/07 16:50:24 DEBUG POST https://127.0.0.1:5601/api/fleet/epm/packages 2023/07/07 16:50:25 DEBUG removing package... 2023/07/07 16:50:25 DEBUG DELETE https://127.0.0.1:5601/api/fleet/epm/packages/entityanalytics_okta-0.1.0 --- Test results for package: entityanalytics_okta - START --- ╭──────────────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├──────────────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤ │ entityanalytics_okta │ │ asset │ dashboard entityanalytics_okta-e5242a60-0f35-11ee-8319-1d33c4a0c7ae is loaded │ PASS │ 3.377µs │ │ entityanalytics_okta │ │ asset │ search entityanalytics_okta-d4f05110-0f7a-11ee-8319-1d33c4a0c7ae is loaded │ PASS │ 754ns │ │ entityanalytics_okta │ user │ asset │ index_template logs-entityanalytics_okta.user is loaded │ PASS │ 731ns │ │ entityanalytics_okta │ user │ asset │ ingest_pipeline logs-entityanalytics_okta.user-0.1.0 is loaded │ PASS │ 943ns │ ╰──────────────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯ --- Test results for package: entityanalytics_okta - END --- Done 
@elasticmachine
Copy link

elasticmachine commented Jul 11, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-07-18T12:27:03.673+0000

  • Duration: 14 min 23 sec

Test stats 🧪

Test Results
Failed 0
Passed 7
Skipped 0
Total 7

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Jul 11, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (1/1) 💚
Classes 100.0% (1/1) 💚
Methods 100.0% (9/9) 💚 33.333
Lines 90.925% (521/573) 👎 -9.075
Conditionals 100.0% (0/0) 💚
@jamiehynds jamiehynds requested a review from a team July 11, 2023 15:42
responses:
- status_code: 200
body: |
[{"id":"00ub0oNGTSWTBKOLGLNR","status":"ACTIVE","created":"2013-06-24T16:39:18.000Z","activated":"2013-06-24T16:39:19.000Z","statusChanged":"2013-06-24T16:39:19.000Z","lastLogin":"2013-06-24T17:39:19.000Z","lastUpdated":"2013-07-02T21:36:25.344Z","passwordChanged":"2013-07-02T21:36:25.344Z","profile":{"firstName":"Isaac","lastName":"Brock","email":"isaac.brock@example.com","login":"isaac.brock@example.com","mobilePhone":"555-415-1337"},"credentials":{"password":{},"recovery_question":{"question":"Who's a major player in the cowboy scene?"},"provider":{"type":"OKTA","name":"OKTA"}}}]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the password field, please add subfields, "value": "tlpWENT2m" and to recovery_question please add "answer": "Annie Oakley". These will be stripped by the provider so the change will test that this is the case and we are not retaining secrets.

Suggested change
[{"id":"00ub0oNGTSWTBKOLGLNR","status":"ACTIVE","created":"2013-06-24T16:39:18.000Z","activated":"2013-06-24T16:39:19.000Z","statusChanged":"2013-06-24T16:39:19.000Z","lastLogin":"2013-06-24T17:39:19.000Z","lastUpdated":"2013-07-02T21:36:25.344Z","passwordChanged":"2013-07-02T21:36:25.344Z","profile":{"firstName":"Isaac","lastName":"Brock","email":"isaac.brock@example.com","login":"isaac.brock@example.com","mobilePhone":"555-415-1337"},"credentials":{"password":{},"recovery_question":{"question":"Who's a major player in the cowboy scene?"},"provider":{"type":"OKTA","name":"OKTA"}}}]
[{"id":"00ub0oNGTSWTBKOLGLNR","status":"ACTIVE","created":"2013-06-24T16:39:18.000Z","activated":"2013-06-24T16:39:19.000Z","statusChanged":"2013-06-24T16:39:19.000Z","lastLogin":"2013-06-24T17:39:19.000Z","lastUpdated":"2013-07-02T21:36:25.344Z","passwordChanged":"2013-07-02T21:36:25.344Z","profile":{"firstName":"Isaac","lastName":"Brock","email":"isaac.brock@example.com","login":"isaac.brock@example.com","mobilePhone":"555-415-1337"},"credentials":{"password":{"value": "tlpWENT2m"},"recovery_question":{"question":"Who's a major player in the cowboy scene?","answer": "Annie Oakley"},"provider":{"type":"OKTA","name":"OKTA"}}}]
Comment on lines 39 to 55
"credentials": {
"password": {
"hash": {
"algorithm": "SHA-1",
"salt": "UEO3wsAsgzQ=",
"saltOrder": "POSTFIX",
"value": "xjrauE6J6kbjcvMjWSSc+PsBBls="
}
},
"recovery_question": {
"question": "Who's a major player in the cowboy scene?"
},
"provider": {
"type": "OKTA",
"name": "OKTA"
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provider does not fill password or recovery_question. If you are finding that it does, this is a defect, so please don't have pipeline tests that include these fields (or if you do, ensure that the secrets are deleted from the document before ingest).

https://github.com/elastic/beats/blob/3de3d53e29437c80ea8a4d54060564ab44fd2360/x-pack/filebeat/input/entityanalytics/provider/okta/internal/okta/okta.go#L92-L96

Looking at sample_event.json, this appears to be true, happily.

- name: credentials
type: group
fields:
- name: password
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This group should never be present.

type: keyword
- name: type
type: keyword
- name: recovery_question
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This group should never be present.

- name: sync_interval
type: text
title: Sync Interval
description: How often full synchronizations should occur. Must be greater than Update Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 24h. NOTE:- Supported units for this parameter are h/m/s.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description: How often full synchronizations should occur. Must be greater than Update Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 24h. NOTE:- Supported units for this parameter are h/m/s.
description: How often full synchronizations should occur. Must be greater than Update Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 24h. Supported units for this parameter are h/m/s.
- name: update_interval
type: text
title: Update Interval
description: How often incremental updates should occur. Must be less than Sync Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 15m. NOTE:- Supported units for this parameter are h/m/s.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description: How often incremental updates should occur. Must be less than Sync Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 15m. NOTE:- Supported units for this parameter are h/m/s.
description: How often incremental updates should occur. Must be less than Sync Interval. Expected value is a duration string (15m, 1h, 1m30, etc), defaults to 15m. Supported units for this parameter are h/m/s.
- name: http_client_timeout
type: text
title: HTTP Client Timeout
description: "Duration before declaring that the HTTP client connection has timed out. NOTE: Valid time units are ns, us, ms, s, m, h."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description: "Duration before declaring that the HTTP client connection has timed out. NOTE: Valid time units are ns, us, ms, s, m, h."
description: "Duration before declaring that the HTTP client connection has timed out. Valid time units are ns, us, ms, s, m, h."
@brijesh-elastic brijesh-elastic requested a review from efd6 July 14, 2023 12:06
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after clarifications.

@@ -0,0 +1,234 @@
- name: asset
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the asset group formally approved?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've added the asset and user group objects to the fields.yml because these fields are coming to the upcoming ECS Schema. (@jamiehynds told us to follow the PR and map this field to the upcoming proposed ECS fields.)

I've one concern: if in the future (let's say 8.9) this proposed field comes in ECS, then it may be a conflict that we've added that in fields.yml and also that it will be added by import mappings (from the build.yml). (Again, that scenario may arise when we have the same fields in agent.yml and ecs.yml)

Copy link
Contributor

@efd6 efd6 Jul 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I had not seen that ECS RFC. When the fields are in we can clean up any issues that arise.

- name: type
type: object
description: user type that determines the schema for the user's profile.
- name: labels
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same query for labels.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For labels, it is a necessary field that I have to add in fields.yml in order to execute a successful system test. If we don't provide that, then it results in labels.identity_source is undefined error. even if the labels field is added by the import_mapping feature.

So, to resolve this issue, I've added labels.identity_source in fields.yml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since labels is an ECS field, please add this to the ecs.yml, its totally fine to define ECS fields in ecs.yml even though dynamic mappings are enabled, especially if they are not covered by it. I will take a note to update the dynamic template.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@P1llus, should we add upcoming proposed ECS fields in ecs.yml too ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brijesh-elastic sure that would be great!

@P1llus P1llus merged commit 64f5d35 into elastic:main Jul 18, 2023
@elasticmachine
Copy link

Package entityanalytics_okta - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=entityanalytics_okta

@andrewkroh andrewkroh added the Integration:entityanalytics_okta Okta Entity Analytics label Jul 22, 2024
@andrewkroh andrewkroh added Crest Contributions from Crest developement team. New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Crest Contributions from Crest developement team. Integration:entityanalytics_okta Okta Entity Analytics New Integration Issue or pull request for creating a new integration package.

5 participants