[Cisco Duo] Integration updates #11200

chemamartinez · 2024-09-20T16:07:08Z

Proposed commit message

This pull request contains several changes for the Cisco Duo integration:

Added new data stream telephony_v2 to support the new v2 API endpoint.
Updated the auth data stream to migrate from the HTTPJSON to CEL.

Both data streams above include a CEL program to make requests to the v2 version of the Cisco Duo API, following the next specifications:
Review documentation and dashboards

The upgrade process has been tested manually to verify that changes in current data streams don't break current users during upgrades.

Data streams that use the new CEL inputs are disabled by default. For the case of the auth data stream, users that are using it will have to enable it again when upgrading the integration, because of the migration from httpjson to CEL.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Added pipeline and system tests for affected data streams.

Run asset tests for the package --- Test results for package: cisco_duo - START --- ╭───────────┬────────────────────┬───────────┬────────────────────────────────────────────────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├───────────┼────────────────────┼───────────┼────────────────────────────────────────────────────────────────────┼────────┼──────────────┤ │ cisco_duo │ │ asset │ dashboard cisco_duo-5a0b80af-49ad-42ee-89b7-c89faa927826 is loaded │ PASS │ 1.167µs │ │ cisco_duo │ │ asset │ dashboard cisco_duo-7a135061-78a3-45d9-951b-4b9b665fa729 is loaded │ PASS │ 625ns │ │ cisco_duo │ │ asset │ dashboard cisco_duo-b386f94c-0856-4508-ba08-a525a2f3b70f is loaded │ PASS │ 417ns │ │ cisco_duo │ │ asset │ dashboard cisco_duo-c3336a66-68ff-4bcd-95ff-fb388793f721 is loaded │ PASS │ 542ns │ │ cisco_duo │ │ asset │ dashboard cisco_duo-e91470e5-2ded-4ff1-8bb5-24e06b949c1d is loaded │ PASS │ 416ns │ │ cisco_duo │ │ asset │ map cisco_duo-158c0e80-148c-11ec-9386-31989719f9db is loaded │ PASS │ 375ns │ │ cisco_duo │ admin │ asset │ index_template logs-cisco_duo.admin is loaded │ PASS │ 291ns │ │ cisco_duo │ admin │ asset │ ingest_pipeline logs-cisco_duo.admin-2.0.0 is loaded │ PASS │ 333ns │ │ cisco_duo │ auth │ asset │ index_template logs-cisco_duo.auth is loaded │ PASS │ 375ns │ │ cisco_duo │ auth │ asset │ ingest_pipeline logs-cisco_duo.auth-2.0.0 is loaded │ PASS │ 250ns │ │ cisco_duo │ offline_enrollment │ asset │ index_template logs-cisco_duo.offline_enrollment is loaded │ PASS │ 417ns │ │ cisco_duo │ offline_enrollment │ asset │ ingest_pipeline logs-cisco_duo.offline_enrollment-2.0.0 is loaded │ PASS │ 334ns │ │ cisco_duo │ summary │ asset │ index_template logs-cisco_duo.summary is loaded │ PASS │ 416ns │ │ cisco_duo │ summary │ asset │ ingest_pipeline logs-cisco_duo.summary-2.0.0 is loaded │ PASS │ 250ns │ │ cisco_duo │ telephony │ asset │ index_template logs-cisco_duo.telephony is loaded │ PASS │ 291ns │ │ cisco_duo │ telephony │ asset │ ingest_pipeline logs-cisco_duo.telephony-2.0.0 is loaded │ PASS │ 250ns │ │ cisco_duo │ telephony_v2 │ asset │ index_template logs-cisco_duo.telephony_v2 is loaded │ PASS │ 416ns │ │ cisco_duo │ telephony_v2 │ asset │ ingest_pipeline logs-cisco_duo.telephony_v2-2.0.0 is loaded │ PASS │ 250ns │ ╰───────────┴────────────────────┴───────────┴────────────────────────────────────────────────────────────────────┴────────┴──────────────╯ --- Test results for package: cisco_duo - END --- Done Run pipeline tests for the package --- Test results for package: cisco_duo - START --- ╭───────────┬────────────────────┬───────────┬────────────────────────────────────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├───────────┼────────────────────┼───────────┼────────────────────────────────────────────────────────┼────────┼──────────────┤ │ cisco_duo │ admin │ pipeline │ (ingest pipeline warnings test-admin.log) │ PASS │ 214.379ms │ │ cisco_duo │ admin │ pipeline │ (ingest pipeline warnings test-empty.log) │ PASS │ 203.127083ms │ │ cisco_duo │ admin │ pipeline │ test-admin.log │ PASS │ 168.91625ms │ │ cisco_duo │ admin │ pipeline │ test-empty.log │ PASS │ 41.405792ms │ │ cisco_duo │ auth │ pipeline │ (ingest pipeline warnings test-auth.log) │ PASS │ 213.616375ms │ │ cisco_duo │ auth │ pipeline │ (ingest pipeline warnings test-empty.log) │ PASS │ 196.946084ms │ │ cisco_duo │ auth │ pipeline │ test-auth.log │ PASS │ 351.136125ms │ │ cisco_duo │ auth │ pipeline │ test-empty.log │ PASS │ 43.347542ms │ │ cisco_duo │ offline_enrollment │ pipeline │ (ingest pipeline warnings test-empty.log) │ PASS │ 196.039ms │ │ cisco_duo │ offline_enrollment │ pipeline │ (ingest pipeline warnings test-offline-enrollment.log) │ PASS │ 222.296084ms │ │ cisco_duo │ offline_enrollment │ pipeline │ test-empty.log │ PASS │ 39.445417ms │ │ cisco_duo │ offline_enrollment │ pipeline │ test-offline-enrollment.log │ PASS │ 44.205709ms │ │ cisco_duo │ summary │ pipeline │ (ingest pipeline warnings test-summary.log) │ PASS │ 220.213792ms │ │ cisco_duo │ summary │ pipeline │ test-summary.log │ PASS │ 39.650167ms │ │ cisco_duo │ telephony │ pipeline │ (ingest pipeline warnings test-empty.log) │ PASS │ 225.43975ms │ │ cisco_duo │ telephony │ pipeline │ (ingest pipeline warnings test-telephony.log) │ PASS │ 198.661417ms │ │ cisco_duo │ telephony │ pipeline │ test-empty.log │ PASS │ 39.71475ms │ │ cisco_duo │ telephony │ pipeline │ test-telephony.log │ PASS │ 43.029333ms │ │ cisco_duo │ telephony_v2 │ pipeline │ (ingest pipeline warnings test-telephony-v2.log) │ PASS │ 198.065125ms │ │ cisco_duo │ telephony_v2 │ pipeline │ test-telephony-v2.log │ PASS │ 43.282834ms │ ╰───────────┴────────────────────┴───────────┴────────────────────────────────────────────────────────┴────────┴──────────────╯ --- Test results for package: cisco_duo - END --- Done Run static tests for the package --- Test results for package: cisco_duo - START --- ╭───────────┬────────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├───────────┼────────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤ │ cisco_duo │ admin │ static │ Verify sample_event.json │ PASS │ 46.128333ms │ │ cisco_duo │ auth │ static │ Verify sample_event.json │ PASS │ 59.536083ms │ │ cisco_duo │ offline_enrollment │ static │ Verify sample_event.json │ PASS │ 40.585333ms │ │ cisco_duo │ summary │ static │ Verify sample_event.json │ PASS │ 37.666792ms │ │ cisco_duo │ telephony │ static │ Verify sample_event.json │ PASS │ 37.640792ms │ │ cisco_duo │ telephony_v2 │ static │ Verify sample_event.json │ PASS │ 37.3455ms │ ╰───────────┴────────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯ --- Test results for package: cisco_duo - END --- Done Run system tests for the package --- Test results for package: cisco_duo - START --- ╭───────────┬────────────────────┬───────────┬───────────┬────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├───────────┼────────────────────┼───────────┼───────────┼────────┼───────────────┤ │ cisco_duo │ admin │ system │ default │ PASS │ 35.644630125s │ │ cisco_duo │ auth │ system │ default │ PASS │ 36.851069708s │ │ cisco_duo │ offline_enrollment │ system │ default │ PASS │ 32.721470209s │ │ cisco_duo │ summary │ system │ default │ PASS │ 31.743466584s │ │ cisco_duo │ telephony │ system │ default │ PASS │ 31.538269666s │ │ cisco_duo │ telephony_v2 │ system │ default │ PASS │ 35.694825375s │ ╰───────────┴────────────────────┴───────────┴───────────┴────────┴───────────────╯ --- Test results for package: cisco_duo - END --- Done

Related issues

Screenshots

Integration page and configuration

Dashboards

elasticmachine · 2024-09-20T16:08:22Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2024-09-22T13:43:31Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/cisco_duo/_dev/build/docs/README.md

efd6 · 2024-09-23T04:59:31Z

packages/cisco_duo/data_stream/auth/agent/stream/cel.yml.hbs

+	state.next_url
+	:
+	state.url.trim_right("/") + "/admin/v2/logs/authentication?" + {
+ "limit": [string(int(state.limit))],


Is the int conversion due to float rendering?

I think so... for some reason, it was formatting these variables as float so I was getting the following error:

failed eval: ERROR: <input>:51:58: no such overload for format_query: type conversion error from Double to 'string'

efd6 · 2024-09-23T05:03:13Z

packages/cisco_duo/data_stream/auth/agent/stream/cel.yml.hbs

+ ).base64()],
+	},


The formatting here looks a little weird; oh, there are mixed spaces and tabs.

Yes, rendering the program with celfmt adds tabs, so probably I have mixed spaces and tabs in further modifications. I will leave spaces for all the code.

efd6 · 2024-09-23T05:10:25Z

packages/cisco_duo/data_stream/auth/agent/stream/cel.yml.hbs

+	"error": {
+	"code": has(body.code) ? string(body.code) : string(resp.StatusCode),
+	"id": string(resp.Status),
+	"message": "GET: " +


Suggested change

"message": "GET: " +

"message": "GET:" +

Don't you think the message looks better with an space in it? an example:

"error": { "code": "40102", "id": "401 Unauthorized", "message": "GET: Invalid identity in request credentials" }

packages/cisco_duo/data_stream/auth/sample_event.json

packages/cisco_duo/data_stream/telephony_v2/_dev/test/system/test-default-config.yml

efd6 · 2024-09-23T05:16:42Z

packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs

+	"error": {
+	"code": has(body.code) ? string(body.code) : string(resp.StatusCode),
+	"id": string(resp.Status),
+	"message": "GET: " +


Suggested change

"message": "GET: " +

"message": "GET:" +

packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs

efd6

Needs an elastic-package build. Otherwise LGTM

elastic-sonarqube · 2024-10-01T08:10:02Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-10-01T08:10:04Z

💚 Build Succeeded

Buildkite Build
Commit: cab6733

History

💔 Build #16606 failed 56a91e7
💚 Build #16262 succeeded fe8c405
💔 Build #16243 failed ba88c6c8c5a1fdd14c6e8e0055e092bb9324fbf9
💔 Build #16240 failed c64c772

cc @chemamartinez

elastic-vault-github-plugin-prod · 2024-10-01T08:35:44Z

Package cisco_duo - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=cisco_duo

Added new data stream telephony_v2 to support the new v2 API endpoint. Updated the auth data stream to migrate from the HTTPJSON to CEL. Review documentation and dashboards

chemamartinez added 6 commits September 20, 2024 17:46

Add telephony_v2 data stream

5fc8a49

Mark telephony v1 as legacy

9660e8e

Migrate from HTTPJSON to CEL for auth data stream

36f1132

Update documentation

dd70e19

Update dashboards

06f9d05

Update screenshots

146186a

chemamartinez added enhancement New feature or request Integration:cisco_duo Cisco Duo Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Sep 20, 2024

chemamartinez self-assigned this Sep 20, 2024

andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Sep 20, 2024

Update changelog

c64c772

chemamartinez marked this pull request as ready for review September 20, 2024 16:08

chemamartinez requested a review from a team as a code owner September 20, 2024 16:08

Fix version constraint for dashboard

fe8c405

chemamartinez force-pushed the cisco_duo-v2-support branch from ba88c6c to fe8c405 Compare September 22, 2024 13:17

efd6 reviewed Sep 23, 2024

View reviewed changes

Requested changes

56a91e7

chemamartinez requested a review from efd6 September 30, 2024 16:22

efd6 approved these changes Oct 1, 2024

View reviewed changes

Update docs

cab6733

chemamartinez merged commit b0c20d8 into elastic:main Oct 1, 2024
3 checks passed

chemamartinez deleted the cisco_duo-v2-support branch February 6, 2025 10:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Cisco Duo] Integration updates #11200

[Cisco Duo] Integration updates #11200

Uh oh!

chemamartinez commented Sep 20, 2024

elasticmachine commented Sep 20, 2024

elastic-vault-github-plugin-prod bot commented Sep 22, 2024

Uh oh!

Uh oh!

Uh oh!

efd6 Sep 23, 2024

chemamartinez Sep 30, 2024

efd6 Sep 23, 2024

chemamartinez Sep 30, 2024

efd6 Sep 23, 2024

chemamartinez Sep 30, 2024

Uh oh!

Uh oh!

efd6 Sep 23, 2024

Uh oh!

efd6 left a comment

elastic-sonarqube bot commented Oct 1, 2024

elasticmachine commented Oct 1, 2024

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 1, 2024

Labels

4 participants

[Cisco Duo] Integration updates #11200

[Cisco Duo] Integration updates #11200

Uh oh!

Conversation

chemamartinez commented Sep 20, 2024

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Sep 20, 2024

elastic-vault-github-plugin-prod bot commented Sep 22, 2024

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

efd6 Sep 23, 2024

Choose a reason for hiding this comment

chemamartinez Sep 30, 2024

Choose a reason for hiding this comment

efd6 Sep 23, 2024

Choose a reason for hiding this comment

chemamartinez Sep 30, 2024

Choose a reason for hiding this comment

efd6 Sep 23, 2024

Choose a reason for hiding this comment

chemamartinez Sep 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efd6 Sep 23, 2024

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

elastic-sonarqube bot commented Oct 1, 2024

Quality Gate passed

elasticmachine commented Oct 1, 2024

💚 Build Succeeded

History

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 1, 2024

Labels

4 participants