[abnormal_security] Initial release of the Abnormal Security #10653

brijesh-elastic · 2024-07-30T09:55:59Z

Proposed commit message

Create New integration package abnormal_security.

Added ai_security_mailbox, audit, case and threat data stream.
Added data collection logic for all the data stream.
Added the ingest pipeline for all the data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for all the data stream.
Added system test cases for all the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/abnormal_security directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: abnormal_security - START --- ╭───────────────────┬─────────────────────┬───────────┬─────────────────────────────────────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├───────────────────┼─────────────────────┼───────────┼─────────────────────────────────────────────────────────┼────────┼──────────────┤ │ abnormal_security │ ai_security_mailbox │ pipeline │ (ingest pipeline warnings test-ai-security-mailbox.log) │ PASS │ 532.894348ms │ │ abnormal_security │ ai_security_mailbox │ pipeline │ test-ai-security-mailbox.log │ PASS │ 250.257522ms │ │ abnormal_security │ audit │ pipeline │ (ingest pipeline warnings test-audit.log) │ PASS │ 445.051287ms │ │ abnormal_security │ audit │ pipeline │ test-audit.log │ PASS │ 335.424619ms │ │ abnormal_security │ case │ pipeline │ (ingest pipeline warnings test-case.log) │ PASS │ 390.989185ms │ │ abnormal_security │ case │ pipeline │ test-case.log │ PASS │ 168.100646ms │ │ abnormal_security │ threat │ pipeline │ (ingest pipeline warnings test-threat.log) │ PASS │ 355.249192ms │ │ abnormal_security │ threat │ pipeline │ test-threat.log │ PASS │ 386.175903ms │ ╰───────────────────┴─────────────────────┴───────────┴─────────────────────────────────────────────────────────┴────────┴──────────────╯ --- Test results for package: abnormal_security - END --- Done --- Test results for package: abnormal_security - START --- ╭───────────────────┬─────────────────────┬───────────┬───────────┬────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├───────────────────┼─────────────────────┼───────────┼───────────┼────────┼───────────────┤ │ abnormal_security │ ai_security_mailbox │ system │ default │ PASS │ 37.814874684s │ │ abnormal_security │ audit │ system │ default │ PASS │ 36.100990035s │ │ abnormal_security │ case │ system │ default │ PASS │ 37.140948377s │ │ abnormal_security │ threat │ system │ default │ PASS │ 35.386249528s │ ╰───────────────────┴─────────────────────┴───────────┴───────────┴────────┴───────────────╯ --- Test results for package: abnormal_security - END --- Done --- Test results for package: abnormal_security - START --- ╭───────────────────┬─────────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├───────────────────┼─────────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤ │ abnormal_security │ ai_security_mailbox │ static │ Verify sample_event.json │ PASS │ 128.51044ms │ │ abnormal_security │ audit │ static │ Verify sample_event.json │ PASS │ 139.387012ms │ │ abnormal_security │ case │ static │ Verify sample_event.json │ PASS │ 128.938014ms │ │ abnormal_security │ threat │ static │ Verify sample_event.json │ PASS │ 164.084638ms │ ╰───────────────────┴─────────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯ --- Test results for package: abnormal_security - END --- Done --- Test results for package: abnormal_security - START --- ╭───────────────────┬─────────────────────┬───────────┬────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├───────────────────┼─────────────────────┼───────────┼────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤ │ abnormal_security │ │ asset │ dashboard abnormal_security-37ed5d19-c753-43a0-b0a2-f8e6437ddfe5 is loaded │ PASS │ 3.37µs │ │ abnormal_security │ │ asset │ dashboard abnormal_security-6a8e53ac-7759-4564-bcd6-03c6a9792eac is loaded │ PASS │ 459ns │ │ abnormal_security │ │ asset │ dashboard abnormal_security-a4364503-ada3-4fe6-a054-d152accf207c is loaded │ PASS │ 327ns │ │ abnormal_security │ │ asset │ dashboard abnormal_security-f6562262-e429-470d-af45-4c80afdcf664 is loaded │ PASS │ 254ns │ │ abnormal_security │ │ asset │ search abnormal_security-8416d710-f7d2-4e62-b6d2-fe3a0b656998 is loaded │ PASS │ 268ns │ │ abnormal_security │ │ asset │ search abnormal_security-ce59df21-762c-4f05-911e-fd9f5ff67e4b is loaded │ PASS │ 283ns │ │ abnormal_security │ │ asset │ search abnormal_security-d2482dd8-c5fa-4f7c-9d5c-8b3f34481a90 is loaded │ PASS │ 292ns │ │ abnormal_security │ │ asset │ search abnormal_security-e34b2986-68c2-4de9-8601-7bdefab429bc is loaded │ PASS │ 318ns │ │ abnormal_security │ ai_security_mailbox │ asset │ index_template logs-abnormal_security.ai_security_mailbox is loaded │ PASS │ 680ns │ │ abnormal_security │ ai_security_mailbox │ asset │ ingest_pipeline logs-abnormal_security.ai_security_mailbox-0.1.0 is loaded │ PASS │ 263ns │ │ abnormal_security │ audit │ asset │ index_template logs-abnormal_security.audit is loaded │ PASS │ 369ns │ │ abnormal_security │ audit │ asset │ ingest_pipeline logs-abnormal_security.audit-0.1.0 is loaded │ PASS │ 203ns │ │ abnormal_security │ case │ asset │ index_template logs-abnormal_security.case is loaded │ PASS │ 249ns │ │ abnormal_security │ case │ asset │ ingest_pipeline logs-abnormal_security.case-0.1.0 is loaded │ PASS │ 178ns │ │ abnormal_security │ threat │ asset │ index_template logs-abnormal_security.threat is loaded │ PASS │ 244ns │ │ abnormal_security │ threat │ asset │ ingest_pipeline logs-abnormal_security.threat-0.1.0 is loaded │ PASS │ 172ns │ ╰───────────────────┴─────────────────────┴───────────┴────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯ --- Test results for package: abnormal_security - END --- Done

Related issues

Closes [New Integration] Abnormal Security #10424

Screenshots

elasticmachine · 2024-07-30T10:09:50Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elasticmachine · 2024-07-30T10:19:45Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/abnormal_security/_dev/deploy/docker/files/config.yml

packages/abnormal_security/data_stream/ai_security_mailbox/agent/stream/cel.yml.hbs

packages/abnormal_security/data_stream/ai_security_mailbox/fields/base-fields.yml

packages/abnormal_security/data_stream/ai_security_mailbox/fields/beats.yml

packages/abnormal_security/data_stream/case/fields/beats.yml

efd6 · 2024-08-01T02:18:51Z

packages/abnormal_security/data_stream/threat/_dev/test/pipeline/test-threat.log-expected.json

+ "https://example.com",
+ "https://example.com",
+ "https://example.com",
+ "https://example.com",
+ "https://example.com"


I'm guessing that these would not all be duplicates in real data. Is that correct?

Yes, it will be unique in real data.

I don't think this should be an array. https://www.elastic.co/guide/en/ecs/current/ecs-url.html#field-url-original.
Better not copy array urls into url.original

packages/abnormal_security/data_stream/threat/agent/stream/cel.yml.hbs

packages/abnormal_security/data_stream/threat/fields/base-fields.yml

packages/abnormal_security/data_stream/threat/fields/beats.yml

packages/abnormal_security/_dev/build/docs/README.md

packages/abnormal_security/data_stream/threat/_dev/test/pipeline/test-threat.log-expected.json

kcreddy · 2024-08-02T07:32:39Z

packages/abnormal_security/data_stream/threat/_dev/test/pipeline/test-threat.log-expected.json

+ "https://example.com",
+ "https://example.com",
+ "https://example.com",
+ "https://example.com",
+ "https://example.com"


I don't think this should be an array. https://www.elastic.co/guide/en/ecs/current/ecs-url.html#field-url-original.
Better not copy array urls into url.original

kcreddy · 2024-08-02T09:42:13Z

packages/abnormal_security/data_stream/audit/sample_event.json

+ "abnormal_security-audit"
+ ],
+ "url": {
+ "extension": "1/messages/email_content/",


Can you apply similar change to this one: #9623 for all datastream's pipeline test config which use uri_parts?

The pipeline test config temporarily fixes issue with url.extension having flaky tests in versions 8.14+.

...ata_stream/ai_security_mailbox/_dev/test/pipeline/test-ai-security-mailbox.log-expected.json

kcreddy

LGTM 👍🏼
Can be merged after @efd6 approval. Thanks!

efd6

There is one outstanding issue.

efd6 · 2024-08-08T22:34:03Z

packages/abnormal_security/data_stream/audit/agent/stream/cel.yml.hbs

+ "code": string(resp.StatusCode),
+ "id": string(resp.Status),
+ "message": "GET:"+(
+ size(resp.Body) != 0 ?
+ string(resp.Body)
+ :
+ string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
+ ),


This is not done.

packages/abnormal_security/data_stream/ai_security_mailbox/agent/stream/cel.yml.hbs

efd6

Thanks

elastic-sonarqube · 2024-08-09T07:31:36Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-08-09T07:31:37Z

💚 Build Succeeded

Buildkite Build
Commit: bbd36c3

History

💚 Build #14459 succeeded 9e9e159
💚 Build #14356 succeeded ea29f81
💚 Build #14140 succeeded 33f8bfc

elasticmachine · 2024-08-09T09:20:54Z

Package abnormal_security - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=abnormal_security

…#10653) * Added ai_security_mailbox, audit, case and threat data stream. * Added data collection logic for all the data stream. * Added the ingest pipeline for all the data stream. * Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files. * Added dashboards and visualizations. * Added test for pipeline for all the data stream. * Added system test cases for all the data stream.

brijesh-elastic added 2 commits July 30, 2024 15:16

Initial release

bb0c4e0

Update changelog entry

33f8bfc

kcreddy added New Integration Issue or pull request for creating a new integration package. Crest Contributions from Crest developement team. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jul 30, 2024

jamiehynds requested a review from a team July 31, 2024 20:25

efd6 reviewed Aug 1, 2024

View reviewed changes

kcreddy reviewed Aug 2, 2024

View reviewed changes

Resolve comments

ea29f81

brijesh-elastic requested review from efd6 and kcreddy August 6, 2024 14:10

kcreddy reviewed Aug 8, 2024

View reviewed changes

Resolve comments

9e9e159

brijesh-elastic requested a review from kcreddy August 8, 2024 07:05

kcreddy approved these changes Aug 8, 2024

View reviewed changes

efd6 reviewed Aug 8, 2024

View reviewed changes

Fix indent

bbd36c3

efd6 approved these changes Aug 9, 2024

View reviewed changes

efd6 merged commit d4f81cc into elastic:main Aug 9, 2024

andrewkroh added the Integration:abnormal_security Abnormal AI label Aug 12, 2024

[abnormal_security] Initial release of the Abnormal Security #10653

[abnormal_security] Initial release of the Abnormal Security #10653

Uh oh!

Conversation

brijesh-elastic commented Jul 30, 2024

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jul 30, 2024

elasticmachine commented Jul 30, 2024

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Aug 1, 2024

Choose a reason for hiding this comment

brijesh-elastic Aug 1, 2024

Choose a reason for hiding this comment

kcreddy Aug 2, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy Aug 2, 2024

Choose a reason for hiding this comment

kcreddy Aug 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

efd6 Aug 8, 2024

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

elastic-sonarqube bot commented Aug 9, 2024

Quality Gate passed

elasticmachine commented Aug 9, 2024

💚 Build Succeeded

History

elasticmachine commented Aug 9, 2024

Labels

5 participants

kcreddy Aug 2, 2024 •

edited

Loading