Keeper Security - New Elastic Integration Package #15089
Conversation
…anifest, changelog, license, and documentation files. This integration allows for direct collection of audit events via Elasticsearch Bulk API without the need for agents.
…changelog, license, manifest, and data stream configurations. This cleanup eliminates unnecessary files for the agentless integration with Elasticsearch.
…ne. Removed commented-out code and unnecessary properties for clarity and maintainability.
| 💚 CLA has been signed |
andrewkroh added New Integration 
jamiehynds added the Team:Security-Service Integrations 
| Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
| @jpkeepersecurity Please sign the CLA with all the email addresses used as author email in the commits in this PR. |
| I just signed @efd6 |
andrewkroh removed the needs CLA 
efd6 left a comment
efd6 left a comment There was a problem hiding this comment.
Please add pipeline and system tests.
There was a problem hiding this comment.
This should be done in _dev/build/docs/README.md and generated to here.
Please also take a look at the guidelines here and make the documentation match that structure.
packages/keeper_security_siem_integration/data_stream/audit/fields/base-fields.yml Outdated Show resolved Hide resolved
...keeper_security_siem_integration/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved
| # If no keeper.timestamp, try to use existing @timestamp or set current time | ||
| - set: | ||
| field: '@timestamp' | ||
| value: "{{_ingest.timestamp}}" |
There was a problem hiding this comment.
| value: "{{_ingest.timestamp}}" | |
| value: "{{{_ingest.timestamp}}}" |
use triple stache throughout
Though this processor does not need to be included since the @timestamp will be set by fleet if it is not present, and including this here means that the product of the pipeline tests will not be resproducible.
There was a problem hiding this comment.
With set you can use copy_from instead of the Mustache templates.
...keeper_security_siem_integration/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
| /test |
efd6 left a comment
efd6 left a comment There was a problem hiding this comment.
Please make the following change:
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index e54e89dd94..0e842c8e83 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -290,6 +290,7 @@ /packages/juniper_srx @elastic/integration-experience /packages/kafka @elastic/obs-infraobs-integrations /packages/kafka_log @elastic/obs-infraobs-integrations +/packages/keeper_security_siem_integration @elastic/security-service-integrations /packages/keycloak @elastic/security-service-integrations /packages/kibana @elastic/stack-monitoring /packages/kubernetes @elastic/obs-ds-hosted-services
| # If no keeper.timestamp, try to use existing @timestamp or set current time | ||
| - set: | ||
| field: '@timestamp' | ||
| value: "{{_ingest.timestamp}}" |
There was a problem hiding this comment.
With set you can use copy_from instead of the Mustache templates.
| value: "{{username}}" | ||
| if: ctx.username != null | ||
| | ||
| # If username looks like email, set user.email |
There was a problem hiding this comment.
These comments can go into the description field. Then they become part of the pipeline specification and are visible in other places that you can access the pipeline definition, like the Ingest Pipeline editor/viewer.
| type: date | ||
| description: Date/time when the audit event occurred | ||
| - name: message | ||
| type: wildcard |
There was a problem hiding this comment.
Fields that are defined in ECS should use the ECS definition.
| dynamic_dataset: true | ||
| dynamic_namespace: true |
There was a problem hiding this comment.
Why are these necessary? The pipeline sets static values for both dataset and namespace. And these are primarily used to give the Elastic Agent permissions to write to a wide swath of logs-*-*, but this integration advertises itself as not needing an Elastic Agent.
| dependencies: | ||
| ecs: | ||
| reference: git@v8.17.0 | ||
| import_mappings: true |
There was a problem hiding this comment.
This shouldn't be needed any more. All integrations receive the ecs@mappings template which provides similar functionality as import_mappings: true.
| @@ -0,0 +1,4 @@ | |||
| dependencies: | |||
| ecs: | |||
| reference: git@v8.17.0 | |||
There was a problem hiding this comment.
There are newer ECS versions. Can we use the latest version?
…asticsearch/ingest_pipeline/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
- Removed unnecessary Elasticsearch dynamic dataset settings from the audit manifest. - Cleaned up the ingest pipeline by adding descriptions for various processing steps and removing redundant comments. - Ensured clarity in the pipeline by maintaining essential functionality while enhancing documentation for future reference.
- Removed deprecated fields and descriptions to streamline the configuration. - Added ECS external references for data stream and event fields to enhance compatibility with Elastic Common Schema.
…and update keeper-dashboard image
...keeper_security_siem_integration/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved
...curity_siem_integration/data_stream/audit/_dev/test/pipeline/test-edge-cases.json-config.yml Outdated Show resolved Hide resolved
...siem_integration/data_stream/audit/_dev/test/pipeline/test-timestamp-formats.json-config.yml Outdated Show resolved Hide resolved
...keeper_security_siem_integration/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved
...keeper_security_siem_integration/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved
...r_security_siem_integration/data_stream/audit/_dev/test/pipeline/test-sample.json-config.yml Outdated Show resolved Hide resolved
…asticsearch/ingest_pipeline/default.yml Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
…ions - Changed the version in manifest.yml from 1.0.0 to 0.1.0. - Removed unnecessary configuration files for test pipelines. - Updated ECS version in expected JSON files and ingest pipeline configuration to 8.17.0. - Simplified test-edge-cases.json by removing redundant user data.
…peline - Deleted unused test files: test-edge-cases.json, test-timestamp-formats.json, and their expected counterparts. - Updated the ingest pipeline to handle timestamp conversion directly from the 'timestamp' field instead of 'keeper.timestamp'. - Simplified the fields definition by removing the 'keeper' group and directly defining 'timestamp' and 'category' fields.
…tions - Updated expected JSON files to include additional event fields such as category, dataset, kind, outcome, and type for various actions. - Modified ingest pipeline conditions to reference the updated category field directly instead of the obsolete keeper.category.
…urrent-changes
| pr is up to date. |
packages/keeper_security_siem_integration/data_stream/audit/_dev/test/pipeline/test-sample.json Outdated Show resolved Hide resolved
...keeper_security_siem_integration/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved
...keeper_security_siem_integration/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved
| /test |
- Introduced a comprehensive README for the Keeper Security integration detailing its architecture, compatibility, data collection, deployment instructions, and troubleshooting steps. - Updated existing README to reflect the new structure and added clarity on integration setup and use cases. - Included information on API usage, ingest pipeline details, and performance considerations. - Added a new test sample JSON file to enhance testing capabilities.
- Removed obsolete fields 'dataset' and 'module' from the expected JSON files to streamline event structure. - Updated ingest pipeline to utilize 'copy_from' for setting user fields and source IP, enhancing clarity and reducing redundancy. - Adjusted error handling messages for timestamp parsing to improve accuracy.
- Introduced a new sample_event.json file to represent the audit event for setting two-factor authentication. - Updated expected JSON structure in test-sample.json-expected.json to include 'dataset' and 'module' fields for consistency with the new event format.
- Changed the audit event from 'set_two_factor_on' to 'change_master_password'. - Updated timestamp, enterprise ID, user details, and remote address to reflect new event context. - Enhanced source information with geographical data for improved event tracking.
| /test |
- Updated Kibana version compatibility in manifest.yml to support versions 8.19.0 and 9.1.0. - Removed obsolete 'data_stream' and 'module' fields from test-sample.json-expected.json to streamline the event structure. - Refactored ingest pipeline by removing unnecessary data stream fields to enhance clarity and reduce redundancy.
| Currently, for agentless integrations of the push type, implementing system tests is not required, since during deployment elastic-package looks for elastic-agent environment variables by default. However, given that there is now a new agentless push integration, I suggest updating elastic-package to support system tests for this type of solution. Once the tool provides this support, the integration can then include system tests as part of its setup. |
| /test |
💚 Build Succeeded
History
|
|
| Package keeper - 0.1.0 containing this change is available at https://epr.elastic.co/package/keeper/0.1.0/ |
andrewkroh added the Integration:keeper_security_siem_integration 5 participants
Proposed commit message
Add Keeper Security agentless integration package
This PR introduces a new Keeper Security integration that enables truly agentless audit event collection by allowing Keeper to push data directly to Elasticsearch via the Bulk API without requiring any Elastic Agent installation.
WHAT:
keeper_security_siem_integrationwith complete ECS field mappingWHY:
The integration processes Keeper audit events (authentication, security actions, user management) and enriches them with geographic location data, user correlation fields, and standardized ECS categorization.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Install the integration assets in Kibana:
Create API key for data ingestion:
POST /_security/api_key { "name": "keeper-integration-test", "expiration": "365d", "role_descriptors": { "keeper-writer": { "cluster": ["monitor"], "indices": [ { "names": ["logs-keeper.audit-*"], "privileges": ["auto_configure", "create_doc"] } ] } } }Enable GeoIP enrichment:
PUT /_cluster/settings { "persistent": { "ingest.geoip.downloader.enabled": true } }Test data ingestion:
Verify in Kibana:
logs-keeper.audit-*Related issues
Screenshots
The integration includes a pre-built dashboard showing: