qualys_vmdr: Add "show_igs" and fingerprint processor UI options #12428

kcreddy · 2025-01-22T17:20:32Z

Proposed commit message

Add show_igs API parameter as a UI option. Also optionally add fingerprint processor. By default, Qualys API only includes detection records with `Confirmed` and `Potential` vulnerabilities. With show_igs option users can enable Qualys API to fetch detections of type `Info` as well. Optionally adding fingerprint processor on event.original field so that any existing vulnerabilities in the current backing index will not be duplicated. I prefixed the document _id value with the timestamp based on recommendations from the "Efficient Duplicate Prevention for Event-Based Data in Elasticsearch" blog post. References - https://qualysguard.qg2.apps.qualys.com/qwebhelp/fo_portal/api_doc/assets/index.htm#t=host_lists%2Fhost_detection.htm - https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Updated system tests with show_igs=1 should run successfully.
Run: cd packages/qualys_vmdr && elastic-package stack down && elastic-package build && elastic-package stack up --version=8.13.0 -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v --data-streams=asset_host_detection
Returns:

2025/01/22 22:22:53 DEBUG found 0 hits in logs-qualys_vmdr.asset_host_detection-71011 data stream 2025/01/22 22:22:54 DEBUG found 4 hits in logs-qualys_vmdr.asset_host_detection-71011 data stream 2025/01/22 22:22:58 DEBUG found 4 hits in logs-qualys_vmdr.asset_host_detection-71011 data stream ...... 2025/01/22 22:23:01 DEBUG assert hit count expected 4, observed 4 ...... --- Test results for package: qualys_vmdr - START --- ╭─────────────┬──────────────────────┬───────────┬───────────┬────────┬───────────────╮ │ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │ ├─────────────┼──────────────────────┼───────────┼───────────┼────────┼───────────────┤ │ qualys_vmdr │ asset_host_detection │ system │ default │ PASS │ 42.686733958s │ ╰─────────────┴──────────────────────┴───────────┴───────────┴────────┴───────────────╯ --- Test results for package: qualys_vmdr - END --- Done

Check tracer logs:
a. Don't select Enable Information Gathered Detections UI option.
Inside tracer logs, the request has:
```
"url.query":"action=list&ids=879484319&show_igs=0&truncation_limit=1000" 
```
b. Select Enable Information Gathered Detections UI option.
Inside tracer logs, the request has:
```
"url.query":"action=list&ids=879484319&show_igs=1&truncation_limit=1000" 
```

Screenshots

elasticmachine · 2025-01-22T17:21:34Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy · 2025-01-22T17:23:11Z

packages/qualys_vmdr/_dev/deploy/docker/files/config.yml

produced by elastic-package

elastic-vault-github-plugin-prod · 2025-01-22T17:40:52Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

andrewkroh · 2025-01-22T17:32:11Z

packages/qualys_vmdr/data_stream/asset_host_detection/manifest.yml

Suggested change

description: By default, Qualys API only includes detection records with `Confirmed` and `Potential` vulnerabilities. When this option is enabled, the Qualys API also sends detections of type `Info` along with `Confirmed` and `Potential` vulnerabilities.

description: By default, Qualys API only includes detection records with `Confirmed` and `Potential` vulnerabilities. When this option is enabled, the Qualys API sends detections of type `Info` in addition to `Confirmed` and `Potential` vulnerabilities.

Updated as suggested in 1f0ea18

andrewkroh · 2025-01-22T19:26:26Z

packages/qualys_vmdr/data_stream/asset_host_detection/elasticsearch/ingest_pipeline/default.yml

Deduplication paired with full syncs each period could make it more difficult to validate all assets and vulnerabilities were collected. Currently the validation process involves reviewing data written during the last sync window. If deduplication is implemented then you need to consider all data in the current backing index to account for vulnerabilities that were indexed during an earlier sync.

Users need to know that deduplication is enabled because it impacts how you should query the data for certain uses. My recommendation is to make deduplication an opt-in feature with an explanation of how it impacts ingestion. Specifically, each sync interval will ingest all detected vulnerabilities. However, if deduplication is enabled, any existing vulnerabilities in the current backing index will not be duplicated. The presence of a vulnerability in the backing index is determined by computing a checksum of the event contents and checking if that checksum already exists in Elasticsearch.

That makes sense.
I added UI option to select Enable Data Deduplication (default: false) in 1f0ea18

Described the option in manifest.yml as suggested. Updated PR description.

andrewkroh · 2025-01-23T17:28:06Z

packages/qualys_vmdr/data_stream/asset_host_detection/agent/stream/input.yml.hbs

Why make this into a string rather than retain the boolean type?

Not intentional. Updated in 6c46311

elasticmachine · 2025-01-24T05:36:09Z

💚 Build Succeeded

Buildkite Build
Commit: 6c46311800ce0890008011bc2a93b277bb41bc63

History

💚 Build #20834 succeeded 1f0ea18da4595c102a5079dc06b9c19c353c60ce
💚 Build #20812 succeeded 856c230d2b3730ff67c309135993970870252a90

cc @kcreddy

elastic-sonarqube · 2025-01-24T05:36:18Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2025-01-28T03:57:57Z

Package qualys_vmdr - 5.9.0 containing this change is available at https://epr.elastic.co/package/qualys_vmdr/5.9.0/

…stic#12428) Add show_igs API parameter as a UI option. Also optionally add fingerprint processor. By default, Qualys API only includes detection records with `Confirmed` and `Potential` vulnerabilities. With show_igs option users can enable Qualys API to fetch detections of type `Info` as well. Optionally adding fingerprint processor on event.original field so that any existing vulnerabilities in the current backing index will not be duplicated. I prefixed the document _id value with the timestamp based on recommendations from the "Efficient Duplicate Prevention for Event-Based Data in Elasticsearch" blog post. References - https://qualysguard.qg2.apps.qualys.com/qwebhelp/fo_portal/api_doc/assets/index.htm#t=host_lists%2Fhost_detection.htm - https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch.

kcreddy marked this pull request as ready for review January 22, 2025 17:21

kcreddy requested a review from a team as a code owner January 22, 2025 17:21

kcreddy self-assigned this Jan 22, 2025

kcreddy added Integration:qualys_vmdr Qualys VMDR enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jan 22, 2025

kcreddy commented Jan 22, 2025

View reviewed changes

packages/qualys_vmdr/_dev/deploy/docker/files/config.yml Outdated

Copy link

Contributor Author

kcreddy Jan 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

produced by elastic-package

andrewkroh reviewed Jan 22, 2025

View reviewed changes

kcreddy requested a review from andrewkroh January 23, 2025 07:22

kcreddy changed the title ~~qualys_vmdr: Add "show_igs" UI option and fingerprint processor~~ qualys_vmdr: Add "show_igs" and fingerprint processor UI options Jan 23, 2025

andrewkroh reviewed Jan 23, 2025

View reviewed changes

kcreddy requested a review from andrewkroh January 24, 2025 05:16

kcreddy mentioned this pull request Jan 27, 2025

qualys_vmdr: Add mappings to improve support for Cloud Detection and Response (CDR) workflow #11999

Merged

5 tasks

andrewkroh approved these changes Jan 27, 2025

View reviewed changes

kcreddy merged this pull request into elastic:main Jan 28, 2025
5 checks passed

kcreddy deleted the qualys-quality-impr branch February 7, 2025 09:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

qualys_vmdr: Add "show_igs" and fingerprint processor UI options #12428

qualys_vmdr: Add "show_igs" and fingerprint processor UI options #12428

Uh oh!

kcreddy commented Jan 22, 2025 •

edited

Loading

elasticmachine commented Jan 22, 2025

kcreddy Jan 22, 2025

elastic-vault-github-plugin-prod bot commented Jan 22, 2025

andrewkroh Jan 22, 2025

kcreddy Jan 23, 2025 •

edited

Loading

andrewkroh Jan 22, 2025

kcreddy Jan 23, 2025 •

edited

Loading

kcreddy Jan 23, 2025

andrewkroh Jan 23, 2025

kcreddy Jan 24, 2025

elasticmachine commented Jan 24, 2025

elastic-sonarqube bot commented Jan 24, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Jan 28, 2025

Labels

3 participants

	description: By default, Qualys API only includes detection records with `Confirmed` and `Potential` vulnerabilities. When this option is enabled, the Qualys API also sends detections of type `Info` along with `Confirmed` and `Potential` vulnerabilities.
	description: By default, Qualys API only includes detection records with `Confirmed` and `Potential` vulnerabilities. When this option is enabled, the Qualys API sends detections of type `Info` in addition to `Confirmed` and `Potential` vulnerabilities.

qualys_vmdr: Add "show_igs" and fingerprint processor UI options #12428

qualys_vmdr: Add "show_igs" and fingerprint processor UI options #12428

Uh oh!

Conversation

kcreddy commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Screenshots

elasticmachine commented Jan 22, 2025

kcreddy Jan 22, 2025

Choose a reason for hiding this comment

elastic-vault-github-plugin-prod bot commented Jan 22, 2025

🚀 Benchmarks report

andrewkroh Jan 22, 2025

Choose a reason for hiding this comment

kcreddy Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

andrewkroh Jan 22, 2025

Choose a reason for hiding this comment

kcreddy Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

kcreddy Jan 23, 2025

Choose a reason for hiding this comment

andrewkroh Jan 23, 2025

Choose a reason for hiding this comment

kcreddy Jan 24, 2025

Choose a reason for hiding this comment

elasticmachine commented Jan 24, 2025

💚 Build Succeeded

History

elastic-sonarqube bot commented Jan 24, 2025

Quality Gate passed

Uh oh!

elastic-vault-github-plugin-prod bot commented Jan 28, 2025

Labels

3 participants

kcreddy commented Jan 22, 2025 •

edited

Loading

kcreddy Jan 23, 2025 •

edited

Loading

kcreddy Jan 23, 2025 •

edited

Loading