Tenable_sc: Update API lastseen parameter format #12114
Conversation
packages/tenable_sc/changelog.yml Outdated
There was a problem hiding this comment.
Incremented major version because batch_size, interval, and initial_interval are moved from package level to data-stream level.
There was a problem hiding this comment.
No longer relevant as the configs are moved back to package level.
kcreddy added Integration:tenable_sc 
| Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
There was a problem hiding this comment.
This doesn't look like it takes units into account. The API expects this to be in days, but we accept h/m/s (and others). Should we say the value is given in days or calculate days from the duration?
There was a problem hiding this comment.
The intention was to change the initial_interval format. I updated it to default of 1 from existing 24h. I also updated the description to mention it should be an integer in days.
There was a problem hiding this comment.
Ah, I missed that because of the change in the package manifest. I think this is likely to cause confusion.
There was a problem hiding this comment.
I think it would be nice to be able to use the same syntax here, to avoid confusion and the conversion from a time.Duration to a day count is now a solved problem, so it should be easy to do.
There was a problem hiding this comment.
Updated in 4a4f1ea. Also moved batch_size, interval, and initial_interval back to package level. Removing the need for breaking change.
There was a problem hiding this comment.
Tested with 2 scenarios:
-
Initial Interval: 48h:- First request:
{\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:2\"}. This returns last event'slastSeenas1734500477. - Second request (after interval):
{\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:1\"}
- First request:
-
Initial Interval: 4h:- First request:
{\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:1\"}. This returns last event'slastSeenas1734500477. - Second request (after interval):
{\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:1\"}
- First request:
There was a problem hiding this comment.
I think this should have an [[else]], probably reaching back initial_interval.
I'm not sure this calculation is right. What we have is equivalent to (now.Unix - .cursor.last_event_ts)/86400. This will truncate the value meaning that we will potentially lose data; if the cursor value indicates a non-integer day, the time interval calculated in the numerator will be the fraction less than a full day and the remainder of that day will be truncated by the division.
Also, is .cursor.last_event_ts a type that results in epoch seconds when converted to an integer? From the docs, and the code below this does not look like a time value. It looks like it is very hard to hold this API correctly.
There was a problem hiding this comment.
I think this should have an [[else]], probably reaching back initial_interval
We have the default block, but I also think it would make sense to avoid template error. I will add the [[else]] case.
I'm not sure this calculation is right.
Also, is .cursor.last_event_ts a type that results in epoch seconds when converted to an integer?
Based on the samples ingested, the value of .cursor.last_event_ts which is derived from .last_event.lastSeen is always in epoch seconds. I verified that lastSeen in the response of each object is epoch seconds. I am not sure if this can make cursor value a non-integer day.
I will remove the int conversion of .cursor.last_event_ts as it is not required.
There was a problem hiding this comment.
Based on the samples ingested, the value of
.cursor.last_event_tswhich is derived from.last_event.lastSeenis always in epoch seconds.
Is the documentation for this?
I am not sure if this can make cursor value a non-integer day.
If the value from the document has come from any time other than 00:00, it will result in a non-integer day before integer truncation, and the truncation will result in the issue that I raised.
There was a problem hiding this comment.
Updated the logic as per ceil division to avoid data loss: a7055b9
Here are the results:
- Re-verified documents counts between exiting package version
1.27.0(Initial Interval: 48h) and new change version2.0.0(Initial Interval: 2). The counts match. - Verified from tracers the lastSeen is correctly getting applied (
Initial Interval: 2):- First request:
{\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:2\"}. This returns last event'slastSeenas1734500477. - Second request (after interval):
{\"filterName\":\"lastSeen\",\"operator\":\"=\",\"value\":\"0:1\"}
- First request:
🚀 Benchmarks reportPackage |
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
plugin | 2832.86 | 2293.58 | -539.28 (-19.04%) | 💔 |
To see the full report comment with /test benchmark fullreport
|
💚 Build Succeeded
History
cc @kcreddy |
| Package tenable_sc - 1.28.0 containing this change is available at https://epr.elastic.co/package/tenable_sc/1.28.0/ |
The API documentation for `vulnerability` mentions the `lastSeen` parameter format should be `#:#` where `#` is in number of days. Although the existing format `@-@` where `@` is epoch seconds is still working, this discrepancy is addressed to avoid any future issues. Other changes: - Updated request's `header.User-Agent` versions in input files.
The API documentation for `vulnerability` mentions the `lastSeen` parameter format should be `#:#` where `#` is in number of days. Although the existing format `@-@` where `@` is epoch seconds is still working, this discrepancy is addressed to avoid any future issues. Other changes: - Updated request's `header.User-Agent` versions in input files.
3 participants
Proposed commit message
The API documentation for
vulnerabilitymentions thelastSeenparameter format should be#:#where#is in number of days.Although the existing format
@-@where@is epoch seconds is still working, this discrepancy is addressed to avoid any future issues.Other changes:
header.User-Agentversions in input files.Ref: #10419
API doc: https://docs.tenable.com/security-center/best-practices/api/Content/RetrieveVulnerabilityDataForSpecificTimeRange.htm
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
a. Initial Interval as
24hb. Initial Interval as
48hc. Initial Interval as
4heval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -veval "$(elastic-package stack shellinit)" && elastic-package test system --generate -vRelated issues