[prisma_cloud] Update the Cursor in the Data Collection of Alert and Audit Data Stream and HTTP Timeout Default Values

[mohitjha-elastic]

Type of change

• BugFix

What does this PR do?

1. Change the default value of HTTP Client Timeout from 30s to 60s
For the larger dataset, Prisma API is taking a longer wait time to send the response that is greater than 30s hence update the default value of timeout to 60s for all the data streams.

2. Update the Cursor Logic of the Alert and Audit Data Stream
There was a bug in the cursor implementation, earlier it was storing the value (now - lastUpdated.max) which resulted in the wrong subsequent requests call. Hence updated the cursor implementation to now() - cursor at the time of request call.
Also, use getMinutes() instead of getHours() to get a more precise time as the Prisma API filter supports the minimum time unit as a minute.

3. Add fingerprint to the Audit Data Stream.
As the getMinutes() returns the floor value results in some data dropping hence added +1 explicity to the requests and added fingerprint to remove the duplicated values due to adding the extra minutes.
The fields added in the fingerprint suggested here

4. Minor Bugfix in Alert Pipeline
Due to a typing mistake, the wrong field was used in the alert pipeline fingerprint processors hence corrected that. Changed it from json._id to json.id.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

All changes

• Change follows the contributing guidelines
• Supported versions of the monitoring target are documented
• Supported operating systems are documented (if applicable)
• Integration or System tests exist
• Documentation exists
• Fields follow ECS and naming conventions
• At least a manual test with ES / Kibana / Agent has been performed.
• Required Kibana version set to: ^8.10.1

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/prisma_cloud directory.
Run the following command to run tests.
elastic-package test -v

Related issues

• Closes [Prisma Cloud] Alert Pipeline not working #8372
• Closes https://github.com/elastic/sdh-beats/issues/4049

Automated Test

prisma_test_file.txt

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-28T20:19:01.228+0000

• Duration: 24 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	36
Skipped	0
Total	36

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[efd6]

Please use consistent indentation to ease logic flow visibility.
Suggest (untested):

 ( state.with(has(state.want_more) && !(state.want_more) ? post_request( state.url + "/login", "application/json", {"username":state.user,"password":state.password}.encode_json() ).do_request().as(resp, bytes(resp.Body).decode_json().as(body, { "access_token": body.token, })) : {} ).as(state, state.with( request("GET", state.url + "/v2/alert?timeType=relative&detailed=true&limit=" + string(state.batch_size) + "&timeAmount=" + ( has(state.want_more) && !(state.want_more) ? ( has(state.cursor) && has(state.cursor.last_time_amount) && state.cursor.last_time_amount != null ? string((now() - timestamp(int(timestamp(0)+duration(string(int(state.cursor.last_time_amount))+"ms")))).getMinutes() + 1) + "&timeUnit=minute" : string(state.time_amount) + "&timeUnit=" + state.time_unit ) : ( has(state.cursor) && has(state.cursor.first_time_amount) && state.cursor.first_time_amount != null ? string((now() - timestamp(int(timestamp(0)+duration(string(int(state.cursor.first_time_amount))+"ms")))).getMinutes() + 1) + "&timeUnit=minute" : "null" ) + (has(state.page_token) ? "&pageToken=" + state.page_token : "") ) ).with({ "Header":{ "x-redlock-auth": [state.access_token], } }).do_request().as(resp, bytes(resp.Body).decode_json().as(inner_body, { "events": inner_body.items.map(e, { "message": e.encode_json(), }), "url": state.url, "page_token": has(inner_body.nextPageToken) ? inner_body.nextPageToken : "", "cursor": { "last_time_amount": ( has(inner_body.items) && inner_body.items.size() > 0 ? ( has(state.cursor) && has(state.cursor.last_time_amount) && inner_body.items.map(e, e.lastUpdated).max() < state.cursor.last_time_amount ? state.cursor.last_time_amount : inner_body.items.map(e, e.lastUpdated).max() ) : ( has(state.cursor) && has(state.cursor.last_time_amount) ? state.cursor.last_time_amount : null ) ), "first_time_amount": ( has(state.cursor) && has(state.cursor.first_time_amount) && state.cursor.first_time_amount != null && has(inner_body.items) ? ( ((int(state.total_rows) + size(inner_body.items)) < inner_body.totalRows) && state.want_more ? state.cursor.first_time_amount : state.cursor.last_time_amount ) : int(timestamp(now())-duration(string( state.time_unit == "year" ? int(state.time_amount)*365*24*60 : state.time_unit == "month" ? int(state.time_amount)*30*24*60 : state.time_unit == "week" ? int(state.time_amount)*7*24*60 : state.time_unit == "day" ? int(state.time_amount)*24*60 : state.time_unit == "hour" ? int(state.time_amount)*60 : int(state.time_amount) )+"m"))*1000 ), }, "want_more": (int(state.total_rows) + size(inner_body.items)) < inner_body.totalRows, "user": state.user, "password": state.password, "batch_size": string(state.batch_size), "time_amount": string(state.time_amount), "time_unit": state.time_unit, "total_rows": (int(state.total_rows) + size(inner_body.items)) < inner_body.totalRows ? int(state.total_rows) + size(inner_body.items) : 0, })) ) ) ) 

Also, since the kibana version is v8.10.1, all the instances of now() can (should) be replaced with now.

[efd6]

/test

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (5/5)	💚
Files	100.0% (5/5)	💚
Classes	100.0% (5/5)	💚
Methods	94.286% (66/70)	👎 -5.714
Lines	95.069% (7056/7422)	👎 -4.931
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Package prisma_cloud - 0.6.0 containing this change is available at https://epr.elastic.co/search?package=prisma_cloud