[atlassian_jira] Add stop condition for Jira Cloud pagination

[taylor-swanson]

What does this PR do?

• Added a stop condition for Jira Cloud pagination. Pagination will now terminate when the offset is greater than or equal to the total count value returned by the API.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

elastic-package system test 

Testing Notes

These are the requests now made by filebeat:

{"level":"debug","ts":"2023-09-21T17:22:54.435Z","caller":"httpserver/httpserver.go:130","msg":"Rule #3 matched: request #1 => Request path: GET /rest/api/3/auditing/record?from=2023-09-20T17%3A22%3A54.432&limit=2&offset=0&to=2023-09-21T17%3A22%3A54.432, Request Headers: 'User-Agent: [Elastic-Filebeat/8.10.2 (linux; amd64; 480bccf4f0423099bb2c0e672a44c54ecd7a805e; 2023-09-18 18:09:06 +0000 UTC)]' 'Accept: [application/json]' 'Authorization: [Basic dGVzdC51c2VyOmFiYzEyMw==]' 'Accept-Encoding: [gzip]' 'Connection: [close]' , Request Body: ","address":":8080"} {"level":"debug","ts":"2023-09-21T17:22:56.400Z","caller":"httpserver/httpserver.go:130","msg":"Rule #2 matched: request #1 => Request path: GET /rest/api/3/auditing/record?from=2023-09-20T17%3A22%3A54.432&limit=2&offset=2&to=2023-09-21T17%3A22%3A54.432, Request Headers: 'User-Agent: [Elastic-Filebeat/8.10.2 (linux; amd64; 480bccf4f0423099bb2c0e672a44c54ecd7a805e; 2023-09-18 18:09:06 +0000 UTC)]' 'Accept: [application/json]' 'Authorization: [Basic dGVzdC51c2VyOmFiYzEyMw==]' 'Accept-Encoding: [gzip]' 'Connection: [close]' , Request Body: ","address":":8080"} 

Here is with the stop condition removed and the extra path added to the system test to induce an infinite loop (causing the hit count assert to fail):

2023/09/21 12:28:14 DEBUG found 50 hits in logs-atlassian_jira.audit-ep data stream 2023/09/21 12:28:19 DEBUG found 1730 hits in logs-atlassian_jira.audit-ep data stream 2023/09/21 12:28:23 DEBUG found 3452 hits in logs-atlassian_jira.audit-ep data stream 2023/09/21 12:28:23 DEBUG check whether or not synthetics is enabled (component template logs-atlassian_jira.audit@package)... 2023/09/21 12:28:23 DEBUG data stream logs-atlassian_jira.audit-ep has synthetics enabled: false 2023/09/21 12:28:23 DEBUG assert hit count expected 4, observed 500 2023/09/21 12:28:23 DEBUG reassigning original policy back to agent... 

Here is what was happening before (note the third request that shouldn't be made, test terminates because we 404-ed, this is not what appears to happen with the real server, it will respond with a 200, causing an never ending increase in the offset).

{"level":"debug","ts":"2023-09-21T17:38:33.985Z","caller":"httpserver/httpserver.go:130","msg":"Rule #3 matched: request #1 => Request path: GET /rest/api/3/auditing/record?from=2023-09-20T17%3A38%3A33.981&limit=2&offset=0&to=2023-09-21T17%3A38%3A33.981, Request Headers: 'User-Agent: [Elastic-Filebeat/8.10.2 (linux; amd64; 480bccf4f0423099bb2c0e672a44c54ecd7a805e; 2023-09-18 18:09:06 +0000 UTC)]' 'Accept: [application/json]' 'Authorization: [Basic dGVzdC51c2VyOmFiYzEyMw==]' 'Accept-Encoding: [gzip]' 'Connection: [close]' , Request Body: ","address":":8080"} {"level":"debug","ts":"2023-09-21T17:38:35.959Z","caller":"httpserver/httpserver.go:130","msg":"Rule #2 matched: request #1 => Request path: GET /rest/api/3/auditing/record?from=2023-09-20T17%3A38%3A33.981&limit=2&offset=2&to=2023-09-21T17%3A38%3A33.981, Request Headers: 'User-Agent: [Elastic-Filebeat/8.10.2 (linux; amd64; 480bccf4f0423099bb2c0e672a44c54ecd7a805e; 2023-09-18 18:09:06 +0000 UTC)]' 'Accept: [application/json]' 'Authorization: [Basic dGVzdC51c2VyOmFiYzEyMw==]' 'Accept-Encoding: [gzip]' 'Connection: [close]' , Request Body: ","address":":8080"} {"level":"debug","ts":"2023-09-21T17:38:35.962Z","caller":"httpserver/httpserver.go:212","msg":"request did not match with any rule: Request path: GET /rest/api/3/auditing/record?from=2023-09-20T17%3A38%3A33.981&limit=2&offset=4&to=2023-09-21T17%3A38%3A33.981, Request Headers: 'Accept-Encoding: [gzip]' 'Connection: [close]' 'User-Agent: [Elastic-Filebeat/8.10.2 (linux; amd64; 480bccf4f0423099bb2c0e672a44c54ecd7a805e; 2023-09-18 18:09:06 +0000 UTC)]' 'Accept: [application/json]' 'Authorization: [Basic dGVzdC51c2VyOmFiYzEyMw==]' , Request Body: ","address":":8080"} 

Related issues

• Closes [atlassian_jira] Various pagination issues with Jira Cloud #7898

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-09-22T14:07:53.637+0000

• Duration: 17 min 45 sec

Test stats 🧪

Test	Results
Failed	0
Passed	10
Skipped	0
Total	10

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (1/1)	💚
Files	100.0% (3/3)	💚 29.412
Classes	100.0% (3/3)	💚 29.412
Methods	100.0% (24/24)	💚 43.182
Lines	98.845% (428/433)	👍 58.885
Conditionals	100.0% (0/0)	💚

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

LGTM.

[taylor-swanson]

/test

[elasticmachine]

Package atlassian_jira - 1.19.1 containing this change is available at https://epr.elastic.co/search?package=atlassian_jira