[AWS S3] Introduce start timestamp and ignore older timespan to AWS S3 based integrations

[Kavindu-Dodan]

Proposed commit message

Introduce Ignore Older Timespan and Start Timestamp properties to integrations backed by AWS S3 input,

• Ignore Older Timespan: Accepts a timespan in which entries are accepted for processing
• Start Timestamp: Accepts a timestamp from which objects are accepted for processing

Configuring these properties allows S3 input to efficiently manage its internal registry. For example, setting Ignore Older Timespan to 2h makes the S3 input registry only track entries within the last 2 hours. Once entries are beyond the timespan, input can remove them from the registry, thus reducing memory consumption.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Closes [AWS] Introduce ignore_older & start_timestamp #11919
• Relates [aws] [s3] Introduce ignore_older & start_timestamp for S3 input allowing better registry cleanups beats#41817

Screenshots

Configuration rendered (Title matching existing format)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[agithomas]

@Kavindu-Dodan , could you please consider backporting the beats changes so that we can bump up the package versions only to 8.16.5 instead of 8.18.0?

cc @zmoog

[zmoog]

We should consider a backporting the Beats change to keep the min stack version in the 8.16.x series.

[Kavindu-Dodan]

Pending backports to 8.16.x ¹ (next - 8.16.5) & 8.17.x ² (next - 8.17.3) tracks. I will update this PR accordingly

Footnotes

1. https://github.com/elastic/beats/pull/42716 ↩

2. https://github.com/elastic/beats/pull/42717 ↩

[agithomas]

@Kavindu-Dodan , we have a great illustration of this feature's behaviour as part of the description of this issue.

Do you think that it will be good to include this illustration in AWS documentation? Reference: Azure Integration I think, if we include this illustration, it would avoid misconfiguration of this setting.

@andrewkroh, could you please share your opinion - if it is a good idea to include the illustration as part of the AWS documentation?

The changes look good to me. As discussed, kindly merge the changes after the 8.16.5 is available. Also, let us get the approval from the security-integrations team.

[Kavindu-Dodan]

@agithomas good point on the documentation. I thought of adding some diagrams, but when checking the current AWS integration documentation, I couldn't find a suitable place to add the details. So I went with detailed descriptions for the integration input fields instead.

Anyway, let me know if there's better placement for detailed documentation.

[andrewkroh]

@andrewkroh, could you please share your opinion - if it is a good idea to include the illustration as part of the AWS documentation?

Personally, I don't think they are necessary to convey the meaning of the configuration options. If we need to provide additional context, then I would suggest linking to the reference docs like, "See the [input reference documentation] for more details." And add images in the reference docs.

[agithomas]

If we need to provide additional context, then I would suggest linking to the reference docs like, "See the [input reference documentation] for more details." And add images in the reference docs.

Thanks @andrewkroh for your inputs. Yes, i agree that it would be the best approach.

@Kavindu-Dodan , it need not be part of this PR, it could be a separate PR by including the documentation team as reviewers.

[elastic-vault-github-plugin-prod]

Package carbon_black_cloud - 2.9.0 containing this change is available at https://epr.elastic.co/package/carbon_black_cloud/2.9.0/

[elastic-vault-github-plugin-prod]

Package cisco_umbrella - 1.30.0 containing this change is available at https://epr.elastic.co/package/cisco_umbrella/1.30.0/

[elastic-vault-github-plugin-prod]

Package cloudflare_logpush - 1.35.0 containing this change is available at https://epr.elastic.co/package/cloudflare_logpush/1.35.0/

[elastic-vault-github-plugin-prod]

Package f5_bigip - 1.27.0 containing this change is available at https://epr.elastic.co/package/f5_bigip/1.27.0/

[elastic-vault-github-plugin-prod]

Package imperva_cloud_waf - 1.9.0 containing this change is available at https://epr.elastic.co/package/imperva_cloud_waf/1.9.0/

[elastic-vault-github-plugin-prod]

Package jamf_protect - 2.11.0 containing this change is available at https://epr.elastic.co/package/jamf_protect/2.11.0/

[elastic-vault-github-plugin-prod]

Package sentinel_one_cloud_funnel - 1.10.0 containing this change is available at https://epr.elastic.co/package/sentinel_one_cloud_funnel/1.10.0/

[elastic-vault-github-plugin-prod]

Package servicenow - 0.11.0 containing this change is available at https://epr.elastic.co/package/servicenow/0.11.0/

[elastic-vault-github-plugin-prod]

Package sublime_security - 1.7.0 containing this change is available at https://epr.elastic.co/package/sublime_security/1.7.0/

[elastic-vault-github-plugin-prod]

Package symantec_endpoint_security - 1.9.0 containing this change is available at https://epr.elastic.co/package/symantec_endpoint_security/1.9.0/

[elastic-vault-github-plugin-prod]

Package tanium - 1.16.0 containing this change is available at https://epr.elastic.co/package/tanium/1.16.0/

[elastic-vault-github-plugin-prod]

Package trellix_edr_cloud - 1.8.0 containing this change is available at https://epr.elastic.co/package/trellix_edr_cloud/1.8.0/