[system tests] Add new settings to wait for documents being ingested #2429
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
mrodm commented Feb 25, 2025
There was a problem hiding this comment.
Would it make sense to keep this package under the other folder? or should we move it to the parallel folder?
At least, this package requires to spin up an Elastic Agent to read the logs. Should we just add packages that their tests are run fast in other?
mrodm commented Feb 25, 2025
mrodm commented Feb 25, 2025
mrodm changed the title | Depends on #2433 |
| test integrations |
| Created or updated PR in integrations repository to test this version. Check elastic/integrations#12900 |
| /test |
jsoriano approved these changes Feb 26, 2025
...stem_test_assert_conditions/data_stream/test/_dev/test/system/test-fields_present-config.yml Show resolved Hide resolved
jsoriano reviewed Feb 27, 2025
test/packages/false_positives/failed_fields_present_assert.expected_errors Outdated Show resolved Hide resolved
jsoriano approved these changes Feb 27, 2025
💚 Build Succeeded
History
cc @mrodm |
jsoriano approved these changes Feb 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Closes #2378
This PR includes new system test settings to allow for different conditions when waiting for the required/needed documents to be ingested into the data stream.
Three new settings have been added:
assert.min_count: minimum number of documents to be ingested in the data stream.assert.fields_present: list of fields that must be present in any document of the given data stream.assert.ingestion_idle_time: minimum period of time that the data stream should not have any new document ingested.this could be useful in those packages that there is no exact number of documents to be ingested, every run could be a different number of documents ingested.Related: [system tests] Add new options to wait for documents before validating fields/mappings #2378 (comment)Examples of the
elastic-packageoutput when these settings are defined:assert.ingestion_idle_time: 20s:assert.fields_present: [target.file, target.expected, target.finish]:assert.min_count: 50:Moreover, these settings can be defined simultaneously. For instance:
In that example, the validation performed by the system test will only run after at least 50 documents have been ingested into the specified data stream, and the two designated fields are prsent in any of the ingested documents.
Author's checklist
How to test this PR locally