Skip to content

Skip iterating DISI when reading metric values #133365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dnhatn merged 9 commits into from
Aug 28, 2025
Merged

Skip iterating DISI when reading metric values #133365

dnhatn merged 9 commits into from
Aug 28, 2025

Conversation

@dnhatn
Copy link
Member

@dnhatn dnhatn commented Aug 22, 2025 •
edited
Loading

Avoids iterating the DISI when reading a sparse metric field if its nulls were already filtered by a WHERE clause (e.g., metrics != NULL). Iterating the DISI on a sparse field can be expensive. From my local benchmark, the query time of

TS my* | WHERE `metrics.system.memory.utilization` IS NOT NULL AND @timestamp >= \"2025-07-25T14:55:59.000Z\" AND @timestamp <= \"2025-07-25T16:25:59.000Z\" | STATS AVG(AVG_OVER_TIME(`metrics.system.memory.utilization`)) BY host.name, BUCKET(@timestamp, 1h) | LIMIT 10000"

decreased from 35ms to 27ms (20%).

Before:
before

After:

after
@dnhatn dnhatn added the WIP label Aug 22, 2025
@dnhatn dnhatn requested review from kkrik-es and martijnvg August 22, 2025 00:08
kkrik-es
kkrik-es approved these changes Aug 22, 2025
View reviewed changes
Copy link
Contributor

@kkrik-es kkrik-es left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, I'm glad my draft has some overlap :)
martijnvg
martijnvg reviewed Aug 25, 2025
View reviewed changes
Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. I left minor comments and we need a test for tryReadDoubles() in ES819TSDBDocValuesFormatTests
@dnhatn dnhatn force-pushed the skip-disi-metric-field branch from 23874ce to 46085ab Compare August 27, 2025 23:41
@dnhatn dnhatn force-pushed the skip-disi-metric-field branch from 46085ab to 6bb2a4a Compare August 28, 2025 00:10
@dnhatn dnhatn force-pushed the skip-disi-metric-field branch from 43831ab to 9fd96a3 Compare August 28, 2025 05:26
@dnhatn
Copy link
Member Author

dnhatn commented Aug 28, 2025

@martijnvg @kkrik-es I've updated this with Kostas's PR. I think it looks better. Can you take a look? Thanks!
@dnhatn dnhatn marked this pull request as ready for review August 28, 2025 06:11
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Aug 28, 2025

Pinging @elastic/es-storage-engine (Team:StorageEngine)
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Aug 28, 2025

Hi @dnhatn, I've created a changelog YAML for you.
martijnvg
martijnvg approved these changes Aug 28, 2025
View reviewed changes
Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Nhat!
kkrik-es
kkrik-es approved these changes Aug 28, 2025
View reviewed changes
Copy link
Contributor

@kkrik-es kkrik-es left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice and clean!
@dnhatn
Copy link
Member Author

dnhatn commented Aug 28, 2025

@martijnvg @kkrik-es Thank you!
@dnhatn dnhatn merged commit d8ea894 into elastic:main Aug 28, 2025
30 of 33 checks passed
@dnhatn dnhatn deleted the skip-disi-metric-field branch August 28, 2025 17:22
@martijnvg martijnvg mentioned this pull request Sep 1, 2025
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>enhancement :StorageEngine/Codec Team:StorageEngine v9.2.0

4 participants

@dnhatn @elasticsearchmachine @martijnvg @kkrik-es