[Transform] Specify use_point_in_time in settings

[prwhelan]

use_point_in_time has been in the docs since before 8.18

Resolve elastic/elasticsearch#134841

[github-actions]

Following you can find the validation changes against the target branch for the APIs.

No changes detected.

You can validate these APIs yourself by using the make validate target.

[pquentin]

@elastic/developer-docs Is this a correct usage of ext_doc_id?

This Transform API can be configured to use the PIT API, and here point-in-time-api links to the point-in-time API docs.

[kosabogi]

Hey @pquentin, the @ext_doc_id looks fine here. I see it’s already in the table.csv file as well, so it should be good. What we usually do to make sure the links appear correctly is test them locally by running these (you should have the bump CLI installed):

make generate make transform-to-openapi bump preview output/openapi/elasticsearch-openapi.json 

[prwhelan]

Seems to have worked as intended:

And it correctly links to https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time

[leemthompo]

FYI we could add a description field to the link definition in

elasticsearch-specification/specification/_doc_ids/table.csv

Line 521 in 34dfe8b

point-in-time-api,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/point-in-time-api.html,

and this would be render nicer link text than just External documentation 💅

[prwhelan]

Do you have an example of that?

[leemthompo]

this could be as simple as Point in Time API

[leemthompo]

can see an example in this PR: https://github.com/elastic/elasticsearch-specification/pull/5331/files#diff-afcc28b0721ba019d194536c1ff7cb6117ac7db92b094fbc8723ec371771385d

[pquentin]

I agree that it would be nice to name this "Open a point in time API" instead of "External documentation."

[prwhelan]

It didn't seem to work, but perhaps I did it wrong:

All the others from that example PR still show External documentation for me so 🤷

[pquentin]

What happens if you set it to false? It uses the scroll API?

[prwhelan]

Nah it just uses a search request without PIT: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/transform/src/main/java/org/elasticsearch/xpack/transform/transforms/TransformIndexer.java#L1154

[leemthompo]

ideally we'd explain the pros/cons differences of each approach, or at least outline why you'd want to use the PIT, rather than just the how :)

[prwhelan]

should that go on the PIT page? in either case, we'd need someone familiar with PIT to go into that detail, I'm just trying to show what users can configure rather than keeping the setting hidden

[pquentin]

I think what I missed is that transforms use composite aggregations, which offer pagination without using PIT. However, transforms use PIT by default, as it returns more correct results. That said, as it can be expensive if many PITs are open at any given time, this can be disabled. Is my understanding correct?

This is not about explaining what PIT does (especially given that we have a link), but making it clear what the setting does. For example, max_page_search_size is clear here!

[leemthompo]

This is not about explaining what PIT does (especially given that we have a link), but making it clear what the setting does.

right Quentin said it much clearer than me 😄

[pquentin]

Thanks, LGTM! Sorry for the delay.

[github-actions]

The backport to 8.19 failed:

The process '/usr/bin/git' failed with exit code 1 

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub git fetch # Create a new working tree git worktree add .worktrees/backport-8.19 8.19 # Navigate to the new working tree cd .worktrees/backport-8.19 # Create a new branch git switch --create backport-5322-to-8.19 # Cherry-pick the merged commit of this pull request and resolve the conflicts git cherry-pick -x --mainline 1 9fe24d15ae7841b87f6e2d2c745374b08d2e55c4 # Push it to GitHub git push --set-upstream origin backport-5322-to-8.19 # Go back to the original working tree cd ../.. # Delete the working tree git worktree remove .worktrees/backport-8.19

Then, create a pull request where the base branch is 8.19 and the compare/head branch is backport-5322-to-8.19.

[github-actions]

The backport to 8.18 failed:

The process '/usr/bin/git' failed with exit code 1 

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub git fetch # Create a new working tree git worktree add .worktrees/backport-8.18 8.18 # Navigate to the new working tree cd .worktrees/backport-8.18 # Create a new branch git switch --create backport-5322-to-8.18 # Cherry-pick the merged commit of this pull request and resolve the conflicts git cherry-pick -x --mainline 1 9fe24d15ae7841b87f6e2d2c745374b08d2e55c4 # Push it to GitHub git push --set-upstream origin backport-5322-to-8.18 # Go back to the original working tree cd ../.. # Delete the working tree git worktree remove .worktrees/backport-8.18

Then, create a pull request where the base branch is 8.18 and the compare/head branch is backport-5322-to-8.18.

[github-actions]

The backport to 9.0 failed:

The process '/usr/bin/git' failed with exit code 1 

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub git fetch # Create a new working tree git worktree add .worktrees/backport-9.0 9.0 # Navigate to the new working tree cd .worktrees/backport-9.0 # Create a new branch git switch --create backport-5322-to-9.0 # Cherry-pick the merged commit of this pull request and resolve the conflicts git cherry-pick -x --mainline 1 9fe24d15ae7841b87f6e2d2c745374b08d2e55c4 # Push it to GitHub git push --set-upstream origin backport-5322-to-9.0 # Go back to the original working tree cd ../.. # Delete the working tree git worktree remove .worktrees/backport-9.0

Then, create a pull request where the base branch is 9.0 and the compare/head branch is backport-5322-to-9.0.