paginator: manual pagination with familiar syntax

[ldanilek]

Create a new convex-helpers function paginator that complements getPage.

See the README and the docstring for usage details.

I would like to ship this so we can recommend it for customers running into the limitations of .paginate without teaching them the new concept of IndexKeys. In particular I want to use it in the migrations component and suggest component authors use it in their components.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[ianmacartney]

some thoughts:

• is it expected to pass in paginationOpts? I suspect we'll not use all those values, just the cursor & numItems? If so, I'd say let's unify the options so everything is in one spot.
• It's fine to require schema always if it makes the types easier - I suspect advanced usage (almost) always uses an index - and it's an easy auto-import param.
• syntax dream:

await paginator(ctx, schema) .query(table) .withIndex(index, q => q.eq(field, value)) .order("desc") .filter((doc) => doc[field] !== value) .paginate(opts)

Could this be possible by returning types like

OrderedQuery<..> & { filter?: (doc: DocumentByName<DataModel, T>) => Promise<boolean>, paginate(opts: { cursor: string | null, numItems: number, endCursor?: string }) }

? I suspect it'd be pretty tough, but that would be an amazing drop-in replacement option (modulo filter types).

[ianmacartney]

Suggested change

	* in small or empty pages.
	* in small or empty pages, though a best effort is made to retrieve `numItems`.

[ldanilek]

i'm not sure what "best effort" means there. we look at numItems documents and then apply the filter

[ianmacartney]

oh interesting. I thought we tried with some limit to fill numItems. If we're truly just fetching numItems then why offer filter at all?

[ldanilek]

as a convenience to make it easy to switch from the native syntax, and because the filter() convex-helper doesn't compose with this helper

[ianmacartney]

I had assumed it would work harder to ensure it returned numItems so I'd rather snip it and just suggest folks do

(await getPageOfQuery(ctx, ...)).filter(doc => doc.isGood)

This also means that the type of paginator(ctx, schema, tableName) could just be the same as db.query (and just throw at runtime for unimplemented functions) since all the syntax would conform as a subset.

[ldanilek]

is it expected to pass in paginationOpts? I suspect we'll not use all those values, just the cursor & numItems

heh PaginationOptions only has those two fields: cursor and numItems (the other fields are internal so they're not in the typescript)

[ianmacartney]

Interesting - the thing is that when I click through to see what the heck is in there, I end up looking at the type that includes the internal fields. Maybe I'm using it wrong, but I often click through when I see opaque types like PaginationOptions b/c auto-complete doesn't do much until you have something you can start tab-completing

…

On Mon, Nov 4, 2024 at 8:35 PM Lee Danilek ***@***.***> wrote: is it expected to pass in paginationOpts? I suspect we'll not use all those values, just the cursor & numItems heh PaginationOptions only has those two fields: cursor and numItems (the other fields are internal so they're not in the typescript) — Reply to this email directly, view it on GitHub <#337 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACZQW34L5QQOEPAATCZQGLZ7A4IXAVCNFSM6AAAAABRDD4CWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJWGIYTIMZRGM> . You are receiving this because your review was requested.Message ID: ***@***.***>

[ldanilek]

Interesting - the thing is that when I click through to see what the heck

is in there, I end up looking at the type that includes the internal

fields.

Maybe I'm using it wrong, but I often click through when I see opaque types

like PaginationOptions b/c auto-complete doesn't do much until you have

something you can start tab-completing

On Mon, Nov 4, 2024 at 8:35 PM Lee Danilek @.***> wrote:

is it expected to pass in paginationOpts? I suspect we'll not use all

those values, just the cursor & numItems

heh PaginationOptions only has those two fields: cursor and numItems (the

other fields are internal so they're not in the typescript)

—

Reply to this email directly, view it on GitHub

#337 (comment),

or unsubscribe

https://github.com/notifications/unsubscribe-auth/AACZQW34L5QQOEPAATCZQGLZ7A4IXAVCNFSM6AAAAABRDD4CWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJWGIYTIMZRGM

.

You are receiving this because your review was requested.Message ID:

@.***>

Makes sense. I was also confused why endCursor wasn't autocompleting, since i saw it in the type, until i realized the actual exported type only has two fields. Regardless, I would like to keep the PaginationOptions arg so you can convert from the existing syntax and use it with usePaginatedQuery seamlessly. There are even cases where usePaginatedQuery will pass endCursor (when page splitting), and that will just work with this helper.

[ldanilek]

• It's fine to require schema always if it makes the types easier - I suspect advanced usage (almost) always uses an index - and it's an easy auto-import param.

• syntax dream:

await paginator(ctx, schema) .query(table) .withIndex(index, q => q.eq(field, value)) .order("desc") .filter((doc) => doc[field] !== value) .paginate(opts)

Could this be possible by returning types like

OrderedQuery<..> & { filter?: (doc: DocumentByName<DataModel, T>) => Promise<boolean>, paginate(opts: { cursor: string | null, numItems: number, endCursor?: string }) }

? I suspect it'd be pretty tough, but that would be an amazing drop-in replacement option (modulo filter types).

Tradeoffs of requiring schema:

• con: if you're using js or don't have a schema, or for some reason it's hard to import, it's nice that plain db.query can work without it.
• pro: if you are using an index range, having the schema required at compile-time means it won't fail at runtime, which is a good pit-of-success.

I like the syntax you suggest. I'll try it.

[ldanilek]

btw i think the type inference might work better if i pass in the db directly instead of a full ctx. Thoughts?

[ianmacartney]

btw i think the type inference might work better if i pass in the db directly instead of a full ctx. Thoughts?

yeah I've found that to be true. One pro of passing in schema is you can pull the data model from that more reliably than ctx. But I agree there's some nice conveniences of not requiring it. but again I think this helper is for more advanced users. so js/ no schema isn't a big thing.

[ianmacartney]

very nice!

[ianmacartney]

looks like this is unused according to github?