Skip to content

Source MongoDB: customize DISCOVER_LIMIT #9284

@alafanechere

Description

@alafanechere

Tell us about the problem you're trying to solve

To make schema discovery faster #8491 limited the schema discovery step to the analysis of 10k records.
Some users might have heterogenous schema spanning on more than this hardcoded limit.

Describe the solution you’d like

We could expose DISCOVER_LIMIT as a parameter in the source configuration to allow users to customize this value, or choose to read all documents in the discovery step.

Describe the alternative you’ve considered or used

Downgrade to version 0.1.8

Additional context

Requested by a user in this Slack conversation

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions