Added runtime validation of sql expressions #625
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| ✅ 406/406 passed, 12 flaky, 1 skipped, 4h45m0s total Flaky tests:
Running from acceptance #2909 |
mwojtyczka changed the title mwojtyczka requested review from alexott, Copilot and ghanse and removed request for grusin-db Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds runtime validation for SQL expressions in the sql_expression check to improve error handling when expressions reference non-existent columns or contain invalid syntax. The enhancement ensures that invalid SQL expressions are detected early and checks are skipped gracefully with descriptive error messages.
Key changes:
- Implemented validation logic to detect invalid SQL expressions before execution
- Enhanced error messages to include SQL expression validation failures
- Updated documentation to clarify validation behavior and limitations
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| src/databricks/labs/dqx/manager.py | Added invalid_sql_expression property and validation logic for SQL expressions |
| tests/integration/test_apply_checks.py | Added test cases for SQL expression validation with missing columns |
| docs/dqx/docs/reference/quality_checks.mdx | Updated documentation for sql_expression check to explain validation behavior |
| docs/dqx/docs/guide/quality_checks_apply.mdx | Updated guide to mention SQL expression validation alongside other field validations |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
mwojtyczka changed the title 
mwojtyczka added a commit that referenced this pull request Nov 6, 2025
* Added Data Quality Summary Metrics ([#553](#553)). The data quality engine has been enhanced with the ability to track and manage summary metrics for data quality validation, leveraging Spark's Observation feature. A new `DQMetricsObserver` class has been introduced to manage Spark observations and track summary metrics on datasets checked with the engine. The `DQEngine` class has been updated to optionally return the Spark observation associated with a given run, allowing users to access and save summary metrics. The engine now supports also writing summary metrics to a table using the `metrics_config` parameter, and a new `save_summary_metrics` method has been added to save data quality summary metrics to a table. Additionally, the engine has been updated to include a unique `run_id` field in the detailed per-row quality results, enabling cross-referencing with summary metrics. The changes also include updates to the configuration file to support the storage of summary metrics. Overall, these enhancements provide a more comprehensive and flexible data quality checking capability, allowing users to track and analyze data quality issues more effectively. * LLM assisted rules generation ([#577](#577)). This release introduces a significant enhancement to the data quality rules generation process with the integration of AI-assisted rules generation using large language models (LLMs). The `DQGenerator` class now includes a `generate_dq_rules_ai_assisted` method, which takes user input in natural language and optionally a schema from an input table to generate data quality rules. These rules are then validated for correctness. The AI-assisted rules generation feature supports both programmatic and no-code approaches. Additionally, the feature enables the use of different LLM models and gives the possibility to use custom check functions. The release also includes various updates to the documentation, configuration files, and testing framework to support the new AI-assisted rules generation feature, ensuring a more streamlined and efficient process for defining and applying data quality rules. * Added Lakebase checks storage backend ([#550](#550)). A Lakebase checks storage backend was added, allowing users to store and manage their data quality rules in a centralized lakabase table, in addition to the existing Delta table storage. The `checks_location` resolution has been updated to accommodate Lakebase, supporting both table and file storage, with flexible formatting options, including "catalog.schema.table" and "database.schema.table". The Lakebase checks storage backend is configurable through the `LakebaseChecksStorageConfig` class, which includes fields for instance name, user, location, port, run configuration name, and write mode. This update provides users with more flexibility in storing and loading quality checks, ensuring that checks are saved correctly regardless of the specified location format. * Added runtime validation of sql expressions ([#625](#625)). The data quality check functionality has been enhanced with runtime validation of SQL expressions, ensuring that specified fields can be resolved in the input DataFrame and that SQL expressions are valid before evaluation. If an SQL expression is invalid, the check evaluation is skipped and the results include a check failure with a descriptive message. Additionally, the configuration validation for Unity Catalog volume file paths has been improved to enforce a specific format, preventing invalid configurations and providing more informative error messages. * Fixed docs ([#598](#598)). The documentation build process has undergone significant improvements to enhance efficiency and maintainability. * Improved Config Serialization ([#676](#676)). Several updates have been made to improve the functionality, consistency, and maintainability of the codebase. The configuration loading functionality has been refactored to utilize the `ConfigSerializer` class, which handles the serialization and deserialization of workspace and run configurations. * Restore use of `hatch-fancy-pypi-readme` to fix images in PyPi ([#601](#601)). The image source path for the logo in the README has been modified to correctly display the logo image when rendered, particularly on PyPi. * Skip check evaluation if columns or filter cannot be resolved in the input DataFrame ([#609](#609)). DQX now skip check evaluation if columns or filters are incorrect allowing other checks to proceed even if one rule fails. The DQX engine validates specified column, columns and filter fields against the input DataFrame before applying checks, skipping evaluation and providing informative error messages if any fields are invalid. * Updated user guide docs ([#607](#607)). The documentation for quality checking and integration options has been updated to provide accurate and detailed information on supported types and approaches. Quality checking can be performed in-transit (pre-commit), validating data on the fly during processing, or at-rest, checking existing data stored in tables. * Improved build process ([#618](#618)). The hatch version has been updated to 1.15.0 to avoid compatibility issues with click version 8.3 and later, which introduced a bug affecting hatch. Additionally, the project's dependencies have been updated, including bumping the `databricks-labs-pytester` version from 0.7.2 to 0.7.4, and code refactoring has been done to use a single Lakebase instance for all integration tests, with retry logic added to handle cases where the workspace quota limit for the number of Lakebase instances is exceeded, enhancing the testing infrastructure and improving test reliability. Furthermore, documentation updates have been made to clarify the application of quality checks to data using DQX. These changes aim to improve the efficiency, reliability, and clarity of the project's testing and documentation infrastructure.
Merged
mwojtyczka added a commit that referenced this pull request Nov 7, 2025
* Added Data Quality Summary Metrics ([#553](#553)). The data quality engine has been enhanced with the ability to track and manage summary metrics for data quality validation, leveraging Spark's Observation feature. A new `DQMetricsObserver` class has been introduced to manage Spark observations and track summary metrics on datasets checked with the engine. The `DQEngine` class has been updated to optionally return the Spark observation associated with a given run, allowing users to access and save summary metrics. The engine now supports also writing summary metrics to a table using the `metrics_config` parameter, and a new `save_summary_metrics` method has been added to save data quality summary metrics to a table. Additionally, the engine has been updated to include a unique `run_id` field in the detailed per-row quality results, enabling cross-referencing with summary metrics. The changes also include updates to the configuration file to support the storage of summary metrics. Overall, these enhancements provide a more comprehensive and flexible data quality checking capability, allowing users to track and analyze data quality issues more effectively. * LLM assisted rules generation ([#577](#577)). This release introduces a significant enhancement to the data quality rules generation process with the integration of AI-assisted rules generation using large language models (LLMs). The `DQGenerator` class now includes a `generate_dq_rules_ai_assisted` method, which takes user input in natural language and optionally a schema from an input table to generate data quality rules. These rules are then validated for correctness. The AI-assisted rules generation feature supports both programmatic and no-code approaches. Additionally, the feature enables the use of different LLM models and gives the possibility to use custom check functions. The release also includes various updates to the documentation, configuration files, and testing framework to support the new AI-assisted rules generation feature, ensuring a more streamlined and efficient process for defining and applying data quality rules. * Added Lakebase checks storage backend ([#550](#550)). A Lakebase checks storage backend was added, allowing users to store and manage their data quality rules in a centralized lakabase table, in addition to the existing Delta table storage. The `checks_location` resolution has been updated to accommodate Lakebase, supporting both table and file storage, with flexible formatting options, including "catalog.schema.table" and "database.schema.table". The Lakebase checks storage backend is configurable through the `LakebaseChecksStorageConfig` class, which includes fields for instance name, user, location, port, run configuration name, and write mode. This update provides users with more flexibility in storing and loading quality checks, ensuring that checks are saved correctly regardless of the specified location format. * Added runtime validation of sql expressions ([#625](#625)). The data quality check functionality has been enhanced with runtime validation of SQL expressions, ensuring that specified fields can be resolved in the input DataFrame and that SQL expressions are valid before evaluation. If an SQL expression is invalid, the check evaluation is skipped and the results include a check failure with a descriptive message. Additionally, the configuration validation for Unity Catalog volume file paths has been improved to enforce a specific format, preventing invalid configurations and providing more informative error messages. * Fixed docs ([#598](#598)). The documentation build process has undergone significant improvements to enhance efficiency and maintainability. * Improved Config Serialization ([#676](#676)). Several updates have been made to improve the functionality, consistency, and maintainability of the codebase. The configuration loading functionality has been refactored to utilize the `ConfigSerializer` class, which handles the serialization and deserialization of workspace and run configurations. * Restore use of `hatch-fancy-pypi-readme` to fix images in PyPi ([#601](#601)). The image source path for the logo in the README has been modified to correctly display the logo image when rendered, particularly on PyPi. * Skip check evaluation if columns or filter cannot be resolved in the input DataFrame ([#609](#609)). DQX now skip check evaluation if columns or filters are incorrect allowing other checks to proceed even if one rule fails. The DQX engine validates specified column, columns and filter fields against the input DataFrame before applying checks, skipping evaluation and providing informative error messages if any fields are invalid. * Updated user guide docs ([#607](#607)). The documentation for quality checking and integration options has been updated to provide accurate and detailed information on supported types and approaches. Quality checking can be performed in-transit (pre-commit), validating data on the fly during processing, or at-rest, checking existing data stored in tables. * Improved build process ([#618](#618)). The hatch version has been updated to 1.15.0 to avoid compatibility issues with click version 8.3 and later, which introduced a bug affecting hatch. Additionally, the project's dependencies have been updated, including bumping the `databricks-labs-pytester` version from 0.7.2 to 0.7.4, and code refactoring has been done to use a single Lakebase instance for all integration tests, with retry logic added to handle cases where the workspace quota limit for the number of Lakebase instances is exceeded, enhancing the testing infrastructure and improving test reliability. Furthermore, documentation updates have been made to clarify the application of quality checks to data using DQX. These changes aim to improve the efficiency, reliability, and clarity of the project's testing and documentation infrastructure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Changes
sql_expressioncheck. Note that this covers a subset of error conditions. Some issues only appear during a Spark action.Tests