- Notifications
You must be signed in to change notification settings - Fork 100
Description
Data quality check not working for Silver. specified expecte_or_drop as product_id IS NULL but those record with product id still inserted
Error Reproduce:
Direct Insert into Flow Spec.
INSERT INTO nonprod.meta.silver_dataflowspec_table SELECT '1031' AS dataFlowId ,'A1' AS dataFlowGroup ,'delta' AS sourceFormat ,MAP("database","bronze","table","t_brz_products") AS sourceDetails ,MAP() AS readerConfigOptions ,'delta' AS targetFormat ,MAP("database","silver","table","t_slv_products_dq_log") AS targetDetails ,MAP() AS tableProperties ,Array("*") AS selectExp ,Array("") AS whereClause ,Array("") AS partitionColumns ,null AS cdcApplyChanges ,'{"expect_or_drop": {"null_product_id": "product_id IS NULL"}}' AS dataQualityExpectations ,null AS appendFlows ,MAP() AS appendFlowsSchemas ,'v1' AS version ,current_timestamp AS createDate ,'xxxxxxx' AS createdBy ,current_timestamp AS updateDate ,'xxxxxx'AS updatedBy;
Data Quality Rule:
{"expect_or_drop": {"null_product_id": "product_id IS NULL"}}


