Skip to content

Silver Data Quality Not working #156

@JunHongPP

Description

@JunHongPP

Data quality check not working for Silver. specified expecte_or_drop as product_id IS NULL but those record with product id still inserted

Error Reproduce:

Direct Insert into Flow Spec.
INSERT INTO nonprod.meta.silver_dataflowspec_table SELECT '1031' AS dataFlowId ,'A1' AS dataFlowGroup ,'delta' AS sourceFormat ,MAP("database","bronze","table","t_brz_products") AS sourceDetails ,MAP() AS readerConfigOptions ,'delta' AS targetFormat ,MAP("database","silver","table","t_slv_products_dq_log") AS targetDetails ,MAP() AS tableProperties ,Array("*") AS selectExp ,Array("") AS whereClause ,Array("") AS partitionColumns ,null AS cdcApplyChanges ,'{"expect_or_drop": {"null_product_id": "product_id IS NULL"}}' AS dataQualityExpectations ,null AS appendFlows ,MAP() AS appendFlowsSchemas ,'v1' AS version ,current_timestamp AS createDate ,'xxxxxxx' AS createdBy ,current_timestamp AS updateDate ,'xxxxxx'AS updatedBy;

Data Quality Rule:
{"expect_or_drop": {"null_product_id": "product_id IS NULL"}}

Bronze data:
Image

DLT Result:
Image

Silver Table Result: whole set of record is inserted
Image

Metadata

Metadata

Labels

help wantedExtra attention is needed

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions