Skip to content
This repository was archived by the owner on May 17, 2024. It is now read-only.
This repository was archived by the owner on May 17, 2024. It is now read-only.

data-diff does not work with Redshift views #590

@oleksii-nikitchuk

Description

@oleksii-nikitchuk

Describe the bug
We have dbt models materialized as views in Redshift and when I try to run diff I get an error:

[10:45:11] INFO - Parsing file dbt_project.yml [10:45:11] INFO - Parsing file target/manifest.json [10:45:12] INFO - Parsing file target/run_results.json Running with data-diff=0.7.7 [10:45:12] INFO - Parsing file /Users/oleksii.nikitchuk/.dbt/profiles.yml [10:45:12] DEBUG - Found PKs via META: ['record_id'] [10:45:12] INFO - [Redshift] Starting a threadpool, size=4. [10:45:12] DEBUG - Running SQL (Redshift): SET TIME ZONE 'UTC' [10:45:13] DEBUG - Running SQL (Redshift): SELECT column_name, data_type, datetime_precision, numeric_precision, numeric_scale FROM warehouse.information_schema.columns WHERE table_name = 'quote' AND table_schema = 'data_staging_dev' [10:45:14] DEBUG - Running SQL (Redshift): SELECT columnname AS column_name , CASE WHEN external_type = 'string' THEN 'varchar' ELSE external_type END AS data_type , NULL AS datetime_precision , NULL AS numeric_precision , NULL AS numeric_scale FROM svv_external_columns WHERE tablename = 'quote' AND schemaname = 'data_staging_dev' AND redshift_database_name = 'warehouse' [10:45:14] ERROR - query_pg_get_cols() missing 1 required positional argument: 'path' Traceback (most recent call last): File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/sqeleton/databases/redshift.py", line 159, in query_table_schema return super().query_table_schema(path) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/sqeleton/databases/base.py", line 416, in query_table_schema raise RuntimeError(f"{self.name}: Table '{'.'.join(path)}' does not exist, or has no columns") RuntimeError: Redshift: Table 'warehouse.data_staging_dev.quote' does not exist, or has no columns During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/sqeleton/databases/redshift.py", line 162, in query_table_schema return self.query_external_table_schema(path) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/sqeleton/databases/redshift.py", line 117, in query_external_table_schema raise RuntimeError(f"{self.name}: Table '{'.'.join(path)}' does not exist, or has no columns") RuntimeError: Redshift: Table 'warehouse.data_staging_dev.quote' does not exist, or has no columns During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/oleksii.nikitchuk/.pyenv/versions/dbt-venv/bin/data-diff", line 8, in <module> sys.exit(main()) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/__main__.py", line 270, in main dbt_diff( File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/dbt.py", line 117, in dbt_diff _local_diff(diff_vars) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/dbt.py", line 194, in _local_diff table1_columns = list(table1.get_schema()) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/table_segment.py", line 152, in get_schema return self.database.query_table_schema(self.table_path) File "/Users/oleksii.nikitchuk/.pyenv/versions/3.9.15/envs/dbt-venv/lib/python3.9/site-packages/data_diff/sqeleton/databases/redshift.py", line 164, in query_table_schema return self.query_pg_get_cols() TypeError: query_pg_get_cols() missing 1 required positional argument: 'path' 

Describe the environment

  • data-diff: v0.7.7
  • dbt-redshift: 1.4.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    --dbtIssues/features related to the dbt integrationLinearbugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions