docs: Liquibase samples and tests (#291)

* docs: add sample for Liquibase * docs: add sample for Liquibase * config: update liquibase config * feat: support show search_path * fix: only include public in search path * feat: add support for set search_path * feat: add support for set search_path * feat: auto-convert explicit DDL transactions to batches Automatically converts explicit transactions that contain blocks of DDL statements into DDL batches. This increases the 'atomicity' of the DDL blocks, and improves the execution speed of the DDL statements. * fix: remove special handling of UNKNOWN We can safely remove the special handling of UNKNOWN statements, as the case where there is no command tag is already handled at the beginning of the method. * test: add more tests + fix test failures * test: add tests for batching errors * docs: add todo to remove later * fix: convert partitioned_table metadata query * test: add test with multiple DDL batches in one transaction * test: add test for running script file with psql * fix: add more translations for Liquibase * feat: set default PG version reported by PGAdapter to 14.1 * docs: add more Liquibase sample change sets * docs: add custom SQL change set sample * test: add integration test * fix: revert liquibase.properties to default values * revert: remove ApplicationDetector * feat: pg_catalog tables as CTEs * feat: add pg_catalog tables * feat: combine pg_settings with pg_catalog * refactor: combine StatementParser and SimpleParser Combines the StatementParser and SimpleParser into one parser to reduce code duplication and to make the SimpleParser more complete, as it now also supports skipping comments and dollar-quoted strings. * test: add more tests + fix bug in update transformer * fix: correctly skip CTEs * fix: only add each CTE once * test: add integration tests * feat: make pg_catalog replacements optional * test: add more tests * fix: parse join statements * test: fix integration test * feat: add ddl_transaction_mode and replace_pg_catalog_tables settings Adds the ddl_transaction_mode and replace_pg_catalog_tables settings from a purely server setting to one that can be set in session state. This makes the features a lot more usable, as they can be set on a per-session basis, and also set in for example the connection string. This will for example enable the use of DDL transaction mode Explicit for Liquibase. * test: add more tests * test: add 14.1 results to tests * test: fix Python tests + add clirr ignore * fix: update Liquibase properties * test: update metadata queries + do not replace pg_catalog in test * fix: JDBC metadata + psql metadata commands * fix: add new patterns for new server version * fix: add more matches * fix: add more matches * fix: further matches * fix: matcher for v11 * chore: remove unused code * fix: remove linux qualifier from version + print version at startup * chore: add more logging + cleanup comments * fix: remove debian reference in expected version * test: add more logging to figure out why the build is stuck * test: try connecting with domain socket * test: try running mvn test * test: try with batch mode * fix: use batch mode + add tags to db change log * fix: invalid rollback command * fix: concerts should have been deleted + change log should remain after rollback * docs: add more samples * fix: remove PG version bump from Liquibase PR * fix: add a result for the correct v1.0 query * test: try with server version 14.1 * fix: guess date type as JDBC skips this * chore: remove unused statement

Author: olavloite

samples/java/liquibase/README.md

@@ -0,0 +1,156 @@
+# PGAdapter and Liquibase
+
+PGAdapter can be used in combination with Liquibase, but with a number of limitations. This sample
+shows the command line arguments and configuration that is needed in order to use Liquibase with
+PGAdapter.
+
+## Liquibase Requirements
+
+* The JDBC Connection URL must include `options=-c%20spanner.ddl_transaction_mode=AutocommitExplicitTransaction`.
+ See [liquibase.properties](liquibase.properties) for an example URL. Cloud Spanner does not
+ support DDL transactions, and this will ensure that DDL transactions are automatically
+ converted to DDL batches. See [DDL options](../../../docs/ddl.md) for more information.
+* The `databasechangeloglock` and `databasechangelog` tables **must** be created manually, as the
+ DDL script that is automatically generated by Liquibase will try to use the data type
+ `timestamp without time zone`, which is not supported by Cloud Spanner. The DDL script to create
+ these tables manually can be found in
+ [create_database_change_log.sql](create_database_change_log.sql).
+
+## Running the Sample
+
+All the steps below assume that the commands are executed from this directory.
+
+1. Start PGAdapter.
+
+```shell
+wget https://storage.googleapis.com/pgadapter-jar-releases/pgadapter.tar.gz && tar -xzvf pgadapter.tar.gz
+java -jar pgadapter.jar -p my-project -i my-instance
+```
+
+2. Manually create the `databasechangeloglock` and `databasechangelog` tables. These need to be manually
+ created because the default table definition that is generated by Liquibase use `timestamp without time zone`
+ instead of `timestamp with time zone`, and it would create a table without a primary key.
+ Replace `my-database` with the actual name of your database.
+
+```shell
+psql -h localhost -d my-database -f create_database_change_log.sql
+```
+
+3. Modify the `liquibase.properties` file in this directory to point to your database:
+
+```
+changeLogFile: dbchangelog.xml
+url: jdbc:postgresql://localhost:5432/my-database?options=-c%20spanner.ddl_transaction_mode=AutocommitExplicitTransaction
+```
+
+4. Update the database to according to the changes in `dbchangelog.xml` by executing:
+
+```shell
+mvn liquibase:update
+```
+
+## Frequently Asked Questions
+See [frequently asked questions](faq.md) for a list of common issues and questions.
+
+## Supported Change Types
+The following change types are supported by PGAdapter (see the [dbchangelog.xml](dbchangelog.xml)
+for examples). Note that some change types have some limitations. See [limitations](#limitations)
+for a specific list.
+
+### Misc
+* sql: https://docs.liquibase.com/change-types/sql.html
+* tagDatabase: https://docs.liquibase.com/change-types/tag-database.html
+
+### Entities
+* createTable: https://docs.liquibase.com/change-types/create-table.html
+* createIndex: https://docs.liquibase.com/change-types/create-index.html
+* createView: https://docs.liquibase.com/change-types/create-view.html
+* addColumn: https://docs.liquibase.com/change-types/add-column.html
+* dropColumn: https://docs.liquibase.com/change-types/drop-column.html
+* dropIndex: https://docs.liquibase.com/change-types/drop-index.html
+* dropTable: https://docs.liquibase.com/change-types/drop-table.html
+* dropView: https://docs.liquibase.com/change-types/drop-view.html
+
+### Constraints
+* addDefaultValue: https://docs.liquibase.com/change-types/add-default-value.html
+* addForeignKeyConstraint: https://docs.liquibase.com/change-types/add-foreign-key-constraint.html
+* addNotNullConstraint: https://docs.liquibase.com/change-types/add-not-null-constraint.html
+* dropDefaultValue: https://docs.liquibase.com/change-types/drop-default-value.html
+* dropForeignKeyConstraint: https://docs.liquibase.com/change-types/drop-foreign-key-constraint.html
+* dropNotNullConstraint: https://docs.liquibase.com/change-types/drop-foreign-key-constraint.html
+
+### Data
+* loadData: https://docs.liquibase.com/change-types/load-data.html
+* insert: https://docs.liquibase.com/change-types/insert.html
+* update: https://docs.liquibase.com/change-types/update.html
+* delete: https://docs.liquibase.com/change-types/update.html
+
+## Limitations
+The Cloud Spanner PostgreSQL dialect supports a [subset of the PostgreSQL DDL dialect](https://cloud.google.com/spanner/docs/reference/postgresql/data-definition-language).
+This means that not all change types that are supported by Liquibase for PostgreSQL can be used with
+Cloud Spanner.
+
+### Supported Change Types with Limitations
+The following change types are supported, but with limitations:
+* createTable: The createTable change set __must include a primary key constraint__, and the name of the
+ primary key constraint must be `pk_<table_name>`. See the examples in [dbchangelog.xml](dbchangelog.xml).
+* createView: The `create view` statement must include a `sql security invoker`. Set `fullDefinition="true"`
+ in the change set and include `create view <view_name> sql security invoker as ...` as thew view
+ definition.
+* dropColumn: Only one column can be dropped per change set.
+* dropTable: Cascade constraints is not supported. All secondary indexes on the table must be dropped
+ before the table can be dropped.
+* add*Constraint: The "disabled" and "validate" properties are not supported. Constraints are always
+ enabled and validated.
+* addForeignKeyConstraint: The "deferrable", "initiallyDeferred", "onDelete", "onUpdate" and "validate"
+ properties are not supported. Foreign key constraints are always validated, non-deferrable and
+ do not support any cascading options.
+* insert, update, delete, loadData: The number of mutations may not exceed the mutation limit
+ (https://cloud.google.com/spanner/quotas#limits_for_creating_reading_updating_and_deleting_data)
+
+### Unsupported Change Types
+
+The following change types are __not__ supported:
+
+#### Entities
+* addAutoIncrement: https://docs.liquibase.com/change-types/add-auto-increment.html
+* alterSequence: https://docs.liquibase.com/change-types/alter-sequence.html
+* createFunction: https://docs.liquibase.com/change-types/create-function.html
+* createPackage: https://docs.liquibase.com/change-types/create-package.html
+* createPackageBody: https://docs.liquibase.com/change-types/create-package-body.html
+* createProcedure: https://docs.liquibase.com/change-types/create-procedure.html
+* createSequence: https://docs.liquibase.com/change-types/create-sequence.html
+* createSynonym: https://docs.liquibase.com/change-types/create-synonym.html
+* createTrigger: https://docs.liquibase.com/change-types/create-trigger.html
+* disableTrigger: https://docs.liquibase.com/change-types/disable-trigger.html
+* dropFunction: https://docs.liquibase.com/change-types/drop-function.html
+* dropPackage: https://docs.liquibase.com/change-types/drop-package.html
+* dropPackageBody: https://docs.liquibase.com/change-types/drop-package-body.html
+* dropProcedure: https://docs.liquibase.com/change-types/drop-procedure.html
+* dropSequence: https://docs.liquibase.com/change-types/drop-sequence.html
+* dropSynonym: https://docs.liquibase.com/change-types/drop-synonym.html
+* dropTrigger: https://docs.liquibase.com/change-types/drop-trigger.html
+* enableTrigger: https://docs.liquibase.com/change-types/enable-trigger.html
+* renameColumn: https://docs.liquibase.com/change-types/rename-column.html
+* renameSequence: https://docs.liquibase.com/change-types/rename-sequence.html
+* renameTable: https://docs.liquibase.com/change-types/rename-table.html
+* renameTrigger: https://docs.liquibase.com/change-types/rename-trigger.html
+* renameView: https://docs.liquibase.com/change-types/rename-view.html
+* setColumnRemarks: https://docs.liquibase.com/change-types/set-column-remarks.html
+* setTableRemarks: https://docs.liquibase.com/change-types/set-table-remarks.html
+
+#### Constraints
+* addNotNullConstraint: https://docs.liquibase.com/change-types/add-not-null-constraint.html
+* addPrimaryKey: https://docs.liquibase.com/change-types/add-primary-key.html
+* addUniqueConstraint: https://docs.liquibase.com/change-types/add-unique-constraint.html
+* disableCheckConstraint: https://docs.liquibase.com/change-types/disable-check-constraint.html
+* dropAllForeignKeyConstraints: https://docs.liquibase.com/change-types/drop-all-foreign-key-constraints.html
+* dropNotNull: https://docs.liquibase.com/change-types/drop-not-null-constraint.html
+* dropPrimaryKey: https://docs.liquibase.com/change-types/drop-primary-key.html
+* dropUniqueConstraint: https://docs.liquibase.com/change-types/drop-unique-constraint.html
+* enableCheckConstraint: https://docs.liquibase.com/change-types/enable-check-constraint.html
+
+#### Data
+* addLookupTable: https://docs.liquibase.com/change-types/add-lookup-table.html
+* loadUpdateData: https://docs.liquibase.com/change-types/load-update-data.html
+* modifyDataType: https://docs.liquibase.com/change-types/modify-data-type.html

samples/java/liquibase/create_database_change_log.sql

@@ -0,0 +1,28 @@
+START BATCH DDL;
+
+CREATE TABLE databasechangeloglock (
+ ID INTEGER NOT NULL,
+ LOCKED BOOLEAN NOT NULL,
+ LOCKGRANTED TIMESTAMPTZ,
+ LOCKEDBY VARCHAR(255),
+ PRIMARY KEY (ID)
+);
+
+CREATE TABLE databasechangelog (
+ ID VARCHAR(255) NOT NULL PRIMARY KEY,
+ AUTHOR VARCHAR(255) NOT NULL,
+ FILENAME VARCHAR(255) NOT NULL,
+ DATEEXECUTED TIMESTAMPTZ NOT NULL,
+ ORDEREXECUTED INTEGER NOT NULL,
+ EXECTYPE VARCHAR(10) NOT NULL,
+ MD5SUM VARCHAR(35),
+ DESCRIPTION VARCHAR(255),
+ COMMENTS VARCHAR(255),
+ TAG VARCHAR(255),
+ LIQUIBASE VARCHAR(20),
+ CONTEXTS VARCHAR(255),
+ LABELS VARCHAR(255),
+ DEPLOYMENT_ID VARCHAR(10)
+);
+
+RUN BATCH;

samples/java/liquibase/data/albums.csv

@@ -0,0 +1,10 @@
+album_id;singer_id;title
+0e3f2ac2-28e9-47b2-ba24-3086b04bd6f3;014a1c27-a8af-4911-bb71-6da16a540df1;Entwined Dreamer
+337d49a3-e427-4148-9c2f-4e7cc33e851e;014a1c27-a8af-4911-bb71-6da16a540df1;Sharp Star
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;014a1c27-a8af-4911-bb71-6da16a540df1;The Final Pirates
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;014a1c27-a8af-4911-bb71-6da16a540df1;Tale of Wizards
+c3f12acb-8311-4383-9a3b-5023c810e435;014a1c27-a8af-4911-bb71-6da16a540df1;The Rainbow's Soul
+4d477fe0-c200-41c1-8046-345febb22359;19cbe87e-2306-452f-b46f-a2822719dfc0;Snow of Rainbow
+0bbfca2e-555b-4385-abd4-e1c85ad92336;a8c06393-a969-4c49-b65b-decc338bf945;The Servant's Force
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;020f9059-e1b4-46b3-8fdd-332dfb8109f4;The Future of the Voyages
+60565717-a4cc-4e35-a3f8-92522b6af64c;710f3b20-285f-434d-a0b1-2b374e95b97f;Search in the Petals

samples/java/liquibase/data/all_types.csv

@@ -0,0 +1,5 @@
+col_bigint;col_bool;col_bytea;col_float8;col_int;col_numeric;col_timestamptz;col_date;col_varchar;col_text
+1;true;cm93MQ==;3.14;100;6.626;2022-08-08T07:51:13.248632+02:00;2022-08-08;"varchar1";"text1's"
+2;false;"";0.0;200;0.0;2000-01-01T00:00:00Z;2000-01-01;"";""
+3;true;cm93Mw==;-3.14;300;-6.626;1953-06-01T20:18:59.123456789-09:00;1953-06-01;varchar3;text3
+4;null;null;null;null;null;null;null;null;null

samples/java/liquibase/data/singers.csv

@@ -0,0 +1,6 @@
+singer_id;first_name;last_name;active
+014a1c27-a8af-4911-bb71-6da16a540df1;Man;Mcconnell;true
+19cbe87e-2306-452f-b46f-a2822719dfc0;Howard;Farmer;true
+a8c06393-a969-4c49-b65b-decc338bf945;Isaias;Bowman;false
+020f9059-e1b4-46b3-8fdd-332dfb8109f4;Alfreda;Pena;true
+710f3b20-285f-434d-a0b1-2b374e95b97f;Jacqueline;Roberson;true

samples/java/liquibase/data/tracks.csv

@@ -0,0 +1,79 @@
+album_id;track_number;title
+0e3f2ac2-28e9-47b2-ba24-3086b04bd6f3;1;Track 1
+0e3f2ac2-28e9-47b2-ba24-3086b04bd6f3;2;Track 2
+0e3f2ac2-28e9-47b2-ba24-3086b04bd6f3;3;Track 3
+0e3f2ac2-28e9-47b2-ba24-3086b04bd6f3;4;Track 4
+0e3f2ac2-28e9-47b2-ba24-3086b04bd6f3;5;Track 5
+0e3f2ac2-28e9-47b2-ba24-3086b04bd6f3;6;Track 6
+0e3f2ac2-28e9-47b2-ba24-3086b04bd6f3;7;Track 7
+337d49a3-e427-4148-9c2f-4e7cc33e851e;1;Track 1
+337d49a3-e427-4148-9c2f-4e7cc33e851e;2;Track 2
+337d49a3-e427-4148-9c2f-4e7cc33e851e;3;Track 3
+337d49a3-e427-4148-9c2f-4e7cc33e851e;4;Track 4
+337d49a3-e427-4148-9c2f-4e7cc33e851e;5;Track 5
+337d49a3-e427-4148-9c2f-4e7cc33e851e;6;Track 6
+337d49a3-e427-4148-9c2f-4e7cc33e851e;7;Track 7
+337d49a3-e427-4148-9c2f-4e7cc33e851e;8;Track 8
+337d49a3-e427-4148-9c2f-4e7cc33e851e;9;Track 9
+337d49a3-e427-4148-9c2f-4e7cc33e851e;10;Track 10
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;1;Track 1
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;2;Track 2
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;3;Track 3
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;4;Track 4
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;5;Track 5
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;6;Track 6
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;7;Track 7
+363e468d-9fc0-41a8-8275-11ce22b0e9fc;8;Track 8
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;1;Track 1
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;2;Track 2
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;3;Track 3
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;4;Track 4
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;5;Track 5
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;6;Track 6
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;7;Track 7
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;8;Track 8
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;9;Track 9
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;10;Track 10
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;11;Track 11
+ff88dd04-2ca9-40df-9478-c3c27f6e97bf;12;Track 12
+c3f12acb-8311-4383-9a3b-5023c810e435;1;Track 1
+c3f12acb-8311-4383-9a3b-5023c810e435;2;Track 2
+c3f12acb-8311-4383-9a3b-5023c810e435;3;Track 3
+c3f12acb-8311-4383-9a3b-5023c810e435;4;Track 4
+c3f12acb-8311-4383-9a3b-5023c810e435;5;Track 5
+c3f12acb-8311-4383-9a3b-5023c810e435;6;Track 6
+4d477fe0-c200-41c1-8046-345febb22359;1;Track 1
+4d477fe0-c200-41c1-8046-345febb22359;2;Track 2
+4d477fe0-c200-41c1-8046-345febb22359;3;Track 3
+4d477fe0-c200-41c1-8046-345febb22359;4;Track 4
+4d477fe0-c200-41c1-8046-345febb22359;5;Track 5
+4d477fe0-c200-41c1-8046-345febb22359;6;Track 6
+4d477fe0-c200-41c1-8046-345febb22359;7;Track 7
+4d477fe0-c200-41c1-8046-345febb22359;8;Track 8
+4d477fe0-c200-41c1-8046-345febb22359;9;Track 9
+0bbfca2e-555b-4385-abd4-e1c85ad92336;1;Track 1
+0bbfca2e-555b-4385-abd4-e1c85ad92336;2;Track 2
+0bbfca2e-555b-4385-abd4-e1c85ad92336;3;Track 3
+0bbfca2e-555b-4385-abd4-e1c85ad92336;4;Track 4
+0bbfca2e-555b-4385-abd4-e1c85ad92336;5;Track 5
+0bbfca2e-555b-4385-abd4-e1c85ad92336;6;Track 6
+0bbfca2e-555b-4385-abd4-e1c85ad92336;7;Track 7
+0bbfca2e-555b-4385-abd4-e1c85ad92336;8;Track 8
+0bbfca2e-555b-4385-abd4-e1c85ad92336;9;Track 9
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;1;Track 1
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;2;Track 2
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;3;Track 3
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;4;Track 4
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;5;Track 5
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;6;Track 6
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;7;Track 7
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;8;Track 8
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;9;Track 9
+35186bd5-2ef5-40c2-bd02-8eb0eceb6304;10;Track 10
+60565717-a4cc-4e35-a3f8-92522b6af64c;1;Track 1
+60565717-a4cc-4e35-a3f8-92522b6af64c;2;Track 2
+60565717-a4cc-4e35-a3f8-92522b6af64c;3;Track 3
+60565717-a4cc-4e35-a3f8-92522b6af64c;4;Track 4
+60565717-a4cc-4e35-a3f8-92522b6af64c;5;Track 5
+60565717-a4cc-4e35-a3f8-92522b6af64c;6;Track 6
+60565717-a4cc-4e35-a3f8-92522b6af64c;7;Track 7