Skip to content

Google Cloud Storage - Field will be removed when all records are NULL #10230

@apiasak

Description

@apiasak

Environment

  • Airbyte version: 0.35.5-alpha
  • OS Version / Instance: Ubuntu 20.04.3 LTS
  • Deployment: Docker
  • Source Connector and version: Postgres 0.3.13
  • Destination Connector and version: Google Cloud Storage (GCS) 0.1.20
  • Severity: Critical
  • Step where error happened: Sync job

Current Behavior

When any fields of the data sources that have all records as NULL value the field is removed from the destination output.

Expected Behavior

All fields should be remain the same as source even they are all NULL

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG
 2022-02-09 09:37:08 INFO i.a.w.DefaultReplicationWorker(run):135 - One of source or destination thread complete. Waiting on the other. 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded. 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 INFO i.a.i.d.g.w.BaseGcsWriter(close):129 - {} - Uploading remaining data for stream 'cmssupplier'. 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 INFO a.m.s.MultiPartOutputStream(close):158 - {} - Called close() on [MultipartOutputStream for parts 1 - 10000] 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 INFO a.m.s.MultiPartOutputStream(close):158 - {} - Called close() on [MultipartOutputStream for parts 1 - 10000] 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 WARN a.m.s.MultiPartOutputStream(close):160 - {} - [MultipartOutputStream for parts 1 - 10000] is already closed 2022-02-09 09:37:10 destination > 2022-02-09 09:37:10 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - {} - [Manager uploading to ofm-data/test/private/cmssupplier/2022_02_09_1644399415821_0.csv with id ABPnzm5gf...enlB5-jIW]: Finished uploading [Part number 1 containing 11.22 MB] 2022-02-09 09:37:10 destination > 2022-02-09 09:37:10 INFO a.m.s.StreamTransferManager(complete):395 - {} - [Manager uploading to ofm-data/test/private/cmssupplier/2022_02_09_1644399415821_0.csv with id ABPnzm5gf...enlB5-jIW]: Completed 2022-02-09 09:37:10 destination > 2022-02-09 09:37:10 INFO i.a.i.d.g.w.BaseGcsWriter(close):131 - {} - Upload completed for stream 'cmssupplier'. 2022-02-09 09:37:10 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. errors: $: null found, object expected 2022-02-09 09:37:10 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: null 2022-02-09 09:37:10 destination > 2022-02-09 09:37:10 INFO i.a.i.b.IntegrationRunner(run):153 - {} - Completed integration: io.airbyte.integrations.destination.gcs.GcsDestination 2022-02-09 09:37:10 INFO i.a.w.DefaultReplicationWorker(run):137 - Source and destination threads complete. 2022-02-09 09:37:10 INFO i.a.w.DefaultReplicationWorker(run):200 - sync summary: io.airbyte.config.ReplicationAttemptSummary@727e6376[status=completed,recordsSynced=4787,bytesSynced=10166739,startTime=1644399404927,endTime=1644399430545,totalStats=io.airbyte.config.SyncStats@43a9ddfd[recordsEmitted=4787,bytesEmitted=10166739,stateMessagesEmitted=0,recordsCommitted=4787],streamStats=[io.airbyte.config.StreamSyncStats@397cdaca[streamName=cmssupplier,stats=io.airbyte.config.SyncStats@18406df6[recordsEmitted=4787,bytesEmitted=10166739,stateMessagesEmitted=<null>,recordsCommitted=4787]]]] 2022-02-09 09:37:10 INFO i.a.w.DefaultReplicationWorker(run):209 - Source did not output any state messages 2022-02-09 09:37:10 WARN i.a.w.DefaultReplicationWorker(run):217 - State capture: No new state, falling back on input state: io.airbyte.config.State@1924c85b[state={"cdc":false,"streams":[{"cursor":"2022-02-08T10:51:22Z","stream_name":"cmssupplier","cursor_field":["updateon"],"stream_namespace":"officemate"}]}] 2022-02-09 09:37:10 INFO i.a.w.t.TemporalAttemptExecution(get):144 - Stopping cancellation check scheduling... 2022-02-09 09:37:10 INFO i.a.w.t.s.ReplicationActivityImpl(replicate):144 - sync summary: io.airbyte.config.StandardSyncOutput@6ced3974[standardSyncSummary=io.airbyte.config.StandardSyncSummary@1b8abfbf[status=completed,recordsSynced=4787,bytesSynced=10166739,startTime=1644399404927,endTime=1644399430545,totalStats=io.airbyte.config.SyncStats@43a9ddfd[recordsEmitted=4787,bytesEmitted=10166739,stateMessagesEmitted=0,recordsCommitted=4787],streamStats=[io.airbyte.config.StreamSyncStats@397cdaca[streamName=cmssupplier,stats=io.airbyte.config.SyncStats@18406df6[recordsEmitted=4787,bytesEmitted=10166739,stateMessagesEmitted=<null>,recordsCommitted=4787]]]],state=io.airbyte.config.State@1924c85b[state={"cdc":false,"streams":[{"cursor":"2022-02-08T10:51:22Z","stream_name":"cmssupplier","cursor_field":["updateon"],"stream_namespace":"officemate"}]}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@7d3df984[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@2b0483e7[stream=io.airbyte.protocol.models.AirbyteStream@2e27396a[name=cmssupplier,jsonSchema={"type":"object","properties":{"poamt":{"type":"number"},"rebate":{"type":"string"},"sfaxno":{"type":"string"},"staxid":{"type":"string"},"country":{"type":"string"},"prepaid":{"type":"number"},"saddrno":{"type":"string"},"sfaxext":{"type":"string"},"sfaxoth":{"type":"string"},"sstatus":{"type":"string"},"altpayee":{"type":"string"},"bankacct":{"type":"string"},"bankname":{"type":"string"},"branchid":{"type":"string"},"createby":{"type":"string"},"createon":{"type":"string"},"currcode":{"type":"string"},"isimport":{"type":"string"},"leadtime":{"type":"number"},"minorder":{"type":"number"},"saddrsoi":{"type":"string"},"sauthfax":{"type":"string"},"sengname":{"type":"string"},"sfaxarea":{"type":"string"},"sfaxcomm":{"type":"string"},"sphoneno":{"type":"string"},"sreftype":{"type":"string"},"stitleid":{"type":"string"},"svatrate":{"type":"number"},"swebsite":{"type":"string"},"updateby":{"type":"string"},"updateon":{"type":"string"},"saddrroad":{"type":"string"},"sauthcomm":{"type":"string"},"sauthname":{"type":"string"},"sinvaddr1":{"type":"string"},"sinvaddr2":{"type":"string"},"sinvaddr3":{"type":"string"},"sinvaddr4":{"type":"string"},"sinvaddr5":{"type":"string"},"smanageid":{"type":"string"},"sphoneext":{"type":"string"},"sphoneno1":{"type":"string"},"sphoneno2":{"type":"string"},"sphoneoth":{"type":"string"},"ssuffixid":{"type":"string"},"lockreason":{"type":"string"},"lockstatus":{"type":"string"},"saddrfloor":{"type":"string"},"sauthphone":{"type":"string"},"semailaddr":{"type":"string"},"sloadaddr1":{"type":"string"},"sloadaddr2":{"type":"string"},"sloadaddr3":{"type":"string"},"sloadaddr4":{"type":"string"},"sloadfaxno":{"type":"string"},"slocalname":{"type":"string"},"sphonearea":{"type":"string"},"ssuppgrpid":{"type":"string"},"stitlename":{"type":"string"},"creditlimit":{"type":"number"},"saddramphur":{"type":"string"},"saddrtumbon":{"type":"string"},"sphonerefno":{"type":"string"},"ssuffixname":{"type":"string"},"ssupplierid":{"type":"string"},"bankaccttype":{"type":"string"},"creditremark":{"type":"string"},"rebateremark":{"type":"string"},"saddrvillage":{"type":"string"},"saddrzipcode":{"type":"string"},"sloadzipcode":{"type":"string"},"spaymentcode":{"type":"string"},"spaymenttype":{"type":"string"},"saddrbuilding":{"type":"string"},"saddrprovince":{"type":"string"},"sauthphoneext":{"type":"string"},"sauthposition":{"type":"string"},"slastsaledate":{"type":"string"},"sloadphoneno1":{"type":"string"},"sloadphoneno2":{"type":"string"},"ssupplieridod":{"type":"string"},"returncriteria":{"type":"string"},"sauthemailaddr":{"type":"string"},"sdiscountrate1":{"type":"number"},"sdiscountrate2":{"type":"number"},"sdiscountrate3":{"type":"number"},"sfirstsaledate":{"type":"string"},"ssupplieridb2s":{"type":"string"},"bankaccountname":{"type":"string"},"consignmentflag":{"type":"string"},"sbusinesstypeid":{"type":"string"},"bankguaranteeamt":{"type":"number"},"deliverylocation":{"type":"string"},"_airbyte_emitted_at":{"type":"string"},"aptrancontrolnature":{"type":"string"},"bankguaranteeremark":{"type":"string"},"aptrancontrolactcode":{"type":"string"},"bankguaranteeexpdate":{"type":"string"},"returncriteriaremark":{"type":"string"},"deliverylocationremark":{"type":"string"},"_airbyte_cmssupplier_hashid":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[updateon],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}]] 2022-02-09 09:37:10 INFO i.a.c.p.ConfigRepository(updateConnectionState):545 - Updating connection eb63f52f-c184-458a-bf61-794c841dcf1b state: io.airbyte.config.State@7de9860b[state={"cdc":false,"streams":[{"cursor":"2022-02-08T10:51:22Z","stream_name":"cmssupplier","cursor_field":["updateon"],"stream_namespace":"officemate"}]}] 

Steps to Reproduce

  1. Create source from Postgres
| COL1 | COL2 | COL3 | ---------------------- | A | NULL | 1 | | B | NULL | 1 | | C | NULL | 1 | 
  1. Set destination as Google Cloud Storage and format is jsonl
  2. Run the job, the COL2 will be missing from the output.

Metadata

Metadata

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions