- Notifications
You must be signed in to change notification settings - Fork 4.9k
Closed
Labels
area/connectorsConnector related issuesConnector related issuescommunityconnectors/destination/gcstype/bugSomething isn't workingSomething isn't working
Description
Environment
- Airbyte version: 0.35.5-alpha
- OS Version / Instance: Ubuntu 20.04.3 LTS
- Deployment: Docker
- Source Connector and version: Postgres 0.3.13
- Destination Connector and version: Google Cloud Storage (GCS) 0.1.20
- Severity: Critical
- Step where error happened: Sync job
Current Behavior
When any fields of the data sources that have all records as NULL value the field is removed from the destination output.
Expected Behavior
All fields should be remain the same as source even they are all NULL
Logs
If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.
LOG
2022-02-09 09:37:08 INFO i.a.w.DefaultReplicationWorker(run):135 - One of source or destination thread complete. Waiting on the other. 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded. 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 INFO i.a.i.d.g.w.BaseGcsWriter(close):129 - {} - Uploading remaining data for stream 'cmssupplier'. 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 INFO a.m.s.MultiPartOutputStream(close):158 - {} - Called close() on [MultipartOutputStream for parts 1 - 10000] 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 INFO a.m.s.MultiPartOutputStream(close):158 - {} - Called close() on [MultipartOutputStream for parts 1 - 10000] 2022-02-09 09:37:08 destination > 2022-02-09 09:37:08 WARN a.m.s.MultiPartOutputStream(close):160 - {} - [MultipartOutputStream for parts 1 - 10000] is already closed 2022-02-09 09:37:10 destination > 2022-02-09 09:37:10 INFO a.m.s.StreamTransferManager(uploadStreamPart):558 - {} - [Manager uploading to ofm-data/test/private/cmssupplier/2022_02_09_1644399415821_0.csv with id ABPnzm5gf...enlB5-jIW]: Finished uploading [Part number 1 containing 11.22 MB] 2022-02-09 09:37:10 destination > 2022-02-09 09:37:10 INFO a.m.s.StreamTransferManager(complete):395 - {} - [Manager uploading to ofm-data/test/private/cmssupplier/2022_02_09_1644399415821_0.csv with id ABPnzm5gf...enlB5-jIW]: Completed 2022-02-09 09:37:10 destination > 2022-02-09 09:37:10 INFO i.a.i.d.g.w.BaseGcsWriter(close):131 - {} - Upload completed for stream 'cmssupplier'. 2022-02-09 09:37:10 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed. errors: $: null found, object expected 2022-02-09 09:37:10 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: null 2022-02-09 09:37:10 destination > 2022-02-09 09:37:10 INFO i.a.i.b.IntegrationRunner(run):153 - {} - Completed integration: io.airbyte.integrations.destination.gcs.GcsDestination 2022-02-09 09:37:10 INFO i.a.w.DefaultReplicationWorker(run):137 - Source and destination threads complete. 2022-02-09 09:37:10 INFO i.a.w.DefaultReplicationWorker(run):200 - sync summary: io.airbyte.config.ReplicationAttemptSummary@727e6376[status=completed,recordsSynced=4787,bytesSynced=10166739,startTime=1644399404927,endTime=1644399430545,totalStats=io.airbyte.config.SyncStats@43a9ddfd[recordsEmitted=4787,bytesEmitted=10166739,stateMessagesEmitted=0,recordsCommitted=4787],streamStats=[io.airbyte.config.StreamSyncStats@397cdaca[streamName=cmssupplier,stats=io.airbyte.config.SyncStats@18406df6[recordsEmitted=4787,bytesEmitted=10166739,stateMessagesEmitted=<null>,recordsCommitted=4787]]]] 2022-02-09 09:37:10 INFO i.a.w.DefaultReplicationWorker(run):209 - Source did not output any state messages 2022-02-09 09:37:10 WARN i.a.w.DefaultReplicationWorker(run):217 - State capture: No new state, falling back on input state: io.airbyte.config.State@1924c85b[state={"cdc":false,"streams":[{"cursor":"2022-02-08T10:51:22Z","stream_name":"cmssupplier","cursor_field":["updateon"],"stream_namespace":"officemate"}]}] 2022-02-09 09:37:10 INFO i.a.w.t.TemporalAttemptExecution(get):144 - Stopping cancellation check scheduling... 2022-02-09 09:37:10 INFO i.a.w.t.s.ReplicationActivityImpl(replicate):144 - sync summary: io.airbyte.config.StandardSyncOutput@6ced3974[standardSyncSummary=io.airbyte.config.StandardSyncSummary@1b8abfbf[status=completed,recordsSynced=4787,bytesSynced=10166739,startTime=1644399404927,endTime=1644399430545,totalStats=io.airbyte.config.SyncStats@43a9ddfd[recordsEmitted=4787,bytesEmitted=10166739,stateMessagesEmitted=0,recordsCommitted=4787],streamStats=[io.airbyte.config.StreamSyncStats@397cdaca[streamName=cmssupplier,stats=io.airbyte.config.SyncStats@18406df6[recordsEmitted=4787,bytesEmitted=10166739,stateMessagesEmitted=<null>,recordsCommitted=4787]]]],state=io.airbyte.config.State@1924c85b[state={"cdc":false,"streams":[{"cursor":"2022-02-08T10:51:22Z","stream_name":"cmssupplier","cursor_field":["updateon"],"stream_namespace":"officemate"}]}],outputCatalog=io.airbyte.protocol.models.ConfiguredAirbyteCatalog@7d3df984[streams=[io.airbyte.protocol.models.ConfiguredAirbyteStream@2b0483e7[stream=io.airbyte.protocol.models.AirbyteStream@2e27396a[name=cmssupplier,jsonSchema={"type":"object","properties":{"poamt":{"type":"number"},"rebate":{"type":"string"},"sfaxno":{"type":"string"},"staxid":{"type":"string"},"country":{"type":"string"},"prepaid":{"type":"number"},"saddrno":{"type":"string"},"sfaxext":{"type":"string"},"sfaxoth":{"type":"string"},"sstatus":{"type":"string"},"altpayee":{"type":"string"},"bankacct":{"type":"string"},"bankname":{"type":"string"},"branchid":{"type":"string"},"createby":{"type":"string"},"createon":{"type":"string"},"currcode":{"type":"string"},"isimport":{"type":"string"},"leadtime":{"type":"number"},"minorder":{"type":"number"},"saddrsoi":{"type":"string"},"sauthfax":{"type":"string"},"sengname":{"type":"string"},"sfaxarea":{"type":"string"},"sfaxcomm":{"type":"string"},"sphoneno":{"type":"string"},"sreftype":{"type":"string"},"stitleid":{"type":"string"},"svatrate":{"type":"number"},"swebsite":{"type":"string"},"updateby":{"type":"string"},"updateon":{"type":"string"},"saddrroad":{"type":"string"},"sauthcomm":{"type":"string"},"sauthname":{"type":"string"},"sinvaddr1":{"type":"string"},"sinvaddr2":{"type":"string"},"sinvaddr3":{"type":"string"},"sinvaddr4":{"type":"string"},"sinvaddr5":{"type":"string"},"smanageid":{"type":"string"},"sphoneext":{"type":"string"},"sphoneno1":{"type":"string"},"sphoneno2":{"type":"string"},"sphoneoth":{"type":"string"},"ssuffixid":{"type":"string"},"lockreason":{"type":"string"},"lockstatus":{"type":"string"},"saddrfloor":{"type":"string"},"sauthphone":{"type":"string"},"semailaddr":{"type":"string"},"sloadaddr1":{"type":"string"},"sloadaddr2":{"type":"string"},"sloadaddr3":{"type":"string"},"sloadaddr4":{"type":"string"},"sloadfaxno":{"type":"string"},"slocalname":{"type":"string"},"sphonearea":{"type":"string"},"ssuppgrpid":{"type":"string"},"stitlename":{"type":"string"},"creditlimit":{"type":"number"},"saddramphur":{"type":"string"},"saddrtumbon":{"type":"string"},"sphonerefno":{"type":"string"},"ssuffixname":{"type":"string"},"ssupplierid":{"type":"string"},"bankaccttype":{"type":"string"},"creditremark":{"type":"string"},"rebateremark":{"type":"string"},"saddrvillage":{"type":"string"},"saddrzipcode":{"type":"string"},"sloadzipcode":{"type":"string"},"spaymentcode":{"type":"string"},"spaymenttype":{"type":"string"},"saddrbuilding":{"type":"string"},"saddrprovince":{"type":"string"},"sauthphoneext":{"type":"string"},"sauthposition":{"type":"string"},"slastsaledate":{"type":"string"},"sloadphoneno1":{"type":"string"},"sloadphoneno2":{"type":"string"},"ssupplieridod":{"type":"string"},"returncriteria":{"type":"string"},"sauthemailaddr":{"type":"string"},"sdiscountrate1":{"type":"number"},"sdiscountrate2":{"type":"number"},"sdiscountrate3":{"type":"number"},"sfirstsaledate":{"type":"string"},"ssupplieridb2s":{"type":"string"},"bankaccountname":{"type":"string"},"consignmentflag":{"type":"string"},"sbusinesstypeid":{"type":"string"},"bankguaranteeamt":{"type":"number"},"deliverylocation":{"type":"string"},"_airbyte_emitted_at":{"type":"string"},"aptrancontrolnature":{"type":"string"},"bankguaranteeremark":{"type":"string"},"aptrancontrolactcode":{"type":"string"},"bankguaranteeexpdate":{"type":"string"},"returncriteriaremark":{"type":"string"},"deliverylocationremark":{"type":"string"},"_airbyte_cmssupplier_hashid":{"type":"string"}}},supportedSyncModes=[full_refresh, incremental],sourceDefinedCursor=<null>,defaultCursorField=[],sourceDefinedPrimaryKey=[],namespace=<null>,additionalProperties={}],syncMode=full_refresh,cursorField=[updateon],destinationSyncMode=append,primaryKey=[],additionalProperties={}]],additionalProperties={}]] 2022-02-09 09:37:10 INFO i.a.c.p.ConfigRepository(updateConnectionState):545 - Updating connection eb63f52f-c184-458a-bf61-794c841dcf1b state: io.airbyte.config.State@7de9860b[state={"cdc":false,"streams":[{"cursor":"2022-02-08T10:51:22Z","stream_name":"cmssupplier","cursor_field":["updateon"],"stream_namespace":"officemate"}]}] Steps to Reproduce
- Create source from Postgres
| COL1 | COL2 | COL3 | ---------------------- | A | NULL | 1 | | B | NULL | 1 | | C | NULL | 1 | - Set destination as Google Cloud Storage and format is jsonl
- Run the job, the COL2 will be missing from the output.
Metadata
Metadata
Assignees
Labels
area/connectorsConnector related issuesConnector related issuescommunityconnectors/destination/gcstype/bugSomething isn't workingSomething isn't working