Skip to content

Schema conflict when storing dataframes with datetime objects using load_table_from_dataframe() #2214

@tatome

Description

@tatome

The output

Traceback (most recent call last): File "/tmp/mwe.py", line 14, in <module> print(job.result()) File "/home/tatome/anaconda3/envs/default_env/lib/python3.6/site-packages/google/cloud/bigquery/job.py", line 688, in result return super(_AsyncJob, self).result(timeout=timeout) File "/home/tatome/anaconda3/envs/default_env/lib/python3.6/site-packages/google/api_core/future/polling.py", line 120, in result raise self._exception google.api_core.exceptions.BadRequest: 400 Provided Schema does not match Table [project+dataset].test. Field test_column has changed type from DATETIME to TIMESTAMP. 

is produced by the following code:

import pandas as pd from google.cloud.bigquery import Client client = Client(project='[project]') table_name = 'test' job = client.query(f'CREATE TABLE [dataset].{table_name} ( idx INT64, test_column DATETIME )') job.result() df = pd.DataFrame({'idx' : [0,1], 'test_column' : ['2018-12-24', '2019-01-01']}) df['test_column'] = pd.to_datetime(df['test_column']).values.astype('datetime64[ms]') dataset_ref = client.dataset('[dataset]') table_ref = dataset_ref.table(table_name) df = df.set_index('idx') job = client.load_table_from_dataframe(df, table_ref, location='EU') print(job.result()) 

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage meI really want to be triaged.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions