BigQuery Storage API sample for reading pandas dataframe #1994
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
tswast added the do not merge 
googlebot added the cla: yes tswast removed the do not merge tswast force-pushed the tswast-bqstorage-pandas branch from 0eaca4e to a3a48c0 Compare 
shollyman approved these changes Feb 5, 2019
| # [START bigquerystorage_pandas_read_query_results] | ||
| import uuid | ||
| | ||
| # Due to a known issue in the BigQuery Storage API (TODO: link to |
There was a problem hiding this comment.
Should we consider simply running a large query that emits enough data to avoid the inline? Pros: better demonstrates the perf of the new API, and avoids us having to revisit the sample. Cons: test time overhead and potential pitfalls for people kicking tires with small results. Part of this is dependent on how the team will be maintaining their KI list.
There was a problem hiding this comment.
Cons: test time overhead.
Long test time is my main reason for avoiding queries that return big results. I guess it's not so bad since this repo can test the different directories independently.
Cons: potential pitfalls for people kicking tires with small results.
The current failure case is rather bad: it returns a successful response, but gives you an empty result set. That's a pretty big pitfall, because catching it requires noticing that you didn't get the data you thought you were going to get.
There was a problem hiding this comment.
Works for me. Let's confirm how the KIs will be maintained so you can link them.
alixhami reviewed Feb 5, 2019
alixhami reviewed Feb 5, 2019
alixhami reviewed Feb 5, 2019
alixhami approved these changes Feb 6, 2019
How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
Blacken sample.
* Move imports inside region tags. * Adjust query indentation to match region tags.
Move duplicate imports out of region tags. Add region tag for the whole sample.
to just above the sample where it is used. This makes the complete source code for the sample make more sense (bigquerystorage_pandas_tutorial_all)
tswast force-pushed the tswast-bqstorage-pandas branch from f064277 to 925fe3b Compare plamut pushed a commit to plamut/python-bigquery-storage that referenced this pull request Sep 2, 2020
…ogleCloudPlatform/python-docs-samples#1994) * BigQuery Storage API sample for reading pandas dataframe How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
plamut pushed a commit to googleapis/python-bigquery-storage that referenced this pull request Sep 10, 2020
…ogleCloudPlatform/python-docs-samples#1994) * BigQuery Storage API sample for reading pandas dataframe How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
Linchin pushed a commit that referenced this pull request Aug 18, 2025
) * BigQuery Storage API sample for reading pandas dataframe How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
9 tasks
parthea pushed a commit to googleapis/google-cloud-python that referenced this pull request Aug 21, 2025
…ogleCloudPlatform/python-docs-samples#1994) * BigQuery Storage API sample for reading pandas dataframe How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
parthea pushed a commit to googleapis/google-cloud-python that referenced this pull request Sep 16, 2025
…ogleCloudPlatform/python-docs-samples#1994) * BigQuery Storage API sample for reading pandas dataframe How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
How to get a pandas DataFrame, fast!
The first two examples use the existing BigQuery client. These examples
create a thread pool and read in parallel. The final example shows using
just the new BigQuery Storage client, but only shows how to read with a
single thread.