feat: Implement ExecuteQuery API for SQL support #2280
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Warning: This pull request is touching the following templated files:
|
product-auto-label bot added the api: bigtable | cc @igorbernstein2 for awareness |
| /gcbrun |
steveniemitz approved these changes Jul 11, 2024
| /gcbrun |
steveniemitz added the owlbot:run 
gcf-owl-bot bot removed the owlbot:run | /gcbrun |
3 similar comments
| /gcbrun |
| /gcbrun |
| /gcbrun |
The ExecuteQuery API will allow users to query Bigtable using SQL PiperOrigin-RevId: 650660213 Source-Link: googleapis/googleapis@f681f79 Source-Link: googleapis/googleapis-gen@3180845 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMzE4MDg0NTQ4NzEzNjc5NDk1MmI4ZjM2NWZlNmM2ODY4OTk5ZDljMCJ9
This is needed to parse ExecuteQuery responses PiperOrigin-RevId: 651386373 Source-Link: googleapis/googleapis@a5be6fa Source-Link: googleapis/googleapis-gen@d467ce8 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZDQ2N2NlODkzYTA0YzQxZTUwNDk4MzM0NmMyMTVkNDFmZDI2MzY1MCJ9
I will re-add these once the client supports the appropriate behavior for readRows Change-Id: I4b5b0861e93f0a8b97d51556c74bccfe0b1da521
This moves things mostly as is. I will move everything to appropriate packages in a follow on (e.g. utils will move to internal). Wanted to keep the changes as minimal as possible Renames ProtoRowsBatchMergingStateMachine to ProtoRowsMergingStateMachine and removes the obsolete ProtoRowsMergingStateMachine This simplifies the internal code by removing support for the old api. It should not change any behavior for he ProtoRowsBatch API Change-Id: I8e8267a9cfeb59e0f8455876d1bccdb7c3778b76
This adds the main interfaces for interacting with data: ResultSet, StructReader, and Struct. StructReader is used by ResultSet, SqlRow, and Struct to provide a standard set of accessors for the data. I'm a little uncertain about the package structure right now. I've put interfaces that users will interact with directly in 'models' and everything else in 'internal'. Internal will include both internal interfaces and the implementations of the public interfaces. Otherwise I have kept the package structure pretty flat but happy to group the sql stuff together more. This includes thorough documentation for the public interfaces. Change-Id: I4535daffc596e9355da64b5655c58a68e5e7a95f
This will be re-used for structs in the following commit. The current design implements this at the Row/Struct level instead of ResultSet & Struct so that result set can operate on rows of different types when we support arrow. ResultSet will then be a light wrapper where each getType call is passed through to the current row. Change-Id: I21fb4cc0cd830110d5455581491c5b1861efbcb1
Change-Id: Ie9d3892fe92c779a6b81d457c31b534bdce20f38
This also removes the metadata helpers from the ResultSet interface. These helpers will live on ResultSetMetadata instead. Change-Id: Iba60b42b4e62f2d72c5a21206edf716c0ad5ea73
Also fix AbstractProtoStructReaderTest which was using the wrong runner and not executing any tests. Fixes a couple test bugs as well. Change-Id: Ia59459f93dd38046e9ac22995191becaad6ea44e
This uses the ExecuteQueryCallContext to pass the metadata back to the user using a future. Note that we need to be very careful here that errors always close this future. That is why watchdog happens higher in the call chain that for other streaming RPCs Also note i decided to unwrap executeExceptions so that when possible the future throws the same exceptions as the row stream Change-Id: Ie9dd523a980efecb2cff010a2ffd7ac52b5f8ac7
Change-Id: Id9113adb24419da18f6e43e47c0c73cddf4b622e
The admin types in admin.v2.models.Type will be migrated to 1) share the same implementation in common, while defining a narrow interface like SqlType 2) be renamed to SchemaType. We aren't going to block the Sql launch on this though. It can happen after. Change-Id: I9dea84fe9b7b2e0b46e8ece4cb666651183dc639
Change-Id: I41a28069865df6aa688ac5bc96181aae9f18deb4
Change-Id: I0cf3bd45a5f2bb3e2ad980d4170049d69e701d7e
Change-Id: Ie08217d025df3ec3c97dc96268f1c4e1d20b33bf
Change-Id: I6073e41b5a5b74b91671002cb93dc42c4023f10f
jackdingilian force-pushed the execute-query-final branch from b9bff1b to 10e69e0 Compare jackdingilian added kokoro:force-run gcf-owl-bot bot removed the owlbot:run 
yoshi-kokoro removed the kokoro:force-run jackdingilian added the kokoro:force-run yoshi-kokoro removed the kokoro:force-run jackdingilian added the kokoro:force-run yoshi-kokoro removed the kokoro:force-run 
ron-gal pushed a commit to ron-gal/java-bigtable that referenced this pull request Jul 23, 2024
* feat: publish the Cloud Bigtable ExecuteQuery API The ExecuteQuery API will allow users to query Bigtable using SQL PiperOrigin-RevId: 650660213 Source-Link: googleapis/googleapis@f681f79 Source-Link: https://github.com/googleapis/googleapis-gen/commit/3180845487136794952b8f365fe6c6868999d9c0 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMzE4MDg0NTQ4NzEzNjc5NDk1MmI4ZjM2NWZlNmM2ODY4OTk5ZDljMCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: publish ProtoRows Message This is needed to parse ExecuteQuery responses PiperOrigin-RevId: 651386373 Source-Link: googleapis/googleapis@a5be6fa Source-Link: https://github.com/googleapis/googleapis-gen/commit/d467ce893a04c41e504983346c215d41fd263650 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZDQ2N2NlODkzYTA0YzQxZTUwNDk4MzM0NmMyMTVkNDFmZDI2MzY1MCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Temporarily allow method level tests to pass for ExecuteQuery I will re-add these once the client supports the appropriate behavior for readRows Change-Id: I4b5b0861e93f0a8b97d51556c74bccfe0b1da521 * Copy SqlRowMerger from internal code This moves things mostly as is. I will move everything to appropriate packages in a follow on (e.g. utils will move to internal). Wanted to keep the changes as minimal as possible Renames ProtoRowsBatchMergingStateMachine to ProtoRowsMergingStateMachine and removes the obsolete ProtoRowsMergingStateMachine This simplifies the internal code by removing support for the old api. It should not change any behavior for he ProtoRowsBatch API Change-Id: I8e8267a9cfeb59e0f8455876d1bccdb7c3778b76 * Implement main interfaces for interacting with ResultSets This adds the main interfaces for interacting with data: ResultSet, StructReader, and Struct. StructReader is used by ResultSet, SqlRow, and Struct to provide a standard set of accessors for the data. I'm a little uncertain about the package structure right now. I've put interfaces that users will interact with directly in 'models' and everything else in 'internal'. Internal will include both internal interfaces and the implementations of the public interfaces. Otherwise I have kept the package structure pretty flat but happy to group the sql stuff together more. This includes thorough documentation for the public interfaces. Change-Id: I4535daffc596e9355da64b5655c58a68e5e7a95f * Implement proto to java conversion via AbstractProtoStructReader This will be re-used for structs in the following commit. The current design implements this at the Row/Struct level instead of ResultSet & Struct so that result set can operate on rows of different types when we support arrow. ResultSet will then be a light wrapper where each getType call is passed through to the current row. Change-Id: I21fb4cc0cd830110d5455581491c5b1861efbcb1 * Add implementation for ProtoStruct Change-Id: Ie9d3892fe92c779a6b81d457c31b534bdce20f38 * Implement ResultSet This also removes the metadata helpers from the ResultSet interface. These helpers will live on ResultSetMetadata instead. Change-Id: Iba60b42b4e62f2d72c5a21206edf716c0ad5ea73 * Implement ResultSetMetadata and use it everwhere we operate on metadata Also fix AbstractProtoStructReaderTest which was using the wrong runner and not executing any tests. Fixes a couple test bugs as well. Change-Id: Ia59459f93dd38046e9ac22995191becaad6ea44e * Create execute query callables and wire up api to data client This uses the ExecuteQueryCallContext to pass the metadata back to the user using a future. Note that we need to be very careful here that errors always close this future. That is why watchdog happens higher in the call chain that for other streaming RPCs Also note i decided to unwrap executeExceptions so that when possible the future throws the same exceptions as the row stream Change-Id: Ie9dd523a980efecb2cff010a2ffd7ac52b5f8ac7 * Add statement API for ExecuteQuery requests Change-Id: Id9113adb24419da18f6e43e47c0c73cddf4b622e * Implement java representation of Type protobuf The admin types in admin.v2.models.Type will be migrated to 1) share the same implementation in common, while defining a narrow interface like SqlType 2) be renamed to SchemaType. We aren't going to block the Sql launch on this though. It can happen after. Change-Id: I9dea84fe9b7b2e0b46e8ece4cb666651183dc639 * Remove unsupported query param types and add doc Change-Id: I41a28069865df6aa688ac5bc96181aae9f18deb4 * Add clirr exclusion for new proto method. Fix dropped import Change-Id: I0cf3bd45a5f2bb3e2ad980d4170049d69e701d7e * Small doc update and remove obsolete TODOs Change-Id: Ie08217d025df3ec3c97dc96268f1c4e1d20b33bf * Minor documentation fix for executeQuery Change-Id: I6073e41b5a5b74b91671002cb93dc42c4023f10f --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
gcf-merge-on-green bot pushed a commit that referenced this pull request Jul 24, 2024
🤖 I have created a release *beep* *boop* --- ## [2.41.0](https://togithub.com/googleapis/java-bigtable/compare/v2.40.0...v2.41.0) (2024-07-24) ### Features * Add MergeToCell to Mutation APIs ([#2279](https://togithub.com/googleapis/java-bigtable/issues/2279)) ([0ce8a2a](https://togithub.com/googleapis/java-bigtable/commit/0ce8a2a38703233da58208655f41f6e81e03576e)) * Add support for MergeToCell API ([#2258](https://togithub.com/googleapis/java-bigtable/issues/2258)) ([191d15c](https://togithub.com/googleapis/java-bigtable/commit/191d15c5284dbb702e11669931272877bf05f44e)) * Add support for new functions ([#2287](https://togithub.com/googleapis/java-bigtable/issues/2287)) ([dd6583a](https://togithub.com/googleapis/java-bigtable/commit/dd6583a22504385b7a1f7dc91b3bc3d2500ea0c5)) * Create new environment variable to toggle directpath scoped to cloud bigtable. ([#2261](https://togithub.com/googleapis/java-bigtable/issues/2261)) ([9062944](https://togithub.com/googleapis/java-bigtable/commit/9062944610277eb7ae77f395dc79ce94239c5bee)) * Implement ExecuteQuery API for SQL support ([#2280](https://togithub.com/googleapis/java-bigtable/issues/2280)) ([25218e8](https://togithub.com/googleapis/java-bigtable/commit/25218e8cc46f9a51d4b6515afdb8931e574b0bb1)) ### Dependencies * Update dependency com.google.truth.extensions:truth-proto-extension to v1.4.4 ([#2282](https://togithub.com/googleapis/java-bigtable/issues/2282)) ([d00a9e0](https://togithub.com/googleapis/java-bigtable/commit/d00a9e01b2b329f3bae50f48a15692d87ad0f3bf)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
lqiu96 pushed a commit that referenced this pull request Jul 26, 2024
* feat: publish the Cloud Bigtable ExecuteQuery API The ExecuteQuery API will allow users to query Bigtable using SQL PiperOrigin-RevId: 650660213 Source-Link: googleapis/googleapis@f681f79 Source-Link: googleapis/googleapis-gen@3180845 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMzE4MDg0NTQ4NzEzNjc5NDk1MmI4ZjM2NWZlNmM2ODY4OTk5ZDljMCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: publish ProtoRows Message This is needed to parse ExecuteQuery responses PiperOrigin-RevId: 651386373 Source-Link: googleapis/googleapis@a5be6fa Source-Link: googleapis/googleapis-gen@d467ce8 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiZDQ2N2NlODkzYTA0YzQxZTUwNDk4MzM0NmMyMTVkNDFmZDI2MzY1MCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * Temporarily allow method level tests to pass for ExecuteQuery I will re-add these once the client supports the appropriate behavior for readRows Change-Id: I4b5b0861e93f0a8b97d51556c74bccfe0b1da521 * Copy SqlRowMerger from internal code This moves things mostly as is. I will move everything to appropriate packages in a follow on (e.g. utils will move to internal). Wanted to keep the changes as minimal as possible Renames ProtoRowsBatchMergingStateMachine to ProtoRowsMergingStateMachine and removes the obsolete ProtoRowsMergingStateMachine This simplifies the internal code by removing support for the old api. It should not change any behavior for he ProtoRowsBatch API Change-Id: I8e8267a9cfeb59e0f8455876d1bccdb7c3778b76 * Implement main interfaces for interacting with ResultSets This adds the main interfaces for interacting with data: ResultSet, StructReader, and Struct. StructReader is used by ResultSet, SqlRow, and Struct to provide a standard set of accessors for the data. I'm a little uncertain about the package structure right now. I've put interfaces that users will interact with directly in 'models' and everything else in 'internal'. Internal will include both internal interfaces and the implementations of the public interfaces. Otherwise I have kept the package structure pretty flat but happy to group the sql stuff together more. This includes thorough documentation for the public interfaces. Change-Id: I4535daffc596e9355da64b5655c58a68e5e7a95f * Implement proto to java conversion via AbstractProtoStructReader This will be re-used for structs in the following commit. The current design implements this at the Row/Struct level instead of ResultSet & Struct so that result set can operate on rows of different types when we support arrow. ResultSet will then be a light wrapper where each getType call is passed through to the current row. Change-Id: I21fb4cc0cd830110d5455581491c5b1861efbcb1 * Add implementation for ProtoStruct Change-Id: Ie9d3892fe92c779a6b81d457c31b534bdce20f38 * Implement ResultSet This also removes the metadata helpers from the ResultSet interface. These helpers will live on ResultSetMetadata instead. Change-Id: Iba60b42b4e62f2d72c5a21206edf716c0ad5ea73 * Implement ResultSetMetadata and use it everwhere we operate on metadata Also fix AbstractProtoStructReaderTest which was using the wrong runner and not executing any tests. Fixes a couple test bugs as well. Change-Id: Ia59459f93dd38046e9ac22995191becaad6ea44e * Create execute query callables and wire up api to data client This uses the ExecuteQueryCallContext to pass the metadata back to the user using a future. Note that we need to be very careful here that errors always close this future. That is why watchdog happens higher in the call chain that for other streaming RPCs Also note i decided to unwrap executeExceptions so that when possible the future throws the same exceptions as the row stream Change-Id: Ie9dd523a980efecb2cff010a2ffd7ac52b5f8ac7 * Add statement API for ExecuteQuery requests Change-Id: Id9113adb24419da18f6e43e47c0c73cddf4b622e * Implement java representation of Type protobuf The admin types in admin.v2.models.Type will be migrated to 1) share the same implementation in common, while defining a narrow interface like SqlType 2) be renamed to SchemaType. We aren't going to block the Sql launch on this though. It can happen after. Change-Id: I9dea84fe9b7b2e0b46e8ece4cb666651183dc639 * Remove unsupported query param types and add doc Change-Id: I41a28069865df6aa688ac5bc96181aae9f18deb4 * Add clirr exclusion for new proto method. Fix dropped import Change-Id: I0cf3bd45a5f2bb3e2ad980d4170049d69e701d7e * Small doc update and remove obsolete TODOs Change-Id: Ie08217d025df3ec3c97dc96268f1c4e1d20b33bf * Minor documentation fix for executeQuery Change-Id: I6073e41b5a5b74b91671002cb93dc42c4023f10f --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
lqiu96 pushed a commit that referenced this pull request Jul 26, 2024
🤖 I have created a release *beep* *boop* --- ## [2.41.0](https://togithub.com/googleapis/java-bigtable/compare/v2.40.0...v2.41.0) (2024-07-24) ### Features * Add MergeToCell to Mutation APIs ([#2279](https://togithub.com/googleapis/java-bigtable/issues/2279)) ([0ce8a2a](https://togithub.com/googleapis/java-bigtable/commit/0ce8a2a38703233da58208655f41f6e81e03576e)) * Add support for MergeToCell API ([#2258](https://togithub.com/googleapis/java-bigtable/issues/2258)) ([191d15c](https://togithub.com/googleapis/java-bigtable/commit/191d15c5284dbb702e11669931272877bf05f44e)) * Add support for new functions ([#2287](https://togithub.com/googleapis/java-bigtable/issues/2287)) ([dd6583a](https://togithub.com/googleapis/java-bigtable/commit/dd6583a22504385b7a1f7dc91b3bc3d2500ea0c5)) * Create new environment variable to toggle directpath scoped to cloud bigtable. ([#2261](https://togithub.com/googleapis/java-bigtable/issues/2261)) ([9062944](https://togithub.com/googleapis/java-bigtable/commit/9062944610277eb7ae77f395dc79ce94239c5bee)) * Implement ExecuteQuery API for SQL support ([#2280](https://togithub.com/googleapis/java-bigtable/issues/2280)) ([25218e8](https://togithub.com/googleapis/java-bigtable/commit/25218e8cc46f9a51d4b6515afdb8931e574b0bb1)) ### Dependencies * Update dependency com.google.truth.extensions:truth-proto-extension to v1.4.4 ([#2282](https://togithub.com/googleapis/java-bigtable/issues/2282)) ([d00a9e0](https://togithub.com/googleapis/java-bigtable/commit/d00a9e01b2b329f3bae50f48a15692d87ad0f3bf)) --- This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
This adds the executeQuery method to the data client in order to support querying data using SQL.
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> ☕️
If you write sample code, please follow the samples format.