A wrapper of the Apache Spark Connect client with additional functionalities that allow applications to communicate with a remote Dataproc Spark Session using the Spark Connect protocol without requiring additional steps.

pip install dataproc_spark_connect

pip uninstall dataproc_spark_connect

This client requires permissions to manage Dataproc Sessions and Session Templates. If you are running the client outside of Google Cloud, you must set following environment variables:

• GOOGLE_CLOUD_PROJECT - The Google Cloud project you use to run Spark workloads
• GOOGLE_CLOUD_REGION - The Compute Engine region where you run the Spark workload.
• GOOGLE_APPLICATION_CREDENTIALS - Your Application Credentials

1. Install the latest version of Dataproc Python client and Dataproc Spark Connect modules:

   pip install google_cloud_dataproc dataproc_spark_connect --force-reinstall

2. Add the required imports into your PySpark application or notebook and start a Spark session with the following code instead of using environment variables:

   from google.cloud.dataproc_spark_connect import DataprocSparkSession from google.cloud.dataproc_v1 import Session session_config = Session() session_config.environment_config.execution_config.subnetwork_uri = '<subnet>' session_config.runtime_config.version = '2.2' spark = DataprocSparkSession.builder.dataprocSessionConfig(session_config).getOrCreate()

The package supports the sparksql-magic library for executing Spark SQL queries directly in Jupyter notebooks.

Installation: To use magic commands, install the required dependencies manually:

pip install dataproc-spark-connect pip install IPython sparksql-magic

1. Load the magic extension:

   %load_ext sparksql_magic

2. Configure default settings (optional):

   %config SparkSql.limit=20

3. Execute SQL queries:

   %%sparksql SELECT * FROM your_table

4. Advanced usage with options:

   # Cache results and create a view %%sparksql --cache --view result_view df SELECT * FROM your_table WHERE condition = true

Available options:

• --cache / -c: Cache the DataFrame
• --eager / -e: Cache with eager loading
• --view VIEW / -v VIEW: Create a temporary view
• --limit N / -l N: Override default row display limit
• variable_name: Store result in a variable

See sparksql-magic for more examples.

Note: Magic commands are optional. If you only need basic DataprocSparkSession functionality without Jupyter magic support, install only the base package:

pip install dataproc-spark-connect

For development instructions see guide.

We'd love to accept your patches and contributions to this project. There are just a few small guidelines you need to follow.

Contributions to this project must be accompanied by a Contributor License Agreement. You (or your employer) retain the copyright to your contribution; this simply gives us permission to use and redistribute your contributions as part of the project. Head over to https://cla.developers.google.com to see your current agreements on file or to sign a new one.

You generally only need to submit a CLA once, so if you've already submitted one (even if it was for a different project), you probably don't need to do it again.

All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult GitHub Help for more information on using pull requests.