Run the Native Query Execution qualification tool

Qualification job

The qualification job simplifies large-scale analysis by programmatically scanning for Serverless for Apache Spark batch workloads and submitting a distributed analysis job. The tool evaluates jobs across your organization, eliminating the need to manually find and specify event log paths.

Grant IAM roles

For the qualification job to access batch workload metadata and read Spark event logs in Cloud Logging, the service account that runs the workload must have the following IAM roles granted in all projects to be analyzed:

• Dataproc Viewer (roles/dataproc.viewer)
• Logs Viewer (roles/logging.viewer)

Submit the qualification job

You submit the qualification job using the gcloud CLI tool. The job includes a PySpark script and a JAR file that are hosted in a public Cloud Storage bucket.

You can run the job in either of the following execution environments:

• As a Serverless for Apache Spark batch workload. This is simple, stand-alone job execution.

• As a job that runs on a Dataproc on Compute Engine cluster. This approach can be useful to integrate the job into a workflow.

Job arguments

Qualification job examples

• A Serverless for Apache Spark batch job to perform simple, ad-hoc analysis. Job arguments are listed after the -- separator.

   gcloud dataproc batches submit pyspark gs://qualification-tool/performance-boost-qualification.py \     --project=PROJECT_ID \     --region=REGION \     --jars=gs://qualification-tool/dataproc-perfboost-qualification-1.2.jar \     -- \     --project-ids=COMMA_SEPARATED_PROJECT_IDS \     --regions=COMMA_SEPARATED_REGIONS \     --limit=MAX_BATCHES \     --output-gcs-path=gs://BUCKET 

• A Serverless for Apache Spark batch job to analyze up to 50 of the most recent batches found in sample_project in the us-central1 region, The results are written to bucket in Cloud Storage. Job arguments are listed after the -- separator.

   gcloud dataproc batches submit pyspark gs://qualification-tool/performance-boost-qualification.py \     --project=PROJECT_ID \     --region=US-CENTRAL1 \     --jars=gs://qualification-tool/dataproc-perfboost-qualification-1.2.jar \     -- \     --project-ids=PROJECT_ID \     --regions=US-CENTRAL1 \     --limit=50 \     --output-gcs-path=gs://BUCKET/ 

• A Dataproc on Compute Engine job submitted to a Dataproc cluster for bulk analysis in a large-scale, repeatable, or automated analysis workflow. Job arguments are placed in an INPUT_FILE that is uploaded to a BUCKET in Cloud Storage. This method is ideal for scanning different date ranges or batch limits across different projects and regions in a single run.

   gcloud dataproc jobs submit pyspark gs://qualification-tool/performance-boost-qualification.py \     --cluster=CLUSTER_NAME \     --region=REGION \     --jars=gs://qualification-tool/dataproc-perfboost-qualification-1.2.jar \     -- \     --input-file=gs://INPUT_FILE \     --output-gcs-path=gs://BUCKET 

  Notes:

  INPUT_FILE: Each line in the file represents a distinct analysis request and uses a format of single-letter flags followed by their values, such as, -p PROJECT-ID -r REGION -s START_DATE -e END_DATE -l LIMITS.

  Example input file content:

   -p project1 -r us-central1 -s 2024-12-01 -e 2024-12-15 -l 100 -p project2 -r europe-west1 -s 2024-11-15 -l 50 

  These arguments direct the tool to analyze the following two scopes:

  • Up to 100 batches in project1 in the us-central1 region created between December 1, 2025 and December 15, 2025.
  • Up to 50 batches in project2 in the europe-west1 region created on or after November 15, 2025.

Qualification script

Use this method if you have the direct Cloud Storage path to a specific Spark event log that you want to analyze. This approach requires you to download and run a shell script, run_qualification_tool.sh, on a local machine or a Compute Engine VM that is configured with access to the event log file in Cloud Storage.

Perform the following steps to run the script against Serverless for Apache Spark batch workload event files.

1.Copy the run_qualification_tool.sh into a local directory that contains the Spark event files to analyze.

1. Run the qualification script to analyze one event file or a set of event files contained in the script directory.

    ./run_qualification_tool.sh -f EVENT_FILE_PATH/EVENT_FILE_NAME \     -o CUSTOM_OUTPUT_DIRECTORY_PATH \     -k SERVICE_ACCOUNT_KEY \     -x MEMORY_ALLOCATEDg \     -t PARALLEL_THREADS_TO_RUN 

   Flags and values:

   -f (required): See Spark event file locations to locate Spark workload event files.

   • EVENT_FILE_PATH (required unless EVENT_FILE_NAME is specified): Path of the event file to analyze. If not provided, the event file path is assumed to be the current directory.

   • EVENT_FILE_NAME (required unless EVENT_FILE_PATH is specified): Name of the event file to analyze. If not provided, the event files found recursively in the EVENT_FILE_PATH are analyzed.

   -o(optional): If not provided, the tool creates or uses an existing output directory under the current directory to place output files.

   • CUSTOM_OUTPUT_DIRECTORY_PATH: Output directory path to output files.

   -k (optional):

   • SERVICE_ACCOUNT_KEY: The service account key in JSON format if needed to access the EVENT_FILE_PATH.

   -x (optional):

   • MEMORY_ALLOCATED: Memory in gigabytes to allocate to the tool. By default, the tool uses 80% of the free memory available in the system and all available machine cores.

   -t(optional):

   • PARALLEL_THREADS_TO_RUN: The number of parallel threads for the tool to execute. By default, the tool executes all cores.

   Example command usage:

    ./run_qualification_tool.sh -f gs://dataproc-temp-us-east1-9779/spark-job-history \     -o perfboost-output -k /keys/event-file-key -x 34g -t 5 

   In this example, the qualification tool traverses the gs://dataproc-temp-us-east1-9779/spark-job-history directory, and analyzes Spark event files contained in this directories and its subdirectories. Access to the directory is provided the /keys/event-file-key. The tool uses 34 GB memory for execution, and runs 5 parallel threads.

Spark event file locations

Perform any of the following steps to find the Spark event files for Serverless for Apache Spark batch workloads:

1. In Cloud Storage, find the spark.eventLog.dir for the workload, then download it.

   1. If you can't find the spark.eventLog.dir, set the spark.eventLog.dir to a Cloud Storage location, and then rerun the workload and download the spark.eventLog.dir.

2. If you have configured Spark History Server for the batch job:

   1. Go to the Spark History Server, then select the workload.
   2. Click Download in the Event Log column.