Error reading job file for Dataflow Flex Template with Unable to Open Template File error #9153
Closed as not planned
Description
Hii @beccasaurus @nicain @lukesneeringer @hfwang I'm encountering an error while running a custom Dataflow job using a flex template in Google Cloud Platform (GCP).
A custom pipeline was created using a custom template in which a JSON file was provided. The pipeline was launched
successfully and scheduled to run. However, at some point during the execution, an error occurred, causing the pipeline to fail.
Environment:
Apache Beam version: apache-beam[gcp]==2.44.0
The error message is as follows:
Failed to read the job file: gs://dataflow-staging-us-central1-713358881388/staging/template_launches/2023-02-20_18_27_45-8498022740013370621/job_object with error message: (c20b1cad16245ca5): Unable to open template file: gs://dataflow-staging-us-central1-713358881388/staging/template_launches/2023-02-20_18_27_45-8498022740013370621/job_object.. I have also verified that the options for the job are set correctly. Here's an example of how I'm setting the options using the PipelineOptions class in Python:
pipeline_options = PipelineOptions.from_dictionary({ 'runner': 'DataflowRunner', 'project': 'testcircle-350611', 'region': 'us-central1', 'staging_location': 'gs://dataflow-staging-us-central1-713358881388/staging/', 'temp_location': 'gs://dataflow-staging-us-central1-713358881388/tmp/', 'template_location': 'gs://dataflow-staging-us-central1-713358881388/staging/template_launches/', 'service_account_email': 'xxxx-compute@developer.gserviceaccount.com' }) here is my JSON file
"resources": { "sdkPipelineOptions": { "description": "Apache Beam SDK pipeline options", "properties": { "saveMainSession": "true", "runner": "DataflowRunner", "project": "testcircle-350611", "region": "us-central1", "staging_location": "gs://dataflow-staging-us-central1-713358881388/staging/", "temp_location": "gs://dataflow-staging-us-central1-713358881388/tmp/", "template_location": "gs://dataflow-staging-us-central1-713358881388/staging/template_launches/", "service_account_email": "-xxxxcompute@developer.gserviceaccount.com" } } }, and here is my docker file code
RUN pip install --upgrade pip RUN apt-get update && apt-get install -y default-jdk postgresql-client ARG WORKDIR=/dataflow/template RUN mkdir -p ${WORKDIR} WORKDIR ${WORKDIR} COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY etl.py . ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt" ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/etl.py" ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"] ```