Mounting a GCSFuse volume with image streaming on Google Batch

Google Cloud Compute Infrastructure
1

I’m trying to reduce the startup time of my Google Batch tasks. My task starts out by running the gcsfuse cli to mount a GCS bucket. When I tried using image streaming (enableImageStreaming=true) I get an error:

Error while mounting gcsfuse: mountWithArgs: mountWithStorageHandle: mount: mount: running /usr/bin/fusermount: exit status 1

This only happens when using image streaming. What could be the cause? Can you suggest a solution or a workaround?

2

Hi @AvishaiW ,

Thanks for trying Batch. Could you please share your project number and one example of your failed job id to help us diagnose?

Thanks,

Wenyan

3

project 642504272574

job id mlapp-inference-v1-ebd3f32c8c9827b1-d5772a52f88811ef8cbe5ace8a6

4

Any updates?

5

I’ve reproduced the same issue when using job.task_groups.task_spec.volumes to let Google Batch set up the mount. (job id mlapp-inference-v1-8ac905ee9dd5b564-490333fcfdda11efb06e4e82ac9)

6

Here’s a simple repro:

{ "taskGroups": [ { "taskSpec": { "runnables": [ { "container": { "imageUri": "us-docker.pkg.dev/xxxxx/xxxxx/avishai-gcsfuse:01", "commands": [ "/bin/sh", "-c", "mkdir /mnt/foo && gcsfuse --log-severity trace --implicit-dirs --only-dir foo <BUCKET_NAME> /mnt/foo && ls -l /mnt/foo" ], "enableImageStreaming": true, "options": "--privileged" } } ], "maxRetryCount": 2, "maxRunDuration": "3600s" }, "taskCount": "1", "parallelism": "0" } ], "allocationPolicy": { "instances": [ { "policy": { "machineType": "n2d-standard-4", "provisioningModel": 1, } } ], "serviceAccount": {"email": "xxxx@xxxx.iam.gserviceaccount.com"} }, "labels": { "foo": "bar1" }, "logsPolicy": { "destination": "CLOUD_LOGGING" } }

Just adding "enableImageStreaming": true causes the failure. The Dockerfile for the gcsfuse image:

FROM --platform linux/amd64 debian USER root # install GCSFuse RUN apt-get update --allow-releaseinfo-change && \ apt-get install sudo gnupg2 curl -y && \ export GCSFUSE_REPO=gcsfuse-bookworm && \ echo "deb https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list && \ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \ sudo apt-get update --allow-releaseinfo-change && \ sudo apt-get -y install gcsfuse # Test that it works RUN gcsfuse --version
7

Thanks for reporting this. There is some issue with gcfuse + image streaming and the image streaming team will need more time to investigate.

If possible, please disable image streaming while gcsfuse is used as a workaround for now.

8

This is due to a limitation in the current Batch image streaming design that it cannot operate in “–priviledge” mode. We’ll treat it as a feature request to address in the future. Please avoid using gcfuse and image streaming together for now.