How to manage external volumes - MaxCompute - Alibaba Cloud Documentation Center

An External Volume is a distributed file system and unstructured data storage solution provided by MaxCompute. This feature lets you directly query and process data in files stored in Object Storage Service (OSS) without importing the data into MaxCompute tables. This process reduces data redundancy and transmission overhead.

Common operations

Type	Features	Role	Operation platform
Create an external volume	Creates an external volume in a project.	External Volume owner. Project owner. Users who are assigned the Super_Administrator or Admin role.	MaxCompute client Connect using DataWorks
View the directory structure of an external volume	Views the directory structure of a target external volume.
Delete an external volume	Deletes a target external volume.

Prerequisites

The MaxCompute client must be v0.43.0 or later. For more information, see Connect using the local client (odpscmd).
Alternatively, you can run commands in the Data Development or SQL Query feature of DataWorks. The version of MaxCompute integrated with these features must be v0.43.2 or later. You can run the Show version; command in DataWorks to view the version. For more information, see Connect using DataWorks.
Note
When you run commands in the Data Development or SQL Query feature of DataWorks, ensure that the MaxCompute version is v0.43.2 or later. You can use the Show version; command to view the version.
If you perform operations using an SDK, the Java SDK must be v0.43.0 or later.
The Alibaba Cloud account or RAM user must have the required permissions to access OSS. For more information, see Authorize access to OSS in STS mode.

Create an external volume

Syntax

vfs -create <volume_name> -storage_provider <oss> -url <oss://oss_endpoint/bucket/path> -acd <true|false> -role_arn <arn:aliyun:xxx/aliyunodpsdefaultrole>

The following table describes the parameters.

Parameter	Required	Description
volume_name	Yes	The name of the external volume to create.
storage_provider	Yes	The storage provider. Only OSS is supported. The value must be `oss`.
url	Yes	The OSS path where the data files are stored. The format is `oss://<oss_endpoint>/<Bucket name>/<OSS directory name>`. Important The OSS path specified for `url` cannot be only an OSS bucket name. A level-2 directory name is required. oss_endpoint: The endpoint of OSS. Use an internal endpoint of OSS to prevent OSS traffic fees. Example: `oss://oss-cn-beijing-internal.aliyuncs.com/xxx`. For more information about OSS internal endpoints, see OSS regions and endpoints. Note Store data files in an OSS bucket that is in the same region as your MaxCompute project. MaxCompute is deployed only in some regions. Cross-region data connectivity may be unstable. Bucket name: The name of the OSS bucket. For more information about how to view bucket names, see List buckets. OSS directory name. Do not specify a file name after the directory.
acd	No	Auto Create Directory. This parameter specifies whether to automatically create the directory if it does not exist. Valid values: false (default): If the directory does not exist, an error is reported and the creation fails. true: The directory is created based on the `role_arn` authorization. Note When you create an external volume with the `acd` parameter configured, MaxCompute first attempts to create the directory using the authorization for the external storage. If the directory is created successfully, MaxCompute will not delete it, regardless of whether the external volume creation succeeds. If the directory already exists, the external volume creation does not fail due to a conflict, even if the `acd` parameter is set to true.
role_arn	Yes	The Alibaba Cloud Resource Name (ARN) of a RAM role that has permissions to access OSS. For more information about how to obtain the ARN, see Use STS temporary credentials to access OSS.

After an external volume is created, the path to the volume in MaxCompute is odps://[project_name]/[volume_name]. In the path, project_name specifies the name of the MaxCompute project and volume_name specifies the name of the external volume. Engines such as Spark and MapReduce can use this path to access the external volume in MaxCompute.

Example

You can create an external volume named test_ext_l.

vfs -create test_ext_l -storage_provider oss -url oss://oss-cn-hangzhou-internal.aliyuncs.com/test/ex_volume/ -role_arn acs:ram::xxxxxxx:role/aliyunodpsdefaultrole;

View the list and directory structure of external volumes

Syntax

-- View the list of external volumes. vfs -ls /; -- View the directory structure of an external volume. vfs -ls [-R] /<volume_name>;

The following table describes the parameters.

Parameter	Required	Description
volume_name	Yes	The name of the external volume to view.

Examples

View the list of external volumes.
```
vfs -ls /;
```
Sample response:
```
> vfs -ls /;	Found 2 items	drwxrwxrwx - 0 2023-03-11 12:06 /test_ext_l -> oss://oss-cn-shanghai-internal.aliyuncs.com/test/ex_volume	drwxrwxrwx - 0 2023-03-21 07:33 /myfirst_volume4 -> oss://oss-cn-shanghai-internal.aliyuncs.com/paristech/data
```
If the current user does not have permission for an external volume, information about that volume is not returned. For example, the user dev01 does not have permission for myfirst_volume4. To query information about myfirst_volume4, run the following command to grant the Read permission on `myfirst_volume4` to `dev01`:
```
grant Read on volume myfirst_volume4 to RAM$xxxxxx:dev01;
```
Note
The supported authorization operations are Read, Write, and CreateVolume.

View the directory structure of the external volume named test_ext_l.

vfs -ls -R /test_ext_l;

Sample response:

drwxrwxrwx - 0 2023-03-27 07:31 /test_ext_l/test -> oss://oss-cn-hangzhou-internal.aliyuncs.com/test/ex_volume/test

Delete an external volume

Syntax

Command 1:
```
vfs -rm -r /<volume_name> 
```
Command 2:
```
vfs -rmv /<volume_name>
```

The following table describes the parameters.

Parameter	Required	Description
volume_name	Yes	The name of the external volume to delete.

Example

Delete the external volume named test_ext_l.

vfs -rm -r /test_ext_l;

References

For more information about how to manage external volumes using an SDK, see External Volume SDK Guide.
In MaxCompute, you can create an external volume to mount an OSS path. This lets you use the MaxCompute permission management system for fine-grained access control and use the MaxCompute engine to process data in the files. For a specific example, see Use a MaxCompute external volume to process unstructured data.