Platform For AI:Secure and encrypted inference services

1. Prepare the model (Optional)

This example uses the modelscope tool to download the Qwen2.5-3B-Instruct model. This requires Python 3.9 or later. You can run the following commands in the terminal to download the model.

pip3 install modelscope importlib-metadata modelscope download --model Qwen/Qwen2.5-3B-Instruct

After the commands run successfully, the model is downloaded to the ~/.cache/modelscope/hub/models/Qwen/Qwen2.5-3B-Instruct/ directory.

2. Encrypt the model

PAI-EAS supports the following two encryption methods. This topic uses Sam as an example.

• Gocryptfs: An encryption mode based on AES256-GCM that complies with the open-source Gocryptfs standard.

• Sam: An Alibaba Cloud trusted AI model encryption format that protects the confidentiality of the model and prevents the license content from being tampered with or used illegally.

3. Upload the model

EAS supports storing encrypted models in various storage backends. When you deploy a service, you can load the decrypted model into the service instance by mounting it. For more information, see Mount storage.

Take OSS as an example. You can create a bucket and a directory named qwen-encrypted, such as oss://examplebucket/qwen-encrypted/. For more information, see Quick Start for the console. Because the model file is large, we recommend that you use ossbrowser to upload the encrypted model to this directory.

The following figure shows the result after you upload the ciphertext model encrypted with Sam. If you use the Gocryptfs encryption method, the file names of the model files are all encrypted into garbled text.

4. Set up the remote attestation service and upload the key

Host the model decryption key in the remote attestation service. The remote attestation service is a user-managed authentication service that is responsible for authenticating the runtime environment of the model and the inference service. The model decryption key is injected to decrypt and mount the model only after the trustworthiness of the EAS model deployment environment meets the specified conditions.

Currently, EAS supports a self-built Trustee remote attestation service hosted on Alibaba Cloud. You can use an Alibaba Cloud ACK Serverless cluster to deploy the remote attestation service and authenticate the model deployment and inference environments. At the same time, you can use Alibaba Cloud KMS to provide professional security for the model decryption key. The procedure is as follows:

1. First, create an Alibaba Cloud KMS instance to use as the key storage backend.

   1. Go to the Key Management Service console. In the navigation pane on the left, choose Resources > Instance Management. On the Software Key Management tab, create and enable an instance. When you enable the instance, select the same VPC as the ACK cluster. For more information, see Purchase and enable a KMS instance.

      Wait about 10 minutes for the instance to start.

   2. After the instance starts, in the navigation pane on the left, choose Resources > Key Management. On the Key Management page, create a customer master key for the instance. For more information, see Step 1: Create a software-protected key.

   3. In the navigation pane on the left, choose Application Access > Access Points. On the Access Points page, create an application access point for the instance. Set Scope to the KMS instance you created. For more information about other configurations, see Method 1: Quick creation.

      After the application access point is created, the browser automatically downloads the ClientKey***.zip file. After you decompress the zip file, it contains:

      • Application Identity Credential Content (ClientKeyContent): The file name is clientKey_****.json by default.

      • Credential Security Token (ClientKeyPassword): The file name is clientKey_****_Password.txt by default.

   4. On the Resources > Instance Management page, click the KMS instance name. In the Basic Information area, click Download next to Instance CA Certificate to export the public key certificate file PrivateKmsCA_***.pem for the KMS instance.

2. Create an ACK service cluster and install the csi-provisioner component.

   1. Go to the Create Cluster page to create an ACK Serverless cluster. The key parameter configurations are described below. For more information about other configurations, see Create a cluster.

      1. Cluster Configurations: Configure the following parameters, then click Next: Component Configurations.

         Key Configuration	Description
         VPC	Select Use Existing and select Configure SNAT For VPC. Otherwise, you cannot pull the Trustee image.
         VSwitch	Make sure that at least two virtual switches are created in the existing VPC. Otherwise, you cannot expose the public ALB.

      2. Component Configurations: Configure the following parameters, then click Next: Confirm Order.

         Key Configuration	Description
         Service Discovery	Select CoreDNS.
         Ingress	Select ALB Ingress. For the source of the ALB cloud-native gateway instance, select Create and select two virtual switches.

      3. Confirm Order: Confirm the configuration information and terms of service, then click Create Cluster.

   2. After the cluster is created, install the csi-provisioner (managed) component. For more information, see Manage components.

3. Deploy the Trustee remote attestation service in the ACK cluster.

   1. First, connect to the cluster over the Internet or an internal network. For more information, see Connect to a cluster.

   2. Upload the downloaded application identity credential (clientKey_****.json), credential security token (clientKey_****_Password.txt), and CA certificate (PrivateKmsCA_***.pem) for the KMS instance to the environment that is connected to the ACK Serverless cluster. Then, run the following command to deploy the Trustee remote attestation service and use Alibaba Cloud KMS as the key storage backend.

      # Install the plugin helm plugin install https://github.com/AliyunContainerService/helm-acr helm repo add trustee acr://trustee-chart.cn-hangzhou.cr.aliyuncs.com/trustee/trustee helm repo update export DEPLOY_RELEASE_NAME=trustee export DEPLOY_NAMESPACE=default export TRUSTEE_CHART_VERSION=1.0.0 # Set the region information of the ACK cluster, for example, cn-hangzhou export REGION_ID=cn-hangzhou # Information about the exported KMS instance # Replace with your KMS instance ID export KMS_INSTANCE_ID=kst-hzz66a0*******e16pckc # Replace with the path to your KMS instance application identity credential export KMS_CLIENT_KEY_FILE=/path/to/clientKey_KAAP.***.json # Replace with the path to your KMS instance credential security token export KMS_PASSWORD_FILE=/path/to/clientKey_KAAP.***_Password.txt # Replace with the path to your KMS instance CA certificate export KMS_CERT_FILE=/path/to/PrivateKmsCA_kst-***.pem helm install ${DEPLOY_RELEASE_NAME} trustee/trustee \ --version ${TRUSTEE_CHART_VERSION} \ --set regionId=${REGION_ID} \ --set kbs.aliyunKms.enabled=true \ --set kbs.aliyunKms.kmsIntanceId=${KMS_INSTANCE_ID} \ --set-file kbs.aliyunKms.clientKey=${KMS_CLIENT_KEY_FILE} \ --set-file kbs.aliyunKms.password=${KMS_PASSWORD_FILE} \ --set-file kbs.aliyunKms.certPem=${KMS_CERT_FILE} \ --namespace ${DEPLOY_NAMESPACE}

      Note

      The first command to install the plugin (helm plugin install...) may take a long time. If the installation fails, you can run the helm plugin uninstall cm-push command to uninstall the plugin, and then run the installation command again.

      The following is an example of the returned result:

      NAME: trustee LAST DEPLOYED: Tue Feb 25 18:55:33 2025 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None

   3. Run the following command in the environment that is connected to the ACK Serverless cluster to obtain the endpoint of Trustee.

      export TRUSTEE_URL=http://$(kubectl get AlbConfig alb -o jsonpath='{.status.loadBalancer.dnsname}')/${DEPLOY_RELEASE_NAME} echo ${TRUSTEE_URL}

      An example of the returned result is http://alb-ppams74szbwg2f****.cn-shanghai.alb.aliyuncsslb.com/trustee.

   4. Run the following command in the environment that is connected to the ACK Serverless cluster to test the connectivity of the Trustee service.

      cat << EOF | curl -k -X POST ${TRUSTEE_URL}/kbs/v0/auth -H 'Content-Type: application/json' -d @- { "version":"0.1.0", "tee": "tdx", "extra-params": "foo" } EOF

      If the Trustee service is running normally, the expected output is as follows:

      {"nonce":"PIDUjUxQdBMIXz***********IEysXFfUKgSwk=","extra-params":""}

4. Configure the Trustee network whitelist.

   Note

   This configuration allows the PAI-EAS model deployment environment to actively access the remote attestation service for an environment security check.

   1. Go to the Alibaba Cloud Application Load Balancer (ALB) console, create an access control policy group, and add the Address/CIDR Block that has permission to access Trustee as an IP entry. For more information, see Access control. The CIDR blocks to add are as follows:

      • The public IP address of the VPC that is attached when you deploy the EAS service.

      • The egress IP address of the inference client.

   2. Run the following command to obtain the ID of the ALB instance used by the Trustee instance on the cluster.

      kubectl get ing --namespace ${DEPLOY_NAMESPACE} kbs-ingress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | cut -d'.' -f1 | sed 's/[^a-zA-Z0-9-]//g'

      The expected output is as follows:

      alb-llcdzbw0qivhk0****

   3. In the Alibaba Cloud ALB console, in the navigation pane on the left, choose Application Load Balancer ALB > Instances. In the region where the cluster is located, search for the ALB instance obtained in the previous step and click the instance ID to go to the instance product page. At the bottom of the page, in the Instance Properties area, click Disable Configuration Read-only Mode.

   4. Switch to the Listeners tab. In the Access Control column of the target listener instance, click Enable and configure the whitelist to be the access control policy group that you created in the previous step.

5. Create a credential to store the model decryption key.

   The model decryption key managed by Trustee is actually stored in KMS. The key can be accessed only after the remote attestation service authenticates the target environment.

   Go to the Key Management Service console. In the navigation pane on the left, choose Resources > Secrets Manager. On the Generic Secret tab, click Create Secret. The key configurations are as follows:

   • Secret Name: A custom name for the secret, used to index the key. For example, model-decryption-key.

   • Set Secret Value: Enter the key used to encrypt the model. For example, 0Bn4Q1wwY9fN3P. Your key may be different.

   • Encryption Master Key: Select the master key that you created in the previous step.