MaxCompute:MaxQA Operation Guide

Public preview schedule by region

Notes

• After you join the public preview, new interactive quota groups that you create will be in MaxQA mode by default. Existing MCQA quota groups are not affected. If you want to create a new MCQA quota group after joining the MaxQA public preview, submit a ticket.

• To help you get started with MaxQA, we provide 100 CUs of computing resources (valid for one month, valued at USD 2,200) to reduce your trial-related costs.

Support channels

If you are participating in the public preview, scan the QR code below to join the official MaxQA user support group (DingTalk group number: 87535025714). MaxCompute technical support can help you with any public preview-related issues.

Activate a MaxQA instance

MaxQA is available only for subscription-based MaxCompute projects. To activate a MaxQA instance, follow these steps:

1. Log on to the MaxCompute console. In the top navigation bar, select a region.

2. In the navigation pane on the left, click Workspace > Quota Management.

3. On the Quota Management page, find the level-1 quota that you want to manage and click Quota Configuration in the Actions column.

4. Configure basic parameters.

   1. On the Quota Configuration page, click the Basic Configurations tab and then click Edit Basic Configurations.

   2. Click Add Level-2 Quota, enter a Quota Name, and select a Type.

      Parameter	Description
      Quota Name	Enter a custom name. You will need to specify this name when you use MaxQA. We recommend that the name reflects the business purpose to help distinguish between multiple instances.
      Type	Select Interactive.

5. Click OK. A MaxQA instance is activated.

After activation, the status of the interactive quota group is displayed as Starting. The process takes 5 to 8 minutes. The instance is ready to use after its status changes to Running.

Delete a MaxQA instance

If you no longer need a MaxQA instance, you can delete it.

1. On the Quota Management page, find the level-1 quota that you want to manage and click Quota Configuration in the Actions column.

2. On the Quota Configuration page, click the Basic Configurations tab. Find the interactive quota group that you want to delete and click Delete in the Actions column.

The deletion is performed in the background. The page does not show status updates or the deletion progress.

Scale a MaxQA instance

You can scale out or scale in an interactive quota group as needed. You can also configure a scheduled scaling plan to automatically scale the instance. For more information, see Configure a time-based scaling rule for MaxQA.

1. On the Quota Management page, find the level-1 quota that you want to manage and click Quota Configuration in the Actions column.

2. On the Quota Configuration page, click the Basic Configurations tab and then click Edit Basic Configurations.

3. Adjust the Reserved CUs [minCU,maxCU] for the interactive quota group. Increase the value to scale out, or decrease it to scale in.

   Note

   • The minimum step size for CUs is 32.

   • The minimum number of CUs must be 32 or greater. If you do not need interactive resources, set the value to 0 or delete the quota group.

   • Elastic reserved CUs are not supported for interactive quotas.

4. Click OK to adjust the reserved CUs for the interactive quota group.

The Actions column for the interactive quota group displays Scaling Out or Scaling In. A scale-out operation takes approximately 5 to 8 minutes. A scale-in operation may take longer because it depends on when the currently running jobs are completed. In a worst-case scenario, the system might wait for jobs to time out before starting the scale-in operation to ensure that all jobs on the scaled-in machines have finished. Neither scale-out nor scale-in operations affect normal job execution.

Configure a time-based scaling rule for MaxQA

You can also configure a time-based scaling plan for scheduled scaling. To do so, follow these steps:

1. On the Quota Management page, find the level-1 quota that you want to manage and click Quota Configuration in the Actions column.

2. On the Quota Configuration page, click the Scaling Configurations tab.

3. Configure the time-based scaling plan.

   1. In the Resource Configuration Plan List section, click Add Configuration Plan.

   2. In the Add Configuration Plan dialog box, adjust the Reserved CUs [minCU,maxCU] for each interactive quota group and click OK.

   3. (Optional) You can also find an existing configuration plan and click Edit in the Actions column to modify it.

4. Configure the time-based plan.

   1. In the Time-based Management section, click Edit Time Plan.

   2. Click Add Effective Period to enable different quota configuration plans at different times of the day. This allows for time-based management of quota configurations. For more information, see Quota management for computing resources.

Scheduling policy

When you submit a job, you must explicitly specify the name of the interactive quota group in the client or the Java Database Connectivity (JDBC) connection string. The job is then scheduled to run on the corresponding MaxQA instance.

Fallback policy

MaxQA does not support an automatic fallback policy. If a MaxQA job fails due to limits or other reasons, you must resubmit it manually or submit it to a batch processing quota group.

MaxQA supports the following connection methods:

• MaxCompute client

• Ad-hoc queries or data development in DataWorks

• SQL analysis in the MaxCompute console

• Use MaxQA in MaxCompute Studio

• JDBC connection

• BI analysis tools

• Java SDK

• Python SDK

Enable MaxQA using the MaxCompute client

1. Install and configure MaxCompute client (odpscmd) V0.51.1 or later. For more information, see Connect using the local client (odpscmd).

2. (Optional) In the conf folder of the client installation folder, you can add the following command to the end of the odps_config.ini file.

   enable_quota_cache=true -- Enables the caching feature for MaxQA connection information.

3. Run the script file in the bin directory of the installation path to start the MaxCompute client.

   Important

   • For Linux or macOS, run the odpscmd script file.

   • For Windows, double-click the odpscmd.bat file.

4. Before you run a job, execute the following command. Subsequent jobs submitted from the client use MaxQA mode and run on the MaxQA instance that corresponds to the specified interactive quota group.

   USE QUOTA <name_of_the_interactive_quota_group>;

   You can also specify the MaxQA instance for the current job at the session level.

   SET odps.task.wlm.quota=<name_of_the_interactive_quota_group>;

Enable MaxQA for data development in DataWorks

1. On the Data Development page in DataWorks, click Debugging Configuration on the right.

2. On the Debugging Configuration page, for Computing Resource, select the attached MaxCompute project resource. For Computing Quota, select the interactive quota group.

For more information, see Create a MaxCompute SQL node in DataWorks.

After you complete the configuration, jobs executed by the SQL node are scheduled to run on the MaxQA instance that corresponds to the interactive quota group.

Enable MaxQA for SQL analysis in the MaxCompute console

1. Log on to the MaxCompute console. In the navigation pane on the left, click Workspace > SQL Analysis.

2. Go to the SQL query page and create a new text file.

3. Click Runtime Parameters on the right. In the tab that appears, select the Project. For Computing Quota, select the computing resource that corresponds to the interactive quota group.

For more information, see SQL analysis.

After you complete the configuration, jobs executed on the SQL analysis page are scheduled to run on the MaxQA instance that corresponds to the interactive quota group.

Use MaxQA in MaxCompute Studio

Ensure that you have installed MaxCompute Studio and connected to a MaxCompute project. You can then execute a script in MaxCompute Studio to use the MaxQA feature. To do so, follow these steps:

1. Install and configure MaxCompute Studio 4.3.2 or later. For more information, see Install MaxCompute Studio.

2. Create a MaxCompute Studio project and connect it to your MaxCompute project. For more information, see Manage project connections.

3. Create a MaxCompute SQL script in the project. For more information, see Develop and submit SQL scripts.

4. In the MaxCompute SQL script, enter the SQL script statements. From the execution mode drop-down list, select MaxQA.

5. Click the specify additional parameters icon , enter the MaxQA quota name, and then click OK.

6. Run the script. The job will use the MaxQA query acceleration feature.

Enable MaxQA using JDBC

When you use Java Database Connectivity (JDBC) to connect to MaxCompute, you can perform the following operations to enable the MaxQA feature. For more information about connecting to MaxCompute using JDBC, see Instructions.

1. Download the source code that supports the MaxQA feature and can be compiled.

2. Configure the Pom dependency in Maven.

   <dependency> <groupId>com.aliyun.odps</groupId> <artifactId>odps-jdbc</artifactId> <version>${Latest Version}</version> </dependency>

   Important

   The Latest Version in the preceding code is a placeholder. You must replace it with a specific SDK version number to successfully compile the project. Visit the Maven Central Repository to view all released versions of odps-jdbc. We recommend that you use the latest version. The version number format is X.X.X.

3. Create a Java program based on the source code and adapt the information as needed. For more information, see MaxCompute JDBC. The following code provides an example.

   import java.sql.Connection; import java.sql.DatabaseMetaData; import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Statement; public class MaxCompute { private static final String DRIVER_NAME = "com.aliyun.odps.jdbc.OdpsDriver"; // An Alibaba Cloud account AccessKey has full permissions on all APIs and poses a high security threat. We strongly recommend that you create and use a RAM user to make API calls or perform routine O&M. Log on to the RAM console to create a RAM user. // This example shows how to store the AccessKey ID and AccessKey secret in environment variables. You can also store them in a configuration file as needed. // We strongly recommend that you do not hard-code the AccessKey ID and AccessKey secret in your code. This can lead to security risks. private static String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"); private static String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"); public static void main(String[] args) throws SQLException { try { Class.forName(DRIVER_NAME); } catch (ClassNotFoundException e) { e.printStackTrace(); System.exit(1); } Connection conn = DriverManager.getConnection( "jdbc:odps:https://service.cn-hangzhou.maxcompute.aliyun.com/api?project=maxqatomax&charset=UTF-8&interactiveMode=true&quotaName=maxfor1&autoSelectLimit=1000000000", MaxCompute.accessId, MaxCompute.accessKey); // create a table Statement stmt = conn.createStatement(); final String tableName = "jdbc_test11"; stmt.execute("DROP TABLE IF EXISTS " + tableName); stmt.execute("CREATE TABLE " + tableName + " (key BIGINT, value STRING)"); // get meta data DatabaseMetaData metaData = conn.getMetaData(); System.out.println("product = " + metaData.getDatabaseProductName()); System.out.println("jdbc version = " + metaData.getDriverMajorVersion() + ", " + metaData.getDriverMinorVersion()); ResultSet tables = metaData.getTables(null, "default", tableName, null); while (tables.next()) { String name = tables.getString("TABLE_NAME"); System.out.println("inspecting table: " + name); ResultSet columns = metaData.getColumns(null, null, name, null); while (columns.next()) { System.out.println( columns.getString("COLUMN_NAME") + "\t" + columns.getString("TYPE_NAME") + "(" + columns.getInt("DATA_TYPE") + ")"); } columns.close(); } tables.close(); stmt.close(); conn.close(); } }

The following code provides an example of a JDBC connection string configuration.

// your_quota_nick_name is the name of the interactive quota group that you want to use. jdbc:odps:<MaxCompute_endpoint>? project=<MaxCompute_project_name>&interactiveMode=true&quotaName=your_quota_nick_name&enableLimit=false&enableOdpsLogger=true"

Connect to MaxQA using BI analysis tools

You can connect to MaxQA using Tableau, Quick BI, Guanyuan BI, SQLWorkBench. To do so, follow these steps.

Enable MaxQA using the Java SDK

For more information about the Java SDK, see Java SDK introduction.

1. Configure the Pom dependency in Maven. The following code provides an example.

   <dependency> <groupId>com.aliyun.odps</groupId> <artifactId>odps-sdk-core</artifactId> <version>${Latest Version}</version> </dependency>

   Important

   The Latest Version in the preceding code is a placeholder. You must replace it with a specific SDK version number to successfully compile the project. Visit the Maven Central Repository to view all released versions of odps-sdk-core. We recommend that you use the latest version. The version number format is X.X.X-public.

2. Create a Java program. The following code provides an example.

   import com.aliyun.odps.Odps; import com.aliyun.odps.OdpsException; import com.aliyun.odps.account.Account; import com.aliyun.odps.account.AliyunAccount; import com.aliyun.odps.sqa.ExecuteMode; import com.aliyun.odps.sqa.SQLExecutor; import com.aliyun.odps.sqa.SQLExecutorBuilder; import java.util.HashMap; import java.util.Map; public class MaxCompute { public static void main(String args[]) { // Set the account and project information. // An Alibaba Cloud account AccessKey has full permissions on all APIs and poses a high security threat. We strongly recommend that you create and use a RAM user to make API calls or perform routine O&M. Log on to the RAM console to create a RAM user. // This example shows how to store the AccessKey ID and AccessKey secret in environment variables. You can also store them in a configuration file as needed. // We strongly recommend that you do not hard-code the AccessKey ID and AccessKey secret in your code. This can lead to security risks. Account account = new AliyunAccount(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"), System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET")); Odps odps = new Odps(account); odps.setDefaultProject("maxqatomax"); odps.setEndpoint("https://service.cn-hangzhou.maxcompute.aliyun.com/api"); // Prepare to build the SQLExecutor. SQLExecutorBuilder builder = SQLExecutorBuilder.builder(); SQLExecutor sqlExecutor = null; try { // Create an executor that uses MCQA 2.0 mode by default. sqlExecutor = builder.odps(odps) .executeMode(ExecuteMode.INTERACTIVE) .quotaName("maxfor1") .enableMcqaV2(true) .build(); // If needed, you can pass special settings for the query. Map<String, String> queryHint = new HashMap<>(); queryHint.put("odps.sql.mapper.split.size", "128"); // Submit a query job. Hints are supported. sqlExecutor.run("select count(1) from test_table;", queryHint); // The logview of the current query job. System.out.println("Logview:" + sqlExecutor.getLogView()); // The InstanceId of the current query job. System.out.println("InstanceId:" + sqlExecutor.getQueryId()); } catch (OdpsException e) { e.printStackTrace(); } finally { if (sqlExecutor != null) { // Close the executor to release resources. sqlExecutor.close(); } } } }

Use MaxQA with PyODPS

PyODPS V0.12.1 and later support the MaxQA feature. The following code provides an example.

import os from odps import ODPS # Set the account and project information. # Make sure that the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable is set to your AccessKey ID and # the ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variable is set to your AccessKey secret. # We do not recommend that you use the AccessKey ID and AccessKey secret strings directly. o = ODPS( os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'), os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'), project='maxto****', endpoint='https://service.cn-hangzhou.maxcompute.aliyun.com/api', ) # If needed, you can pass special settings for the query. hints = {"odps.sql.mapper.split.size": "128"} inst = o.execute_sql_interactive( 'select count(1) from test_table', use_mcqa_v2=True, quota_name='maxfor1', # Set quota_name to the name of the interactive quota group that corresponds to the MaxQA instance. hints=hints, ) # The logview of the current query job. print(inst.get_logview_address()) # The InstanceId of the current query job. print(inst.id) # Read data row by row. with inst.open_reader() as reader: result = [res.values for res in reader]

odps://<access_id>:<ACCESS_KEY>@<project>/?endpoint=<endpoint>&quota_name=<YOUR_INTERACTIVE_QUOTA_NICKNAME>&interactive_mode=v2&reuse_odps=true

Check the Logview

After a job is complete, you can go to the Logview page and check the following three items to confirm that the job is in MaxQA mode.

• Check whether the MaxCompute InstanceId ends with _mcqa.

• Check whether the Quota Nickname is the specified interactive quota name.

• On the Summary tab, check whether the Job run mode is mcqa 2.0.

Check the observability page in the console

1. Log on to the MaxCompute console. In the navigation pane on the left, click Workspace > Job O&M.

2. On the Job O&M page, set Job Type to MCQA2 to filter for MaxQA jobs within a specified time range.