Consumption monitoring alerts - MaxCompute - Alibaba Cloud Documentation Center

This topic describes how to manage consumption for pay-as-you-go computing jobs in MaxCompute.

Functionality

MaxCompute's pay-as-you-go resources are scalable, allowing computing jobs to utilize resources without restrictions. To prevent unexpected consumption, it's essential to monitor resource usage. MaxCompute offers the following alert methods for consumption monitoring:

Consumption monitoring alert method	Consumption control measure	Description
Max resources consumed by an SQL statement	Resource consumption limit	MaxCompute enables you to estimate the resources an SQL statement will consume before execution. If the estimated consumption exceeds the set limit, the statement will not execute, and a failure message is returned. This feature helps prevent unexpected charges from an SQL statement.
Daily cumulative SQL consumption limit	Expense alert	Monitor the daily cumulative consumption of standard SQL in a project. If the day's cumulative consumption plus the estimated consumption of new standard SQL exceeds the threshold, the execution of the new SQL is restricted, and a failure message is returned. This feature helps prevent daily cumulative consumption of standard SQL from exceeding expectations. Note Daily cumulative SQL consumption refers to metered usage that is unbilled and excludes any discounts or promotions.

Max resources consumed by an SQL statement

Setting methods

You can set a limit for the resources that an SQL statement can consume for a project or a session:

Project-level setting
The project owner or a user with the Super_Administrator role must execute the following command to set or remove the limit on resources consumed by an SQL statement:
```
--Impose the upper limit on the resources consumed by an SQL statement. SETPROJECT odps.sql.metering.value.max=<m_value>; --Remove the upper limit on the resources consumed by an SQL statement. SETPROJECT odps.sql.metering.value.max;
```
m_value represents the maximum threshold for resources consumed by an SQL statement. It is calculated as SQL read volume (GB) × SQL complexity, not the actual consumption amount.
Note
The resources consumed by an SQL statement are limited by m_value. If the estimated consumption exceeds m_value, the statement will not execute.
Session-level setting
Submit the following command with the SQL statement you want to execute to set the limit for resources consumed by the statement. If the command is not added before the SQL statement, the limit is removed. This configuration only applies to the current statement.
```
SET odps.sql.metering.value.max=<m_value>;
```

Recommendations

In practice, we recommend that you:

Determine the m_value for a project based on historical SQL consumption and your expected usage.
- Download detailed daily metering information and use the project as the granularity to calculate the resource consumption of SQL tasks which are classified as ComputationSql. This is done by calculating SQL read volume (GB) × SQL complexity. Choose an appropriate value based on the highest normal daily consumption and business fluctuations.
- Alternatively, convert the computing consumption into a fee . Determine an appropriate fee based on the business situation, and then convert it into a consumption amount .
First, set the consumption limit for a project. Then, set the m_value limit for specific SQL statements at the session level.
Note
- Before setting a session-level consumption limit, make sure that the corresponding SQL indeed requires a session-level limit to prevent the misuse of this function.
- If you set limits for both a project and a session, the session configuration takes precedence over the project configuration.
  For example, if the project's m_value is 100, and the session's m_value for an SQL statement is 200, the statement's execution is restricted only if the estimated consumption exceeds 200. SQL statements without a session-level limit are subject to the project-level limit and will not execute if the estimated resource consumption exceeds 100.
Configuring an upper limit for either a project or a session can impact task execution. We recommend that you utilize the task failure alert feature, such as DataWorks' intelligent monitoring, to promptly detect task failures and intervene manually, ensuring that critical tasks are not hindered and business operations remain unaffected.

Daily cumulative SQL consumption limit

Setting method

To set the daily cumulative SQL consumption limit for a project, use the following command. Only the project owner or a user with the Super_Administrator role can execute this command.

SETPROJECT odps.costcontrol.rule={"byDate":{"sql":<Limit>}};

byDate: Daily cumulative. The cumulative consumption from 00:00:00 to 23:59:59 on the current day is calculated based on UTC+8. The cumulative consumption is reset the following day.
Limit: The daily cumulative consumption threshold for standard SQL in a project, measured in USD. This threshold does not account for any discounts or promotions.
Note
Consumption of a single SQL statement = Scanned data volume (GB) × Complexity × 0.0438. In the SAU (Riyadh - Partner Region), the standard billing price for SQL is 0.05256 USD/GB.

Considerations

After executing the command above, note the following rules:

Rule type	System behavior	Example and alert recommendations
First-time setting	The system calculates the cumulative consumption of all standard SQL statements in a project for the current day and estimates the consumption of new standard SQL statements.	Example: When you first set `SETPROJECT odps.costcontrol.rule={"byDate":{"sql":100}};` at the project level, the system begins calculating the daily cumulative consumption and estimating the consumption of new tasks.
Non-first-time setting (modifying existing value)	The estimated consumption of new SQL statements is verified against the cumulative consumption and the latest value to determine if execution is restricted. The cumulative consumption is not reset.	If the original daily cumulative SQL consumption limit is 100, the current day's accumulated consumption is 99, and you change the limit to 150 before initiating a new standard SQL statement. The system will first estimate the cost of this SQL (assume it's 20). If the sum of the estimated consumption and accumulated consumption (20+99) is less than or equal to 150, the SQL can execute normally. Otherwise, it will be blocked.
When the cumulative consumption and the estimated consumption exceed the Limit	New standard SQL tasks are blocked and cannot be executed. An error message is returned.	Error example: If you execute the command `setproject odps.costcontrol.rule={"byDate":{"sql":100}};` to set the consumption limit, an error is returned when the daily cumulative consumption of standard SQL statements in a project exceeds the limit. The error message is as follows: `Exceed Cost Limit : {"AlreadyCost":"100.1","InstanceId":"xxx","Limit":"100","Project":"xxx","TaskType":"SQL","ThisTaskWillCost":"0","TimeWindow":"BYDATE"}` Recommendation: We recommend using the task failure alert feature, such as DataWorks' intelligent monitoring, to promptly detect task failures and enable manual intervention. This helps prevent critical tasks from being hindered and impacting your business operations.