Resource groups provide the computing power needed to run your DataWorks tasks. Starting June 10, 2024, serverless resource groups became the standard solution for all new users. They provide unified computing resources for all use cases, including task scheduling, data integration, and DataService Studio. This guide covers serverless resource groups and how they compare to legacy alternatives.
We recommend using serverless resource groups. Legacy resource groups are being phased out and are no longer available to new users.
New users: Start directly with serverless resource groups.
Existing users: We strongly recommend that you upgrade your existing legacy resource groups to benefit from improved performance, elasticity, and cost-effectiveness.
Serverless resource groups
Serverless resource groups are the default and recommended solution in DataWorks. They eliminate the separation between legacy resource groups, which were previously divided for scheduling, data integration, and DataService Studio, to provide a unified, general-purpose pool of computing resources.
Core advantages
One solution for everything: Handle data integration, task scheduling, and DataService Studio with a single resource group.
Scale instantly: Add or remove resources on demand without affecting running tasks. The minimum purchase granularity is as low as 2 CUs, which minimizes resource waste.
Flexible billing: Both pay-as-you-go and subscription billing methods are supported. You can select a billing method based on the stability of your business workloads to optimize costs.
Enhanced security: Resources are dedicated, and network policies are fully under your control, which provides enterprise-grade security isolation.
Powerful features: Advanced features such as custom images are supported to meet your requirements for custom runtime environments.
Billing
Serverless resource groups are billed based on Compute Units (CUs). 1 CU ≈ 1-core CPU, 4 GiB memory.
Example: Serverless resource group vs. exclusive resource group
Consider this scenario: You need to sync data from MySQL to MaxCompute using 20 daily jobs. Each job runs for 1 hour and uses 1 CU.
Using a pay-as-you-go serverless resource group:
Fees = 20 tasks × 1 CU/task × 1 hour × USD 0.077399/CU-hour = USD 1.54798/day.
No fees are incurred for the remaining 23 hours when the resources are idle.
Using a subscription exclusive resource group for data integration:
The minimum required specification is 4 vCPUs and 8 GB of memory, which costs approximately USD 2.541/day. The resources remain idle for most of the day.
Conclusion: In this scenario, the serverless resource group not only minimizes resource waste but also saves nearly 40% on costs.
Resource groups comparison: Serverless vs. legacy
Supported legacy resource groups include exclusive and public options for data integration, scheduling, and DataService Studio.
Item | Serverless resource group | Legacy resource groups |
Purpose | Works with all DataWorks features. | Requires separate purchases for different features. |
Billing models | Subscription and pay-as-you-go. | Exclusive: subscription only. |
Elasticity | Seamless scaling without affecting running tasks. Simply adjust the number of CUs for your resource group. | Scaling disrupts running tasks. Scaling actions include upgrading/downgrading specifications or adding/removing nodes. |
Resource utilization | High resource efficiency with on-demand CU purchases and no resource fragmentation. Minimum purchase: 2 CUs; minimum scaling increment: 1 CU. | Fixed specifications lead to resource fragmentation and waste. Minimum purchase: one 4 vCPUs and 8 GB of memory node; minimum scaling increment: one 4 vCPUs and 8 GB of memory node. |
Network security | Bound to your VPC with full network control and high security. | DataWorks-managed public endpoints shared across multiple tenants. |
Future development | Continuously developed as the standard resource group type. | Being phased out with no new features planned. |
Legal and regulatory matters
You're responsible for all code executed on your resource groups. Please ensure your usage complies with applicable laws and regulations.
Next steps
New to DataWorks: Use a Serverless resource group
Planning your budget: Billing for serverless resource groups
Need data access: Network connectivity solutions