Real-time message subscription feature of OpenEvent - DataWorks

The OpenEvent module allows you to subscribe to event messages. You can receive notifications about various change events in DataWorks and respond to the events based on your configurations at the earliest opportunity. For example, you can subscribe to table change events to receive notifications about the changes to core tables in real time. You can also subscribe to task change events to implement custom data monitoring of a dashboard that displays the status of real-time synchronization tasks. This topic describes the types of events for subscription by using OpenEvent and performing the subscription configuration procedure.

Prerequisites

DataWorks Enterprise Edition is activated in a supported region. The OpenEvent module is available for DataWorks Enterprise Edition users in the following regions: China (Beijing), China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), and Singapore.
EventBridge is activated. The OpenEvent module uses EventBridge to subscribe to and consume messages. For more information, see What is EventBridge?

Permission description

The Open Platform administrator and tenant administrator have read and write permissions on the developer backend. Other tenant-level roles and workspace-level roles have read-only permissions on the developer backend. For more information, see Manage permissions on global-level services.
Only Alibaba Cloud accounts or RAM users to which the AliyunDataWorksFullAccess policy is attached can be used to configure settings on pages in the DataWorks console. For more information, see Manage permissions on the DataWorks services and the entities in the DataWorks console by using RAM policies.

Background information

DataWorks allows you to subscribe to messages of status change events in DataWorks key processes and can push the messages to the OpenEvent module of DataWorks Open Platform. You can use the OpenEvent module to push event messages in a workspace and use EventBridge to filter and consume the messages.

Concepts

DataWorks events are classified into regular events and extension point events based on whether the operation process related to an event in DataWorks can be blocked.

Regular events: You can develop an on-premises program to receive messages of this event type and configure custom responses to the event messages based on your business requirements.
Extension point events: This type of event has the characteristics and capabilities of regular events. In addition, this type of event allows you to register your on-premises program as a DataWorks extension to process event messages generated by the operations you perform in DataWorks based on custom processing logic, such as blocking the operations.

DataWorks events are classified into workspace-level events and tenant-level events based on the effective scope of events.

Workspace-level events: Events that are generated in workspace-level services. For example, events that are generated when nodes are run in DataStudio and events that are generated when the status of instances is changed in Operation Center are workspace-level events.
Tenant-level events: Events that are generated in tenant-level services. For example, the event that is generated when a user deletes a workspace in the DataWorks console is a tenant-level event. The following tables describe the events supported by each DataWorks service, whether an event is a regular event or an extension point event, and the message format of an event.

Supported event types

The following tables list the extension point events and regular events that are supported at the workspace level and tenant level.

Workspace-level events

DataStudio

For information about message formats of events in DataStudio, see the Events of DataStudio section in the "Development reference: Event lists and event message formats" topic.

Event category	Event	Regular event	Extension point event	Example (event-generation operation)
Node change event	Create a node			Create a node and deploy the node to the production environment.
Node change event	Update a node			Modify a node in the development environment. Modify a node and deploy the node to the production environment.
File change event Node Resource Function	Delete a file			Delete a file in the development environment. Move the deleted file to the recycle bin.
	Commit a file			Commit a node, resource, or function in DataStudio.
	Deploy a file			Deploy a node, resource, or function in DataStudio.
	Run the code of a node			Run the code of a node in DataStudio.
Table change event	Commit a table to the development environment			Commit a table on the Workspace Tables page of DataStudio.
Table change event	Deploy a table to the production environment			Deploy a table on the Workspace Tables pane of DataStudio. Note You can deploy tables in the Workspace Tables pane only in a workspace in standard mode. Therefore, table deployment events can be generated only in workspaces that are in standard mode.

Data Integration

For information about message formats of events in Data Integration, see the Events of Data Integration section in the "Development reference: Event lists and event message formats" topic.

Event category	Event	Regular event	Extension point event	Example (event-generation operation)
Node change event	Pre-event for Di-Job running			Start a real-time task in Data Integration.
Node change event	Pre-event for Di-Job batch running			Start a batch of real-time tasks in Data Integration.

Operation Center

For information about message formats of events in Operation Center, see the Events of Operation Center section in the "Development reference: Event lists and event message formats" topic.

Event category	Event	Regular event	Extension point event	Example (event-generation operation)
Node changing	Delete a node			Deploy the delete operation to the production environment to undeploy the node in the production environment. Delete a node from the recycle bin.
	Undeploy a node			Undeploy an auto triggered node in the production environment. Note This operation deletes the node in DataStudio and moves the node to the recycle bin.
	Freeze a node			Freeze a node on the Auto Triggered Nodes page of Operation Center.
	Unfreeze a node			Unfreeze a node on the Auto Triggered Nodes page of Operation Center.
Data backfilling for nodes	Backfill data for a node			Backfill data for an auto triggered node. Backfill data for an auto triggered node and the descendant nodes of the node. Backfill data for a large number of nodes. Backfill data in advanced mode.
Node status changing	Change the status of an auto triggered node			A scheduling task status change event can be generated in the following scenarios: A scheduling task is sent to Data Quality for data quality check. A scheduling task is checked based on branch conditions. A scheduling task is triggered as scheduled, manually triggered, or triggered by an event. The task status changes from Waiting for Resources to Running. A scheduling task finishes running no matter whether the running succeeds or fails. The task status changes from Running to Successful. A scheduling task is rerun after the scheduling task fails to run. The task status changes from Run Failed to Running. A scheduling task is scheduled to run. The task status changes to Waiting for Scheduling Time. The scheduling time is determined by the DueTime or cycleTime parameter.
Instance change event	Freeze an instance			When you perform any of the following operations on an auto triggered instance in the Operation Center, the corresponding event is triggered: Freeze an instance. Unfreeze an instance. Terminate an instance. Rerun an instance. Set the instance status to success. Delete a specified ancestor instance of an instance.
	Unfreeze an instance
	Terminate an instance
	Rerun an instance Rerun an instance Rerun the descendant instances of an instance
	Set the instance status to success
	Delete a specified ancestor instance of an instance
	Pre-event for instance frozen			When you perform any of the following operations on an auto triggered instance in the Operation Center, the corresponding event is triggered. Freeze an instance. Unfreeze an instance. Rerun an instance. Set the instance status to success. Terminate an instance.
	Pre-event for instance unfrozen
	Pre-event for instance rerun
	Pre-event for set instance success
	Pre-event for instance kill
	Delete an expired instance
Workflow status changing	Change the status of a workflow Backfill data for an instance Run a manually triggered workflow			Backfill data for a node. Run a manually triggered workflow.
Monitoring and alerting	Monitor objects and send alerts			Baseline alerts: If a node in a baseline does not finish running before the estimated completion time, an alert is triggered. Event-based alerts: If a node in a baseline fails to run, an alert is triggered. Rule-based alerts: If you create an alert rule for isolated nodes, an alert is triggered after an isolated node is identified in a workflow.

Security Center

For information about message formats of events in Security Center, see the Events of Security Center section in the "Development reference: Event lists and event message formats" topic.

Event category	Event	Regular event	Extension point event	Example (event-generation operation)
Request order creation or processing	Pre-event for approval order created			Create a request order for a function, table, or resource on the Permission Application tab of the Data Access Control page in Security Center.
Request order creation or processing	Complete a request order			Process a request order for tables, resources, or functions on the Permission Application Processing tab of the Data Access Control page in Security Center.
Table permission changing	Request permissions on tables			Apply for the read, write, and modify permissions on a specific table based on the business requirements.

Data Quality

For information about message formats of events in Data Quality, see the Events of Data Quality section in the "Development reference: Event lists and event message formats" topic.

Event category	Event	Regular event	Extension point event	Example (event-generation operation)
Data quality check	Provide feedback on data quality check results			Provide feedback on a check result in Data Quality. For example, you enable a monitoring rule that is used to check whether the number of rows in a table exceeds 0 for a specific table in Data Quality. After the rule is triggered and the check is complete, users can provide their feedback on the check result. After the feedback is provided, the system sends the feedback that contains information such as the monitoring rule ID and feedback content to a subscriber.
Data quality check	Report that a data quality check is complete			Send data quality check results, including whether the check is passed and the sampling information. For example, you enable a monitoring rule that is used to check whether the number of rows in a table exceeds 0 for a specific table in Data Quality. If the rule is triggered and the check is complete, the system sends the check result to a subscriber. The check result shows whether the data passes the quality check and contains samples that are used.
Pre-events for data quality rules	Pre-event for data quality rules batch create			For example, in the Data Quality module, go to the Configure Rules > Configure by Table page and open the details page of a specific table. On the Rule Management tab, performing the following operations triggers the corresponding pre-events: Create multiple data quality monitoring rules. Modify multiple data quality monitoring rules. Delete multiple data quality monitoring rules. Modify a single data quality monitoring rule.
	Pre-event for data quality rules batch update
	Pre-event for data quality rules batch delete
	Pre-event for data quality rules update
Pre-events for data quality evaluation task	Pre-event for data quality evaluation task create			For example, in the Data Quality module, go to the Rule Configuration > Configure by Table page and open the details page of a specific table. On the Monitor tab, performing the following operations triggers the corresponding pre-events: Create a data quality monitor. Modify a data quality monitor. Clone a data quality monitor. Delete a batch of data quality monitors.
	Pre-event for data quality evaluation task update
	Pre-event for data quality evaluation task clone
	Pre-event for data quality evaluation task batch delete
Pre-events for data quality evaluation task notification	Pre-event for data quality evaluation task notification create			For example, in the Data Quality module, go to the Rule Configuration > Configure by Table page and open the details page of a specific table. On the Monitor tab, subscribe to alerts for a specific data quality monitor.
	Pre-event for data quality evaluation task notification update			For example, in the Data Quality module, go to the Rule Configuration > Configure by Table page and open the details page of a specific table. On the Monitor tab, modify the alert subscription settings of a specific data quality monitor.
	Pre-event for data quality evaluation task notification delete			For example, in the Data Quality module, go to the Rule Configuration > Configure by Table page and open the details page of a specific table. On the Monitor tab, delete the alert subscription of a specific data quality monitor.

Tenant-level events

DataWorks console

For information about message formats of events in the DataWorks console, see the Events of the DataWorks console section in the "Development reference: Event lists and event message formats" topic.

Event category

Event

Regular event

Extension point event

Example (event-generation operation)

Workspace deletion

Delete a workspace (pre-event)

A pre-event for workspace deletion is generated before a user deletes a workspace on the Workspaces page. The pre-event for workspace deletion can trigger the related extension to block the process.

Delete a workspace (post-event)

Upload and Download

For information about message formats of events in Upload and Download, see the Events of Upload and Download section in the "Development reference: Event lists and event message formats" topic.

Event category	Event	Regular event	Extension point event	Example (event-generation operation)
Data download and upload	Download data (pre-event) (file generation)			A data download or upload event is generated when you download or upload data in Upload and Download. Before you download data, a pre-event for data download is generated to obtain information about the download operation.
	Download data (pre-event) (file download)
	Upload data (pre-event)

Application benefits

The OpenEvent module provides the following benefits:

Low O&M costs:
After you enable the message subscription feature in DataWorks Open Platform, you can subscribe to DataWorks events with ease.
Improved message publishing and receiving efficiency:
You need to only configure the parameters for message subscription and make preparations for SDK development. This reduces and simplifies manual operations.
Fulfillment of various needs:
You can subscribe to various types of DataWorks events. This allows you to identify important DataWorks changes and respond to the changes at the earliest opportunity.