DataWorks:Elasticsearch data source

Parameter

Description

Required

Default value

datasource

The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor.

Yes

No default value

index

The name of the index from which you want to read data.

Yes

No default value

type

The name of the index type in the Elasticsearch cluster.

No

Index name

search

The query parameter of Elasticsearch.

Yes

No default value

pageSize

The number of data records to read at a time.

No

100

scroll

Elasticsearch paging parameter that sets the cursor storage time.

• If you set this parameter to an excessively small value and the interval at which two pages of data is obtained exceeds the value of the scroll parameter, the scroll may expire and data may be lost.

• If you set this parameter to an excessively large value and the number of query requests initiated at the same point in time exceeds the value of the max_open_scroll_context parameter of the server side, an error may be reported for data queries.

Yes

No default value

strictMode

Specifies whether to read data from the Elasticsearch cluster in strict mode. In strict mode, if a shard.failed error is reported, Elasticsearch Reader stops reading data to prevent some data from failing to be read.

No

true

sort

The field based on which the returned results are sorted.

No

No default value

retryCount

The maximum number of retries allowed after a failure.

No

300

connTimeOut

The connection timeout period of the client.

No

600,000

readTimeOut

The data read timeout period of the client.

No

600,000

multiThread

Specifies whether to use multiple threads for an HTTP request.

No

true

preemptiveAuth

Specifies whether to use the preemptive request mode for an HTTP request.

No

false

retrySleepTime

The interval between retries after a failure.

No

1000

discovery

Specifies whether to enable the node discovery mechanism.

• true: The node discovery mechanism is enabled. Data Integration connects to a random node in the Elasticsearch cluster. If node discovery is enabled, the server list in the client is polled and regularly updated, and the nodes that are discovered are queried.

• false: Initiates a query request to the configured endpoint.

No

false

compression

Specifies whether to compress a request body in the GZIP format. If you set this parameter to true, you must enable the http.compression settings on the Elasticsearch cluster.

No

false

dateFormat

The dateFormat parameter is required if the fields to be synchronized include fields of a date data type and the format setting is not configured for mappings of the fields. You must specify all formats that are required to synchronize the fields of a date data type in this parameter, such as "dateFormat" : "yyyy-MM-dd||yyyy-MM-dd HH:mm:ss".

No

No default value

full

Specifies whether to synchronize all fields in a document in an Elasticsearch cluster to the destination as a field, and use the queried data in an Elasticsearch cluster as a field. For more information, see Scenario 1: Extract all data.

No

None

multi

You can configure this parameter to enable the advanced feature that supports five solutions to help Elasticsearch Reader convert semi-structured data into structured data. The two child properties are multi.key and multi.mult. For more information about the configurations of the advanced feature, see Advanced features in the table.

No

No default value

Parameter

Description

Required

Default value

datasource

The name of the Elasticsearch data source that you want to synchronize. If no data sources are available, add an Elasticsearch cluster to DataWorks as a data source. For more information, see Add an Elasticsearch data source.

Yes

No default value

index

The name of the index to which you want to write data.

Yes

No default value

indexType

The name of the index type in the destination Elasticsearch cluster.

No

Elasticsearch

cleanup

Specifies whether to delete the existing data from the index before data write. Valid values:

• true: Deletes the original index and creates an index whose name is the same as the original index. This way, the existing data in the index is deleted.

• false: Retains the existing data in the index.

No

false

batchSize

The number of data records to write at a time.

No

1,000

trySize

The maximum number of retries that can be performed after a failure occurs.

No

30

timeout

The connection timeout of the client.

No

600,000

discovery

Specifies whether to enable node discovery.

• true: The node discovery mechanism is enabled. Data Integration connects to a random node in the Elasticsearch cluster. The server list in the client is polled and regularly updated.

• false: The node discovery mechanism is disabled. In this case, Data Integration connects to the Elasticsearch cluster.

No

false

compression

Specifies whether to enable compression for an HTTP request.

No

true

multiThread

Specifies whether to use multiple threads for an HTTP request.

No

true

ignoreWriteError

Specifies whether to ignore write errors and proceed with data write operations without retries.

No

false

ignoreParseError

Specifies whether to ignore format parsing errors and proceed with data write operations.

No

true

alias

The alias of the index to which you want to write data. The alias feature of Elasticsearch is similar to the view feature of a database. For example, if you create an alias named my_index_alias for the index my_index, the operations that are performed on my_index_alias also take effect on my_index.

If you configure the alias parameter, the alias that you specify in this parameter is created for the index after data is written to the index.

No

No default value

aliasMode

The mode in which an alias is added after data is written to the index. Valid values: append and exclusive:

• If you set the aliasMode parameter to append, an alias is added for the index. One alias maps to multiple indexes.

• If you set the aliasMode parameter to exclusive, the existing alias of the index is deleted and a new alias is added for the index. One alias maps to one index.

Elasticsearch Writer can convert aliases to actual index names. You can use aliases to migrate data from one index to another index, search for data across multiple indexes in a unified manner, and create a view on a subset of data in an index.

No

append

settings

The settings of the index. The settings must follow the specifications of open source Elasticsearch.

No

No default value

column

The column parameter is used to configure the fields of the document. For each field, you can configure basic parameters such as name and type, and advanced parameters such as Analyzer, Format, and Array.

- id // The id type corresponds to the _id type in Elasticsearch and can be considered as the unique primary key. When Elasticsearch Writer writes data to Elasticsearch, data that has the same ID is overwritten and not indexed. - string - text - keyword - long - integer - short - byte - double - float - date - boolean - binary - integer_range - float_range - long_range - double_range - date_range - geo_point - geo_shape - ip - token_count - array - object - nested

If you need to configure properties other than the type in the column parameter, you can use the other_params parameter. This parameter is configured in the column parameter and is used to describe Elasticsearch property information other than the type when updating mappings.

 { "name": "guid", "other_params": { "doc_values": false }, "type": "text" }

If you want to write source data to Elasticsearch as arrays, you can enable Elasticsearch Writer to parse the source data in the JSON format or based on a specified delimiter. For more information, see Appendix 2: Write data to Elasticsearch as arrays.

Yes

No default value

dynamic

Specifies whether to use the dynamic mapping mechanism of Elasticsearch to establish mappings for fields that are written to the index.

• true: Uses the dynamic mapping mechanism of Elasticsearch to establish mappings.

• false: Establishes field mappings and updates the mapping configurations of the index based on the setting of the column parameter. This is the default value. If you leave the dynamic parameter empty, the default value is used.

In Elasticsearch V7.X, the default value of the type parameter is _doc. If you use the dynamic mapping mechanism of Elasticsearch, set the type parameter to _doc and the esVersion parameter to 7.

You need to switch to the code editor and add a version parameter: "esVersion": "7".

No

false

actionType

Indicates the action type when Elasticsearch writes data. Currently, Data Integration supports two actionType values: index and update. The default value is index:

• index: Elasticsearch Writer uses Index.Builder of an Elasticsearch SDK to construct a request for writing multiple data records at a time. In index mode, Elasticsearch Writer first checks whether an ID is specified for the document that you want to insert:

  • If no ID is specified, Elasticsearch Writer generates a unique ID. In this case, the document is directly inserted into the Elasticsearch cluster.

  • If an ID is specified, the existing document is replaced with the document that you want to insert. You cannot modify specific fields in the document.

    Note

    This update operation is different from the update operation in Elasticsearch (which replaces only specific columns).

• update: Elasticsearch Writer updates the existing document based on the ID that you specify. If the specified ID does not exist in the index, a new document is inserted. If the specified ID exists in the index, Elasticsearch Writer updates the fields that are specified in the column parameter to the document. Other fields in the document remain unchanged. Each time Elasticsearch Writer updates the document, Elasticsearch Writer obtains the information of the whole document to update specific fields in the document. The update operation does not support filter conditions and can be performed based only on a specified ID. Because each update requires obtaining the original document, this may have a significant impact on performance.

  Note

  If you set the actionType parameter to update, you must configure the primaryKeyInfo parameter.

No

index

primaryKeyInfo

Specifies the value assignment method of the _id column that is used as the primary key to write data to Elasticsearch.

• Business Primary Key (pk): The value of the _id column is a specific field.

  "parameter":{ "primaryKeyInfo":{ "type":"pk", "column":["id"]} }

• Composite Primary Key (specific): The value of the _id column is obtained by concatenating the values of specific fields. The delimiter that is used to concatenate the values is the same as the delimiter that is specified by using the Primary Key Separator fieldDelimiter.

  Note

  The field names are the names of fields to which Elasticsearch Writer writes data. When you configure the batch synchronization task by using the codeless UI, the system extracts only fields that exist in the Elasticsearch index as the Configurations Of Primary Key Columns.

  "parameter":{ "primaryKeyInfo":{ "type":"specific", "fieldDelimiter":",", "column":["col1","col2"]} }

• No Primary Key (nopk): The value of the _id column is automatically generated when data is written to Elasticsearch.

  "primaryKeyInfo":{ "type":"nopk" }

Yes

specific

esPartitionColumn

Specifies whether to enable partitioning for the index when data is written to Elasticsearch. This parameter is used to change the routing setting of the Elasticsearch cluster.

• true: Enables partitioning. In this case, the value of a specific column is used to insert a document to a specific shard or update a document in a specific shard. If you set this parameter to true, you must specify the partition key columns.

  { "esPartitionColumn": [ { "name":"col1", "comment":"xx", "type":"STRING" } ], }

• false: Disables partitioning. If you leave this parameter empty, the values of the _id column are used to evenly distribute documents to shards. This prevents data skew.

No

false

enableWriteNull

Specifies whether source fields whose values are NULL can be synchronized to Elasticsearch. Valid values:

• true: Source fields whose values are NULL can be synchronized to Elasticsearch. After the fields are synchronized, the values of the fields in Elasticsearch are also NULL.

• false: Source fields whose values are NULL cannot be synchronized to Elasticsearch. The fields are not displayed in Elasticsearch.

No

true