clickhouse-kafka-connect is the official Kafka Connect sink connector for ClickHouse.

The Kafka connector delivers data from a Kafka topic to a ClickHouse table.

See the ClickHouse website for the full documentation entry.

For a full overview of the design and how exactly-once delivery semantics are achieved, see the design document.

For additional help, please file an issue in the repository or raise a question in ClickHouse public Slack.

We've created a transformation that allows you to convert a Kafka message key into a value. This is useful when you want to store the key in a separate column in ClickHouse - by default, the column is _key and the type is String.

CREATE TABLE your_table_name ( `your_column_name` String, ... ... ... `_key` String ) ENGINE = MergeTree()

Simply add the transformation to your connector configuration:

transforms=keyToValue transforms.keyToValue.type=com.clickhouse.kafka.connect.transforms.KeyToValue transforms.keyToValue.field=_key

There is a dedicated gradle project in this repo - benchmark for performance testing. Please see its README for more information and how to run.