Transforming Data Streams with Kafka Connect: An Introduction to Single Message Transforms Ranjan Rao
❏ Kafka Connect Overview ❏ Single Message Transforms ❏ Common Use Cases ❏ Built-in SMTs ❏ Example SMT Configuration ❏ Alternatives @ranjan_rao
Kafka Connect Overview ➔ Tool for streaming data from various sources to Kafka and from Kafka to various sink systems. @ranjan_rao Source Source Connector Sink Sink Connector
Single Message Transforms ➔ Provides a powerful and flexible way to transform data ➔ Simple function that accepts one record as an input and outputs a modified record ➔ Apache Kafka 0.10.2.0 @ranjan_rao
@ranjan_rao Single Message Transforms Database Source Connector Transform #1 Transform #N Converter Database Sink Connector Transform #N Converter Transform #1
✓ Kafka Connect Overview ✓ Single Message Transforms ❏ Common Use Cases ❏ Built-in SMTs ❏ Examples of SMT Configurations ❏ Alternatives @ranjan_rao
Common Use Cases ● Data enrichment/enhancement ● Partitioning ● Field masking ● Event routing @ranjan_rao
✓ Kafka Connect Overview ✓ Single Message Transforms ✓ Common Use Cases ❏ Built-in SMTs ❏ Example of SMT Configurations ❏ Alternatives @ranjan_rao
Built-in SMTs ● InsertField ● ReplaceField ● MaskField ● ValueToKey ● ExtractField @ranjan_rao ● TimestampRouter ● RegexRouter ● InsertHeader ● HeadersFrom ● DropHeaders
✓ Kafka Connect Overview ✓ Single Message Transforms ✓ Common Use Cases ✓ Built-in SMTs ❏ Example of SMT Configurations ❏ Alternatives @ranjan_rao
Example SMT configuration {“Firstname”: “John”, “LastName”: “Doe”,”Phone”: “000-000-0000” } {“Firstname”: “John”, “LastName”: “Doe”, “Phone”: “123-456-7890”} InsertField MaskField {“Firstname”: “John”, “LastName”: “Doe”,”Phone”: “ 000-000-0000”, “Origin”: “Source Connector”, “RecordTimestamp”: 1707940094 }
Example SMT configuration "transforms": "MaskField,InsertField", "transforms.MaskField.type": "org.apache.kafka.connect.transforms.MaskField$Value" , "transforms.MaskField.fields": "Phone", "transforms.MaskField.replacement" : "000-000-0000" "transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Value", "transforms.InsertField.static.field": "Origin", "transforms.InsertField.static.value": "Source Connector", "transforms.InsertField.timestamp.field": "RecordTimestamp" {“Firstname”: “John”, “LastName”: “Doe”,”Phone”:“000-000-0000”, “Origin”: “Source Connector”, “RecordTimestamp”: 1707940094 }
● Single Message Transforms for Confluent Platform ● Single Message Transforms for Kafka Connectors in Confluent Cloud @ranjan_rao Example SMT configuration
✓ Kafka Connect Overview ✓ Single Message Transforms ✓ Common Use Cases ✓ Built-in SMTs ✓ Example of SMT Configurations ❏ Alternatives @ranjan_rao
Alternatives ➢ Kafka Streams ➢ ksqlDB ➢ Apache Flink @ranjan_rao
Thank You!

Transforming Data Streams with Kafka Connect: An Introduction to Single Message Transforms

  • 1.
    Transforming Data Streamswith Kafka Connect: An Introduction to Single Message Transforms Ranjan Rao
  • 2.
    ❏ Kafka ConnectOverview ❏ Single Message Transforms ❏ Common Use Cases ❏ Built-in SMTs ❏ Example SMT Configuration ❏ Alternatives @ranjan_rao
  • 3.
    Kafka Connect Overview ➔Tool for streaming data from various sources to Kafka and from Kafka to various sink systems. @ranjan_rao Source Source Connector Sink Sink Connector
  • 4.
    Single Message Transforms ➔Provides a powerful and flexible way to transform data ➔ Simple function that accepts one record as an input and outputs a modified record ➔ Apache Kafka 0.10.2.0 @ranjan_rao
  • 5.
  • 6.
    ✓ Kafka ConnectOverview ✓ Single Message Transforms ❏ Common Use Cases ❏ Built-in SMTs ❏ Examples of SMT Configurations ❏ Alternatives @ranjan_rao
  • 7.
    Common Use Cases ●Data enrichment/enhancement ● Partitioning ● Field masking ● Event routing @ranjan_rao
  • 8.
    ✓ Kafka ConnectOverview ✓ Single Message Transforms ✓ Common Use Cases ❏ Built-in SMTs ❏ Example of SMT Configurations ❏ Alternatives @ranjan_rao
  • 9.
    Built-in SMTs ● InsertField ●ReplaceField ● MaskField ● ValueToKey ● ExtractField @ranjan_rao ● TimestampRouter ● RegexRouter ● InsertHeader ● HeadersFrom ● DropHeaders
  • 10.
    ✓ Kafka ConnectOverview ✓ Single Message Transforms ✓ Common Use Cases ✓ Built-in SMTs ❏ Example of SMT Configurations ❏ Alternatives @ranjan_rao
  • 11.
    Example SMT configuration {“Firstname”:“John”, “LastName”: “Doe”,”Phone”: “000-000-0000” } {“Firstname”: “John”, “LastName”: “Doe”, “Phone”: “123-456-7890”} InsertField MaskField {“Firstname”: “John”, “LastName”: “Doe”,”Phone”: “ 000-000-0000”, “Origin”: “Source Connector”, “RecordTimestamp”: 1707940094 }
  • 12.
    Example SMT configuration "transforms":"MaskField,InsertField", "transforms.MaskField.type": "org.apache.kafka.connect.transforms.MaskField$Value" , "transforms.MaskField.fields": "Phone", "transforms.MaskField.replacement" : "000-000-0000" "transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Value", "transforms.InsertField.static.field": "Origin", "transforms.InsertField.static.value": "Source Connector", "transforms.InsertField.timestamp.field": "RecordTimestamp" {“Firstname”: “John”, “LastName”: “Doe”,”Phone”:“000-000-0000”, “Origin”: “Source Connector”, “RecordTimestamp”: 1707940094 }
  • 13.
    ● Single MessageTransforms for Confluent Platform ● Single Message Transforms for Kafka Connectors in Confluent Cloud @ranjan_rao Example SMT configuration
  • 14.
    ✓ Kafka ConnectOverview ✓ Single Message Transforms ✓ Common Use Cases ✓ Built-in SMTs ✓ Example of SMT Configurations ❏ Alternatives @ranjan_rao
  • 15.
    Alternatives ➢ Kafka Streams ➢ksqlDB ➢ Apache Flink @ranjan_rao
  • 16.