Skip to content

Support extracting schema details (size/type) of BINARY column type #480

@ckolbitsch-work

Description

@ckolbitsch-work

Users of the binlog syncer/streamer may need to know the type and size of table columns, but schema.go currently does not expose detailed type/size information (especially for BINARY column types).

This type of information is useful when having to replay DML events for BINARY columns, as MySQL will pad fixed size columns with 0-bytes, and padded values do not match non-padded values. This is described in the documentation here:

https://dev.mysql.com/doc/refman/5.7/en/binary-varbinary.html

Particularly this section is important:

When BINARY values are stored, they are right-padded with the pad value to the specified length. The pad value is 0x00 (the zero byte). Values are right-padded with 0x00 for inserts, and no trailing bytes are removed for retrievals. All bytes are significant in comparisons, including ORDER BY and DISTINCT operations. 0x00 and space differ in comparisons, with 0x00 sorting before space.

Since 0-bytes may not need appear in the binlogs and need to be added by the caller of the binlog syncer/streamer, but the schema analysis in go-mysql is currently not extracting the needed information. This led to cases where callers were not correctly applying DML events, as seen - for example - here:

Shopify/ghostferry#157

To handle this properly, we need to extract BINARY columns as dedicated type and extract the size metadata for a column

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions