MySQL Charset NULL Error

[eitamring]

MySQL Charset NULL Error - Complete Analysis & Fix

panic: failed to set charset from rows: failed to scan charset row: sql: Scan error on column index 1, name "CHARACTER_SET_NAME": converting NULL to string is unsupported 

This error occurred in the RiveryIO/go-mysql fork at line 521 in canal.go.

🔍 Root Cause Analysis

What Was Happening

1. The Application Flow:

   • CDC application uses the go-mysql library to connect to MySQL
   • When starting up, it calls GetColumnsCharsets() to load charset information for tables
   • This function queries MySQL's INFORMATION_SCHEMA.COLUMNS table to get charset data
   • The query tries to scan CHARACTER_SET_NAME values into Go variables

2. The Fatal Flaw:

   // Original problematic code func (c *Canal) setColumnsCharsetFromRows(tableRegex string, rows *sql.Rows) error { for rows.Next() { var ordinal int var charset, columnName string // ❌ PROBLEM: Using string type if err := rows.Scan(&ordinal, &charset, &columnName); err != nil { return errors.Annotate(err, "failed to scan charset row") } // ... rest of code } }

3. Why It Failed:

   • MySQL's CHARACTER_SET_NAME column can contain NULL values
   • Go's database/sql package cannot convert NULL database values directly to Go string types
   • When rows.Scan() encounters a NULL value, it throws the error: "converting NULL to string is unsupported"

The MySQL Data Issue

In MySQL, charset information can be NULL in these scenarios:

• Binary data types (BINARY, VARBINARY, BLOB types) don't have charsets
• Some MySQL versions/configurations may have incomplete charset metadata
• Corrupted or incomplete INFORMATION_SCHEMA data
• Custom MySQL distributions with missing charset information

1. SQL-Level Protection

Before (vulnerable to NULLs):

SELECT c.ORDINAL_POSITION, c.CHARACTER_SET_NAME, -- ❌ Can be NULL c.COLUMN_NAME FROM information_schema.COLUMNS c

After (NULL-safe):

SELECT c.ORDINAL_POSITION, COALESCE( c.CHARACTER_SET_NAME, -- Try column charset first col.CHARACTER_SET_NAME, -- Try table charset second 'utf8mb4' -- Always provide fallback ) AS CHARACTER_SET_NAME, c.COLUMN_NAME FROM information_schema.COLUMNS c LEFT JOIN information_schema.TABLES t ON t.TABLE_SCHEMA = c.TABLE_SCHEMA AND t.TABLE_NAME = c.TABLE_NAME LEFT JOIN information_schema.COLLATIONS col ON col.COLLATION_NAME = t.TABLE_COLLATION

2. Go-Level Protection

Before (crashes on NULL):

var charset string // ❌ Cannot handle NULL err := rows.Scan(&ordinal, &charset, &columnName)

After (handles NULL gracefully):

var charset sql.NullString // ✅ Can handle NULL values err := rows.Scan(&ordinal, &charset, &columnName) // Additional safety with fallback logic charsetValue := "utf8mb4" // default charset if charset.Valid && charset.String != "" { charsetValue = charset.String }

3. Error Handling Improvement

Before (crashes application):

func() { defer rows.Close() if err := c.setColumnsCharsetFromRows(tableRegex, rows); err != nil { panic(fmt.Errorf("failed to set charset from rows: %w", err)) // ❌ CRASHES } }()

After (graceful error handling):

defer rows.Close() if err := c.setColumnsCharsetFromRows(tableRegex, rows); err != nil { return fmt.Errorf("failed to set charset from rows for table %s: %w", tableRegex, err) // ✅ PROPER ERROR }

Handles All Edge Cases

• ✅ NULL charset → Uses utf8mb4 default
• ✅ Empty string charset → Uses utf8mb4 default
• ✅ Valid charset → Uses actual charset value
• ✅ Missing table metadata → Graceful fallback
• ✅ Database connection issues → Proper error messages

Tested and Verified

I created and ran tests that confirm:

NULL charset handled as: utf8mb4 Valid charset handled as: latin1 Empty string charset handled as: utf8mb4 ✅ All tests passed - NULL charset handling works correctly! 

[snyk-io]

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)