MySQL Cluster -> Galera MariaDB Cluster Replication not working, but no errors

Question

I am setting up a database migration between a 2-node master cluster running MySQL Cluster 7.2 (MySQL 5.5) and a 2-node slave cluster (plus arbitrator) running MariaDB 10.3 with Galera clustering. I currently have all but one of the Galera cluster's nodes turned off to simplify the choreography.

Configuration appears correct, and everything in SHOW ALL SLAVES STATUS \G appears to indicate successful replication. Read_ and Exec_Master_Log_Pos match each other and what the master server reports, no errors are logged, etc. Indeed, adding a database to the master node we're replicating from will create a database in the slave node. However, none of the data rows are being replicated.

The only oddity in mariadb.err is as follows:

2019-04-11 18:31:19 101 [Warning] Master '[redacted]': Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Internal MariaDB error code: 1193 

Master Replication Config:

server-id=13 binlog-format = row log-bin=mysql-bin sync_binlog = 1 

Slave Replication Config:

server-id = 1 log_bin = /var/log/mysql/mariadb-bin log_bin_index = /var/log/mysql/mariadb-bin.index expire_logs_days = 10 max_binlog_size = 100M relay_log = /var/log/mysql/relay-bin relay_log_index = /var/log/mysql/relay-bin.index relay_log_info_file = /var/log/mysql/relay-bin.info slave_sql_verify_checksum = 0 log_slave_updates read_only binlog_format=row 

SHOW MASTER STATUS;

File: mysql-bin.000049 Position: 4494988 Binlog_Do_DB: '' Binlog_Ignore_DB: '' 

SHOW ALL SLAVES STATUS \G

*************************** 1. row *************************** Connection_name: [redacted]13 Slave_SQL_State: Slave has read all relay log; waiting for the slave I/O thread to update it Slave_IO_State: Waiting for master to send event Master_Host: [redacted] Master_User: replicant Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000049 Read_Master_Log_Pos: 4494988 Relay_Log_File: relay-bin-[redacted].000002 Relay_Log_Pos: 2982721 Relay_Master_Log_File: mysql-bin.000049 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 4494988 Relay_Log_Space: 2983028 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 13 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: No Gtid_IO_Pos: Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: conservative SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it Slave_DDL_Groups: 0 Slave_Non_Transactional_Groups: 0 Slave_Transactional_Groups: 0 Retried_transactions: 0 Max_relay_log_size: 104857600 Executed_log_entries: 28995 Slave_received_heartbeats: 57 Slave_heartbeat_period: 30.000 Gtid_Slave_Pos: 1 row in set (0.000 sec) 

Initiating the replication

On master, I execute the following:

mysqldump -S /data/mysql/mysql.sock -u [redacted] -p --opt --skip-lock-tables --single-transaction --flush-logs --master-data=2 --databases [redacted] > replication.sql 

I then copy the sql to the slave server, and pipe it into the Galera node. This brings the slave up-to-date with the point of the dump.

I then run the following on the slave server:

head -n100 replication.sql | grep "MASTER_LOG_POS" 

to get the correct starting file and position for replication, then run the following in SQL:

CHANGE MASTER '[redacted]' TO MASTER_HOST='[redacted]', MASTER_USER='replicant', MASTER_PASSWORD='[redacted]', MASTER_LOG_FILE='[file]', MASTER_LOG_POS=[position]; 

I've already reviewed dozens of posts on this site and others over the past few days, and haven't found a solution yet. Here are some examples to demonstrate diligence:

MySQL replication not working - no errors - I turned off all filtering to ensure this wasn't the problem.

mysql replication fail (no error) - We're using row-based bin-logging

MySQL master-slave replication not updating - No real solutions here. I confirmed that all the positive indicators listed by the first answer are true for me as well.

MySQL says replication is fine but data is not copied - Not using filtering, server IDs are different, etc.

Answer 1

After much experimentation with various versions of MariaDB and MySQL in docker containers, I have determined that MariaDB versions of 10.0 or greater will silently fail and continue when they encounter unreadable row events.

Using MariaDB 5.5's mysqlbinlog in a docker container revealed the following errors from replication:

ERROR: Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 42, event_type: 30 ERROR: Could not read entry at offset 938: Error in log format or read error. 

This error falls outside the scope of the question as asked.