MySQL replication lag increasing

Question

I have simple master-slave replication setup. Everything worked with no problems for more than a year.

Few days ago replication lag started to rise and it's still rising with no obvious reason.

Seconds_Behind_Master is now more than 4 days and 6 hours.

Both SQL and IO slave threads are working, there is no problem with communication.

SHOW SLAVE STATUS\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: s1.my.pl Master_User: root Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.001674 Read_Master_Log_Pos: 10948252 Relay_Log_File: mysqld-relay-bin.000509 Relay_Log_Pos: 63370742 Relay_Master_Log_File: mysql-bin.001638 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 63370596 Relay_Log_Space: 3523067396 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 371329 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 1 row in set (0.00 sec) 

Show full processlist produces strange output in one row "exit mysqld_lock_tables()", I'm not sure what it means

 SHOW FULL PROCESSLIST; +-----+-------------+-----------+------+---------+--------+----------------------------------+-----------------------+-----------+---------------+-----------+ | Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined | Rows_read | +-----+-------------+-----------+------+---------+--------+----------------------------------+-----------------------+-----------+---------------+-----------+ | 808 | system user | | NULL | Connect | 892 | Waiting for master to send event | NULL | 0 | 0 | 1 | | 809 | system user | | NULL | Connect | 371477 | exit mysqld_lock_tables() | NULL | 0 | 0 | 1 | | 810 | root | localhost | NULL | Query | 0 | NULL | SHOW FULL PROCESSLIST | 0 | 0 | 401 | +-----+-------------+-----------+------+---------+--------+----------------------------------+-----------------------+-----------+---------------+-----------+ 

SHOW_OPEN_TABLES command shows that no table is locked

Any hints what can I do to fix this or how to get more detailed diagnosis?

Answer 1

One of the problems with MySQL replication is the serialization of I/O in the slave. While the master performs several I/O tasks at once, they get written to a single relay log, which is then executed by the slave, hence serializing the I/O. Therefore your slave should always be capable of more I/O throughput than your master.

To see if you've actually got a I/O problem, install ioprompt (homepage).

In your case, there could be another problem, you might have run into a bug. Have you tried

stop slave; start slave; 

to see if that fixes the problem? Next thing try stopping and starting your MySQL slave process.