MySQL RBR (Row Based Replication) /w GTID halts with Error_code: 1032; handler error HA_ERR_END_OF_FILE

MySQL RBR (Row Based Replication) /w GTID halts with Error_code: 1032; handler error HA_ERR_END_OF_FILE

I want to start off by saying that there are a lot of causes for the error in the title of this article, most of which are easy to fix conditions centering around misconfiguration. However none of them applied to our situation here.

After weeks/months of troubleshooting and isolating on a customer’s (unfortunately production) servers we believe we’ve identified a key bug in MySQL’s row based replication method (RBR) which produces errors like this on the “slave” server (we’re using GTID multi-master so both servers are slaved to each other):

Could not execute Update_rows event on table customer1.table1; Can't find record in 'table1', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log db1-bin.000001, end_log_pos 344884
        
        Could not execute Update_rows event on table customer1.table1; Can't find record in 'table1', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log db1-bin.000001, end_log_pos 9685352
        
        Could not execute Update_rows event on table customer2.table1; Can't find record in 'table1', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log db1-bin.000001, end_log_pos 28653123
        
        Could not execute Delete_rows event on table customer3.table2; Can't find record in 'table2', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log db1-bin.000001, end_log_pos 179801477

Our eventual conclusion was that these tables did not have any indexes which had a UNIQUE value set, which apparently causes big problems for RBR. Once we added an index which did have UNIQUE set, the problem vanished. In our case we created an extra column specifically for this and indexed that so it didn’t interfere with the existing data layout.

We’ve opened a MySQL bug report for this but have yet to see any movement on it:

MySQL Bugs: #78061 GTID replication halts with Error_code: 1032; handler error HA_ERR_END_OF_FILE

We also created a script which identifies all tables in all databases that meet this condition:

no-unique-key-scan.sh

Hopefully this helps others become aware of this and work around this issue.