MySQL change event producer

The MySql Producer relies on Open Replicator. It uses the Open Replicator to read MySql bin logs and convert them into change events. The Open Replicator project has been forked to https://github.com/Flipkart/open-replicator for the following reasons:

• Enable support for MySql version 5.6 Bin Log while the original version supports only version 5.5 event formats.
• Support for parsing Checksum introduced in version 5.6.

The implementation details of this producer is as follows:

• MySql Producer registers a Replication Listener with Open Replicator. A Call back method is invoked on the Replication Listener when events are processed by the Open Replicator.
• Various Event Processors are registered in MySql producer to handle specific event types viz. Insert, Update, Delete, InsertV2, UpdateV2, DeleteV2 for DML and various type of DDL events.
• For DML events
  • Transaction Manager handles these events. The Transaction Manager groups events under a Transaction and manages life cycle of the Transaction until it is committed. Once the Transaction is committed, the Transaction Manager adds the events into Event Buffer.
• For DDL (Currently only Alter is supported) events
  • The AVRO Schema Generator is called to regenerate the Schema for the table and update the schema registry in Relay.
  • Propagation of the DDL events to Clients is not yet supported.

The MySql producer generates SCN by a combination of the bin log sql file number and the location within the bin log. In the resulting SCN, the binlog file number occupies the high 32 bits and the location occupies the low 32 bits.

• Example if the bin log file is 1 and the location within the bin log file is 4 (from start of bin log file) then SCN would be (1<<32)|4