MySQL change event producer

Jump to bottom

jagadeesh-huliyar edited this page Feb 23, 2015 · 15 revisions

MySql Producer relies on Open Replicator. Open Replicator reads the MySql bin log and converts them into events.
Open Replicator has been forked and changes made. Open Replicator with changes can be found at https://github.com/Flipkart/open-replicator
Open Replicator reads bin log and convert it into events.
- Open Replicator supported MySql version 5.5.
- Enhancements have been made to Open Replicator to support MySql version 5.6.
- Bin Log format for MySql version 5.6 is different from that of 5.5 for certain data types. Support has been added to handle the new format.
- Other enhancements include support for handling checksum feature in bin log.
MySql Producer registers a listener with Open Replicator. When events are processed in Open Replicator the Replication Listener in MySql producer is given a call back.
Various Event Processors are registered in MySql producer to handle specific event types (Insert, Update, Delete, InsertV2, UpdateV2, DeleteV2 for DML and various type of DDL events).
The event processors are called to execute business logic based on the type of event.
- For events of type DML
  - Transaction Manager is called to handle the event. The Transaction Manager maintains the events and handles the life cycle of the Transaction until the Transaction has been committed.
  - Once the Transaction is committed the Transaction Manager adds the events into Event Buffer.
- For events of type DDL (Currently for Alter)
  - The AVRO Schema Generator is called to regenerate the Schema for the table and update the schema registry in Relay.
  - This event (DDL) needs to be propagated the Client. This needs to be done. Not yet coded.
MySql producer generates SCN by a combination of the bin log sql file number and the location with in the bin log.
The binlog file number is left shifted by 32 bits and the location with in it added to it.
Example if the bin log file is 1 and the location within the bin log file is 4 (start of bin log file) then SCN would be ((1<<32)+4).