Last week I set up my first MySQL multi-master replication in a load-balanced WebGUI cluster. Since then, the databases have desynced three times in the middle of the night because of duplicate key conflicts, causing a chain of even more duplicate key conflicts due to users performing the same action on both databases while they are out of sync.

I've since fixed my load balancer's session affinity, but it brings up a problem: WebGUI's database writes are not very sync-friendly.

I've altered a couple of the writes to use REPLACE INTO instead of INSERT INTO, specifically in groupings, and I plan to add some features to lineage to stagger the "next child lineage" to avoid lineage key conflicts if the sync lags a bit, but is this enough of a solution? Is it a solution at all?

The changes I'm making fix groupings and lineage, but what about username, which is a unique key input directly by the user. How can I prevent sync problems when the user is giving me the unique data? Since the replication happens async, the user is long-gone when the problem shows up, so I can't just catch an exception and inform the user. 

Is requiring a load balancer with session affinity the best solution to prevent desyncing? Should we require all db writes to perform uniqueness integrity checks with all database masters? Or should we start moving towards being DB agnostic so we can use some DBM that performs synchronous replication? (though I hear MySQL 6.0 has a "semi-sync replication" feature built-in)

What do we need to do to make WebGUI perform well in larger database clusters? What problems have you run into doing this and how did you correct them?