Score:0

Master DB latency with AWS RDS cross region replication

mg flag

I have my app setup in 3 regions (EU, AP, US) and the master MySQL RDS resides in eu-west-1, with read replicas in the other regions.

This works great for read queries, with the region specific app connecting to it's local read replica RDS very quickly, but when my app needs to perform a write query, it has to connect to the master DB in eu-west-1.

When writing to the master DB from US or AP, the latency is huge, usually taking around 2.5s to complete the insert.

I've been really struggling to find any info on how to overcome this, Aurora comes up a lot in forums and tutorials for global databases, but it requires replication of the instance type, with db.r5 being the minimum, which soon becomes very expensive when running multiple instances.

Has anybody faced this issue with slow cross-region writes to the master DB? Would VPC peering help speed this up?

Wilson Hauck avatar
jp flag
What is result of SELECT @@binlog_format; ? What is your instance type? The same in all regions? What is result of SELECT @@version; ?
Score:0
gp flag
Tim

This is not a full answer, more some thoughts to try and questions that won't fit in a comment box. Please try to resist the urge to downvote :)

VPC peering is definitely worth a shot, that way traffic stays on the AWS backbone which should reduce latency a bit. I don't know how much that will help. Those three areas are 200 - 300ms ping apart, so you will always have some delays.

I suspect the conversation between client and DB is multiple requests for one insert - eg create connection, connect to specific DB, insert, commit, close. If that's the case reducing latency helps, but cutting out some of the steps is more important. Are you using connection pooling so the connections are already open? I suspect VPC Peering and general optimization this will be a better solution that either of the ideas below.

If there any way you could make the updates asynchronous? If you can put writes into an SQS queue processed in a single region it'll probably be done within a second or two. This might be an optimization over direct database connections, depending how fast it is.

Multi-master is another option, using database native replication features. I'm not entirely sure if you can do this in RDS, but it's maybe worth a look at if it's possible and the advantages / disadvantages. If you expect people to update the same record at the same time you will have to protect against that.

Another option could be sharding, with specific users data on specific databases. That's going to make your application logic more complex though.

mg flag
Thanks a lot @Tim, some good tips here for me to do more research on....
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.