Score:0

What is the best AWS infrastructure to run a migration of data from one SQL server to mongoDB?

id flag

I have some terabytes of data in our legacy system which runs SQL server. Our newer version runs on MongoDB. We are migrating this data to MongoDB. We have python scripts written and verfied, all data movement happens properly.

we did this on a lower machine which 4 cores, if we do it on bigger machine, its going to be very expensive. AWS Lambda has 15 minutes processing time, this takes more than 24 hrs for one iteration to finish. AWS step functions promises it, but not sure if it is the right one.

in flag
Does this answer your question? [Can you help me with my capacity planning?](https://serverfault.com/questions/384686/can-you-help-me-with-my-capacity-planning)
J Bourne avatar
id flag
Thanks for response, but Im not looking for capacity planning, I have those details, I need what is the best infra (mostly from AWS cloud) to run my scripts uniterruptedly for few days, which would migrate data of SQL to mongo, so that they are finished on time.
Score:2
gp flag
Tim

Can you not do "mongoexport" locally, export to S3 (or a physical AWS Snowcone device), use an EC2 instance to "mongoimport", then run your script to do any updates since the dump?

As for how to run it, you would probably get away with using a spot EC2 instance, particularly if you use it outside peak hours for the region - perhaps a weekend. If your job can't be interrupted then on-demand EC2. An m5.xlarge with 4 cores / 16GB RAM is $0.20 per hour, a couple of days of that is $10.

I'll also point out that say 3TB at 100Mbps will take 2.6 days to send, but at 800Mbps will take 7 hours - but sustaining that bandwidth may be difficult without DirectConnect. You might be best off using an AWS Snowcone which is a physical device you copy data to then ship to AWS.

I would suggest using AWS Database Migration Service to migrate from MongoDB to AWS DocumentDB, which is their version of MongoDB with a different name. DMS will migrate the data, then you just point your application at the new instance and turn the old one off.

Oscar De León avatar
la flag
For "only" 3 TB a Snowball maybe overkill. Perhaps consider a Snowcone?
Tim avatar
gp flag
Tim
Yeah, I was thinking the Snow family, I'll edit to be more precise.
J Bourne avatar
id flag
Im already using m5.xlarge, its not helping much. but thanks for snowcone and snow family, will take a look at it.
Tim avatar
gp flag
Tim
You're best placed to consider instance requirements since you have access to metrics - do you need more CPU, more RAM, more or faster storage, or something else? I do think a snow family device will be a good choice for this.
J Bourne avatar
id flag
Yes, I need more CPU and RAM, highspeed, Im not doing offline datamigration, Im doing from one live DB server to another live DB server.
Tim avatar
gp flag
Tim
The cost of a larger instance for a day or two to migrate a huge database shouldn't be much. I would look at using AWS Database Migration Service to migrate to AWS DocumentDB which is basically MongoDB with a different label, instead of using custom scrips and your own EC2 instance. Answer updated.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.