Database Migration: A Marvel Movie

Yes, so I am back with yet another blog.

So, I've been MIA on the blogging scene for a hot minute. To be honest, life hasn't exactly been a rollercoaster of excitement lately. But hold up, because this blog is all about database migration. Apologies for the detour, let's dive right back in.

Now, when it comes to wrangling data from one place to another, there's a golden rule: don't do it while your customers are peeking. Trust me, you don't want to accidentally make their precious data disappear into thin air. So, the easiest trick is to schedule some downtime.

Another way is to start writing the data into both databases and on the same hand migrate the data, without disturbing the balance. Will it increase complexity? Yes. Will there be a few edge cases that make the user journey annoying? Yes. So should you have taken a downtime? Maybe.

Speaking of migration adventures, let's talk about how I moved our stuff from Mongo Atlas to our very own self-hosted Mongo setup. My job? Well, it was like being the designated mover. Imagine lugging around 60 gigs of content – links, text, and whatnot. Yeah, that's a pretty hefty collection. But fear not, we had a couple of commands up our sleeve to help us out: mongodump and mongorestore.

mongodump --host <hostname> --port <port> --db <database_name> --out <output_directory>

mongorestore --host <hostname> --port <port> --db <database_name> <input_directory>

FYI

Mongodump doc Mongorestore doc

The rest were collections ranging from 1MB to 200MB, so the scripts were okay. For the smaller ones, we just let the scripts work their magic. But for the bigger collections, we decided to take it slow. Instead of throwing everything into the migration blender at once, we approached it step by step, moving collections one by one.

Before we kicked things into gear, we used the stats command to snoop around and get a feel for the kind of data we were wrangling. A little reconnaissance never hurt anybody. When the time came to shift documents, we did it in batches of 1000. Oh, and here's a nifty trick I used: exporting some of the less-frequently used collections as good old JSON files. Surprisingly effective.

The majority of the process was pretty procedural – keeping an eye on what's moved, giving collections a thorough once-over, and then putting everything to the test after the switch. The real MVP, you ask? It was the stress of anticipating what might go wrong. But guess what? Things went kinda smoothly, I had some help with changing the secrets on github and re-triggering the workloads, changed secrets across the cluster because we have two microservices + some cron jobs. I'm convinced this entire migration dance was less about typing lines of code and more about planning, with a keen eye for quality.

And there you have it, folks! Until our next tech escapade, stay grounded and keep it real.

Database Migration: A Marvel Movie

Comments (1)

More from this blog

Don't you worry about Integrations

Do you understand your Code?

What I did in April

What did I do in March?

What did I do in February

Command Palette

Comments (1)

More from this blog