1

I have a website that's running on a production server, backed by mongodb. Every hour or so, some new data becomes available that I want to add to the db, inserting new documents, and updating old ones.

I get the new data on a development server, and update the dev server's copy of the db. If everything looks good, I want to send the new info to the production db, so that production now looks the same as dev's db. If tests fail, don't send changes to production until they pass. The changes I have in mind are not schema changes, but just new values for old fields, and new documents with the same fields as the others.

Several options occur to me for how to do this.

  1. With every update, do

{$set: {data_field: new_value, modified: ISODate()})

and then copy everything with appropriate modified to production.

  1. With every update, write the update command to a log, then run all of these commands on production.

  2. With every update, tail the oplog, and run all of the commands in the update on production.

Are all of these common approaches? I'd like to make this as low-maintenance as possible within standard practice. How can I decide between these methods?

Improve this question
3
  • 2
    Is this "development server" the same place you put freshly minted code that's not had the full Q/A treatment yet? Commented Oct 26, 2016 at 20:54
  • @DanPichelman The software is tested separately elsewhere, but the integrity of the database is tested by itself after each update on what I'm calling the development server (perhaps it's more accurately a 'staging' database). Commented Oct 26, 2016 at 20:57
  • 1
    Thanks. Perhaps it's a "production data staging box" :-). Commented Oct 26, 2016 at 21:00

1 Answer 1

Reset to default
2

We have to do something similar, and we use AWS services to help us out. If AWS isn't part of your picture, I'm sure something similar is possible in your infrastructure.

Risking potentially oversimplifying things, here's what we've got.

There are two fundamental parts of our solution.

  1. An S3 bucket where the new data lives.
  2. A script (which is part of our code base) that is capable of taking data and applying it to a MongoDB. It accepts a parameter: the location of the data in S3.

The script, since it is part of our code base, gets deployed to every environment. Included in the deploy is a configuration file used to specify the MongoDB server we connect to.

So our steps are:

  1. Update dev database to look like production database
  2. Run migration script (passing in the S3 path of the data) in the development environment.
  3. If development data looks good, continue. Otherwise go back to Step 1.
  4. Run migration script (passing in the S3 path of the data) in the production environment.

Hope this helps!

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.