Transferring Large User Data with Node.js and MongoDB: An Effective Approach

Fetching and inserting 200,000 users data from one database to another can be a significant task, especially if performance is a concern. A good approach would be to use asynchronous operations with proper batching to ensure we don’t overwhelm the system.

Here, I'll provide an example using Node.js with the MongoDB database. This will include reading users from one MongoDB database and inserting them into another MongoDB database in batches.

First, make sure you have the mongodb package installed:

npm install mongodb

Here's a script to achieve this:

const { MongoClient } = require('mongodb');

async function transferUsers(batchSize = 1000) {
  const sourceUri = 'mongodb://sourceDbUri';
  const targetUri = 'mongodb://targetDbUri';

  const sourceClient = new MongoClient(sourceUri, { useNewUrlParser: true, useUnifiedTopology: true });
  const targetClient = new MongoClient(targetUri, { useNewUrlParser: true, useUnifiedTopology: true });

  try {
    // Connect to both databases
    await sourceClient.connect();
    await targetClient.connect();

    const sourceDb = sourceClient.db('sourceDbName');
    const targetDb = targetClient.db('targetDbName');
    
    // Collections
    const sourceCollection = sourceDb.collection('users');
    const targetCollection = targetDb.collection('users');

    const totalUsers = await sourceCollection.countDocuments();
    console.log(`Total users to transfer: ${totalUsers}`);

    let offset = 0;

    while (offset < totalUsers) {
      const users = await sourceCollection.find().skip(offset).limit(batchSize).toArray();
      if (users.length === 0) break;

      await targetCollection.insertMany(users);
      console.log(`Transferred batch of ${users.length} users`);

      offset += batchSize;
    }

    console.log('Transfer complete');
  } catch (err) {
    console.error('Error during transfer:', err);
  } finally {
    await sourceClient.close();
    await targetClient.close();
  }
}

// Run the transfer with a batch size of 1000
transferUsers(1000);

Explanation:

1. Setup: We first import the mongodb package and setup connection URIs for the source and target databases.
2. Connection: We create clients for both source and target databases and connect to them.
3. Batch Process:
   • Determine the total number of users in the source database.
   • Iterate over the user records in batches. The batch size is configurable (defaulted to 1000 in this example).
   • Fetch a batch of users from the source collection, insert them into the target collection, and log the progress.
4. Error Handling and Cleanup: Catch errors during the process and ensure the database connections are closed properly.

This method batches the fetch and insert operations to avoid overloading memory and database operations, ensuring a smoother and faster transfer process. Adjust the batchSize as necessary to optimize for your specific use case.