monSQLize
  • English

      • insertBatch - batch insertion (supports automatic retry)

      API parameter description

      Method signature

      collection(name: string).insertBatch(
  documents: object[],
  options?: InsertBatchOptions
): Promise<InsertBatchResult>

      Detailed explanation of parameters

      First parameter: documents (required)

      • Type: object[]
      • Description: Array of documents to be inserted

      Second parameter: options (optional)

      ParametersTypeDefault valueDescription
      batchSizenumber1000Number of documents inserted in each batch
      concurrencynumber1Number of concurrent batches (1=serial, >1=parallel)
      orderedbooleanfalseWhether the batch is inserted in order
      onProgressFunction-Progress callback function (progress) => {}
      onErrorstring'stop'Error handling strategy: 'stop'/'skip'/'collect'/'retry'
      retryAttemptsnumber3Maximum number of retries for failed batches (valid when onError='retry')
      retryDelaynumber1000Retry delay time (milliseconds)
      onRetryFunction-Retry callback function (retryInfo) => {}
      writeConcernobject{ w: 1 }Write confirmation level
      bypassDocumentValidationbooleanfalseWhether to bypass document validation
      commentstring-Operation comments (for log tracking)

      Return value

      {
  acknowledged: boolean,      //Is it confirmed
  totalCount: number,          //Total number of documents
  insertedCount: number,       //Number of successful insertions
  batchCount: number,          //Total number of batches
  errors: Array<Object>,       //error list
  retries: Array<Object>,      //Retry record list (new)
  insertedIds: Object          //Inserted document _id mapping table
}

      Progress callback parameters

      {
  currentBatch: number,    //Current batch number (starting from 1)
  totalBatches: number,    //Total number of batches
  inserted: number,        //Inserted quantity
  total: number,           //total quantity
  percentage: number,      //Complete percentage (0-100)
  errors: number,          //number of errors
  retries: number          //Number of retries (new)
}

      Usage example

      1. Basic usage - automatic batch insertion

      const largeDataset = Array.from({ length: 10000 }, (_, i) => ({
  name: `User ${i + 1}`,
  email: `user${i + 1}@example.com`,
  createdAt: new Date()
}));

const result = await collection('users').insertBatch(largeDataset, {
  batchSize: 1000  // 1000 items per batch, automatically divided into 10 batches
});

console.log(`Successfully inserted ${result.insertedCount}/${result.totalCount} documents`);
console.log(`Total ${result.batchCount} batches, ${result.errors.length} errors`);

      2. Progress monitoring

      await collection('products').insertBatch(largeDataset, {
  batchSize: 500,
  onProgress: (progress) => {
    console.log(
      `Progress: ${progress.percentage}%` +
      `(Batch ${progress.currentBatch}/${progress.totalBatches})`
    );
  }
});

//Output:
//Progress: 20% (Batch 1/5)
//Progress: 40% (Batch 2/5)
//Progress: 60% (Batch 3/5)
//Progress: 80% (Batch 4/5)
//Progress: 100% (Batch 5/5)

      3. Automatic retry mechanism ⭐ New features

      3.1 retry strategy - automatic retry on failure

      const result = await collection('items').insertBatch(unstableData, {
  batchSize: 1000,
  onError: 'retry',      //Automatically retry on failure
  retryAttempts: 3,      //Retry up to 3 times
  retryDelay: 1000,      //Delay 1 second for each retry
  onRetry: (retryInfo) => {
    console.log(
      `Batch ${retryInfo.batchIndex + 1} Retrying...` +
      `(${retryInfo.attempt}/${retryInfo.maxAttempts} times)`
    );
  }
});

console.log(`Success: ${result.insertedCount}`);
console.log(`Retries: ${result.retries.length} batches`);
console.log(`Final failure: ${result.errors.length} batches`);

//View retry details
result.retries.forEach(retry => {
  console.log(
    `Batch ${retry.batchIndex + 1}: Retry ${retry.attempts} times,` +
    `${retry.success ? 'success' : 'failed'}`
  );
});

      4. Comparison of error handling strategies

      4.1 stop strategy (default) - stop on error

      try {
  await collection('items').insertBatch(dataWithDuplicate, {
    batchSize: 1000,
    onError: 'stop'  //Stop immediately when encountering an error
  });
} catch (error) {
  console.log('Insertion failed:', error.message);
  //The previous successful batch has been inserted, but subsequent batches have not been executed.
}

      4.2 skip strategy - skip failed batches

      const result = await collection('items').insertBatch(dataWithErrors, {
  batchSize: 1000,
  onError: 'skip'  //Skip failed batches and continue with subsequent batches
});

console.log(`Success: ${result.insertedCount}, failed batch: ${result.errors.length}`);
//Output: Success: 8000, Failure batch: 2

      4.3 collect strategy - collect all errors

      const result = await collection('items').insertBatch(dataWithErrors, {
  batchSize: 1000,
  onError: 'collect'  //Collect all errors and complete execution
});

if (result.errors.length > 0) {
  console.log('Error details:');
  result.errors.forEach((err, idx) => {
    console.log(`Batch ${err.batchIndex + 1}: ${err.message}`);
  });
}

      5. Concurrent insertion (accelerate big data import)

      //serial insert (default)
await collection('data').insertBatch(largeDataset, {
  batchSize: 1000,
  concurrency: 1  //Insert batch by batch
});

//Concurrent inserts (faster)
await collection('data').insertBatch(largeDataset, {
  batchSize: 1000,
  concurrency: 3  //3 batches of parallel inserts
});

//⚠️ Note: Too large a concurrency may overwhelm the database. Recommended value: 2-5

      6. Combined with comment parameter (production environment)

      await collection('logs').insertBatch(logData, {
  batchSize: 2000,
  comment: 'DataMigration:logs:v2.0',  //Easy to track
  onProgress: (progress) => {
    if (progress.percentage % 10 === 0) {
      console.log(`Migration progress: ${progress.percentage}%`);
    }
  }
});

      Comparison of error handling strategies

      StrategyBehaviorApplicable ScenariosPerformance
      stopStop immediately when an error occursHigh data consistency requirementsFastest (stop when an error occurs)
      skipSkip failed batchesAllow partial failuresMedium
      collectCollect all errorsFull error report requiredSlower (execute all)
      retry ⭐Automatically retry failed batchesNetwork instability, temporary failureSlowest (with retry delay)

      Strategy Selection Guide

      //Data import - using skip or retry
await collection('products').insertBatch(importData, {
  onError: 'retry',  //Automatically retry temporary failures
  retryAttempts: 3,
  onProgress: (p) => console.log(`${p.inserted} items imported`)
});

//Data migration - using stop
await collection('users').insertBatch(migrationData, {
  onError: 'stop',  //Stop when encountering an error to ensure data integrity
  writeConcern: { w: 'majority', j: true }
});

//Data validation - using collect
const result = await collection('test').insertBatch(testData, {
  onError: 'collect'  //Collect all errors and generate a complete report
});

console.log(`Verification completed: ${result.insertedCount} successful, ${result.errors.length} failed`);

      Performance optimization suggestions

      1. Batch size (batchSize)

      //❌ Too small - too many batches, large network overhead
await collection('data').insertBatch(data, { batchSize: 10 });

//❌ Too big - possible memory overflow or timeout
await collection('data').insertBatch(data, { batchSize: 100000 });

//✅ Recommended range: 500-2000
await collection('data').insertBatch(data, { batchSize: 1000 });

      Selection Guide:

      • Small document (< 1KB): batchSize: 1000-2000
      • Chinese Document (1-10KB): batchSize: 500-1000
      • Large Document (> 10KB): batchSize: 100-500

      2. Concurrency control (concurrency)

      //✅ Local database - allows for higher concurrency
await collection('data').insertBatch(data, {
  batchSize: 1000,
  concurrency: 5  //The local network is fast and can be set to high speed
});

//✅ Remote database - be conservative
await collection('data').insertBatch(data, {
  batchSize: 1000,
  concurrency: 2  //The remote network is slow, avoid timeouts
});

//✅ Production environment - more conservative
await collection('data').insertBatch(data, {
  batchSize: 500,
  concurrency: 1,  //Serial is the most stable
  writeConcern: { w: 'majority', j: true }
});

      3. Retry policy configuration

      //✅ Unstable network environment
await collection('data').insertBatch(data, {
  onError: 'retry',
  retryAttempts: 5,      //Try multiple times
  retryDelay: 2000,      //longer delay
  onRetry: (info) => {
    console.log(`Batch ${info.batchIndex + 1} retry ${info.attempt}`);
  }
});

//✅ Stable environment, fast failure
await collection('data').insertBatch(data, {
  onError: 'stop',       //No retry, immediate failure
  retryAttempts: 0
});

      FAQ

      Q: How to choose insertBatch vs insertMany?

      A: Select based on data volume:

      • < 5K items: Use insertMany (easier)
      • 5K-50K items: Use insertBatch (safer)
      • > 50K items: Must use insertBatch (to avoid timeouts)

      Q: How to set batchSize?

      A: Consider the following factors:

      1. Document size: The larger the document, the smaller batchSize
      2. Network speed: The slower the network, the smaller the batchSize
      3. Database Performance: The weaker the database, the smaller batchSize
      4. Recommended starting point: Use 1000 first, and adjust according to the actual situation

      Q: When should the retry mechanism be used?

      A: The following scenarios are suitable for retrying:

      • ✅ Network instability (WiFi, mobile network)
      • ✅ High database load (temporary connection failure)
      • ✅ Lock conflict (may succeed after waiting)
      • ❌ Data error (retry will not succeed)
      • ❌ Permission problem (retry will not succeed)

      Q: Will concurrency lead to data inconsistency?

      A: No. insertBatch ensures:

      • Each batch is inserted independently
      • insertedIds is mapped in original order
      • Error handling is associated with batches
      • Explicit cache invalidation is supported

      Q: How to deal with partial failure?

      A: Use collect or retry strategy:

      const result = await collection('data').insertBatch(data, {
  onError: 'retry', // or'collect'
  retryAttempts: 3
});

//Retry failed batches
for (const err of result.errors) {
  const failedDocs = data.slice(
    err.batchStartIndex,
    err.batchStartIndex + err.batchSize
  );

  //Analyze the cause of the failure, clean the data and try again
  console.log(`Batch ${err.batchIndex + 1} failed: ${err.message}`);
  console.log(`Retried ${err.attempts} times`);
}

      References