monSQLize
  • English

      • deleteBatch - delete documents in batches

      API parameter description

      Method signature

      collection(name: string).deleteBatch(
  filter: object,
  options?: DeleteBatchOptions
): Promise<DeleteBatchResult>

      Detailed explanation of parameters

      First parameter: filter (required)

      • Type: object
      • Description: Deletion condition, same as deleteMany

      Second parameter: options (optional)

      ParametersTypeDefault valueDescription
      batchSizenumber1000Number of documents deleted in each batch
      estimateProgressbooleantrueWhether to pre-count the total number (for progress percentage)
      onProgressFunction-Progress callback function (progress) => {}
      onErrorstring'stop'Error handling strategy: 'stop'/'skip'/'collect'/'retry'
      retryAttemptsnumber3The maximum number of retries for a failed batch (when onError='retry')
      retryDelaynumber1000Retry delay time (milliseconds)
      onRetryFunction-Retry callback function (retryInfo) => {}
      writeConcernobject{ w: 1 }Write confirmation level
      commentstring-Operation comments (for log tracking)

      Return value

      {
  acknowledged: boolean,      //Is it confirmed
  totalCount: number | null,  //Total number of documents (valid when estimateProgress=true)
  deletedCount: number,       //Number of successful deletions
  batchCount: number,         //Total number of batches
  errors: Array<Object>,      //error list
  retries: Array<Object>      //Retry record list
}

      Progress callback parameters

      {
  currentBatch: number,    //Current batch number (starting from 1)
  totalBatches: number,    //Total number of batches
  deleted: number,         //Deleted quantity
  total: number | null,    //Total quantity (valid when estimateProgress=true)
  percentage: number | null, //Complete percentage (0-100, has value when estimateProgress=true)
  errors: number,          //number of errors
  retries: number          //Number of retries
}

      Usage example

      1. Basic usage - clear expired logs

      //Delete logs older than 90 days
const expireDate = new Date(Date.now() - 90 * 24 * 60 * 60 * 1000);

const result = await collection('logs').deleteBatch(
    { createdAt: { $lt: expireDate } },
    { batchSize: 5000 }
);

console.log(`Delete ${result.deletedCount} expired logs`);

      2. With progress monitoring - cleaning large amounts of data

      const result = await collection('logs').deleteBatch(
    { level: 'debug' },
    {
        batchSize: 5000,
        estimateProgress: true,  //Pre count, display percentage
        onProgress: (progress) => {
            console.log(`Progress: ${progress.percentage}% (${progress.deleted}/${progress.total} items)`);
        }
    }
);

      Example output:

      Progress: 20% (100000/500000 items)
Progress: 40% (200000/500000 items)
Progress: 60% (300000/500000 items)
Progress: 80% (400000/500000 items)
Progress: 100% (500000/500000 items)

      3. No pre-count - avoid performance overhead

      //When the amount of data is particularly large, counting in advance is very slow, so you don’t need to count.
const result = await collection('logs').deleteBatch(
    { status: 'archived' },
    {
        batchSize: 5000,
        estimateProgress: false,  //No pre-count
        onProgress: (progress) => {
            //percentage is null, but you can still see the deleted count
            console.log(`Deleted: ${progress.deleted} items (batch ${progress.currentBatch})`);
        }
    }
);

      4. Error handling - stop strategy (default)

      const result = await collection('logs').deleteBatch(
    { userId: { $in: userIds } },
    {
        batchSize: 1000,
        onError: 'stop'  //Stop immediately when encountering an error
    }
);

if (result.errors.length > 0) {
    console.error('Delete failed:', result.errors[0].message);
}

      5. Error handling - skip strategy

      const result = await collection('temp_data').deleteBatch(
    { category: 'test' },
    {
        batchSize: 5000,
        onError: 'skip'  //Skip failed batches and continue with subsequent batches
    }
);

console.log(`Successfully deleted: ${result.deletedCount} items`);
console.log(`Failure batch: ${result.errors.length}`);

      6. Error handling - retry strategy (recommended)

      const result = await collection('logs').deleteBatch(
    { status: 'expired' },
    {
        batchSize: 5000,
        onError: 'retry',      //Automatically retry on failure
        retryAttempts: 3,      //Retry up to 3 times
        retryDelay: 1000,      //1 second between retries
        onRetry: (info) => {
            console.log(`Batch ${info.batchIndex + 1} retry ${info.attempt}...`);
        }
    }
);

console.log(`Number of retries: ${result.retries.length}`);

      7. Error handling - collect strategy

      const result = await collection('logs').deleteBatch(
    { type: 'temp' },
    {
        batchSize: 1000,
        onError: 'collect'  //Collect all errors and continue execution
    }
);

//View all errors
result.errors.forEach((err, idx) => {
    console.log(`Batch ${err.batchIndex + 1} Error: ${err.message}`);
});

      8. Complex query conditions

      //Delete documents that match multiple criteria
const result = await collection('orders').deleteBatch(
    {
        status: 'cancelled',
        createdAt: { $lt: new Date('2024-01-01') },
        $or: [
            { paymentStatus: 'unpaid' },
            { amount: { $eq: 0 } }
        ]
    },
    {
        batchSize: 5000,
        estimateProgress: true,
        onProgress: (p) => {
            console.log(`Delete progress: ${p.percentage}%`);
        }
    }
);

      9. Use comment to track operations

      const result = await collection('logs').deleteBatch(
    { level: 'debug' },
    {
        batchSize: 5000,
        comment: 'cleanup-debug-logs'  //It will be displayed in the slow query log
    }
);

      Performance optimization suggestions

      1. Batch size selection

      Data volumeRecommended batchSizeReason
      < 100,0001000-2000Small batch, quick response
      100,000-1 million3000-5000Balance performance and memory
      > 1 million5000-10000Large batches, reducing network overhead
      //Example: Dynamically adjust based on data volume
const totalCount = await collection('logs').count({ status: 'expired' });
const batchSize = totalCount > 1000000 ? 10000 : 5000;

await collection('logs').deleteBatch(
    { status: 'expired' },
    { batchSize }
);

      2. Whether to count in advance

      //✅ Small data volume: count in advance, display progress
if (estimatedCount < 1000000) {
    await collection('logs').deleteBatch(filter, {
        estimateProgress: true  //Show percentage
    });
}

//✅ Large data volume: no count to avoid performance overhead
else {
    await collection('logs').deleteBatch(filter, {
        estimateProgress: false  //Doesn't show percentage, but faster
    });
}

      3. Index optimization

      //Make sure there is an index before deleting
await collection('logs').createIndex({ createdAt: 1 });

//and then delete
await collection('logs').deleteBatch(
    { createdAt: { $lt: expireDate } },
    { batchSize: 5000 }
);

      4. Wrong strategy selection

      ScenarioRecommended strategyReason
      Production environment cleanupretryAutomatically retry to reduce failures
      Test data cleaningskipQuick cleaning, skip failures
      Deletion of critical datastopStop immediately when an error occurs to ensure consistency
      Batch cleaning taskscollectCollect all errors and process them afterwards

      FAQ

      Q1: What is the difference between deleteBatch and deleteMany?

      Compare itemsdeleteBatchdeleteMany
      Applicable data volume> 10000 items< 10000 items
      Memory usageConstant (streaming)Linear growth
      Progress Monitoring✅ Supported❌ Not Supported
      Error Handling✅ 4 Strategies❌ Only Fail All
      Auto Retry✅ Supported❌ Not Supported
      PerformanceBetter for large data volumesFaster for small data volumes

      Suggestions:

      • Data volume < 10000 items → use deleteMany
      • Data volume ≥ 10000 → use deleteBatch

      Q2: Will deleteBatch cause data inconsistency?

      Answer: deleteBatch processes matching _id values through a cursor and deletes them in batches. It does not create a MongoDB transaction or guarantee snapshot isolation by itself. If you need a transactional snapshot, run it inside an explicit transaction and pass the transaction session.

      //✅ Security: Even if other operations insert new data at the same time, it will not be deleted accidentally.
await collection('logs').deleteBatch(
    { createdAt: { $lt: expireDate } },
    { batchSize: 5000 }
);

      Q3: How do I know which documents have been deleted?

      //Method 1: Query before deleting
const toDelete = await collection('logs').find({ status: 'expired' });
console.log('Will be deleted:', toDelete.map(d => d._id));

//then delete
await collection('logs').deleteBatch({ status: 'expired' });

//Method 2: Use soft delete
await collection('logs').updateBatch(
    { status: 'expired' },
    { $set: { deleted: true, deletedAt: new Date() } }
);

      Q4: Will deleteBatch trigger slow query logs?

      Answer: Yes. If the delete operation exceeds the threshold (default 500ms), slow query logs will be recorded.

      //You can see it in the slow query log
//[WARN] [deleteBatch] Slow operation warning {
//   ns: 'mydb.logs',
//   duration: 25000,
//   deletedCount: 500000,
//   batchCount: 100
// }

      Q5: Can deleteBatch be used in a transaction?

      Answer: Yes, but please note:

      const session = await msq.startSession();

try {
    await session.withTransaction(async () => {
        //✅Supports use in transactions
        await collection('orders').deleteBatch(
            { status: 'cancelled' },
            { batchSize: 1000 }
        );

        await collection('payments').deleteBatch(
            { orderId: { $in: cancelledIds } },
            { batchSize: 1000 }
        );
    });
} finally {
    await session.endSession();
}

      Q6: How to estimate the deletion time?

      //Performance Reference (In-Memory Database)
//Deletion speed: about 30000-50000 items/second

const totalCount = 1000000;
const estimatedTime = totalCount / 40000;  //about 25 seconds

console.log(`Estimated time: ${Math.ceil(estimatedTime)} seconds`);

      References