#Production Rollout: Data and Index Sync

Use this guide before moving a monSQLize service into production or before deploying a version that changes data shape, sync targets, Model index declarations, or collection indexes.

monSQLize provides the database runtime building blocks for this path:

• Change Stream sync for asynchronous CDC after writes reach MongoDB.
• Model index preflight through ensureIndexes() and ensureModelIndexes().
• Collection index APIs such as listIndexes(), createIndex(), createIndexes(), and dropIndex().
• Data tasks through msq.dataTasks and monsqlize data-task for reviewed index sync, filtered data sync, field transforms, snapshots, and verification.

It does not replace MongoDB native import/export, a full backup/restore policy, or an exactly-once data pipeline. Treat dataTasks as bounded release tasks: explicit filter, dry-run, snapshot, controlled run, and verification.

#When to Use This Page

Use this page when you need any of these release steps:

#Production Release Sequence

1. Create a database backup or restore point.
2. Deploy code with autoIndex: false for production services.
3. Run the index dry-run against the target environment.
4. Resolve index conflicts before creating missing indexes.
5. Run msq.dataTasks.dryRun() / monsqlize data-task dry-run, then execute run with production confirmation if the release needs historical data changes.
6. Enable Change Stream sync only for ongoing CDC after prerequisites are ready.
7. Check counts, sample records, sync stats, slow queries, and error logs.
8. Switch traffic only after the release gates are green.

#Index Sync

For Model-declared indexes, disable automatic index creation in production and use explicit preflight.

Automatic indexing also preflights with listIndexes(), skips existing indexes, and creates only missing indexes. Production services should still prefer autoIndex: false because startup-time asynchronous index creation is not a release gate: you still need a deliberate dry-run, conflict review, duplicate-data checks for unique indexes, a maintenance window for heavy builds, and an operator-visible rollback plan.

const msq = new MonSQLize({
  type: 'mongodb',
  databaseName: 'app',
  config: { uri: process.env.MONGODB_URI },
  autoIndex: false
});

await msq.connect();

const plan = await msq.ensureModelIndexes({
  models: ['users', 'orders'],
  dryRun: true
});

console.log(plan.totals);

if (plan.totals.conflicts === 0) {
  await msq.ensureModelIndexes({
    models: ['users', 'orders'],
    throwOnError: true
  });
}

ensureModelIndexes() and ModelInstance.ensureIndexes() compare declared indexes with existing database indexes. Dry-run mode only reports existing, missing, and conflicts. Execution mode creates missing indexes only. It does not drop, rename, or rebuild conflicting indexes.

For non-Model collections, use collection index APIs directly:

const users = msq.collection('users');

const existing = await users.listIndexes();
console.log(existing.map(index => index.name));

await users.createIndexes([
  { key: { email: 1 }, unique: true, name: 'users_email_unique' },
  { key: { status: 1, createdAt: -1 }, name: 'users_status_createdAt' }
]);

Production index checklist:

• Run dry-run first when using Model indexes.
• Review unique index changes against existing duplicate data.
• Avoid creating heavy indexes during peak traffic.
• Record the current index list before destructive changes.
• Use dropIndex() only when rollback and query impact are understood.
• Recheck slow-query logs and explain() output after traffic moves.

#Data Migration Sync

Use dataTasks for release-scoped historical data changes. A task should be bounded by filter, match records by stable business keys, snapshot affected target documents before writes, and verify the result before traffic moves.

const task = {
  name: 'sync-active-users',
  source: { collection: 'sourceUsers' },
  target: { collection: 'targetUsers' },
  filter: { status: 'active' },
  matchBy: ['email'],
  snapshot: { dir: '.monsqlize/snapshots' },
  steps: [
    {
      type: 'ensureIndexes',
      indexes: [
        { key: { email: 1 }, options: { unique: true }, name: 'target_users_email_unique' }
      ]
    },
    { type: 'syncData', strategy: 'upsert' },
    { type: 'transformFields', update: { $set: { schemaVersion: 2 } } },
    { type: 'verify', count: true, fields: ['schemaVersion'], indexes: true }
  ]
};

const plan = await msq.dataTasks.plan(task);
const dryRun = await msq.dataTasks.dryRun(task);

if (plan.errors.length === 0 && dryRun.errors.length === 0) {
  await msq.dataTasks.run(task, { confirmProduction: true });
  const verify = await msq.dataTasks.verify(task);
  if (!verify.passed) {
    throw new Error('data task verification failed');
  }
}

CLI form:

monsqlize data-task plan --task ./tasks/sync-active-users.cjs --json
monsqlize data-task dry-run --task ./tasks/sync-active-users.cjs
monsqlize data-task run --task ./tasks/sync-active-users.cjs --confirm-production
monsqlize data-task verify --task ./tasks/sync-active-users.cjs

Task recommendations:

• Use a stable filter and set allowFullCollection: true only after an explicit review.
• Use business matchBy fields for cross-endpoint sync. Source _id matching is blocked by default.
• Keep target writes idempotent; reruns should update or skip already synced documents.
• Review the snapshot path and checksum before continuing a production run.
• Use MongoDB native tooling or an application-owned job for full database copy, backup, restore, or very large batch pipelines.

#Change Stream Sync After Backfill

Change Stream sync is for ongoing CDC, backup targets, projections, cache invalidation callbacks, and other asynchronous target updates. It is not a substitute for the first full historical backfill.

const msq = new MonSQLize({
  type: 'mongodb',
  databaseName: 'app',
  config: {
    uri: process.env.MONGODB_URI,
    replicaSet: 'rs0'
  },
  sync: {
    enabled: true,
    targets: [
      {
        name: 'backup-main',
        uri: process.env.BACKUP_MONGODB_URI,
        collections: ['users', 'orders']
      }
    ],
    resumeToken: {
      storage: 'redis',
      redis,
      strictSave: true,
      strictLoad: true,
      saveRetries: 3
    },
    idempotency: {
      store: durableStore,
      markMode: 'success'
    }
  }
});

Operational boundaries:

• MongoDB must be a replica set for Change Streams.
• Sync is at-least-once, not exactly-once.
• Built-in MongoDB targets are idempotent for replacement/upsert style writes.
• Custom apply targets should deduplicate by change event _id.
• Monitor getSyncStats().isRunning, errorCount, lastError, target errors, and token save errors.
• If a token is lost, do not assume the historical gap is covered; run a repair or full comparison job.

#Traffic Switch Checklist

Before switching traffic:

• npm run release:preflight has passed for package release readiness.
• Target database has the expected collections and indexes.
• Model index dry-run reports no conflicts.
• Missing indexes have been created or intentionally deferred.
• Migration/backfill counts match the expected scope.
• Sync stats are healthy and no target is stopped.
• Slow-query logs do not show new index misses.
• Rollback path and backup restore point are documented.

After switching traffic:

• Watch sync stats and application error logs.
• Compare source and target counts for migrated collections.
• Sample critical records across old and new read paths.
• Run representative queries with explain() when latency changes.
• Keep the old rollback point until the release window closes.

#Related Documents

• Data Tasks
• Change Stream Sync
• Distributed Deployment
• Create indexes in bulk
• List indexes
• Drop an index
• Model Overview
• Slow Query Logging