Distributed Metadata Migration

How to move large-scale metadata systems without losing correctness.

Migrating a metadata system sounds straightforward:

1. Copy existing records to the new system.
2. Replay writes that happened during the copy.
3. Validate the result.
4. Cut traffic over.

At small scale, that may work. At cloud scale, the migration itself becomes a distributed system.

The source may be sharded. The destination may use a different partitioning model. Objects may be small, large, versioned, or spread across many records. Writes may continue while the migration is running. Validation may take hours or days. A failed migration may leave partial state that must be cleaned up safely.

The hard part is not only copying data. The hard part is proving that the destination is complete, ordered, and safe to become authoritative.

Core Risks

A large metadata migration has four common risks:

• Drift: source keeps changing while the destination is being copied.
• Partial copy: only some records, pages, or chunks reach the destination.
• Stale replay: old migration writes arrive after newer ones.
• Unsafe cutover: traffic moves before validation is strong enough.

Any of these can create missing records, duplicate records, broken listing behavior, stale metadata, or cleanup bugs.

Testing Comes First

Before building the happy path, define the failures the migration must survive:

• copy worker crashes mid-batch
• replay worker sends the same batch twice
• stale write arrives after a newer write
• validation finds a missing object
• cutover starts while writes are still arriving
• cleanup sees a failed partial migration
• large objects require many metadata pages
• small buckets and large buckets need different strategies

These tests become the real design requirements.

A Safer Migration Pattern

snapshot -> copy -> replay -> validate -> cutover -> cleanup -> recover

The destination is built and validated before it becomes authoritative. The source remains authoritative until cutover is safe.

Source System       Migration State       Destination System       Clients
    |                     |                      |                    |
    |---- snapshot ------>|                      |                    |
    |                     |                      |                    |
    |---- copy batch ---->|---- write batch ---->|                    |
    |---- copy batch ---->|---- write batch ---->|                    |
    |                     |                      |                    |
    |---- change logs --->|---- replay logs ---->|                    |
    |                     |                      |                    |
    |                     |---- validate ------->|                    |
    |                     | count/list/checksum  |                    |
    |                     |                      |                    |
    |                     |==== CUTOVER ========>|---- active ------->|
    |                     |  authority boundary  |                    |
    |                     |                      |                    |
    |<--- cleanup old ----|                      |                    |
    |                     |                      |                    |
    |<--------------- recover / retry from migration state ---------->|

External authority should stay simple:

Before cutover: source = authoritative, destination = shadow
After cutover:  source = retired,       destination = authoritative

Step Summary

• Snapshot: define the migration start point and the set of records to copy.
• Copy: move existing metadata in batches, often partitioned by shard or virtual cell.
• Replay: apply writes that happened after the snapshot using generation logs or change streams.
• Validate: compare source and destination using list validation, counters, sizes, versions, or checksums.
• Cutover: stop or narrow writes, run final validation, and make the destination authoritative.
• Cleanup: remove partial, failed, or retired migration state only after it is safe.
• Recover: make every batch, replay, validation, and cleanup step retry-safe.

Ordering Matters

During migration, the destination may receive repeated or delayed writes. Each partition should reject stale work.

One practical pattern is:

• assign a monotonically increasing sequence number per destination partition
• allow only one outstanding request per partition when ordering matters
• accept only writes with a strictly newer sequence
• treat stale sequence responses as safe duplicate/retry outcomes

This prevents old replay traffic from overwriting newer destination state.

Validation Matters

Validation is not a single check. Different buckets or datasets may require different validation levels.

Small datasets can sometimes be validated fully during a short write pause. Large datasets often need staged validation:

• validate copied data
• validate replayed logs
• validate list output
• validate per-partition counters
• perform final validation during cutover

The goal is to fail early when possible and avoid discovering correctness bugs only after days of copying.

Where This Pattern Applies

This pattern applies to:

• storage metadata migrations
• database shard migrations
• search-index rebuilds
• tenant moves between cells
• cache ownership migration
• table-partition migration
• control-plane metadata upgrades
• object-store namespace migration

In each case, the system is not just moving records. It is changing which system is authoritative.

Production Test Cases

Validate the migration against failure and scale cases:

• copy succeeds but validation fails
• replay sends duplicate batches
• stale sequence arrives after a newer sequence
• worker crashes after partial batch write
• final validation fails during cutover
• cleanup sees partial destination state
• small dataset cutover completes with full validation
• large dataset requires staged validation
• recovery runs twice for the same migration
• destination rejects stale writes safely

Expected behavior is always the same: no missing metadata, no stale overwrite, no unsafe cutover, no cleanup of needed state, and safe retry from recorded migration state.

Core Lesson

Large-scale metadata migration is not a copy job. It is a correctness protocol.

Copy in batches -> replay changes -> validate continuously -> cut over once -> clean up safely