Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[CBG-4554] Sync Gateway Import Failure During Migration to Non-Empty Bucket

Yuvraj
Updated 
Product: Sync Gateway
Component: sync-gateway
Issue Link: CBG-4554
Affects Version: 3.2.x
Fix Version: 3.3.0

Summary

  • This issue was discovered during migration from self-hosted Couchbase Mobile Clusters to Capella App Service, specifically while importing/migrating data into a non-empty bucket. However, it is not limited to Capella and can impact any Sync Gateway migration.
  • Newly generated documents did not receive Sync Gateway (_sync) metadata in their xattrs (part of the document metadata). This will block synchronization of impacted documents to Couchbase Lite devices.
  • The issue appears after forced checkpoint rollbacks to zero, causing a subset of DCP streams to fail to re-open and causing specific vbuckets to skip import.
  • The issue was observed in a currently unsupported scenario: Any kind of migration or restoration into a non-empty bucket is not supported (at the time of writing this article). This will cause duplicate sequence numbers in documents in cases where the destination bucket has received write operations.

Symptoms 

  • Newly imported documents are missing _sync metadata.
  • Couchbase Lite devices do not receive updates for the documents.
  • Following rollback, all DCP streams may fail to reopen, leaving some vbuckets idle (and skipping import).

Triggers

  • Migration or restoration into a non-empty bucket, especially with prior Sync Gateway history in the destination bucket.
  • Generating new documents directly into the bucket (rather than through App Service API or Sync Gateway API)

Verification

Identify affected documents by running:

SELECT META().id FROM `_default` WHERE META().xattrs._sync IS MISSING;

Indicators of issue from logs:

  • Rollback events during migration, e.g.:
2025-02-28T00:25:02.043Z [WRN] c:SGI db:endpoint DCP RollbackEx request - rolling back DCP feed for: vbucketId: 768, rollbackSeq: 0. -- base.(*DCPCommon).rollbackEx() at dcp_common.go:166
2025-02-28T00:25:02.047Z [INF] DCP: c:SGI db:endpoint RollbackEx:769, 74061953563387, 0
2025-02-28T00:25:02.047Z [WRN] c:SGI db:endpoint DCP RollbackEx request - rolling back DCP feed for: vbucketId: 769, rollbackSeq: 0. -- base.(*DCPCommon).rollbackEx() at dcp_common.go:166
2025-02-28T00:25:02.052Z [INF] DCP: c:SGI db:endpoint RollbackEx:770, 249582766442309, 0
  • Rollbacks are accompanied by janitor: rollbackPIndex for pindex returning with err: manager: registered pindex exists
2025-02-28T00:25:03.545Z [INF] DCP: janitor: rollbackPIndex for pindex db0xb8cc1d82_index_7997cab3efe92946_103fc5fb, err: janitor: error registering pindex: db0xb8cc1d82_index_7997cab3efe92946_103fc5fb, err: manager: registered pindex exists, name: db0xb8cc1d82_index_7997cab3efe92946_103fc5fb
2025-02-28T00:25:03.545Z [INF] DCP: c:SGI db:endpoint Starting sharded feed for SGI.  Total partitions:6
2025-02-28T00:25:03.545Z [INF] DCP: c:SGI db:endpoint Using DCP Logging Receiver
2025-02-28T00:25:03.545Z [INF] DCP: janitor: rollbackPIndex for pindex db0xb8cc1d82_index_7997cab3efe92946_103fc5fb, err: janitor: error registering pindex: db0xb8cc1d82_index_7997cab3efe92946_103fc5fb, err: manager: registered pindex exists, name: db0xb8cc1d82_index_7997cab3efe92946_103fc5fb
  • Increase in Starting sharded feed for SGI partition counts during/after rollback events:
2025-02-28T00:25:03.545Z [INF] DCP: c:SGI db:endpoint Starting sharded feed for SGI.  Total partitions:5
2025-02-28T00:25:03.545Z [INF] DCP: c:SGI db:endpoint Using DCP Logging Receiver
2025-02-28T00:25:03.545Z [INF] DCP: janitor: rollbackPIndex for pindex db0xb8cc1d82_index_7997cab3efe92946_103fc5fb, err: janitor: error registering pindex: db0xb8cc1d82_index_7997cab3efe92946_103fc5fb, err: manager: registered pindex exists, name: db0xb8cc1d82_index_7997cab3efe92946_103fc5fb
2025-02-28T00:25:03.545Z [INF] DCP: c:SGI db:endpoint Starting sharded feed for SGI.  Total partitions:6
2025-02-28T00:25:03.545Z [INF] DCP: c:SGI db:endpoint Using DCP Logging Receiver
2025-02-28T00:25:03.545Z [INF] DCP: janitor: rollbackPIndex for pindex db0xb8cc1d82_index_7997cab3efe92946_103fc5fb, err: janitor: error registering pindex: db0xb8cc1d82_index_7997cab3efe92946_103fc5fb, err: manager: registered pindex exists, name: db0xb8cc1d82_index_7997cab3efe92946_103fc5fb
2025-02-28T00:25:03.545Z [INF] DCP: c:SGI db:endpoint Starting sharded feed for SGI.  Total partitions:7
2025-02-28T00:25:03.545Z [INF] DCP: c:SGI db:endpoint Using DCP Logging Receiver
  • Observed Import Partitions on any Sync Gateway node are significantly higher than configured (import_partitions). For example, if we see the following graph with import_partitions configured to the default value of 16:

e2b4d9f8-3927-4187-ae11-0018b79bc464.png

4528fa39-1297-417c-b5ce-71cca8641324.png

  • The import_partitions count growing aligns with the following errors:
72076:2025-02-28T00:25:47.533Z [INF] DCP: planner: could not SplitIndexDefIntoPlanPIndexes, indexDef.Name: db0x877eecaa_index, server: couchbases://cb.xxxxxx.cloud.couchbase.com?idle_http_connection_timeout=90000&kv_pool_size=1&max_idle_http_connections=64000&max_perhost_idle_http_connections=256, err: planner: could not get partitions, indexDef.Name: db0x877eecaa_index, server: couchbases://cb.xxxxxx.cloud.couchbase.com?idle_http_connection_timeout=90000&kv_pool_size=1&max_idle_http_connections=64000&max_perhost_idle_http_connections=256, err: gocbcore_utils: createAgents (1), setup err: agent setup failed, err: EOF | {"document_key":"{\"a\":\"gocbcore/v10.5.1 sg:stats\",\"i\":\"ac49fa15078c5901/eede7150a354efed\"}","bucket":"default","last_dispatched_to":"svc-d-node-003.xxxxxx.cloud.couchbase.com:11207","last_dispatched_from":"x.x.x.x:37116","last_connection_id":"ac49fa15078c5901/eede7150a354efed"}
72117:2025-02-28T00:25:47.564Z [INF] DCP: planner: could not SplitIndexDefIntoPlanPIndexes, indexDef.Name: db0x877eecaa_index, server: couchbases://cb.xxxxxx.cloud.couchbase.com?idle_http_connection_timeout=90000&kv_pool_size=1&max_idle_http_connections=64000&max_perhost_idle_http_connections=256, err: planner: could not get partitions, indexDef.Name: db0x877eecaa_index, server: couchbases://cb.xxxxxx.cloud.couchbase.com?idle_http_connection_timeout=90000&kv_pool_size=1&max_idle_http_connections=64000&max_perhost_idle_http_connections=256, err: gocbcore_utils: createAgents (1), setup err: agent setup failed, err: EOF | {"document_key":"{\"a\":\"gocbcore/v10.5.1 sg:stats\",\"i\":\"c9ff3b449e725b57/b2484f00db961df0\"}","bucket":"default","last_dispatched_to":"svc-d-node-003.xxxxxx.cloud.couchbase.com:11207","last_dispatched_from":"x.x.x.x:37126","last_connection_id":"c9ff3b449e725b57/b2484f00db961df0"}
711347:2025-02-28T00:25:47.533Z [INF] DCP: planner: could not SplitIndexDefIntoPlanPIndexes, indexDef.Name: db0x877eecaa_index, server: couchbases://cb.xxxxxx.cloud.couchbase.com?idle_http_connection_timeout=90000&kv_pool_size=1&max_idle_http_connections=64000&max_perhost_idle_http_connections=256, err: planner: could not get partitions, indexDef.Name: db0x877eecaa_index, server: couchbases://cb.xxxxxx.cloud.couchbase.com?idle_http_connection_timeout=90000&kv_pool_size=1&max_idle_http_connections=64000&max_perhost_idle_http_connections=256, err: gocbcore_utils: createAgents (1), setup err: agent setup failed, err: EOF | {"document_key":"{\"a\":\"gocbcore/v10.5.1 sg:stats\",\"i\":\"ac49fa15078c5901/eede7150a354efed\"}","bucket":"default","last_dispatched_to":"svc-d-node-003.xxxxxx.cloud.couchbase.com:11207","last_dispatched_from":"x.x.x.x:37116","last_connection_id":"ac49fa15078c5901/eede7150a354efed"}

Errors such as EOF returned from the server indicate disruption in partitioning, causing the import issues, and Sync Gateway miscalculating the number of assigned import partitions.

Workarounds

  • Mutate affected documents to trigger import processing.
  • Pause and resume the App Service Endpoint or restart Sync Gateway to force reprocessing/reinitialization.

Related to

Related articles

Comments

0 comments

Article is closed for comments.