Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[CBG-4309] Missing janitor kick in cbgt rollback

Vilius
Updated 
Product: Couchbase Sync Gateway
Component: mobile
Issue Link: CBG-4309
Affects Version(s): 3.1.11, 3.2.1
Fix Version(s): 3.2.2

Summary

  • When this issue happens, some vBuckets stay without an active DCP stream, leading to intermittent and partial import feed processing.
  • vBuckets are grouped into pIndexes and sharded across nodes; failover reassigns shards to surviving nodes.
  • Rollback should close all pIndex streams, roll back, and then reopen streams.
  • After rollback of some vBuckets in a pIndex, the Janitor doesn’t restart the DCP streams, causing intermittent import feed gaps.
  • The root cause of the issue is the coordination gap between rollback completion and Janitor’s mechanism that should kick off stream reinitialization for the affected pIndex/vBuckets.
  • Janitor should reliably reopen all pIndex streams after rollback, but that doesn’t happen.

Symptoms 

  • A subset of documents is not successfully imported by the Sync Gateway.
    • Sync Gateway metadata will be missing from the document’s xattrs in the bucket on the Couchbase Server
    • The effect can be limited to only a few documents.

Triggers

  • Sync Gateway checkpoints rolled back to 0 on a subset of vBuckets.

Verification

Issue can be definitively confirmed only if the Sync Gateway logs cover the start of the import feed. The easiest way to force this is to restart Sync Gateway.

If DEBUG logs are covering the start of the import feed:

Look for rollbacks to zero and take a pIndex as an example:

2025-02-05T11:05:32.626+01:00 [DBG] DCP+: feed_dcp_gocbcore: [db0x1a1ef1c9_index_3bbf73bfad701bf6_15a93994] OpenStream received rollback, for vb: 643, streamOptions: &{StreamID:25}, seqno requested: 1553

Follow the expected steps as per https://jira.issues.couchbase.com/browse/CBG-4309.

2025-02-05T11:05:34.980+01:00 [DBG] DCP+: janitor: fully rolling back pindex db0x1a1ef1c9_index_3bbf73bfad701bf6_15a93994

During steps 5 and 6, the pIndex should get successfully removed and closed as such (expected):

2025-02-05T11:05:35.033+01:00 [DBG] DCP+: janitor: pindexes to remove: 8
2025-02-05T11:05:35.033+01:00 [DBG] DCP+:   pindex: db0x1a1ef1c9_index_3bbf73bfad701bf6_15a93994; UUID: 3bbf73bfad701bf6
...
2025-02-05T11:05:35.033+01:00 [DBG] DCP+: pindex: db0x1a1ef1c9_index_3bbf73bfad701bf6_15a93994 Close started with remove: true
2025-02-05T11:05:35.033+01:00 [DBG] DCP+: pindex: db0x1a1ef1c9_index_3bbf73bfad701bf6_15a93994 Close completed successfully

However, after this point we do not see this same pIndex db0x1a1ef1c9_index_3bbf73bfad701bf6_15a93994 being added again. This indicates that the issue is present.

On the other hand, the correct flow from this point should look like Step 7:

2025-02-05T11:05:35.050+01:00 [DBG] DCP+: janitor: pindexes to add: 8
...
2025-02-05T11:05:35.050+01:00 [DBG] DCP+:   pindex: db0x1a1ef1c9_index_3bbf73bfad701bf6_15a93994; UUID: 3bbf73bfad701bf6
...
2025-02-05T11:05:35.050+01:00 [DBG] DCP+: janitor: feeds to add: 8
...
2025-02-05T11:05:35.050+01:00 [DBG] DCP+:   db0x1a1ef1c9_index_3bbf73bfad701bf6_15a93994
...

If only INFO logs are covering the start of the import feed:

You will not be able to definitively confirm the issue. However, the following may give a good idea that this is in fact occurring.

Normal startup of the DCP import feed:

2025-01-21T11:51:34.540+01:00 [INF] Import: db:formx_config Starting DCP import feed for bucket: "<ud>formx_config</ud>"

This is then followed by some logging indicating errors with pIndex rollbacks:

2025-01-21T11:51:34.544+01:00 [INF] DCP: bucket_scope_tracker: streamingEndpointListener hasn't been set up, bucket scope info won't be cached
2025-01-21T11:51:34.563+01:00 [INF] DCP: c:SGI db:formx_config Creating cbgt index "db0x58b3ec9a_index" for db "formx_config"
...
2025-01-21T11:51:37.479+01:00 [INF] DCP: janitor: rollbackPIndex for pindex db0x58b3ec9a_index_634da3927ad0712b_103fc5fb, err: janitor: error registering pindex: db0x58b3ec9a_index_634da3927ad0712b_103fc5fb, err: manager: registered pindex exists, name: db0x58b3ec9a_index_634da3927ad0712b_103fc5fb

This is then followed by lines of logging indicating issues with the cbgt library:

2025-01-21T11:51:37.480+01:00 [INF] DCP: db:formx_config Handling EOF on cbgt feed - notifying manager to trigger reconnection to feed for bucketName:formx_config, bucketUUID:6546563abaef1217c788b26a34cb70d4, err: feed_dcp_gocbcore: [db0x58b3ec9a_index_634da3927ad0712b_1076212c], End, vb: 389, err: EOF | {"bucket":"formx_config","last_dispatched_to":"svcbsnodtst04.nonprod.domain:11207","last_dispatched_from":"10.154.64.227:52416","last_connection_id":"e72ae491b3ab70df/e3ccfd95cec938d4"}

In addition to the above, vBucket rollbacks where the checkpoints were rolled back to zero at some point in the past:

2025-01-21T11:51:35.057+01:00 [WRN] c:SGI db:formx_config DCP RollbackEx request - rolling back DCP feed for: vbucketId: 797, rollbackSeq: 0. -- base.(*DCPCommon).rollbackEx() at dcp_common.go:190
2025-01-21T11:51:35.122+01:00 [WRN] c:SGI db:formx_config DCP RollbackEx request - rolling back DCP feed for: vbucketId: 817, rollbackSeq: 0. -- base.(*DCPCommon).rollbackEx() at dcp_common.go:190
2025-01-21T11:51:35.124+01:00 [WRN] c:SGI db:formx_config DCP RollbackEx request - rolling back DCP feed for: vbucketId: 819, rollbackSeq: 0. -- base.(*DCPCommon).rollbackEx() at dcp_common.go:190
...

Workarounds

  1. Stop all Sync Gateway nodes from the cluster.
  2. Delete the following two documents from all relevant buckets in your Couchbase Server, for which you see documents not being successfully imported:
    1. _sync:cfgindexDefs
    2. _sync:cfgplanPIndexes

  3. Delete all documents prefixed with _sync:dcp_ck: from the relevant buckets in the Couchbase Server.  Full format should be structured as so {{_sync:dcp_ck: {m_$}

    :

    {groupID:} {version:}

    }}.

  4. Start all Sync Gateway nodes on this cluster again.

Important Notes:

  • Missed documents will be re-imported into the database. Sync Gateway will check all documents to find missed ones, but will not re-import documents already imported. Built-in logic prevents reprocessing of already imported documents.
  • The workaround still adds a small overhead since all bucket documents are being scanned.

Related to

Related articles

Comments

0 comments

Article is closed for comments.