[MB-65511] An XDCR replication can stuck or become invisible at the time of bootup

Product: Couchbase Server
Component: xdcr
Issue Link: MB-65511
Affects Version(s): 7.0.x, 7.1.x, 7.2.x, 7.6.x
Fix Version(s): 7.2.8, 7.6.6, 8.0.0

Summary

• An XDCR replication gets stuck/blocked when a particular set of internal circumstances related to the XDCR process(goXDCR) is encountered. Please see the Triggers section of this article below for an extended description.

Symptoms 

• XDCR is stuck/hanging/blocked, especially after a restart.
• XDCR replications not visible in the UI:

Triggers

The issue is triggered when the following conditions are met:

• The cluster manager process(ns_server) regularly sends notifications to the XDCR process(goXDCR) to ensure it is kept up to date with bucket and cluster topology changes. These messages are received by the BucketTopologySvc component within goXDCR and then enqueued into dedicated notification channels (notificationCh), which subscribe to these notifications. If multiple “subscribe” calls happen in succession for the same subscriberId (with no unsubscribing occurring), a scenario can occur where only the most recent subscribe call is stored for one notification channel, while other notification channels become blocked indefinitely.
• When this happens, the goroutine within goXDCR becomes stuck, resulting in XDCR becoming completely stuck/blocked and/or invisible in the UI.
• The affected code path is triggered every 10s due to the way goXDCR checks for topology changes in a repeating manner.

Verification

• Collect a set of Couchbase Server logs while XDCR is stuck. If the issue is not occurring while the logs are collected, the logs will not provide insights into whether or not this issue is occurring. The goroutine process profiling information for the XDCR process(goxdcr) in the logs only provides a 'snapshot' of the goroutines at the log collection time.
• Then look for the presence of the following line in goxdcr_cprof.log, goxdcr_mprof.log, goxdcr_pprof.log, which may indicate that a goroutine is stuck:

github.com/couchbase/goxdcr/metadata_svc.(*CollectionsManifestAgent).getCurrentCollectionManifestUid(0xc0006a2000)

Workarounds

• Restart XDCR on every affected node using the instructions below:

First, verify the process ID(PID) of the goxdcr process on the node.

This can be done with a utility like pidof or ps, e.g.:

pidof goxdcr

or

ps -aux | grep goxdcr | grep -v grep

Then kill the process with a command like pkill or killall, e.g.:

pkill goxdcr

Once terminated, the goxdcr process will be automatically restarted by the Couchbase Server. You can then verify the process was successfully restarted by checking that the process is now present with a different PID with the same commands as before:

pidof goxdcr

ps -aux | grep goxdcr | grep -v grep