Product: Couchbase Server Component: XDCR Issue Link: MB-68952 Affects Versions: 7.2.x, 7.6.x Fix Versions: 7.2.9, 7.6.9, 8.0.1, 8.1.0
Summary
When security-related fields (Username, Password, or Certificate) are changed in an XDCR remote cluster reference, we set the connection pool of the main pipeline as stale. This ensures that the connection pool is recreated with the updated credentials. In the affected versions, the backfill pipeline connection pool is not set to stale. This leads to the xdcr_changes_left_total metric to grow or be stuck on a non-zero value, and XDCR Checked Ratio to drop due to the backfill pipeline using stale credentials or certificates.
Symptoms
- Grow or remain on a non-zero level of the{{ xdcr_changes_left_total}} metric after changing Username, Password, or Certificate in a remote cluster reference.
- Drop in XDCR Checked Ratio in XDCR Stats:
Triggers
- Change of security-related fields like Username, Password, or Certificate (including updating or changing the certificate) triggers this issue when the affected version of Couchbase is in use.
Verification
- If you observe that some documents are not replicated after changing the security-related fields in the XDCR remote reference, check whether the xdcr_changes_left_total metric grows or remains non-zero, and whether the XDCR Checked Ratio in XDCR Stats drops.
- If the xdcr_changes_left_total metric grows or remains non-zero, and the XDCR Checked Ratio in XDCR Stats drops, you might have been affected by this issue.
Workarounds
- The workaround for this issue is to restart the goxdcr process on all Data Service nodes. You can do it by running the following command in a terminal of the data nodes:
sudo pkill goxdcr
After killing the goxdcr process, the babysitter process will restart it automatically. This action recreates the backfill pipeline connection pool with the updated credentials or certificate that fixes the issue.
Comments
0 comments
Article is closed for comments.