[MB-64783] Cannot process rebalance when partitions topology is never bootstrapped

Product: Couchbase Server
Component: analytics-service
Issue Link: MB-64783
Affects Version(s): 7.2.6, 7.6.3
Fix Version(s): 7.2.7, 7.6.6

Summary

If the Analytics master node (CC) is failed over before its initial bootstrap completes, subsequent rebalances will fail. This failure occurs because the system cannot find the partition topology needed to identify candidate replacement nodes.

Symptoms 

• Rebalance operations fail.
• The Analytics Service (cbas) repeatedly restarts.

Triggers

• The Analytics Service master node (CC) is failed over before the first bootstrap is completed.
• A rebalance operation that adds an Analytics Service master node (CC) is manually stopped before the bootstrap is complete.

Verification

Check diag.log to determine whether the affected Analytics (CC) node was failed over and to determine whether a rebalance was manually stopped.

Look for signs that the Analytics Service is constantly restarting with Service 'cbas' exited with status 1 in diag.log:

2025-01-07T20:52:59.047+08:00, ns_log:0:info:message(ns_1@10.10.15.15) - Service 'cbas' exited with status 1. Restarting. Messages:

Look for signs that the Analytics Service is constantly restarting with cbas process aborting with exit code 1 in the Analytics logging:

07T20:52:59.046+08:00 ERRO CBAS.cbas cbas process aborting with exit code 1 due to Failed to load metadata replicas

Workarounds

To recover the cluster from this state, you must manually delete the incomplete cbas topology state from metakv. This will allow the service to reinitialize correctly during the next rebalance attempt.

1. SSH to the problematic node.
2. Back up the current cbas metakv state by running the following command. Note that this API requires administrator credentials.

curl -v -X GET -u Administrator:password http://localhost:8091/_metakv/cbas/ > cbas_metakv_backup.json

1. Delete the current cbas metakv state

curl -v -X DELETE -u Adminstrator:password http://localhost:8091/_metakv/cbas/

1. If the cbas process is still running on any nodes, stop it by executing the following command:

killall -9 cbas

1. After waiting a few minutes for the service to bootstrap, perform a rebalance from the Couchbase Web Console to confirm the issue is resolved.