Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-64982] When file based rebalance is enabled, ALTER INDEX statement that specifies the placement of replicas using "nodes" clause can lead to subsequent rebalances to fail

David
Updated 
Product: Couchbase Server
Component: index-service
Issue Link: MB-64982
Affects Version(s): 7.6.x
Fix Version(s): 7.6.6, 8.0.0

Summary

  • If an index is altered using an ALTER INDEX statement that specifies the placement of replicas using a "nodes" clause while File-Based Rebalancing is active, this could result in subsequent rebalance failures.

Symptoms 

  • Rebalance failures due to RestoreShard error:
2025-01-13T23:43:05.054+03:00, ns_orchestrator:0:info:message(ns_1@node-1.cluster.local) - Starting rebalance, KeepNodes = ['ns_1@node-2.cluster.local',
...
2025-01-14T00:53:39.495+03:00, ns_orchestrator:0:critical:message(ns_1@node-1.cluster.local) - Rebalance exited with reason {service_rebalance_failed,index,
                              {worker_died,
                               {'EXIT',<0.13065.7100>,
                                {task_failed,rebalance,
                                 {service_error,
                                  <<"RestoreShard error :alternateId(741766320988112600-1-0) already exists (shardId15431294857910525589)">>}}}}}.

Triggers

  • Executing an ALTER INDEX statement that contains a nodes clause while File-Based Rebalance is enabled. This may cause subsequent rebalances to fail.
  • Since the planning of a shard placement is randomised, the issue can be encountered on a random rebalance after a compromised ALTER INDEX command was issued.

Verification

Collect cluster logs and check for the following:

  • Check the rebalance failure message for RestoreShard error in diag.log:
2025-01-13T23:43:05.054+03:00, ns_orchestrator:0:info:message(ns_1@node-1.cluster.local) - Starting rebalance, KeepNodes = ['ns_1@node-2.cluster.local',
...
2025-01-14T00:53:39.495+03:00, ns_orchestrator:0:critical:message(ns_1@node-1.cluster.local) - Rebalance exited with reason {service_rebalance_failed,index,
                              {worker_died,
                               {'EXIT',<0.13065.7100>,
                                {task_failed,rebalance,
                                 {service_error,
                                  <<"RestoreShard error :alternateId(741766320988112600-1-0) already exists (shardId15431294857910525589)">>}}}}}.
  • Look for the following log message in ns_server.indexer.log indicating an Alter Index command was issued:
2024-12-23T12:30:33.809+03:00 [Info] AlterReplicaCount 7040652249195519604 replica_count ...
  • Look for the following message in ns_server.indexer.log indicating the Alter Index command has caused some nodes to be skipped from planning:
2024-12-23T12:30:34.723+03:00 [Info] Planner:Skip node node-09.cluster.local:8091 since it is not in the given host list [node-06.cluster.local:8091 node-08.cluster.local:8091]

Workarounds

  • Identify the affected index instance, drop and recreate it. If this involves a replica, execute the ALTER INDEX command to remove the replica and then recreate it using "num_replica" (NOT "nodes"). Find more details in the ALTER INDEX article of our documentation.

Related to

Related articles

Comments

0 comments

Article is closed for comments.