Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-64824] A lot of "error unlocking shard messages" due to failed removal of a metaKV token

Callum
Updated 
Product: Couchbase Server
Component: index-service
Issue Link: MB-64824
Affects Version(s): 7.6.x
Fix Version(s): 7.6.6, 8.0.0

Summary

  • If a node is failed over immediately after an incomplete rebalance, there may not be enough time to remove a metaKV token. This results in a lot of the error unlocking shard messages despite no functional impact:
2024-12-20T21:37:32.902+03:00 [Error] ShardTransferManager::handleUnlockShardsCommand Error observed while unlocking shard: 17518768334138769269, err: error unlocking shard, shardId :17518768334138769269, err :cannot drop conflict (4) before adding, conflictOps :0
2024-12-20T21:37:32.902+03:00 [Error] ShardTransferManager::handleUnlockShardsCommand Error observed while unlocking shard: 17064706283871973095, err: error unlocking shard, shardId :17064706283871973095, err :cannot drop conflict (4) before adding, conflictOps :0

Symptoms 

  • No functional impact.
  • A lot of error unlocking shard messages that might be concerning:
2024-12-20T19:12:22.630+03:00 [Error] ShardTransferManager::handleUnlockShardsCommand Error observed while unlocking shard: 17518768334138769269, err: error unlocking shard, shardId :17518768334138769269, err :cannot drop conflict (4) before adding, conflictOps :0
2024-12-20T19:12:22.631+03:00 [Error] ShardTransferManager::handleUnlockShardsCommand Error observed while unlocking shard: 17064706283871973095, err: error unlocking shard, shardId :17064706283871973095, err :cannot drop conflict (4) before adding, conflictOps :0

Triggers

  • If a failover occurs immediately after an incomplete rebalance, there might not be enough time to remove a metaKV token, which results in the described issue.

Verification

  • The logs are consistently flooded with the error unlocking shard messages similar to below:
2024-12-20T21:37:32.902+03:00 [Error] ShardTransferManager::handleUnlockShardsCommand Error observed while unlocking shard: 17518768334138769269, err: error unlocking shard, shardId :17518768334138769269, err :cannot drop conflict (4) before adding, conflictOps :0
2024-12-20T21:37:32.902+03:00 [Error] ShardTransferManager::handleUnlockShardsCommand Error observed while unlocking shard: 17064706283871973095, err: error unlocking shard, shardId :17064706283871973095, err :cannot drop conflict (4) before adding, conflictOps :0

Workarounds

  • Search for the ShardTokendd in log messages, for example ShardTokendd:44:b0:e6:43:bf:e6:d7:

    2024-12-22T10:48:39.988+03:00 [Info] RebalanceServiceManager::cleanupShardTransferTokens Cleaning Up ShardTokendd:44:b0:e6:43:bf:e6:d7  MasterId: bc802399c87efba51bb3f586db11a543 SourceId: aaa8129fe26c1581b1497ff7999e41e0 (ubdprdcouch07.deniz.denizbank.com:8091) DestId: a16e324b18ff6e61322cf370cd1ec7b2 (ubdprdcouch09.deniz.denizbank.com:8091) RebalId: f3d679e88834bd9faaaef6a21e5fa769 ShardTokenState: ShardTokenReady BuildSource: Peer TransferMode: Move SiblingTokenId:  Shards: [17518768334138769269 17064706283871973095]
  • To stop these log messages from flooding the logs, remove the token from metaKV manually by issuing the following REST API request:
curl -X DELETE -v -u <username>:<password> http://localhost:8091/_metakv/indexing/rebalance/ShardTokendd:[tokenID]
  • For example, for the above ShardTokendd:44:b0:e6:43:bf:e6:d7 the command should be the following:
curl -X DELETE -v -u <username>:<password> http://localhost:8091/_metakv/indexing/rebalance/ShardTokendd:44:b0:e6:43:bf:e6:d7

Related to

Related articles

Comments

0 comments

Article is closed for comments.