[MB-62547] Update shard.path if existing shard directory renamed with new shardId during recovery

Product: Couchbase Server
Component: index-service
Issue Link: MB-62547
Affects Version(s): 7.6.x
Fix Version(s): 7.6.2, 8.0.0

Summary

When upgrading from 7.1.x or 7.2.x to 7.6.0 or 7.6.1, index corruption occurs, resulting in an inability to ingest mutations into indexes and rebalance failures.

This issue is documented in the Release Notes for 7.6.2:

A critical issue occurs when performing an off-line-upgrade, graceful failover, or delta-recovery upgrade methods on index service nodes from Couchbase Server versions 7.1 through 7.2 to versions 7.6.0 or 7.6.1. The upgrade corrupted existing indexes requiring you to drop and rebuild them.

The issue can be seen on 7.6.2 [or a higher version] if a duplicate shard(s) from 7.6.1 continues to exist on the node before the upgrade to 7.6.2.

At the time of upgrade, the current shards will be renamed to the new ones (with UUID). Due to the MB, it stores both new and old shards.

On recovery, one shard is recovered, while another becomes "dead" and is deleted along with all its instances. However, these are the same instances that were already recovered with the first shard, which means their files are now incorrectly deleted.

As a result, the instances attempt to access deleted files, which prevents mutations from flushing to disk. These operations are continuously retried, leading to a high number of pending mutations.

Symptoms 

• Rebalance failures.
• Indexes are unable to ingest items, leading to a buildup of pending/queued mutations.

Triggers

• Upgrading from 7.1.x or 7.2.x to 7.6.0 or 7.6.1.
  • This issue is documented in the Release Notes for 7.6.2.

Verification

If rebalance failures are occurring, check the rebalance failure message in diag.log for PlasmaId found for new instance and Error creating slice, which can indicate this issue occurred:

2024-08-05T07:21:15.346-04:00, ns_orchestrator:0:critical:message(ns_1@xxxx) - Rebalance exited with reason {service_rebalance_failed,index,
{worker_died,
{'EXIT',<0.19642.4178>,
{task_failed,rebalance,
{service_error,
<<"Error creating slice Unable to initialize /opt/xxxx/COUCHBASE/data/cbindex/@2i/xxxx.index/mainIndex, err = PlasmaId found for new instance. Path=/opt/xxxx/COUCHBASE/data/cbindex/@2i/xxxx.index/mainIndex, plasmaId=23">>}}}}}.
Rebalance Operation Id = ae63dc08760bb2ef4b422ecbb2a7166e

The error here means that GSI is trying to create a new instance, but Plasma already finds the instance in one of the shards' metadata. An instance was created and later dropped, but due to MB-62547 Closed , it remains in the shard metadata after the upgrade to 7.6.1. This persistence is what leads to the error.

If indexes are unable to ingest mutations, look for log messages containing Unable to write in the indexer.log:

2024-07-12T07:51:41.654-04:00 [Info] LSS /opt/xxxx/COUCHBASE/data/cbindex/@2i/xxxx.index/mainIndex/recovery(shard16810821006246096725) : recoveryCleaner: completed... frag 50, data: 8341784, used: 16765387, disk: 17262078, relocated: 731, retries: 0, skipped: 120, cleaned:347508, log:(2393114486 - 2410029056), disk:(2392766978-2410029056) aheadOff: 0, activeFragRatio:30, activeMaxFragRatio:80, run:1 duration:14 ms elapsed: 14 ms
Plasma: (/opt/xxxx/COUCHBASE/data/cbindex/@2i/xxxx.index/docIndex/recovery) Unable to write - err open /opt/xxxx/COUCHBASE/data/cbindex/@2i/xxxx.index/docIndex/recovery/log.00000000000001.data: no such file or directory
...
Plasma: (/opt/xxxx/COUCHBASE/data/cbindex/@2i/xxxx.index/mainIndex/recovery) Unable to write - err open /opt/xxxx/COUCHBASE/data/cbindex/@2i/xxxx.index/mainIndex/recovery/log.00000000000001.data: no such file or directory
2024-07-15T10:41:17.171-04:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 117000 Milliseconds KeyspaceId xxxx Vbucket 1005
2024-07-15T10:41:17.242-04:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 120000 Milliseconds KeyspaceId xxxx Vbucket 323
2024-07-15T10:41:17.426-04:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 117000 Milliseconds KeyspaceId xxxx Vbucket 354
2024-07-15T10:41:17.476-04:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 117000 Milliseconds KeyspaceId xxxx Vbucket 144
2024-07-15T10:41:17.535-04:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 117000 Milliseconds KeyspaceId xxxx Vbucket 733
2024-07-15T10:41:17.561-04:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 117000 Milliseconds KeyspaceId xxxx Vbucket 943

The mutation queue is forced to wait because the system cannot flush to disk—it is unable to create a new segment in a deleted directory. This waiting is the cause of the high number of pending mutations.

Workarounds

The workaround is to drop and rebuild all the affected indexes.