[MB-63193] Projector may miss few partitions of an index during file transfer rebalance

Product: Couchbase Server
Component: index-service
Issue Link: MB-63193
Affects Version(s): 7.6.2
Fix Version(s): 7.6.3

Summary

• During a file based rebalance, it is possible for partitions to be missed by the Projector process, leading to these partitions failing to be updated with new mutations.

Symptoms 

• Partitions falling behind within an index may indicate this issue.
• For a single index, it's hard to detect unless queries return incorrect results.
• With multiple indexes, if one index (main and/or replica) lags behind others on some partitions, despite no pending mutations, this can be a sign.
• If no rebalances have occurred, or File Based Rebalancing isn’t enabled, these symptoms are unrelated to this issue.

Triggers

• Any rebalance involving File Based Rebalancing (FBR) and partitioned indexes can trigger this behavior.
  • A race condition may occur if a partition merge doesn’t finish before the next build starts.
  • Partitions not merged in time can be overwritten and lost within the projector.
• This issue only arises during partition rebalances with FBR.

Verification

• Identify problematic partitions by comparing item counts against replica partitions since the last rebalance (problematic partition will not have increased in size).
• Once identified, locate the instance ID (InstId) for the affected partitions in the logs.

InstId: 15269458424720392459 RealInstId: 4199289883770430120 Name: idx_forms Bucket: Forms ScopeId: data/8 Collection/Id: templateVersions/b Partitions: [2] Versions: [3]
...
        InstId: 5428832503800560075 RealInstId: 4199289883770430120 Name: idx_forms Bucket: Forms ScopeId: data/8 Collection/Id: templateVersions/b Partitions: [1] Versions: [3] 
...
        InstId: 3763790366616279914 RealInstId: 4199289883770430120 Name: idx_forms Bucket: Forms ScopeId: data/8 Collection/Id: templateVersions/b Partitions: [6] Versions: [2]
...
        InstId: 13186856225341759055 RealInstId: 4199289883770430120 Name: idx_forms Bucket: Forms ScopeId: data/8 Collection/Id: templateVersions/b Partitions: [7] Versions: [2]
...
        InstId: 13817877923667796265 RealInstId: 4199289883770430120 Name: idx_forms Bucket: Forms ScopeId: data/8 Collection/Id: templateVersions/b Partitions: [4] Versions: [5]

• In this example, partitions 2, 1, and 7 are being moved (15269458424720392459, 5428832503800560075, and 13186856225341759055).
• During rebalance, affected partitions will begin to be merged after a handleBuildRecoveredIndexes call in the logs.

2024-06-27T15:27:52.149-07:00 [Info] Indexer::handleBuildRecoveredIndexes [4199289883770430120 13186856225341759055 5428832503800560075]
...
2024-06-27T15:28:54.703-07:00 [Info] MutationMgr::handleAddIndexListToStream
        Message: MsgStreamUpdate
        Type: ADD_INDEX_LIST_TO_STREAM
        Stream: MAINT_STREAM
        KeyspaceId: Forms
        BuildTS: []
        IndexList: [
        InstId: 13186856225341759055
        Defn: DefnId: 6660006021741324720 Name: idx_forms_data_templateVersions_meta_id Using: plasma Bucket: Forms Scope/Id: data/8 Collection/Id: templateVersions/b IsPrimary: false NumReplica: 1 InstVersion: 2
...

• Notice the InstId matching one of the 3 partitions.
• After this merge begins but has not yet completed, another build will begin for the moved partitions:

2024-06-27T15:29:02.362-07:00 [Info] Indexer::handleBuildRecoveredIndexes Added Index: [13817877923667796265 3763790366616279914 5771314579540276138 18365234790335285545] to Stream: INIT_STREAM State: INDEX_STATE_INITIAL

• At this stage, several partitions of the original merge are now lost (1 and 7 in this example)
• Afterwards, it's seen that the merge completes for the original merge:

2024-06-27T15:29:17.648-07:00 [Info] MergePartition: Merge instance 5428832503800560075 to instance 4199289883770430120
2024-06-27T15:29:17.729-07:00 [Info] MergePartition: Merge instance 13186856225341759055 to instance 4199289883770430120

• If the merges complete BEFORE any new handleBuildRecoveredIndexes appears within the logs, then it is unlikely this issue is present.

Workarounds

• Restarting the projector will allow it to recover the lost partitions; however, mutations that would have been added to them have already been processed and will not be retroactively added.
• As a result, dropping and recreating the indexes from scratch is the best way forward, as it brings the indexes to a consistent state.