[MB-63600] Race Condition Between Partition Prune and Merge Can Cause Incomplete Partition List in Projector

Product: Couchbase Server
Component: index-service
Issue Link: MB-63600
Affects Version(s): 6.x, 7.0.x, 7.1.x, 7.2.x, 7.6.x
Fix Version(s): 7.6.4

Summary

• A race condition during a rebalance can cause the projector process to have an incomplete list of partitions for an index.
• As a result, some partitions are not updated because the projector is unaware of them.
• This can lead to millions of missed mutations, compromising the accuracy of scan results.
• This has been present for all versions since 5.0.0.

Symptoms 

• Query results are either incorrect or inconsistent between query executions.
• Main or Replica indexes can be behind by a considerable number of items, causing query execution results to be inconsistent depending on which index is scanned.
• If no replicas are present, the index is always missing items.

Triggers

• Any rebalance with partitioned indexes.
• File based rebalancing is not required, but it increases the likelihood of seeing this.

Verification

• If 2 copies of the index exist at a minimum (Main + 1 Replica), then it can be identified with the following steps:
  • Identify the location of all partitions for all indexes. This is needed to identify which partitions are likely not increasing.
  • If a node has an exact copy of another’s partitions (For example, Node 1 has Main 1, 2, 3, and Node 2 has partition 1, 2, and 3 of the replica), then compare the total item counts between the two
  • If significant incoming traffic outweighs the item count differences, it indicates this issue is present.
• Verification is not reasonably possible with only a single copy of an index.

Workarounds

• Recreate the index entirely. Without it, the projector will never be made aware of the missing partitions.