Product: Couchbase Server Component: Index-Service Issue Link: MB-63261 Affects Versions: 7.1.x, 7.2.x, 7.6.x Fix Versions: 7.2.7, 7.6.3
Summary
A race condition in the index recovery process can introduce duplicate items into existing indexes, causing the mainstore and backstore to become unsynchronized. This issue has existed in Couchbase Server since version 7.1.0, and it is more likely to occur when file-based rebalancing is used, as it increases the number of recovery operations.
Symptoms
- After a rebalance, item counts within an index may increase even without any new mutations.
- In some cases, the Main and Replica indexes report different item counts despite having 0 items pending to be indexed.
Triggers
- The primary trigger for this issue is the index recovery process. Prior to version 7.6 and the introduction of file-based rebalancing, occurrences were rare, and any rollback would typically resolve the mismatch.
- With file-based rebalancing, however, the recovery process is triggered repeatedly, increasing the likelihood of inconsistencies.
Verification
This issue is very difficult to verify without impacting the cluster.
Initial steps to investigate include:
- File-based rebalancing is enabled.
- Rebalance has been performed prior to the item count difference.
- Some items are pending to be indexed for any index.
These checks can suggest the presence of the issue, but they are not conclusive.
Definitive evidence can be gathered by using the plasma_dump utility with the rpitemcountcheck option, following the instructions given below. The tool is typically located in /opt/couchbase/bin/ directory, and it checks item count consistency for the persisted snapshot of mainIndex and docIndex. It will report Index is good for healthy indexes, or FAIL - instance corrupted for affected ones.
Important caveats:
- The tool is IO-intensive, performs random read operations, and can significantly impact query latencies and drain rates.
- It is strongly recommended to run the tool only during low-traffic or downtime windows.
- For large indexes (e.g., 1B items), the check may take a long time to complete.
- Never run this tool on production clusters. It is almost always better to assume the index is affected and recreate it, rather than risk any further impact.
For these reasons, it is strongly recommended to run plasma_dump tool after raising a ticket with the Couchbase Technical Support Team regarding this issue. If you are an entitled customer, please submit a ticket through support.couchbase.com. If not, we encourage you to reach out to our Sales team at couchbase.com/pricing.
Usage:
export LD_LIBRARY_PATH=/opt/couchbase/bin ./plasma_dump rp_item_count_check <path to verify_config.json>
Configuration:
It is driven by a JSON-based configuration file. Below is a default verify_config.json
{
"StorageDir": "",
"Bucket": "",
"Index": "",
"Partn": 0,
"InstId": 0,
"Quota": 0,
"LogLevel": 0,
"MaxRetries": 10,
"ReverseScan": true,
"ExtraDelete": true,
"Concurrency": 1
}
Sample Test:
cat /tmp/verify_config.json :
{
"StorageDir": "data/n_2/data/@2i",
"Bucket": "default",
"Index": "idx_3",
"Partn": 0,
"InstId": 0,
"Quota": 0,
"MaxRetries": 10,
“HolePunching”: true,
"ReverseScan": true,
"ExtraDelete": true,
"Concurrency": 1
}
./plasma_dump rp_item_count_check /tmp/verify_config.json
- Good Index Output:
DocIndex default_idx_3_17545078156939020683_0 LSS: data/n_2/data/@2i/shards/shard13168071746743611069/data PlasmaId: 3 MainIndex default_idx_3_17545078156939020683_0 LSS: data/n_2/data/@2i/shards/shard16980509676804285825/data PlasmaId: 4 data/n_2/data/@2i/shards/shard16980509676804285825/data numPids: 9565 numPages 9565 buildPidList::Done duration :6.140861ms RP version 23 Sn: 16059 instPath: /Users/user/ws/couchbase/trinity/ns_server/data/n_2/data/@2i/default_idx_3_17545078156939020683_0.index/mainIndex item count match 1000000 with persisted snapshot no: 16059 rpversion: 23 scanPages duration 4.294083374s memoryInUse 4045405 STATUS:PASS - instance ok data/n_2/data/@2i/shards/shard13168071746743611069/data numPids: 15369 numPages 15369 buildPidList::Done duration :14.874236ms RP version 59 Sn: 16059 instPath: /Users/user/ws/couchbase/trinity/ns_server/data/n_2/data/@2i/default_idx_3_17545078156939020683_0.index/docIndex item count match 1000000 with persisted snapshot no: 16059 rpversion: 59 scanPages duration 3.098356475s memoryInUse 5124272 STATUS:PASS - instance ok STATUS: Index:idx_3 is good
- Bad Index Output:
DocIndex default_idx_3_17545078156939020683_0 LSS: data/n_2/data/@2i/shards/shard13168071746743611069/data PlasmaId: 3 MainIndex default_idx_3_17545078156939020683_0 LSS: data/n_2/data/@2i/shards/shard16980509676804285825/data PlasmaId: 4 data/n_2/data/@2i/shards/shard16980509676804285825/data buildPidList::Done duration :6.549557ms RP version 23 Sn: 16059 scanPages duration 1.995648ms memoryInUse 4044966 Error /Users/user/ws/couchbase/trinity/ns_server/data/n_2/data/@2i/default_idx_3_17545078156939020683_0.index/mainIndex err = fatal error - item count mismatch, expected :13 got :12 FAIL - instance corrupted
Workarounds
There is no direct workaround for this issue; mitigation focuses on reducing its likelihood and impact:
- Disable File-Based Rebalancing to prevent repeated triggering of the problem.
- Drop and recreate affected indexes to restore consistency.
- For non-partitioned indexes, with File-Based Rebalancing disabled, moving the index to another node will rebuild it from scratch via DCP.
Upgrading to a fixed version will not repair already impacted indexes unless they have previously been rebuilt or moved with File-Based Rebalancing disabled. The most efficient upgrade path for affected clusters is to drop the problematic indexes, perform a rapid upgrade, and then recreate the indexes. This is typically faster and safer than rebalance operations with File-Based Rebalancing disabled.
Comments
0 comments
Article is closed for comments.