[MB-63246] Full Text Search (FTS) indexes made on Couchbase Server earlier than 7.0 cannot ingest data post-upgrade to 7.6.x

Product: Couchbase Server
Component: FTS-Service
Issue Link: MB-63246
Affects Version: 7.6.x
Fix Versions: 7.6.6, 8.0.0

Summary

After upgrading a cluster that contains legacy Full Text Search (FTS) indexes created with Couchbase Server versions prior 7.x (that use gocouchbase for data ingestion), these indexes fail to ingest data due to a change introduced in Couchbase Server 7.6.2. This can lead to increased CPU usage and/or failing or stuck rebalances.

In this section of code, a gocbcore-specific check was added without accounting for other feed types. As a result, when a legacy index attempts to ingest data through the gocouchbase feed, it triggers the code path that includes this check. This causes an error because gocouchbase cannot interpret a gocbcore-specific construct.

Symptoms 

• After upgrading Couchbase Server nodes to version 7.6.2 or later, a cluster rebalance may become stuck if it contains Full Text Search (FTS) indexes created on versions earlier than 7.x.
• In some cases, high CPU utilization may occur due to continuous FTS index rollbacks.

Triggers

• The issue occurs when a rebalance starts after upgrading cluster nodes to version 7.6.2 or later. At the start of the rebalance, affected indexes begin ingesting data, which triggers the issue and causes the rebalance process to become stuck.

Verification

• Verify whether any Full Text Search (FTS) indexes currently on the cluster were originally created using Couchbase Server versions earlier than 7.x.

• You can use the /api/nsstats REST API endpoint to monitor whether the document count (index.doc_count) for certain FTS indexes is not increasing over time, despite incoming new data that should be indexed. This can indicate a problem with data ingestion for those indexes. An example of using the endpoint:

  ~ curl -X GET -u Administrator:password http://localhost:9200/api/nsstats | jq | grep -e doc_count
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100 21766    0 21766    0     0  5321k      0 --:--:-- --:--:-- --:--:-- 7085k
    "travel-sample:travel-sample._default.travel-legacy:doc_count": 0,
    "travel-sample:travel-sample._default.travel-sample-index-copy:doc_count": 31591,
    "travel-sample:travel-sample._default.travel-sample-index:doc_count": 31591,
    "travel-sample:travel-sample.inventory.travel-sample-index-copy:doc_count": 24211,

Workarounds

• There is no workaround for this issue. All indexes created using Couchbase Server versions earlier than 7.x must be re-created using the upgraded version of Couchbase Server.