[MB-64863] Incorrect timeout setting for restGetStatsFromIndexerNodes and MoveIndex can cause queries to hang indefinitely

Product: Couchbase Server
Component: index-service
Issue Link: MB-64863
Affects Version(s): 7.6.x
Fix Version(s): 7.6.6, 8.0.0

Summary

As part of the Plan phase of a query, indexer stats are obtained for the indexes that could be used for that query. The query node maintains a cache of indexer stats and calls the GSI interface (restGetStatsFromIndexerNodes) if the requested stats for an index are not in the cache.

If the stats are not in the cache and restGetStatsFromIndexerNodes is called, the timeout for this function may be incorrectly set due to an overflow, which results in the timeout being set to 0 (no timeout). Since there is no timeout, restGetStatsFromIndexerNodes may run indefinitely if the indexer does not respond, resulting in the query getting stuck in the Plan phase.

Since the query is in the Plan phase and has not yet picked up a servicer, the query won’t timeout because query timeouts are only measured from when execution of the plan begins.

Symptoms 

• Queries get stuck/hang indefinitely.
  • Once queries are terminated, the query plan, in the completed requests log, shows almost all the time spent in the Plan operator.

Triggers

• Query is executed when no indexer stats are cached (meaning restGetStatsFromIndexerNodes is called).
  • The cache may be invalidated if UPDATE STATISTICS gets run (either explicitly or implicitly), eg:
    • If a new index gets created on the keyspace.
    • If the indexer is restarted.
• The Index Service is unresponsive, causing no response to restGetStatsFromIndexerNodes, which will never timeout.

Verification

Check completed requests.

Look for all the time spent during the Plan operator, for example:

{
  "clientContextID": "f7d3b626-bc38-4527-acf5-478ec9bbe868",
  "elapsedTime": "4h18m40.795306918s",
  "errorCount": 1,
  "errors": [
    {
      "_level": "exception",
      "caller": "server:1433",
      "code": 1202,
      "key": "service.no_client",
      "message": "Client disconnected"
    }
  ],
  "n1qlFeatCtrl": 16460,
  "phaseTimes": {
    "instantiate": "19.827µs",
    "parse": "1.159186ms",
    "plan": "4h18m40.792656193s",
    "plan.index.metadata": "53.249571ms",
    "plan.keyspace.metadata": "4.535µs"
  },
  "remoteAddr": "10.5.93.229:59859",
  "requestId": "95aaf562-33d9-4d68-84ff-801d8d17f1de",
  "requestTime": "2024-12-05T11:49:26.101Z",
  "resultCount": 0,
  "resultSize": 0,
  "scanConsistency": "scan_plus",
  "serviceTime": "4h18m40.793878412s",
  "state": "fatal",
  "statement": "...,{\"#operator\":\"InitialProject\",\"discard_original\":true,\"preserve_order\":true,\"result_terms\":[{\"expr\":\"cover ((`b`.`metricId`))\"},{\"expr\":\"cover ((`b`.`metricName`))\"}]}]},{\"#operator\":\"Limit\",\"expr\":\"1\"},{\"#operator\":\"Stream\",\"serializable\":true}]},\"~versions\":[\"7.6.3-N1QL\",\"7.6.3-4211-enterprise\"]}",
  "useCBO": true,
  "userAgent": "couchbase-java/3.7.2 (Linux 5.10.226-214.880.amzn2.x86_64 amd64; OpenJDK 64-Bit Server VM 17.0.12+7-LTS)",
  "users": "local:master",
  "~qualifier": "threshold"
}

Workarounds

Upgrade to fix versions.