[MB-55100] When memcached process is killed forcibly or due to OOM, SDK fails to get the cluster config and bootstrap the cluster

Product: Couchbase Server
Component: ns-server
Issue Link: MB-55100
Affects Versions: 7.0.x, 7.1.x
Fix Versions: 7.1.6, 7.2.0, 7.6.0, 8.0.0

Summary

• When the memcached process is forcefully terminated by the OS due to an OOM kill or by a user issuing the kill -9 command, connected Java-based SDKs start producing Failed to open global config messages in the SDK logs. If the application is restarted afterward, it fails to bootstrap the cluster, logging the following message:

com.couchbase.client.core.error.UnambiguousTimeoutException: WaitUntilReady timed out

This issue persists until the Couchbase Server process is restarted on the affected node.

Symptoms 

• SDK logs show Failed to open global config messages followed by GlobalConfigNotFoundException.
• After restarting the application, the logs show UnambiguousTimeoutException with WaitUntilReady timed out messages.
• The issue stops after a restart of Couchbase Server on the affected node.

Triggers

• This happens due to a force kill of the memcached process on a cluster node. Both OOM kills and explicit kill -9 <PID> commands have been observed to produce the same effect.

Verification

• Check the SDK logs for Failed to open global config messages followed by either GlobalConfigNotFoundException or UnambiguousTimeoutException related to WaitUntilReady timed out messages.
• Confirm that the memcached process on any cluster node was forcefully terminated shortly before these issues appeared.

Workarounds

• The workaround for this issue is to restart the Couchbase Server process on the node where the memcached process was forcefully killed.