[GOCBC-1569] Dispatching an operation to a particular node can infinite loop if the pipeline is closed – Couchbase Support

Product: Go SDK
Component: sdk
Issue Link: GOCBC-1569
Affects Version(s): 7.0.x, 7.1.x, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4
Fix Version(s): 7.2.5, 7.6.0

Summary

While dispatching a request to the Data Service by calling "DispatchDirectToAddress()", the Go SDK can end up in an infinite loop if the pipeline is closed. This leads to processes hanging and CPU usage steadily climbing until the process is killed.

Symptoms

This issue manifests differently depending on which component is affected, due to the Go SDK being used heavily as a dependency.

Go SDK

With the Go SDK as an application library, this will manifest as a high CPU usage for the customers application, only going down after a restart

cbbackupmgr

High CPU usage similar to the Go SDK on its own
Backup/Restore processes are hung indefinitely due to infinite loop

Triggers

Issuing a request with the Go SDK that makes use of "DispatchDirectToAddress" such as:

Starting a backup with cbbackupmgr
Pinging the KV service with the Diagnostics API

Verification

This can be easily verified by reviewing a goroutine dump of the Go process that is hung/consuming high CPU. This manifests differently, but we are looking for instances of DispatchDirectToAddress within the stack trace:

8 @ 0x4043af 0x8d5ec5 0x8c5545 0x8c5533 0x8ae26b 0x468bc1
#       0x8d5ec4        github.com/couchbase/gocbcore/v10.(*memdPipeline).sendRequest+0x44              /go/pkg/mod/github.com/couchbase/gocbcore/v10@v10.1.4/memdpipeline.go:89
#       0x8c5544        github.com/couchbase/gocbcore/v10.(*memdPipeline).SendRequest+0xc4              /go/pkg/mod/github.com/couchbase/gocbcore/v10@v10.1.4/memdpipeline.go:105
#       0x8c5532        github.com/couchbase/gocbcore/v10.(*kvMux).DispatchDirectToAddress+0xb2         /go/pkg/mod/github.com/couchbase/gocbcore/v10@v10.1.4/kvmux.go:403
#       0x8ae26a        github.com/couchbase/gocbcore/v10.(*diagnosticsComponent).pingKV.func1.1+0x28a  /go/pkg/mod/github.com/couchbase/gocbcore/v10@v10.1.4/diagnosticscomponent.go:141

If a CPU profile is available, this function will appear as a significant consumer of CPU:

File: rohandler
Type: cpu
Time: Jun 3, 2024 at 6:49am (BST)
Duration: 60.11s, Total samples = 340.94s (567.17%)
Showing nodes accounting for 338.90s, 99.40% of 340.94s total
Dropped 126 nodes (cum <= 1.70s)
      flat  flat%   sum%        cum   cum%
   221.76s 65.04% 65.04%    306.12s 89.79%  github.com/couchbase/gocbcore/v10.(*memdOpQueue).Push
    78.86s 23.13% 88.17%     80.10s 23.49%  sync.(*Mutex).Unlock
    13.97s  4.10% 92.27%     14.07s  4.13%  runtime.ifaceeq
    12.06s  3.54% 95.81%    325.66s 95.52%  github.com/couchbase/gocbcore/v10.(*memdPipeline).sendRequest
     5.08s  1.49% 97.30%      5.08s  1.49%  runtime.asyncPreempt
     4.34s  1.27% 98.57%    340.26s 99.80%  github.com/couchbase/gocbcore/v10.(*kvMux).DispatchDirectToAddress
     2.83s  0.83% 99.40%    328.67s 96.40%  github.com/couchbase/gocbcore/v10.(*memdPipeline).SendRequest (inline)
         0     0% 99.40%    340.26s 99.80%  github.com/couchbase/gocbcore/v10.(*diagnosticsComponent).pingKV.func1.1

Workarounds

Restart clients
Avoid using the SDKs Diagnostics API

Fixed versions

SDK:

2.7.1 (Gocbcore v10.3.1)

Server versions:

7.2.5
7.6.0

Related to