[MB-51318] Potential Request Stall When Stream Operator Fails to Notify of Termination – Couchbase Support

Product: Couchbase Server
Component: query-service
Issue Link: MB-51318
Affects Version(s): 6.6.5, 7.0.4, 7.1.2
Fix Version(s): 6.6.6, 7.0.5, 7.1.3, 7.2.0, 8.0.0

Summary

The Query Service can become unresponsive, causing the request queue to fill up and eventually reject new queries.
This issue may occur if the ServeHTTP request handler does not detect that the Stream operator has failed (e.g., after a network close, timeout, or termination).
Without proper notification, this leads to a hung state, as observed in at least one customer environment—even though no explicit code defect or test case currently demonstrates it.

Symptoms

All new Query Service operations on the affected node are failing or timing out.
Existing Query Service operations remain stuck in the "submitted" state.
The UI displays a 503 error when attempting to access the Query Service page.
Sync Gateway queries are failing.

Triggers

If the Stream operator fails to send a notification for any reason, this code path can result in a hang.

Verification

1. Verify Servicer Counts

On each Query Service node, check the number of servicers using the formula: servicers = number of CPUs × 4
If using scan_consistency of request_plus or at_plus, calculate plus-servicers as: plus-servicers = number of CPUs × 16
Alternatively, examine the ns_server.query.log for lines like:

2023-05-04T02:12:36.987-05:00 [INFO] cbq-engine started version=7.1.0-N1QL datastore=http://127.0.0.1:8091 max-concurrency=4 loglevel=INFO servicers=16 plus-servicers=64 scan-cap=512 pipeline-cap=512 pipeline-batch=16 request-cap=256 request-size-cap=67108864 max-index-api=4 max-parallelism=1 n1ql-feat-ctrl=76 use-cbo=true timeout=0s txtimeout=0s gc-percent=200

Here, servicers=16 and plus-servicers=64 indicate the configured values.

2. Check Request Queue Size

The expected request queue size is: queue size = number of CPUs × 256

3. Analyze Profiler Logs

Review query_pprof.log for signs of issues:
- Look for stack traces where the number of goroutines matches (queue size − 1) - this indicates the request queue is full and the node cannot process new queries.
- We should also be able to see (*HttpEndpoint).ServeHTTP in this stack.

For example, with a 32 CPU node (queue size = 8192), a stack trace showing “8191” indicates a full queue.

8191 @ 0xe12370 0xe227f0 0xe227db 0xe22442 0xe60434 0x1ec5963 0x1ec9581 0x1ec9321 0x1ec8983 0x1ec7d67 0x1f6e52d 0x141c972 0x1130044 0x
112b8e5 0xe40141
#       0xe22441        sync.runtime_Semacquire+0x41                                            /home/couchbase/.cbdepscache/exploded/
x86_64/go-1.13.7/go/src/runtime/sema.go:56
#       0xe60433        sync.(*WaitGroup).Wait+0x63                                             /home/couchbase/.cbdepscache/exploded/
x86_64/go-1.13.7/go/src/sync/waitgroup.go:130
#       0x1ec5962       github.com/couchbase/query/server.(*BaseRequest).sleep+0x32             /home/couchbase/jenkins/workspace/couc
hbase-server-unix/goproj/src/github.com/couchbase/query/server/request.go:1040
#       0x1ec9580       github.com/couchbase/query/server.(*runQueue).addRequest+0x190          /home/couchbase/jenkins/workspace/couc
hbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:791
#       0x1ec9320       github.com/couchbase/query/server.(*runQueue).enqueue+0x80              /home/couchbase/jenkins/workspace/couc
hbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:743
#       0x1ec8982       github.com/couchbase/query/server.(*Server).handleRequest+0x42          /home/couchbase/jenkins/workspace/couc
hbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:618
#       0x1ec7d66       github.com/couchbase/query/server.(*Server).ServiceRequest+0x76         /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:561
#       0x1f6e52c       github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP+0x35c  /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:323
#       0x141c971       github.com/gorilla/mux.(*Router).ServeHTTP+0xe1                         /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210
#       0x1130043       net/http.serverHandler.ServeHTTP+0xa3                                   /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802
#       0x112b8e4       net/http.(*conn).serve+0x874                                            /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890

Similarly, for a 4 CPU node (queue size = 1024), “1023” goroutines means the queue is full.

1023 @ 0xd757c5 0xd87345 0xd8732e 0xda9325 0xdc9045 0x1fadaf3 0x1fb1a8a 0x1fb1825 0x1fb0f65 0x1fb0127 0x20ea88a 0x14f83f3 0x10ac3c3 0x10adf8d 0x107e98b 0xdad3c1
#       0xda9324        sync.runtime_Semacquire+0x44                                            /home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.6/go/src/runtime/sema.go:56
#       0xdc9044        sync.(*WaitGroup).Wait+0x64                                             /home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.6/go/src/sync/waitgroup.go:130
#       0x1fadaf2       github.com/couchbase/query/server.(*BaseRequest).sleep+0x32             /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/request.go:1145
#       0x1fb1a89       github.com/couchbase/query/server.(*runQueue).addRequest+0x1a9          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:833
#       0x1fb1824       github.com/couchbase/query/server.(*runQueue).enqueue+0x84              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:785
#       0x1fb0f64       github.com/couchbase/query/server.(*Server).handlePlusRequest+0x44      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:672
#       0x1fb0126       github.com/couchbase/query/server.(*Server).PlusServiceRequest+0x86     /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:602
#       0x20ea889       github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP+0x329  /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:445
#       0x14f83f2       github.com/gorilla/mux.(*Router).ServeHTTP+0xd2                         /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
#       0x10ac3c2       net/http.serverHandler.ServeHTTP+0xa2                                   /home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.6/go/src/net/http/server.go:2867
#       0x10adf8c       net/http.initALPNRequest.ServeHTTP+0x8c                                 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.6/go/src/net/http/server.go:3439
#       0x107e98a       net/http.(*http2serverConn).runHandler+0x8a                             /home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.6/go/src/net/http/h2_bundle.go:5723

Servicer Saturation: If the queue is full, this means the Query Service node cannot accept any new requests, which can result in query operations failing.
- Also check for stack traces where the number of goroutines equals the number of servicers or plus-servicers: this means all workers are blocked/stuck.
- We should also be able to see (*HttpEndpoint).ServeHTTP in this stack.

For example, with 32 CPUs: servicers = 128 (32 × 4), so "128" in the trace means all are stuck.

128 @ 0xe12370 0xde60d8 0xde60ae 0xde5d9b 0x1f7b921 0x1eca136 0x1ec89ac 0x1ec7d67 0x1f6e52d 0x141c972 0x1130044 0x112b8e5 0xe40141
#       0x1f7b920       github.com/couchbase/query/server/http.(*httpRequest).Execute+0x280     /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_response.go:156
#       0x1eca135       github.com/couchbase/query/server.(*Server).serviceRequest+0x7a5        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:1037
#       0x1ec89ab       github.com/couchbase/query/server.(*Server).handleRequest+0x6b          /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:622
#       0x1ec7d66       github.com/couchbase/query/server.(*Server).ServiceRequest+0x76         /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:561
#       0x1f6e52c       github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP+0x35c  /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:323
#       0x141c971       github.com/gorilla/mux.(*Router).ServeHTTP+0xe1                         /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210
#       0x1130043       net/http.serverHandler.ServeHTTP+0xa3                                   /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:2802
#       0x112b8e4       net/http.(*conn).serve+0x874                                            /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/net/http/server.go:1890

With 4 CPUs and at_plus mode: plus-servicers = 64 (4 × 16).

64 @ 0xd757c5 0xd411ef 0xd40e2b 0x20f93df 0x1fb2832 0x1fb10c7 0x1fb0127 0x20ea88a 0x14f83f3 0x10ac3c3 0x10adf8d 0x107e98b 0xdad3c1
#       0x20f93de       github.com/couchbase/query/server/http.(*httpRequest).Execute+0x53e     /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_response.go:164
#       0x1fb2831       github.com/couchbase/query/server.(*Server).serviceRequest+0x8d1        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:1112
#       0x1fb10c6       github.com/couchbase/query/server.(*Server).handlePlusRequest+0x1a6     /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:687
#       0x1fb0126       github.com/couchbase/query/server.(*Server).PlusServiceRequest+0x86     /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/server.go:602
#       0x20ea889       github.com/couchbase/query/server/http.(*HttpEndpoint).ServeHTTP+0x329  /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/service_endpoint.go:445
#       0x14f83f2       github.com/gorilla/mux.(*Router).ServeHTTP+0xd2                         /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210
#       0x10ac3c2       net/http.serverHandler.ServeHTTP+0xa2                                   /home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.6/go/src/net/http/server.go:2867
#       0x10adf8c       net/http.initALPNRequest.ServeHTTP+0x8c                                 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.6/go/src/net/http/server.go:3439
#       0x107e98a       net/http.(*http2serverConn).runHandler+0x8a                             /home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.6/go/src/net/http/h2_bundle.go:5723
We should be able to see (*httpRequest).Execute and (*HttpEndpoint).ServeHTTP in the above stack.

If you observe either a full request queue or all servicers/plus-servicers blocked (as above) in the logs/profiles of any Query Service node, this strongly indicates the known issue is present and the node is unable to accept or process new queries.

Workarounds

Restart the Query Service on the affected nodes.

Related to

Summary

Symptoms

Triggers

Verification

Workarounds

Related articles