[MB-68028] Panic in reporting changed settings – Couchbase Support

Product: Couchbase Server
Component: Query-Service
Issue Link: MB-68028
Affects Version: 7.2
Fix Version: 7.2.8

Summary

Trapping multiple errors in query settings may result in a Query Service crash when submitting the settings.

Symptoms

Query Service crashes with a panic on all Query Service nodes in the cluster when settings are submitted.

runtime error: comparing uncomparable type []interface {}:

http: panic serving 10.132.132.176:58664: runtime error: comparing uncomparable type []interface {}
goroutine 98993 [running]:
net/http.(*conn).serve.func1()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:1903 +0xbe
panic({0x28179e0?, 0xc000432500?})
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/runtime/panic.go:770 +0x132
github.com/couchbase/query/server.compareMaps(...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:371
github.com/couchbase/query/server.reportChangedValues(0xc0015cd648, 0xc0015cd508)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:387 +0x611
github.com/couchbase/query/server.ProcessSettings(0xc00150c5d0, 0xc0004d46c8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:360 +0x905
github.com/couchbase/query/server/http.settingsWorkHorse(0xc00150c5d0, 0xc0004d46c8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_clustering_endpoint.go:436 +0x74
github.com/couchbase/query/server/http.doSettings(0xc000a3fc00, {0x31780a0?, 0x0?}, 0xc001593560, 0xc001593680)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_clustering_endpoint.go:418 +0x239
github.com/couchbase/query/server/http.(*HttpEndpoint).wrapAPI(0xc000a3fc00, {0x316c7d0, 0xc00047a2a0}, 0xc001593560, 0x2f4bac0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_endpoint.go:38 +0x1e4
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_en
dpoint.go:38 +0x1e4
github.com/couchbase/query/server/http.(*HttpEndpoint).registerClusterHandlers.func8({0x316c7d0?, 0xc001895340?}, 0xc0013f
2cc0?)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_cl
ustering_endpoint.go:60 +0x34
net/http.HandlerFunc.ServeHTTP(0xc001b77c20?, {0x316c7d0?, 0xc001895340?}, 0x1?)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:2171 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000a78000, {0x316c7d0, 0xc001895340}, 0xc001b77b00)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1e2
net/http.serverHandler.ServeHTTP({0x3169ba8?}, {0x316c7d0?, 0xc001895340?}, 0x6?)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:3142 +0x8e
net/http.(*conn).serve(0xc000256ab0, {0x31702a8, 0xc00143e060})
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:2044 +0x5e8
created by net/http.(*Server).Serve in goroutine 421
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:3290 +0x4b4

Triggers

Submitting Query Service settings trapping multiple errors.
Clicking Save in the UI settings may also trigger the issue even if no changes are made via the UI, if the problematic settings are already present.
Submitting multiple errors sequentially will trigger the issue as well as submitting multiple errors in one command.

Sequentially:

curl -u Administrator:password http://localhost:8093/admin/settings -XPOST -d  '{"completed": { "+error": 12015}, "distribute":true}'
curl -u Administrator:password http://localhost:8093/admin/settings -XPOST -d  '{"completed": { "+error": 12014}, "distribute":true}'
curl -u Administrator:password http://localhost:8093/admin/settings -XPOST -d  '{"completed": { "+error": 12013}, "distribute":true}'

2025-07-21T10:24:09.894+00:00 [INFO] Query Configuration changed for completed-threshold. New value is 100
2025-07-21T10:24:09.903+00:00 [INFO] Query Configuration changed for completed. New value is map[+error:3000]
2025-07-21T10:24:09.912+00:00 [INFO] Query Configuration changed for completed. New value is map[+error:4000]
2025-07-21T10:24:09.921+00:00 [INFO] Query Configuration changed for completed. New value is map[+error:4010]
2025/07/21 10:24:09 http: panic serving 10.132.132.176:40680: runtime error: comparing uncomparable type []interface {}
goroutine 615692 [running]:
net/http.(*conn).serve.func1()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:1903 +0xbe
panic({0x28179e0?, 0xc00105ec00?})
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/runtime/panic.go:770 +0x132
github.com/couchbase/query/server.compareMaps(...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:371
github.com/couchbase/query/server.reportChangedValues(0xc000ff7648, 0xc000ff7508)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:387 +0x611
github.com/couchbase/query/server.ProcessSettings(0xc0046f6990, 0xc00019b8c8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:360 +0x905
github.com/couchbase/query/server/http.settingsWorkHorse(0xc0046f6990, 0xc00019b8c8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_clustering_endpoint.go:436 +0x74
github.com/couchbase/query/server/http.doSettings(0xc000988700, {0x31780a0?, 0x0?}, 0xc001987560, 0xc001987680)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_clustering_endpoint.go:418 +0x239
github.com/couchbase/query/server/http.(*HttpEndpoint).wrapAPI(0xc000988700, {0x316c7d0, 0xc001e48000}, 0xc001987560, 0x2f4bac0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_endpoint.go:38 +0x1e4
github.com/couchbase/query/server/http.(*HttpEndpoint).registerClusterHandlers.func8({0x316c7d0?, 0xc001e48000?}, 0xc0046f6930?)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_clustering_endpoint.go:60 +0x34
net/http.HandlerFunc.ServeHTTP(0xc001987440?, {0x316c7d0?, 0xc001e48000?}, 0x1?

Concurrently:

{"atrcollection":"","auto-prepare":false,"cleanupclientattempts":true,"cleanuplostattempts":true,"cleanupwindow":"1m0s","completed":{"aborted":null,"error":[3000,4000,4010,4020,4025,4030,4040,4050,4060,4070,4080,4090,4100,4110,4210,4300,4310,4320,4321,5000,5001,5002,5003,5010,5015,5020,5030,5035,5040,5050,5060,5070,5072,5075,5078,5080,5090,5100,5110,5120,5180,5190,5200,5210,5220,5230,5240,5250,5260,5270,5280,9999,11000,11001,11002,11003,11004,11005,11006,11007,11008,11009,11010,11011,11012,12000,12001,12002,12003,12004,12005,12006,12007,12008,12009,12010,12011,12012,12013,12014,12015,12016,12017,12018,15000,15001,15002,15003,15004,15005,15006,15007,15008,15009,15010,15011],"threshold":100},"completed-limit":4000,"completed-max-plan-size":262144,"completed-threshold":100,"controls":false,"cpuprofile":"","debug":false,"functions-limit":16384,"gc-percent":200,"keep-alive-length":16384,"loglevel":"INFO","max-index-api":4,"max-parallelism":1,"memory-quota":0,"memprofile":"","mutexprofile":false,"n1ql-feat-ctrl":76,"numatrs":1024,"pipeline-batch":16,"pipeline-cap":512,"plus-servicers":256,"prepared-limit":16384,"pretty":false,"profile":"off","request-error-limit":16,"request-size-cap":67108864,"scan-cap":512,"servicers":64,"timeout":0,"txtimeout":"0s","use-cbo":true}

Verification

Look for signs of a Query Service crash, eg the following log message will be printed in diag.log:

2025-08-08T07:08:15.768Z, ns_log:0:info:message(ns_1@10.132.171.52) - Service 'n1ql' exited with status 2. Restarting. Messages:
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:389 +0x52
github.com/couchbase/query/server/settings/couchbase.Subscribe.func1.1(0x0?, {0x0?, 0x0?})
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/settings/couchbase/metkv_settings.go:50 +0x134
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc00140ffa8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x7b
github.com/couchbase/query/server/settings/couchbase.Subscribe.func1()
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/settings/couchbase/metkv_settings.go:59 +0xa5
created by github.com/couchbase/query/server/settings/couchbase.Subscribe in goroutine 1
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/settings/couchbase/metkv_settings.go:44 +0x90

Look for evidence of Query Service settings changes in ns_server.query.log that traps multiple errors (either sequentially or concurrently):

2025-07-21T10:24:09.903+00:00 [INFO] Query Configuration changed for completed. New value is map[+error:3000]
2025-07-21T10:24:09.912+00:00 [INFO] Query Configuration changed for completed. New value is map[+error:4000]
2025-07-21T10:24:09.921+00:00 [INFO] Query Configuration changed for completed. New value is map[+error:4010]

On the node where the Query Service crashed, look for a panic in ns_server.query.log that aligns with the following stack (search for comparing uncomparable type):

http: panic serving 10.132.132.176:58664: runtime error: comparing uncomparable type []interface {}
goroutine 98993 [running]:
net/http.(*conn).serve.func1()
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:1903 +0xbe
panic({0x28179e0?, 0xc000432500?})
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/runtime/panic.go:770 +0x132
github.com/couchbase/query/server.compareMaps(...)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:371
github.com/couchbase/query/server.reportChangedValues(0xc0015cd648, 0xc0015cd508)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:387 +0x611
github.com/couchbase/query/server.ProcessSettings(0xc00150c5d0, 0xc0004d46c8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/set_params.go:360 +0x905
github.com/couchbase/query/server/http.settingsWorkHorse(0xc00150c5d0, 0xc0004d46c8)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_clustering_endpoint.go:436 +0x74
github.com/couchbase/query/server/http.doSettings(0xc000a3fc00, {0x31780a0?, 0x0?}, 0xc001593560, 0xc001593680)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_clustering_endpoint.go:418 +0x239
github.com/couchbase/query/server/http.(*HttpEndpoint).wrapAPI(0xc000a3fc00, {0x316c7d0, 0xc00047a2a0}, 0xc001593560, 0x2f4bac0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_endpoint.go:38 +0x1e4
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_en
dpoint.go:38 +0x1e4
github.com/couchbase/query/server/http.(*HttpEndpoint).registerClusterHandlers.func8({0x316c7d0?, 0xc001895340?}, 0xc0013f
2cc0?)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/server/http/admin_cl
ustering_endpoint.go:60 +0x34
net/http.HandlerFunc.ServeHTTP(0xc001b77c20?, {0x316c7d0?, 0xc001895340?}, 0x1?)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:2171 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000a78000, {0x316c7d0, 0xc001895340}, 0xc001b77b00)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1e2
net/http.serverHandler.ServeHTTP({0x3169ba8?}, {0x316c7d0?, 0xc001895340?}, 0x6?)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:3142 +0x8e
net/http.(*conn).serve(0xc000256ab0, {0x31702a8, 0xc00143e060})
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:2044 +0x5e8
created by net/http.(*Server).Serve in goroutine 421
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.22.5/go/src/net/http/server.go:3290 +0x4b4

Workarounds

Avoid submitting Query Service settings changes trapping multiple errors.
No mitigation steps are required. When the Query Service crashes and restarts, the problematic settings are discarded.

Summary

Symptoms

Triggers

Verification

Workarounds

Related articles