Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-62923] Analytics leaking connection on credential refresh

Callum
Updated 
Product: Couchbase Server
Component: Analytics-Service
Issue Link: MB-62923
Affects Version: 7.6.x
Fix Version: 7.6.3

Summary

A connection leak between the Analytics Service (cbas) and its Java driver fails to close TCP connections properly. Over time, this exhausts the available ephemeral ports on the node, which can lead to service unavailability, connection failures, and other customer-facing impacts.

Symptoms 

  • The Analytics Service stops ingesting new data into collections.
  • In the UI, the status of Analytics Collections changes to amber, indicating a warning state.
    c0c9fa8c-d287-4039-ab10-64858b8cf01d-20250910-161935.png
  • Queries executed against Analytics Collections return empty results instead of the expected data.
  • Analytics queries fail and return a connection error.
[
  {
    "code": 25000,
    "msg": "Internal error"
  }
]
  • Analytics queries run from the Couchbase Server UI will fail with Analytics Service is temporarily unavailable.
  • In the ns_server.analytics_error.log log file, look for a combination of ingestion job failures (e.g., ingestion job JID:0.10611 failed) and Ingestion failed while reading a new record and Unauthorized - Incorrect credentials or bucket <bucket> does not exist, which indicate that the Analytics Service cannot access the required bucket:
2024-07-19T08:03:11.648+00:00 ERRO CBAS.active.ActiveEntityEventsListener [ActiveNotificationHandler] ingestion job JID:0.10611 failed
org.apache.hyracks.api.exceptions.HyracksDataException: ASX3110: Ingestion failed while reading a new record
    at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:70) ~[hyracks-api-7.6.1-3202.jar:7.6.1-3202]
    at org.apache.hyracks.api.util.ExceptionUtils.setNodeIds(ExceptionUtils.java:70) ~[hyracks-api-7.6.1-3202.jar:7.6.1-3202]
    at org.apache.hyracks.control.nc.Task.run(Task.java:398) ~[hyracks-control-nc-7.6.1-3202.jar:7.6.1-3202]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: com.couchbase.client.dcp.error.AuthorizationException: Unauthorized - Incorrect credentials or bucket vidya does not exist
    at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api-7.6.1-3202.jar:7.6.1-3202]
    at com.couchbase.analytics.adapter.CouchbaseConnector.init(CouchbaseConnector.java:661) ~[cbas-connector-7.6.1-3202.jar:7.6.1-3202]
    at com.couchbase.analytics.adapter.CouchbaseConnector.hasNext(CouchbaseConnector.java:625) ~[cbas-connector-7.6.1-3202.jar:7.6.1-3202]
    at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.hasNext(FeedRecordDataFlowController.java:156) ~[asterix-external-data-7.6.1-3202.jar:7.6.1-3202]
    at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.start(FeedRecordDataFlowController.java:87) ~[asterix-external-data-7.6.1-3202.jar:7.6.1-3202]
    at org.apache.asterix.external.dataset.adapter.FeedAdapter.start(FeedAdapter.java:41) ~[asterix-external-data-7.6.1-3202.jar:7.6.1-3202]
    at org.apache.asterix.common.external.IDataSourceAdapter.start(IDataSourceAdapter.java:75) ~[asterix-common-7.6.1-3202.jar:7.6.1-3202]
    at com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:55) ~[cbas-connector-7.6.1-3202.jar:7.6.1-3202]
    at org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:101) ~[asterix-active-7.6.1-3202.jar:7.6.1-3202]
    at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:233) ~[hyracks-api-7.6.1-3202.jar:7.6.1-3202]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    ... 3 more
Caused by: com.couchbase.client.dcp.error.AuthorizationException: Unauthorized - Incorrect credentials or bucket vidya does not exist
    at com.couchbase.client.dcp.conductor.RequestConfigHandler.channelRead0(RequestConfigHandler.java:98) ~[cbas-dcp-client-7.6.1-3202.jar:7.6.1-3202]
    at com.couchbase.client.dcp.conductor.RequestConfigHandler.channelRead0(RequestConfigHandler.java:41) ~[cbas-dcp-client-7.6.1-3202.jar:7.6.1-3202]
    at com.couchbase.client.core.deps.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[core-io-2.5.2.jar:?]
    at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[core-io-2.5.2.jar:?]
    ... 1 more
  • Backup operations fail with errors indicating that the Backup Service cannot connect to the Analytics Service.

Triggers

This issue is most often triggered by a credential rotation event.

Verification

Check the Analytics Service logs for a sequence of specific errors. The problem typically begins with connection failures, appearing as connect: cannot assign requested, which indicates ephemeral port exhaustion. As the issue progresses and credentials expire, these will be followed by authentication failures, such as Unexpected response from getHttpServiceAuth: HTTP/1.1 401 Unauthorized in the Analytics Service logging. You can also monitor the number of leaked connections between the cbas and Java processes on the loopback interface using a command like lsof.

Workarounds

As a temporary workaround, you can restart the Analytics Service on the affected nodes. While this does not provide a permanent fix, it will clear the leaked connections. Use the Cluster Restart API to perform the restart:

curl -v -u Administrator:password -X POST http://localhost:8095/analytics/node/restart
curl -v -u Administrator:password -X POST http://localhost:8095/analytics/cluster/restart

You could alternatively kill the cbas process (automatically restarted by the babysitter):

killall -9 cbas

Related to

Related articles

Comments

0 comments

Article is closed for comments.