Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-63876] ArrayIndexOutOfBoundsException on connect link after pointing to a new cluster or bucket with differing number of vbuckets

Nam
Updated 
Product: Couchbase Server
Components: Analytics-Service, Columnar
Issue Link: MB-63876
Affects Versions: 7.2.x, 7.6.x
Fix Versions: 7.2.7, 7.6.6

Summary

The Analytics Service supports collections surviving the drop/recreation of a bucket, or updating a remote link to point to a new cluster. In these cases, we do a full rollback and ingest from 0, as the vbucket UUIDs are changed and the failover logs will not match.

The logic processing this rollback does not consider that the number of vbuckets in the new bucket and the old bucket may not match. If the number of vbuckets has been reduced, a runtime exception will occur.

This issue is possible on environments with Analytics enabled whenever a bucket is recreated with a higher vbucket count.

Symptoms 

  • Logs similar to the following can be seen:
2024-10-10T18:12:25.322-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:9600)-8] handleException: unexpected exception: uuid=null, clientContextID=c926591c-f447-4a2b-a479-f5c362885dca
com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0029: Connect link failed {"Default.externalcb.beer-sample":"Internal error"}
	at com.couchbase.analytics.lang.ConnectLinkStatement.doHandle(ConnectLinkStatement.java:464) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
	at com.couchbase.analytics.lang.ConnectionStatement.handle(ConnectionStatement.java:83) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
	at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:523) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
	at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java:181) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
	at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java:64) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
	at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$0(ApplicationMessageWork.java:74) ~[hyracks-control-cc-8.0.0-0000.jar:8.0.0-0000]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
	Suppressed: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.ArrayIndexOutOfBoundsException: Index 3072 out of bounds for length 3072
		at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api-8.0.0-0000.jar:8.0.0-0000]
		at org.apache.asterix.messaging.CCMessageBroker.sendSyncRequestToNCs(CCMessageBroker.java:141) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.util.DcpUtil.requestShadowStates(DcpUtil.java:262) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.util.DcpUtil.aggregateDcpState(DcpUtil.java:212) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.coordinate(CouchbaseConnectorFactory.java:556) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.lambda$submitShadowStateFutures$5(CouchbaseConnectorFactory.java:510) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.util.BlockingUnwindFuture.lambda$submit$1(BlockingUnwindFuture.java:50) ~[cbas-common-8.0.0-0000.jar:8.0.0-0000]
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
		at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
	Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 3072 out of bounds for length 3072
		at org.apache.hyracks.data.std.primitive.LongPointable.getLong(LongPointable.java:75) ~[hyracks-data-std-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.runtime.DcpBucketState.get(DcpBucketState.java:93) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.runtime.DcpBucketState.getSeq(DcpBucketState.java:78) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.adapter.ShadowStatesRequest.handle(ShadowStatesRequest.java:152) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at org.apache.asterix.messaging.NCMessageBroker.lambda$receivedMessage$0(NCMessageBroker.java:108) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
		... 4 more
  • This can result in the connect link will fail with internal error.

Triggers

  • To reproduce, an analytics collection must be based on a bucket that gets recreated with a higher vbucket count. e.g.,:
    • create analytics collection on a remote magma bucket w/ 128 vbuckets
    • connect link if not connected, allow for some ingestion to occur
    • disconnect link
    • drop remote bucket, recreate bucket with same name w/ 1024 vbuckets (e.g. couchstore bucket)
    • connect link

Verification

  • Log lines similar to the following:
2024-10-10T18:12:25.322-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:9600)-8] handleException: unexpected exception: uuid=null, clientContextID=c926591c-f447-4a2b-a479-f5c362885dca
com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0029: Connect link failed {"Default.externalcb.beer-sample":"Internal error"}
	at com.couchbase.analytics.lang.ConnectLinkStatement.doHandle(ConnectLinkStatement.java:464) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
	at com.couchbase.analytics.lang.ConnectionStatement.handle(ConnectionStatement.java:83) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
	at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:523) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
	at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java:181) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
	at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java:64) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
	at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$0(ApplicationMessageWork.java:74) ~[hyracks-control-cc-8.0.0-0000.jar:8.0.0-0000]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
	Suppressed: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.ArrayIndexOutOfBoundsException: Index 3072 out of bounds for length 3072
		at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api-8.0.0-0000.jar:8.0.0-0000]
		at org.apache.asterix.messaging.CCMessageBroker.sendSyncRequestToNCs(CCMessageBroker.java:141) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.util.DcpUtil.requestShadowStates(DcpUtil.java:262) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.util.DcpUtil.aggregateDcpState(DcpUtil.java:212) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.coordinate(CouchbaseConnectorFactory.java:556) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.lambda$submitShadowStateFutures$5(CouchbaseConnectorFactory.java:510) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.util.BlockingUnwindFuture.lambda$submit$1(BlockingUnwindFuture.java:50) ~[cbas-common-8.0.0-0000.jar:8.0.0-0000]
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
		at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
	Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 3072 out of bounds for length 3072
		at org.apache.hyracks.data.std.primitive.LongPointable.getLong(LongPointable.java:75) ~[hyracks-data-std-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.runtime.DcpBucketState.get(DcpBucketState.java:93) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.runtime.DcpBucketState.getSeq(DcpBucketState.java:78) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at com.couchbase.analytics.adapter.ShadowStatesRequest.handle(ShadowStatesRequest.java:152) ~[cbas-connector-8.0.0-0000.jar:8.0.0-0000]
		at org.apache.asterix.messaging.NCMessageBroker.lambda$receivedMessage$0(NCMessageBroker.java:108) ~[asterix-app-8.0.0-0000.jar:8.0.0-0000]
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
		... 4 more
2025-01-13T16:06:33.755+00:00 WARN CBAS.runtime.DcpBucketState [ShadowStatesRequest-(Default/externalcb/beer-sample)[-1]:StateRequest-0x0] number of bucket partitions has changed (was 128 now 1024); rolling back to 0
2025-01-13T16:06:33.757+00:00 WARN CBAS.adapter.ShadowRollbackPredicate [ShadowStatesRequest-(Default/externalcb/beer-sample)[-1]:StateRequest-0x0] rolling back memory component {"class":"LSMBTreeMemoryComponent", "state":"INACTIVE", "writers":0, "readers":0, "pendingFlushes":0, "id":"null"}
2025-01-13T16:06:33.758+00:00 WARN CBAS.runtime.DcpBucketState [ShadowStatesRequest-(Default/externalcb/beer-sample)[-1]:StateRequest-0x0] number of bucket partitions has changed (was 128 now 1024); rolling back to 0
2025-01-13T16:06:33.758+00:00 WARN CBAS.adapter.ShadowRollbackPredicate [ShadowStatesRequest-(Default/externalcb/beer-sample)[-1]:StateRequest-0x0] rolling back disk component {"class":LSMBTreeWithBloomFilterDiskComponent", "index":{"class":"DiskBTree","file":"storage/partition_0/Default/beers/0/beers/0_0_b"}} since its state has branched off of source history
2025-01-13T16:06:33.763+00:00 WARN CBAS.adapter.ShadowStatesRequest [ShadowStatesRequest-(Default/externalcb/beer-sample)[-1]:StateRequest-0x0] reallocating dcp bucket state due to bucket partitions mismatch (ours: 128 theirs: 1024)

Workarounds

Aside from upgrading to a version containing the fix, the only workaround for this issue is to drop and recreate all analytics collections and secondary indexes.

Related to

Related articles

Comments

0 comments

Article is closed for comments.