Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-62633] Ingestion hung post scale down operation

Nam
Updated 
Product: Couchbase Server
Components: Analytics-Service, Columnar
Issue Link: MB-62633
Affects Versions: 7.2.5, 7.6.2
Fix Versions: 7.2.6, 7.6.3

Summary

This issue can happen when Analytics ingestion starts, connects to the Data service, and gets a rollback response. When a rollback occurs, the Analytics Service reconnects by first disconnecting and then connecting again. The nature of this issue is that a race condition may be triggered during the disconnect that causes it to get stuck.

Symptoms 

  • CBAS link process hangs.
  • CBAS queries unexpectedly show empty results, or inaccurate results.

Triggers

  • Full DCP rollback occurs for a CBAS link during Analytics ingestion.
  • Race condition occurs for the bucket disconnect, see below:

Verification

Find the log messages indicating that ingestion starts in ns_server.analytics_info.log, e.g.:

2024-12-28T05:05:37.851+00:00 INFO CBAS.active.RecoveryTask [RecoveryTask ({bucket}/{scope}/{collection})] Actual Recovery task has started
2024-12-28T05:05:41.679+00:00 INFO CBAS.active.RecoveryTask [RecoveryTask ({bucket}/{scope}/{collection}] Recovery completed successfully

Verify in ns_server.analytics_info.log, that rollbacks are required by looking for the following log message after ingestion has started:

2024-12-28T05:05:42.954+00:00 INFO CBAS.adapter.CouchbaseConnector [SAO:JID:1.1:TAID:TID:ANID:ODID:298:0:30:0:({link})[30]:BO] notifying CC of the following required rollbacks: [{"sid":1,"vbucket":661,"open-stream-response":"Rollback required (0x23)","rollback-seq":0}, {"sid":1,"vbucket":662,"open-stream-response":"Rollback required (0x23)","rollback-seq":0},

Check that CBAS handles the rollback response and tries to reconnect the bucket by disconnecting and then connecting again:

2024-12-28T05:05:42.982+00:00 INFO CBAS.metadata.BucketEventsListener [ActiveNotificationHandler] {link} handling RollbackEvent{fullRollback=true, fullRollbackStreams=[<13,{14}>,
2024-12-28T05:05:42.998+00:00 INFO CBAS.metadata.BucketEventsListener [ActiveNotificationHandler] Reconnecting {"class":"BucketEventsListener", "entityId":"{entity_id}", "state":"RUNNING", "prev state":"RECOVERING", "suspended":false}
...
2024-12-28T05:05:43.927+00:00 INFO CBAS.metadata.BucketEventsListener [Executor-64:ClusterController] Disconnecting the bucket {bucket} to perform full rollback
2024-12-28T05:05:43.927+00:00 INFO CBAS.active.ActiveEntityEventsListener [Executor-64:ClusterController : wait-for-ingestion-completion: JID:1.1] sending stop messages

Look for evidence of the race condition during the disconnect causing it to enter a “stuck” state. The following snippet from ns_server.analytics_debug.log is an example of this, where on one of the partitions, you can see the disconnect process hangs:

, "Executor-172:43c526c117020fc722f748b9e085a5b8" daemon prio=5 Id=331 BLOCKED on com.couchbase.analytics.adapter.CouchbaseConnector@252695e4 owned by "SAO:JID:1.1:TAID:TID:ANID:ODID:298:0:5:0:({link})[5]:BO" Id=188
	at app//com.couchbase.analytics.adapter.CouchbaseConnector.stop(CouchbaseConnector.java:995)
	-  blocked on com.couchbase.analytics.adapter.CouchbaseConnector@252695e4
	at app//org.apache.asterix.external.dataflow.FeedRecordDataFlowController.stop(FeedRecordDataFlowController.java:249)
	at app//org.apache.asterix.external.dataset.adapter.FeedAdapter.stop(FeedAdapter.java:50)
	at app//com.couchbase.analytics.runtime.BucketOperatorNodePushable.abort(BucketOperatorNodePushable.java:92)
	-  locked com.couchbase.analytics.runtime.BucketOperatorNodePushable@6ee71d9e
	at app//org.apache.asterix.active.ActiveSourceOperatorNodePushable.stop(ActiveSourceOperatorNodePushable.java:69)
	-  locked com.couchbase.analytics.runtime.BucketOperatorNodePushable@6ee71d9e
	at app//org.apache.asterix.active.ActiveManager.stopIfRunning(ActiveManager.java:208)
	at app//org.apache.asterix.active.ActiveManager.lambda$stopRuntime$3(ActiveManager.java:196)
	at app//org.apache.asterix.active.ActiveManager$$Lambda$648/0x000000008086c898.run(Unknown Source)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@58c6f90

, "SAO:JID:1.1:TAID:TID:ANID:ODID:298:0:5:0:(Local/onea)[5]:BO" daemon prio=5 Id=188 WAITING on java.lang.Thread@4797753f
	at java.base@17.0.10/java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Thread@4797753f
	at java.base@17.0.10/java.lang.Thread.join(Thread.java:1313)
	at java.base@17.0.10/java.lang.Thread.join(Thread.java:1381)
	at app//com.couchbase.client.dcp.conductor.Conductor.disconnect(Conductor.java:111)
	at app//com.couchbase.client.dcp.Client.disconnect(Client.java:311)
	-  locked com.couchbase.client.dcp.Client@36a1d306
	at app//com.couchbase.analytics.adapter.CouchbaseConnector.disconnect(CouchbaseConnector.java:1016)
	at app//com.couchbase.analytics.adapter.CouchbaseConnector$$Lambda$656/0x000000008086e6b0.run(Unknown Source)
	at app//org.apache.hyracks.api.util.InvokeUtil.doUninterruptibly(InvokeUtil.java:62)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@671dc2df

Workarounds

  • Restart CBAS using the Restart API.
  • Upgrade to a version containing the fix 7.2.6+ or 7.6.3+, ideally the latest 7.2.x or 7.6.x release.

Related to

Related articles

Comments

0 comments

Article is closed for comments.