[MB-67367] Ingestion, connect race between ingestion partitions can result in repeated/perpetual rollbacks

Product: Couchbase Server
Component: Analytics-Service
Issue Link: MB-67367
Affects Versions: 7.2.6, 7.2.7, 7.6.2, 7.6.3, 7.6.4, 7.6.5, 7.6.6, 7.6.7
Fix Versions: 7.2.8, 7.6.8

Summary

• A race condition occurs when some partitions start processing mutations while others indicate rollback, causing perpetual rollback cycles.
• This cycle can repeat indefinitely until all lagging collections indicate a rollback from a single ingestion partition in a given cycle. At this point, ingestion will restart from 0 on all lagging collections, and the cycle will terminate.
• The likelihood of the cycle stopping is going to vary based on the number of ingestion partitions and the number of distinct KV collections being ingested. (The larger the number of each, the lower the chances of the cycle breaking spontaneously).

Symptoms 

• Perpetual rollbacks on existing CBAS links.

Triggers

• A race at the cluster-level that can occur during ingestion bootstrap between the Analytics and Data Services. Once a partition successfully opens all of its assigned streams, it is then able to start processing mutations.
• Other partitions may have indicated that rollback is required on one or more collections. If any of the partitions received any mutations during this time, these will be written to storage once the cluster stops the ingestion across all partitions to accommodate the rollback.
• Once the Analytics Service collections are rolled back, ingestion will bootstrap again. Depending on how historical the bucket is, it becomes likely that the mutations that were processed by the successful partitions will fail to reach the purge point.
• Finally, as a result, the Data Service will instruct the collections to require rollback, starting the cycle again.

Verification

• Unfortunately, there are currently no specific statistics that would help us understand that this issue is being encountered. Generally, though, if we can see the notifying CC of the following required rollback messages on the same bucket over an extended period of time:

2025-09-19T03:25:13.130+00:00 INFO CBAS.adapter.CouchbaseConnector [SAO:JID:1.729287:TAID:TID:ANID:ODID:218:0:2:0:({link_name})[2]:BO] notifying CC of the following required rollbacks: [{"sid":2,"vbucket":147,"open-stream-response":"Rollback required (0x23)","rollback-seq":0}, {"sid":2,"vbucket":148,"open-stream-response":"Rollback required (0x23)","rollback-seq":0},...

• In addition, we should also be able to see messages such as Performing full rollback of datasets...for the bucket... within a short period of time. E.g.:

2025-08-18T00:10:59.264-05:00 INFO CBAS.metadata.BucketEventsListener [Executor-3370:ClusterController] Performing full rollback of datasets [{dataset_names}] for the bucket {bucket_details}

• It is possible that it can self-recover in a handful of cycles, but this depends on how many Analytics Service nodes there are, and how many distinct KV collections are being ingested, and it is difficult to know for certain.

Workarounds

A safe workaround would be to disconnect the link, drop, and recreate all collections on the bucket that are observed to be rolling back in the cycles.

This issue can be largely avoided by preventing partial ingestion that does not reach the purge point by coordinating operations that cause ingestion to start from 0 to be performed while the link is disconnected. This includes:

1. Creating new analytics collections on KV collections
2. Dropping or recreating KV collections

If the link is disconnected before these actions and only reconnected once these are complete, this issue can be avoided.