[MB-64077] cbbackupmgr will fail and not retry when hitting ETMPFAIL

Product: Couchbase Server
Component: tools
Issue Link: MB-64077
Affects Versions: 7.2.x, 7.6.x
Fix Versions: 7.6.6, 8.0.0

Summary

The backup process (cbbackupmgr) will fail immediately and without retrying if it receives a temporary failure error{{ETMPFAIL}} from the Data Service. This can occur during common scenarios such as when a bucket's quota is exceeded or when the node experiences a temporary out-of-memory (tmpOOM) event.

Symptoms 

A backup using cbbackupmgr will fail without being automatically retried if the process is interrupted by a temporary server-side error, such as the bucket's memory quota being exceeded.

Triggers

This issue is triggered when a cbbackupmgr backup operation is interrupted by a temporary failure error (ETMPFAIL) from the Data Service, which can happen if a bucket's memory quota is reached or a temporary out-of-memory (tmpOOM) event occurs.

Verification

Check cbbackupmgr log for the following message:

2024-12-27T13:02:50.671+00:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'logs': failed to transfer bucket data for bucket 'logs': failed to transfer key value data: failed to transfer key value data: failed to open stream: %w: failed to stream vBucket 949: client received unexpected error 'temporary failure | {"status_code":134,"bucket":"logs","error_name":"ETMPFAIL","error_description":"Temporary failure. Try again","opaque":25,"last_dispatched_to":"svc-d-node-xxx.xxx.cloud.couchbase.com:11207","last_dispatched_from":"10.0.0.20:43158","last_connection_id":"2cd6c792fbc452f7/5e0be5b6d9b79f4a"}'

Check memcached.log for the following message (Stream request failed because memory usage is above quota):

2024-12-27T13:02:50.634409+00:00 WARNING 6613: (logs) DCP (Producer) eq_dcpq:cbbackupmgr:2024-12-27T13:02:49Z_5349_0 - (vb:949) Stream request failed because memory usage is above quota

Workarounds

To prevent the backup from failing due to temporary out-of-memory errors, you can use one of the following strategies:

• Increase Bucket Quota: Temporarily increase the memory quota for the bucket being backed up to ensure the operation has enough resources to complete. If the cluster has limited memory, you can achieve this by temporarily reducing the quota for other, non-essential buckets.
• Reduce Data Size: If possible, delete unnecessary data from the bucket before starting the backup. This reduces the overall memory pressure on the node and lowers the risk of a temporary out-of-memory (tmpOOM) event, which would cause the backup to fail.