[MB-65094] AWS SDK no longer send content hash, causing upload to object lock buckets to fail

Product: Couchbase Server
Component: cbbackupmgr
Issue Link: MB-65094
Affects Version: 7.6.x
Fix Version: 7.6.6

Summary

Backups to an S3 bucket where an Object Lock has been configured on the S3 bucket (with a defined retention period) will fail.

Symptoms 

Backup operations will fail every time with the following error message:

operation error S3: PutObject, https response error StatusCode: 400, RequestID: ****, HostID: ****, api error InvalidRequest: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameter 

Triggers

• Must be using cbbackupmgr versions from CB 7.6.1 to 7.6.5 inclusive (note: that this could be indirectly via the Backup Service).
• Must have enabled Object Lock on their S3 bucket with a defined retention period.
• Currently, no released version of the operator-backup image uses an affected version of cbbackupmgr, so Operator-managed backups should not be affected.

Verification

Look for the following error in the cbbackupmgr output/logs:

operation error S3: PutObject, https response error StatusCode: 400, RequestID: ****, HostID: ****, api error InvalidRequest: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameter 

Workarounds

Using cbbackupmgr directly, you can use the CB 7.6.0 version of cbbackupmgr. This is included in the CB 7.6.0 server tools package. The links are no longer published on the Couchbase website, but they do still work and can be found on the older version of the website.

In this cases where the backup service is in use, we recommend turning off Object Lock on S3 buckets or upgrading to a version of Couchbase Server with the fix.