[MB-64441] Support reading compressed values from DCP

Product: Couchbase Server
Component: Analytics-Service
Issue Link: MB-64441
Affects Versions: 6.5.x, 6.6.x, 7.0.x, 7.1.x, 7.2.x, 7.6.x
Fix Versions: 7.2.7, 7.6.6, 8.0.0

Summary

The Analytics Service currently does not support the reading of compressed values via DCP. This is mostly seen when reading from the Data Service.

For these compressed documents to be ingested properly by the Analytics Service, the producer (typically KV) will have to duplicate all of the values that are stored compressed and decompress them, due to the lack of support for reading compressed values in current versions of the Analytics Service.

Symptoms 

• The cbas process has an elevated kv_dcp_ready_queue_size_bytes value, as seen by the following example:
  
• This typically causes memory consumption to rise on Data Service nodes, which can be detrimental if there are many compressed documents being read.
• Since the Data Service is suffering from high memory usage, the Data Service will reject requests from:

1. Applications, where KV temporary failure exceptions can be seen.

com.couchbase.client.core.error.AmbiguousTimeoutException: UpsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x4b5a44510000001c","idempotent":false,"lastChannelId":"4B5A44510000001C/00000000C12B5220","lastDispatchedFrom":"169.193.70.84:47214","lastDispatchedTo":"LNJOLYALCB9P.nam.nsroot.net:11210","reason":"TIMEOUT","requestId":81812785,"requestType":"UpsertRequest","retried":67,"retryReasons":["KV_TEMPORARY_FAILURE"],"service":{"bucket":"bucket_name","collection":"colelction_name","documentId":"docID","errorCode":{"description":"Temporary failure. Try again","name":"ETMPFAIL"},"opaque":"0x4e071f0","scope":"scope_name","type":"kv","vbucket":995},"timeoutMs":60000,"timings":{"dispatchMicros":590832,"encodingMicros":61,"totalDispatchMicros":30380660,"totalServerMicros":2,"totalMicros":60001482,"serverMicros":0}}

2. Eventing Service, where backlogs and errors can be seen. An example of the problematic logs can be seen below:

2024-12-17T16:57:13.683+00:00 [Error] Consumer::getOpCallback [bucket_name/scope_name/collection_name:/tmp/127.0.0.1:8091_3_1205098686.sock:940693] Bucket fetch failed for key: <ud>eventing::1205098686::Cx6FM1::vb::607</ud>, err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"Get","Opaque":"0xa90ec5","TimeObserved":2500389022,"RetryReasons":["KV_ERROR_MAP_RETRY_INDICATED"],"RetryAttempts":13,"LastDispatchedTo":"node_name","LastDispatchedFrom":"node_address","LastConnectionID":"f5edb92f1d5b745b/d89226f585a3ef66"}

2024-12-17T16:57:13.692+00:00 [Error] Producer::getOpCallback [bucket_name/scope_name/collection_name:1] Bucket get failed for key: <ud>eventing::825308046::cAfNA3::vb::67</ud> , err: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"Get","Opaque":"0xa90ec2","TimeObserved":2500799862,"RetryReasons":["KV_ERROR_MAP_RETRY_INDICATED"],"RetryAttempts":13,"LastDispatchedTo":"node_name","LastDispatchedFrom":"node_address","LastConnectionID":"f5edb92f1d5b745b/d89226f585a3ef66"}

The impact of this issue will always come down to the node configurations and data load. If the Data Service nodes have enough capacity or if the load is low enough, the increased memory usage due to this isn’t going to be an issue. It may also have to do with the connection buffer size configured. It is possible that duplicate copies are going to be limited to what can fit in the connection buffer in use for DCP between KV + CBAS.

Triggers

• This can occur for any DCP producer from which the Analytics Service is reading from, but the impact will vary depending on the amount of compressed data being read.
• The impact will be higher when ingesting a large volume of data into the Analytics Service, such as during a rollback or building from scratch.

Verification

It can be validated that compression is being used by these two KV metrics:

"kv_dcp_total_data_size_bytes"
"kv_dcp_total_uncompressed_data_size_bytes"

These should be different in the event that analytics is handling compressed values – ensure that cbas is the only DCP client in order for this to be definitive.

The above is usually the best way to verify that the issue is being met, but you’ll get a hint that this is happening if we see stats like kv_dcp_ready_queue_size_bytes being elevated for the connection type cbas.

Workarounds

• First workaround is increasing the number of analytics nodes in the cluster, as this should enable the DCP queues to drain faster and hopefully avoid the OOM condition.
• Alternatively, avoiding the usage of compressed documents will result in this issue being avoided as well.
• Finally, the issue is fixed in 7.2.7 and 7.6.6, so upgrading to those versions when available should also be good, and is the ideal.