[MB-59368] An ephemeral bucket with auto_delete enabled can trigger the item pager constantly with no memory reduction.

Product: Couchbase Server
Component: Data-Service
Issue Link: MB-59368
Affects Versions: 7.0.x, 7.1.x, 7.2.x
Fix Versions: 7.2.4, 7.6.0

Summary

Eviction does not function as intended for Ephemeral buckets on affected versions if the bucket memory quota is less than 250MB.

This issue is happening due to a mismatch in the quantifiers used to trigger memory recovery (item eviction) and then to stop memory recovery. An ephemeral bucket can be sized and utilised in such a way that the memory recovery is constantly triggered, only to be canceled, leading to noticeable CPU usage and to no reduction in the amount of memory utilized by the bucket.

Symptoms 

• High CPU utilisation for the memcached process on Data Service nodes.
  • “Unbalanced” CPU usage for Data Service nodes, some Data Service nodes have unexplained higher CPU usage than other Data Service nodes in the same cluster.
  • Memory usage does not decrease despite the items' eviction being triggered.

Triggers

• The adjusted high watermark (approximately half of the high watermark) is reached for Ephemeral buckets, but the memory usage is lower than the low watermark.
• The ephemeral bucket memory quota is low (below ~250MB).

Verification

Check whether bucket memory usage exceeds the adjusted high watermark, which is approximately half of the high watermark.

Check that the Eject data when RAM is full setting is enabled in settings for the affected Ephemeral bucket.

Check whether CPU usage for Data Service nodes is imbalanced, meaning the CPU usage is higher on some Data Service nodes compared to others.

In case cluster metrics are available, check kv_vb_auto_delete_count to determine if auto-deletes are occurring, as shown in the following image(the line representing kv_vb_auto_delete_count should be increasing):

Workarounds

Increase the bucket quota for the affected Ephemeral bucket above 250MB.