[MB-57221] Efficient estimation of free memory in Plasma for cgroups

Product: Couchbase Server
Component: Index-Service
Issue Link: MB-57221
Affects Version(s): 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4
Fix Version(s): 7.1.5, 7.2.5, 7.6.0 

Summary

In environments that use cgroups, the amount of free memory may be underestimated, resulting in the Plasma mem-tuner behaving more aggressively than it should. 

The Plasma mem-tuner uses the mem_actual_free value to tune its quota. With cgroups, the pagecache is included as 'used memory' when calculating mem_actual_free. This results in mem_actual_free being much smaller. Since mem_actual_free is much smaller, this results in the mem-tuner being activated earlier and more aggressively.

Since the Plasma mem-tuner sees that there is less free memory than there actually is, the mem-tuner will reduce Plasma's internal memory quota:

2024-04-06T17:13:18.431+00:00 [Info] Plasma: Adaptive memory quota tuning (decrementing): RSS:46635016192, freePercent:5, currentQuota=22484676154, newQuota=22111070118, netGrowth=373606036, percent=1

Reducing the quota can cause burst eviction to occur in an attempt to free memory. Evicting items out of memory results in the resident ratio decreasing. Burst eviction is also a CPU intensive process per this comment.

Reducing the quota can also result in lower throughput as Plasma will throttle the incoming mutations to keep memory usage in check per this comment. 

Symptoms

• Low index resident ratios (this can trigger MB-58147 Closed if the RR reaches 0%).
• High Index Service CPU utilisation. 
• Slow Indexer mutation ingestion. 
• Query timeouts due to slow index scans as a result of low index resident ratios.
• Poor index build and rebalance performance as a result of slow mutation ingestion.

Triggers

• Running Couchbase Server in an environment that uses cgroups, eg:
  • Docker containers
  • Couchbase Autonomous Operator clusters
  • RHEL 9.x+
• Running low on free memory / high pagecache usage.

Verification

First, high active_file / inactive_file usage (couchbase.log):

/sys/fs/cgroup/memory/memory.stat:inactive_file 2715095040
/sys/fs/cgroup/memory/memory.stat:active_file 50757259264

The above example has active_file using 50GB of memory which the Indexer assumes is in use.

Alongside the above, Plasma reporting low memory available and shrinking quota:

2024-02-06T22:49:28.285+00:00 [Info] Plasma: Adaptive memory quota tuning RSS:32571850752, freePercent:0, currentQuota=19134079303, heapSz=29257052160, limit=false 
..
2024-02-06T22:49:53.712+00:00 [Info] Plasma: Warning: system free memory < 5%!

Finally, verifying the amount of actual free memory (which includes free+available) is significantly higher than free memory reported by Plasma.

Workarounds

Clearing the page cache on every Index node on a frequent basis is the only workaround. This should be deployed as a bash script with a cron job if it occurs frequently enough. The frequency differs for each cluster and will require tuning.

Linux command to clear the page cache:

sudo sync; echo 3 > /proc/sys/vm/drop_caches

This can be used to drop the cache (after sync) when active_file and inactive_file exceeds 50% of the cgroup memory limit.