[CBG-4540] Avoid retention of DCP sync.Pool instances during large documents import processing

Product: Couchbase Sync Gateway
Component: sync-gateway
Issue Link: CBG-4540
Affects Version(s): 3.2.x
Fix Version(s): 3.2.4, 3.3.0

Summary

• This improvement involves addressing excessive memory retention in Sync Gateway, primarily because sync.Pool allocations would persist during import in memory on Sync Gateways, particularly in revision cache persistence of imported documents, especially large ones.
• This results in high system memory usage that is slow to release, causing sustained pressure on the Go runtime’s garbage collector.

Symptoms 

• Consistent, high memory usage by Sync Gateway during or after import operations
• Heap In Use frequently spikes and tracks closely with the internal GOMEMLIMIT (set internally to 90% of system memory minus 1 GB)
• Continuous increase in go_memstats_pausetotalns (prolonged or frequent Garbage Collection (GC) pauses, indicating GC thrashing)
• Notable concurrent spikes in CPU usage, coinciding with GC overhead
• May observe system slowness or instability if not addressed

Triggers

• Heavy or continuous DCP-based import of documents (particularly large document imports)
• Steady document loads that create persistent revision cache entries for imported large documents

Verification

Reviewing logs and Sync Gateway stats for memory and GC patterns during periods of document import activity:

• go_memstats_pausetotalns continuously rises (indicating frequent GCs)

• CPU usage spikes correspond with a rise in go_memstats_pausetotalns stat and memory pressure

• Heap In Use climbs and stays near GOMEMLIMIT during the time of GC:

Workarounds

This behavior is fundamentally a sizing issue:

• Temporarily increasing node/system memory will lift the GOMEMLIMIT, easing pressure on the Go garbage collector and making the issue less severe.
• Adjusting revision cache limits will also temporarily alleviate the issue.
• If feasible, upgrade Sync Gateway to 3.2.4 version or greater to reduce heap allocations for DCP-imported documents, where we have addressed some of these inefficiencies