Experiencing high CPU usage while running the XDCR diff tool on non-KV nodes.

Product: Couchbase Server
Component: XDCR
Affects Versions: 7.2.x, 7.6.x
Fix Version: TBD

Summary

Running an XDCR diff can lead to high CPU utilization due to the large number of items that need to be downloaded for preprocessing (e.g., removing duplicate entries and sorting documents by keys) before comparison.

Symptoms

High CPU usage observed during the execution of an XDCR diff on a non-KV node.

Triggers

The issue is triggered by:

• A large number of items must be downloaded and processed to eliminate duplicates and perform key-based sorting.
• A large number of tombstones are present in the dataset..

Verification

Try one of the two possible workarounds below and verify whether the issue is resolved.

Workarounds

1. Run compaction on both the source and target buckets so that differ could work on a comparatively smaller number of items, as mentioned in the README section of xdcrDiffer.
2. By default, GOMAXPROCS is set to the total number of logical CPU cores available on the system. If we want to limit CPU usage, we can set GOMAXPROCS to a smaller value than the default.
    
   Below are sample steps to configure GOMAXPROCS during an XDCR run on an 8-core CPU:
   1. Set GOMAXPROCS to a smaller value: 
      This limits the Go runtime to use only 4 threads for parallel execution, helping to reduce CPU load.

      export GOMAXPROCS=4

   2. Run xdcrDiffer, execute the tool with the required parameters.

      ./runDiffer.sh -u <username> -p <password> -h <host:port> -r <remoteRefName> -s <SourceBucket> -t <TargetBucket> -c

   3. Unset GOMAXPROCS This removes the environment variable, allowing future Go programs to default to using all available CPU cores.

      unset GOMAXPROCS

3. This ensures the Go runtime will only schedule work across 4 cores instead of all 8, thereby reducing CPU pressure.
   Note: Reducing GOMAXPROCS limits parallel execution, so xdcrDiffer may take longer to complete.
4. If running xdcrDiffer through a cron job, we recommend updating the cron job to explicitly include the GOMAXPROCS variable so that the change takes effect.

   /30 * * * * GOMAXPROCS=4 ~/runDiffer.sh -u Administrator -p <pass> -h <ip>:9000 -r Target -s B1 -t B1 -m both -c

Note: Making the above change will only affect the goxdcr process and will not impact other processes that use GOMAXPROCS. Please feel free to reach out to us for further details on this change.