Troubleshooting Slow Couchbase Backup and Restore Processes

Overview

This guide addresses common causes and solutions for slow backup and restore processes in Couchbase, providing steps to diagnose and resolve potential bottlenecks.

Identifying Bottlenecks

Couchbase Server Logs: Check CPU/Memory metrics on Data Service nodes to identify overload. As most of the resources used by the backup service are used for backing up the actual data, CPU/Memory pressure due to the overall load on the Cluster contributes the most to the backup/restoration slowness.
Backup Manager (cbbackupmgr) Logs: First of all you have to collect backup logs by running

the cbbackupmgr collect-logs command to collect the backup manager logs:

cbbackupmgr collect-logs --archive <archive_dir> --output-dir <dir>

Specify the name of your "archive"(--archive <archive_dir>) and the directory to store the logs (--output-dir <dir>).

Then collect a fresh set of your Cluster logs. Cluster logs can be collected via WebUI (on the Logs page), or via the command line using our cbcollect_info tool (requires sudo on Linux) with a command like the following (replacing <FILENAME> with a suitable name):

sudo /opt/couchbase/bin/cbcollect_info <FILENAME>.zip

This will produce a .zip archive with logs.

Determine the timeframe of when the slow backup/restoration was happening. For example in the backup logs below we can see the backup process started at 2024-04-16T12:00:01.758-04:00 and finished successfully at 2024-04-16T12:00:09.442-04:00:

2024-04-16T12:00:01.758-04:00 (Cmd) backup --archive /var/opt/couchbase/data/backup --repo My_repo_name --host couchbase://localhost --username Administrator --password ********** --threads 1 --no-progress-bar 
...
2024-04-16T12:00:09.442-04:00 (Cmd) Backup successfully completed

Then we need to analyze memcached.log for slow runtimes or connection issues at the time when the backup was taken, and overall. We can do it by utilizing the grep command:

The command below will look for any mentions of Slow Operations and Slow Runtimes on the Couchbase side
$ grep -i 'slow op.*\|slow run.*' memcached.log -c
1

We can remove -c flag to show the messages instead of the number of messages

$ grep -i 'slow op.*\|slow run.*' memcached.log
...

Also, we can specify the exact date, hour, or minute to narrow down the timeframe of the search. Let's search for all Slow Operations and Slow Runtimes that happened on 2024-04-16 at 12 PM

$ grep -i '2024-04-16T12.*slow op.*\|2024-04-16T12.*slow run.*' memcached.log

If you found any of the Slow Operations and Slow Runtimes it may indicate that your cluster is undersized or experiencing some other issues affecting the speed of the Data Service. We would recommend getting in touch with Couchbase Support to analyze the slowness further.

Common Issues

There are a few common issues that you might want to check.

Resource Limitation: Ensure the backup manager is not limited by CPU. You can specify the number of threads dedicated to cbbackupmgr. Please, check if the number of threads (--threads) is less than the number of CPUs available to your cluster).
Network Issues: For backups to Network File Systems (NFS), verify network speed and stability. Eliminating network issues is crucial for determining the root cause of the backup slowness.
DCP Packet Handling: Look for "Notifying paused connection" in logs which indicates slow acknowledging (TCP ACK packets) of DCP packets.

Memory Utilization

High memory usage might be due to the number of KV nodes, threads, and document throughput(traffic load on the cluster).
Collect memory profiles to pinpoint high memory consumption. You can utilize several tools and methods in Linux:

valgrind with massif: Run your application under valgrind using the massif tool to analyze memory usage over time.

gperftools: Install Google Performance Tools which include tcmalloc and use its heap profiler for detailed memory usage analysis.

memprof: A lightweight tool that tracks memory allocation and reports leaks and growth.

perf: Use Linux's built-in perf tool to collect memory-related performance counters.

System tools: Utilities like top, htop, free, and vmstat provide quick insights into memory usage.

Each tool offers different levels of detail and requires varying degrees of setup or overhead. Choose based on your specific needs, such as whether you need detailed allocation tracking or just overall usage statistics.

CPU profiles

To collect CPU profiles on Linux, you can use a variety of tools depending on your needs:

perf: This powerful tool is part of the Linux kernel's performance tools suite. It can trace system calls, sample CPU usage, and more.

gprof: This GNU profiler measures the time spent in each function of a program to help identify sections of code that are consuming the most CPU time.

top and htop: These are interactive command-line tools that show a dynamic real-time view of a running system, particularly CPU usage.

sysstat package (mpstat, iostat): Includes utilities to report CPU activity and system statistics.

pidstat: Part of the sysstat utilities, it is useful for monitoring individual processes.

Each tool offers different features and outputs, allowing you to analyze CPU usage effectively.

Solutions and Workarounds

Here is the summary of the above including possible workarounds and further actions.

Adjust the number of threads (--threads) to balance memory usage and backup speed.
Collect and analyze memory and CPU profiles to determine specific issues and adjust configurations accordingly.
If issues persist, consider opening a Couchbase Support Engineering case.

Contacting Couchbase Support

Collect logs from all relevant nodes, including Couchbase Server and cbbackupmgr logs, to assist in troubleshooting, utilizing the methods highlighted in Identifying Bottlenecks section of this article.
Upload profiles and logs to the support portal to the opened ticket.