Understanding mutations during ongoing backups

Question:

How does cbbackupmgr handle mutations that happen during the backup process?

• For example, if a cluster's backup takes 1 hour. How many of the mutations will end up in the backup? 

Prerequisites:

Before deep diving into the answer, it is important to understand what sequence numbers are, each bucket in Couchbase will have 1024 vBuckets and for each mutation (create, update, and delete) to a vBucket it maintains and increments a value monotonically called sequence number. This can be referenced by interested processes. The later the mutation, the higher the number. The sequence is strictly relevant to the vBucket, and does not provide a cluster-wide ordering of events. 

Answer:

At the start of a backup, cbbackupmgr fetches the sequence number range it wants from each vBucket. Cbbackupmgr then begins transferring data from as many vBuckets as it can at one time. Cbbackupmgr can transfer data from one vBucket per thread available to it. For each vBucket cbbackupmgr starts by asking for the mutations from the initially calculated range of sequence numbers calculated at the start of the backup process. So, extra mutations that come in during a backup are not intended to be backed up, as they will correspond to a higher sequence number than the range cbbackupmgr originally asked for. Although these extra mutations are not originally intended to be backed up, some of these new mutations could be backed up as we will cover below in Edge Cases.

Full Backup:

For a full backup, all mutations from sequence number 0 to the current sequence number at the start of the backup process will be backed up.

Full Backup example logs:

For vb:0, a stream was created from sequence number 0 to the latest 3306.

2024-06-26T12:33:24.360-07:00 (Archive) (Repo) Creating new backup '2024-06-26T12_33_24.36071-07_00'
...
2024-06-26T12:33:26.272-07:00 (DCP) (pillow) (vb 0) Creating DCP stream | {"uuid":0,"start_sequence number":0,"end_sequence number":3306,"snap_start":0,"snap_end":0,"retries":0}
...
2024-06-26T12:33:30.327-07:00 (DCP) (pillow) (vb 0) Stream closed because all items were streamed | {"uuid":130950556931958,"snap_start":0,"snap_end":3306,"last_sequence number":3306,"retries":0}

Incremental Backup:

In an incremental backup, the starting sequence number will be the end sequence number of the previous backup, and the new ending sequence number will be the current highest sequence number at the start of the incremental backup process. 

In our previous example of the full backup, we saw the full backup for vBucket 0 covered the following sequence numbers: "start_sequence number":0,"end_sequence number":3306

In this incremental backup taken of the same bucket minutes later, we can confirm that the new start_sequence number matches the end_sequence number from the previous backup.

Incremental Backup example logs:

2024-06-26T12:41:56.966-07:00 (Archive) (Repo) Creating new backup '2024-06-26T12_41_56.966956-07_00'
...
2024-06-26T12:41:58.848-07:00 (DCP) (pillow) (vb 0) Creating DCP stream | {"uuid":130950556931958,"start_sequence number":3306,"end_sequence number":5768,"snap_start":0,"snap_end":3306,"retries":0}
...
2024-06-26T12:42:03.158-07:00 (DCP) (pillow) (vb 0) Stream closed because all items were streamed | {"uuid":130950556931958,"snap_start":0,"snap_end":5768,"last_sequence number":5768,"retries":0}

Edge Cases:

Some extra mutations beyond the initial sequence number range might be backed up to keep vBuckets consistent with disk snapshots. This can happen because cbbackupmgr doesn’t necessarily start streaming the mutations from each vBucket as soon as the backup process starts. As previously mentioned, cbbackupmgr can only transfer data from the number of vBuckets equal to the number of threads available to the cbbackupmgr process at a time.

If the Data Service has mutations coming in while the backup is going on, it could create disk snapshots that differ from the sequence number range that cbbackupmgr initially calculated for each vBucket. 

Let’s say we are running a full backup. For a particular vBucket, the initially calculated sequence number was 0-150. However, this particular vBucket is the last vBucket backed up during this full backup run, when cbbackupmgr gets to the point that it starts backing up this particular vBucket, the snapshot on disk could contain sequence numbers 0-200. Disk snapshots are consistent, they represent a real state that existed on disk. Since we want a consistent backup, the Data Service will send us the entirety of that snapshot, even though it contains more mutations than we originally requested.

This situation is most likely to occur when:

• The cluster contains a lot of data, as the backup will take longer, meaning it’s more likely for new mutations to come in between the sequence number calculation and the actual start of the vBucket being streamed
• The cluster is very active meaning there is a higher chance for new mutations to occur in between the sequence number calculation and the actual start of the vBucket being streamed
• The number of threads allocated to the backup process is low or the backup node is slow

Example of this from cbbackupmgr logs:

The first line shows the sequence number range we initially asked for in vBucket 62, 0-15252. The second line shows the mutations we actually backed up for vBucket 62: 0-16722.

2024-06-25T10:48:09.730+01:00 (DCP) (travel-sample) (vb 62) Creating DCP stream | {"uuid":0,"start_sequence number":0,"end_sequence number":15252,"snap_start":0,"snap_end":0,"retries":0}
...
2024-06-25T10:48:10.682+01:00 (DCP) (travel-sample) (vb 62) Stream closed because all items were streamed | {"uuid":158537472749432,"snap_start":0,"snap_end":16722,"last_sequence number":16722,"retries":0}