[MB-62984] ActiveStream Monotonicity Failure: Premature SeqnoAdvance on Filtered Memory Snapshots for First SnapshotMarker

Product: Couchbase Server
Component: Data-Service
Issue Link: MB-62984
Affects Versions: 7.0.x, 7.1.x, 7.2.x, 7.6.x
Fix Versions: 7.2.6, 7.6.3

Summary

• A failover or unclean shutdown can lead to a situation where the same sequence number is processed twice on a DCP stream to a client, causing the Data Service to crash.
• This happens because processing the duplicated mutation breaks the monotonicity of lastReadSeqno.
• The issue typically arises when a client has already received a SnapshotMarker for a snapshot that was lost during the crash. On reconnect, the client requests to continue from that snapshot.
• If rollback is not required, the stream request succeeds.
• If the node was not failed over (just restarted), the first item for that vBucket is a meta item describing the vBucket topology. This meta item may have a seqno within the client’s requested snapshot range. In that case, the server sends a SeqnoAdvanced DCP message to indicate that a seqno exists in the range, allowing the snapshot to be completed without sending the meta item itself. The problem occurs if, after SeqnoAdvanced is sent, a new write is processed for that stream. At that point, the system encounters the monotonicity break described in MB-62984.

Symptoms 

• Data Service (memcached) crashes with Linux status 134.

2024-08-05T14:39:01.443Z, ns_log:0:info:message(ns_1@node1) - Service 'memcached' exited with status 134. Restarting. Messages: #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fd09bd43000+0xb85b7] #11 /opt/couchbase/bin/memcached() [0x400000+0x146c8d] #12 /opt/couchbase/bin/memcached() [0x400000+0x4dfdfc] #13 /opt/couchbase/bin/memcached() [0x400000+0x4e262d] #14 /opt/couchbase/bin/memcached() [0x400000+0x4eb3d5] #15 /opt/couchbase/bin/memcached() [0x400000+0x4ebadc] #16 /opt/couchbase/bin/memcached() [0x400000+0x4ebc04] #17 /opt/couchbase/bin/memcached() [0x400000+0x4f2a89] #18 /opt/couchbase/bin/memcached() [0x400000+0x6c6819] #19 /opt/couchbase/bin/memcached() [0x400000+0x6bfefa] #20 /opt/couchbase/bin/memcached() [0x400000+0x6c7bce] #21 /opt/couchbase/bin/memcached() [0x400000+0x825d10] #22 /opt/couchbase/bin/memcached() [0x400000+0x8107aa] #23 /opt/couchbase/bin/memcached() [0x400000+0x828cc9] #24 /opt/couchbase/bin/memcached() [0x400000+0x6bfbf4] #25 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fd09bd43000+0xe4aa3] #26 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7fd09d8a1000+0x8609] #27 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fd09b9e6000+0x11f353]

Triggers

• A failover or unclean shutdown of the Data Service.

Verification

Look for signs that memcached crashed with status 134 in diag.log:

3665:2024-08-05T14:39:01.443Z, ns_log:0:info:message(ns_1@node1) - Service 'memcached' exited with status 134. Restarting. Messages:
3744:2024-08-05T14:54:40.581Z, ns_log:0:info:message(ns_1@node1) - Service 'memcached' exited with status 134. Restarting. Messages:
3942:2024-08-06T06:37:57.310Z, ns_log:0:info:message(ns_1@node2) - Service 'memcached' exited with status 134. Restarting. Messages:

Look for the following log message in memcached.log on the node experiencing the memcached crashes.

2024-08-06T06:34:38.110896+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Monotonic<m> (ActiveStream(eq_dcpq:"i":"76A3465A3502663D/FE65FC87ECC7186F","a":"kafka-connector/4.1.13 (whiskey) java-dcp-client/0.47.0 Java/1.8.0_422 (; OpenJDK 64-Bit Server VM; 25.422-b05) OS (Linux; 5.10.220-209.8" vb:855)::last
ReadSeqno) invariant failed: new value (51077) breaks invariant on current value (51077)

Workarounds

To work around this issue, the goal is to ensure that the client sends a snapshot range in its stream request that is valid for the vBucket. One can achieve this through the following steps:

1. Disable the connector client.
2. Increase the high sequence number (highSeqno) of all vBuckets in the affected bucket. The highSeqno is incremented by Couchbase every time a write operation occurs, such as adding or deleting a document or performing collection management tasks. This can be done in two ways:
   1. Option A (script-based): Run a script (e.g., using cbc-pillowfight) to increase the highSeqno for all vBuckets.
   2. Option B (collection-based):
      a. Create a temporary dummy collection.
      b. Drop the dummy collection.
      These two operations increment the highSeqno of all vBuckets by 2.
3. Re-enable the connector client.