Product: Couchbase Server
Component: data-service
Issue Link: MB-62678
Affects Version(s): 6.6.x, 7.0.x, 7.1.x, 7.2.x, 7.6.x
Fix Version(s): 7.2.6, 7.6.4, 8.0.0Summary
- When an operation is performed on a document, Couchbase first checks whether the document has a TTL (time-to-live) set. If a TTL exists and has already expired, the document is expired as part of the process.
- However, when the operation is an explicit delete request (CMD_DEL) on a document with an expired TTL, the expiration code path can sometimes generate a corrupted tombstone.
- This corrupted tombstone, in turn, leads to a crash of the data service (memcached) process.
Symptoms
- memcached crashes with status 134
Triggers
- Corrupted tombstones occur when documents with expired TTLs are explicitly deleted.
- The crash is triggered when a corrupted tombstone is streamed to a DCP client.
Verification
Key error is Unexpected body (size 1166) in deletion found in babysitter.log:
[ns_server:info,2024-07-10T05:18:31.830Z,babysitter_of_ns_1@cb.local:<0.5601.25>:ns_port_server:log:226]memcached<0.5601.25>: terminate called after throwing an instance of '2024-07-10T05:18:31.630390+00:00 CRITICAL *** Fatal error encountered during exception handling ***
memcached<0.5601.25>: terminate called recursively
memcached<0.5601.25>: 2024-07-10T05:18:31.630395+00:00 CRITICAL *** Fatal error encountered during exception handling ***
memcached<0.5601.25>: 2024-07-10T05:18:31.630436+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Item::removeUserXattrs: Unexpected body (size 1166) in deletion: <ud>Item[0x7fa5458cfd80] with key:cid:0x0:3342337-0,bcProfile
memcached<0.5601.25>: value:Blob[0x7fa573845600] with size:1357 age:0 data: <00 00 00 bb 00 00 00 b7 m v D 00 { " A P P _ O I D " : " 3 3 4 2 3 3 7 - 0 - 6 1 7 - 3 6>
memcached<0.5601.25>: metadata:ItemMetaData[0x7fa5458cfd90] with cas:1720588711630143488 revSeqno:4345 flags:33554435 exptime:1720588711
memcached<0.5601.25>: bySeqno:129126 queuedTime:975320836 vb:129 op:mutation datatype:4 deleted:true(TTL)</ud>
memcached<0.5601.25>: std::logic_error'
memcached<0.5601.25>: what(): Item::removeUserXattrs: Unexpected body (size 1166) in deletion: <ud>Item[0x7fa5458cfd00] with key:cid:0x0:3342337-0,bcProfile
memcached<0.5601.25>: value:Blob[0x7fa573845600] with size:1357 age:0 data: <00 00 00 bb 00 00 00 b7 m v D 00 { " A P P _ O I D " : " 3 3 4 2 3 3 7 - 0 - 6 1 7 - 3 6>
memcached<0.5601.25>: metadata:ItemMetaData[0x7fa5458cfd10] with cas:1720588711630143488 revSeqno:4345 flags:33554435 exptime:1720588711
memcached<0.5601.25>: bySeqno:129126 queuedTime:975320836 vb:129 op:mutation datatype:4 deleted:true(TTL)</ud>Workarounds
The easiest workaround is actually upgrading the cluster to a version with the fix. Possible mitigations include:
-
Avoiding creation of corrupted tombstones:
- Ensure the TTL has not expired before deleting a document.
- Slightly easier:
- Issue a GET before DELETE. If the TTL has expired, KV will remove the document during the GET (a code path unaffected by this issue).
- Note: there remains a small window where the document may expire after the GET but before the DELETE, still triggering the bug.
- Avoid explicitly deleting documents with TTLs.
-
Removing corrupted tombstones:
- Lower the metadata purge interval to the minimum (0.04 days = 1 hour) and wait for one hour.
Comments
0 comments
Article is closed for comments.