Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-59310] Data Service crashes due to handling empty snapshots from checkpoints

Callum
Updated 
Product: Couchbase Server
Component: data-service
Issue Link: MB-59310
Affects Version(s): 7.2.5, 7.6.3
Fix Version(s): 7.2.6, 7.6.4

Summary

  • If a client streams from a replica vBucket, this may trigger logic that causes the Data Service(memcached) to crash. In versions containing the fix for this issue, the Data Service handles this scenario more gracefully.

Symptoms 

  • Data Service(memcached) crashes on the cluster.
  • The crash results in the OS producing a full core dump of the memcached memory image.
    • Note that the core dumps can result in high disk usage. Core dumps can be disabled; see our documentation for further instructions.

Triggers

  • A client wrongly streams from a replica vBucket. E.g.:
    • Java DCP clients (all connectors) may stream from replica vBuckets (fixed per JDCP-245).
    • FTS Service may stream from replica vBuckets per MB-57443.
    • Analytics Service may stream from replica vBuckets per MB-59282.
    • XDCR may stream from replica vBuckets per MB-62962.

Verification

  • Log messages similar to the ones below should be present in memcached.log:
2024-06-11T06:41:34.554533+00:00 CRITICAL *** Fatal error encountered during exception handling ***
2024-06-11T06:41:34.554569+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveStream::getOutstandingItems: stream:eq_dcpq:"i":"BB90D155899DD765/8065AD0999BE5350","a":"kafka-connector/4.1.11 (productorder-0141266b368ca2d3fe0678a6a2c9eae9) java-dcp-client/0.44.0 Java/17.0.9 (Company Inc.; OpenJDK 64-Bit Server VM; 17" vb:524 processing checkpoint type:Disk, CheckpointHistorical::No, ranges:0, HCS:15931649, MVS:15931841, items:0
  • Breakpad caught a crash and Stack backtrace of crashed thread messages are present in memcached.log as well:
2024-06-11T06:41:43.080055+00:00 INFO 2530: Client {"ip":"100.81.32.30","port":42521} authenticate2024-06-11T07:10:44.083161+00:00 INFO ---------- Opening logfile: 
2024-06-11T07:10:44.084702+00:00 INFO Couchbase version 7.2.5-7596 starting.
2024-06-11T07:10:44.084725+00:00 INFO Process identifier: 136223
2024-06-11T07:10:44.084740+00:00 INFO recalculate_max_connections: {"engine_fds":982531,"max_connections":65000,"max_fds":1048576,"system_connections":5000}
2024-06-11T07:10:44.084794+00:00 INFO Breakpad enabled. Minidumps will be written to '/opt/couchbase/var/lib/couchbase/crash'
2024-06-11T07:10:44.084810+00:00 CRITICAL Detected previous crash
2024-06-11T07:10:44.084867+00:00 CRITICAL Breakpad caught a crash (Couchbase version 7.2.5-7596). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/67ff031c-b7dc-4a68-99709691-cf847998.dmp before terminating.
2024-06-11T07:10:44.084874+00:00 CRITICAL Stack backtrace of crashed thread:
2024-06-11T07:10:44.084874+00:00 CRITICAL    #0  /opt/couchbase/bin/memcached() [0x400000+0x64e02c]
2024-06-11T07:10:44.084874+00:00 CRITICAL    #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x330) [0x400000+0x685ee0]
2024-06-11T07:10:44.084875+00:00 CRITICAL    #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x686258]
2024-06-11T07:10:44.084875+00:00 CRITICAL    #3  linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff96153000+0x850]
2024-06-11T07:10:44.084875+00:00 CRITICAL    #4  /lib/aarch64-linux-gnu/libc.so.6(gsignal+0xe0) [0xffff95246000+0x33d78]
2024-06-11T07:10:44.084876+00:00 CRITICAL    #5  /lib/aarch64-linux-gnu/libc.so.6(abort+0x114) [0xffff95246000+0x20aac]
2024-06-11T07:10:44.084876+00:00 CRITICAL    #6  /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x188) [0xffff95495000+0xa0368]
2024-06-11T07:10:44.084877+00:00 CRITICAL    #7  /opt/couchbase/bin/memcached() [0x400000+0x65907c]
2024-06-11T07:10:44.084878+00:00 CRITICAL    #8  /opt/couchbase/bin/../lib/libstdc++.so.6() [0xffff95495000+0x9deac]
2024-06-11T07:10:44.084878+00:00 CRITICAL    #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0xffff95495000+0x9df10]
2024-06-11T07:10:44.084878+00:00 CRITICAL    #10 /opt/couchbase/bin/../lib/libstdc++.so.6(__cxa_rethrow+0) [0xffff95495000+0x9e200]
2024-06-11T07:10:44.084878+00:00 CRITICAL    #11 /opt/couchbase/bin/memcached() [0x400000+0x410f58]
2024-06-11T07:10:44.084879+00:00 CRITICAL    #12 /opt/couchbase/bin/memcached() [0x400000+0x41aaa0]
2024-06-11T07:10:44.084879+00:00 CRITICAL    #13 /opt/couchbase/bin/memcached() [0x400000+0x41ac08]
2024-06-11T07:10:44.084880+00:00 CRITICAL    #14 /opt/couchbase/bin/memcached() [0x400000+0x4213f0]
2024-06-11T07:10:44.084880+00:00 CRITICAL    #15 /opt/couchbase/bin/memcached() [0x400000+0x5c2a2c]
2024-06-11T07:10:44.084880+00:00 CRITICAL    #16 /opt/couchbase/bin/memcached() [0x400000+0x5bc968]
2024-06-11T07:10:44.084881+00:00 CRITICAL    #17 /opt/couchbase/bin/memcached() [0x400000+0x5c4534]
2024-06-11T07:10:44.084881+00:00 CRITICAL    #18 /opt/couchbase/bin/memcached() [0x400000+0x6f0508]
2024-06-11T07:10:44.084882+00:00 CRITICAL    #19 /opt/couchbase/bin/memcached() [0x400000+0x6dcbc8]
2024-06-11T07:10:44.084882+00:00 CRITICAL    #20 /opt/couchbase/bin/memcached() [0x400000+0x6f376c]
2024-06-11T07:10:44.084883+00:00 CRITICAL    #21 /opt/couchbase/bin/memcached() [0x400000+0x5bc638]
2024-06-11T07:10:44.084883+00:00 CRITICAL    #22 /opt/couchbase/bin/../lib/libstdc++.so.6() [0xffff95495000+0xc8bcc]
2024-06-11T07:10:44.084884+00:00 CRITICAL    #23 /lib/aarch64-linux-gnu/libpthread.so.0() [0xffff95f73000+0x7624]
2024-06-11T07:10:44.084884+00:00 CRITICAL    #24 /lib/aarch64-linux-gnu/libc.so.6() [0xffff95246000+0xd162c]
  • There are signs that vBuckets were transitioned from active to dead and from dead to replica:
2024-06-11T04:27:51.160232+00:00 INFO (bucket-1) VBucket::setState: transitioning vb:524 with high seqno:1849074 from:active to:dead
2024-06-11T04:27:51.242926+00:00 INFO (bucket-1) VBucket::setState: transitioning vb:524 with high seqno:1849074 from:dead to:replica
  • There should be a stream opened on the replica vBuckets:
2024-06-11T04:27:51.294083+00:00 INFO 6724: (bucket-1) DCP (Producer) eq_dcpq:"i":"9C230E5A273B6203/3790392D9C85C121","a":"elasticsearch-connector/4.4.6 (pod-2) java-dcp-client/0.46.0 Java/17.0.7 (Red Hat, Inc.; OpenJDK 64-Bit Server VM; 17.0.7+7-LTS) OS (Linux" - (vb:524) Creating stream with start seqno 1848424 and end seqno 18446744073709551615; requested end seqno was 18446744073709551615, snapshot:{1848424,1848424} collections-filter-size:1 sid:none

Workarounds

  • If you are impacted by the core dumps using too much disk space issue, the core dumps can be disabled as per our documentation.
  • There are no workarounds for the crashes within the Data Service(memcached).
  • Stopping the clients that are trying to read from the replica vBucket should prevent the issue.
  • To fix this issue permanently, either the Couchbase Server version should be upgraded to receive the fix for https://jira.issues.couchbase.com/browse/MB-59310 or clients should be upgraded to receive the fix for https://jira.issues.couchbase.com/browse/JDCP-245.

Related to

Related articles

Comments

0 comments

Article is closed for comments.