Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-64353] DCP client will always rollback to 0 for ephemeral bucket on hard-failover (despite the newly promoted active having all the mutations)

Callum
Updated 
Product: Couchbase Server
Component: data-service
Issue Link: MB-64353
Affects Version(s): 7.2.x, 7.6.x
Fix Version(s): 7.2.7, 7.6.5, 8.0.0

Summary

When a replica vBucket of an Ephemeral bucket is promoted to active after a hard-failover, this will always result in a rollback for clients to seqno 0.

This can result in index rollbacks, which can cause temporary missing query results.

Symptoms 

  • DCP streams rollback to 0 (re-stream all data).
    • Indexes rollback to 0 (rebuild index from scratch).
      • Temporary empty query results until the indexes are rebuilt.
      • Temporary missing query results until the indexes are rebuilt.
      • Index scans will fail while indexes are in the process of rolling back to 0.

Triggers

  • The ephemeral bucket is in use.
  • The ephemeral bucket has at least 1 replica.
  • A hard failover is performed on a Data Service node (replica vBucket promoted to active)
    • Causes DCP clients to rollback to 0.

Verification

  • Verify whether an Ephemeral bucket is in use.
  • Verify whether a Hard Failover was performed (check diag.log).
  • Look for the messages that contain Stream request requires rollback to seqno:0 because consumer ahead of producer, which indicate a rollback in memcached.log for the relevant bucket:
2024-11-14T06:34:52.287933+00:00 WARNING 282: (bucket_name) DCP (Producer) eq_dcpq:secidx:proj-MAINT_STREAM_TOPIC_ceafe4d4b868580d5511d45755f06d05-18219150613156370419/1 - (vb:490) Stream request requires rollback to seqno:0 because consumer ahead of producer - producer upper at 0. Client requested seqnos:{11389,18446744073709551615} snapshot:{11389,11389} uuid:90228356588047
2024-11-14T06:34:52.289226+00:00 WARNING 283: (bucket_name) DCP (Producer) eq_dcpq:secidx:proj-MAINT_STREAM_TOPIC_ceafe4d4b868580d5511d45755f06d05-18219150613156370419/2 - (vb:492) Stream request requires rollback to seqno:0 because consumer ahead of producer - producer upper at 0. Client requested seqnos:{10375,18446744073709551615} snapshot:{10375,10375} uuid:242205542029499
  • Check whether any index scans failed due to a rollback, like in the following log message in ns_server.query.log.
2024-11-14T06:34:54.391+00:00 [Error] [GsiScanClient:"node_name"] Scan3(72845722-36bc-4f60-b2c0-461d23cf3df3) response failed `Indexer rollback or warmup`
2024-11-14T06:34:54.391+00:00 [Warn] scan failed: requestId 72845722-36bc-4f60-b2c0-461d23cf3df3 queryport node_name inst 5138920749074781508 partition [0]
2024-11-14T06:34:54.391+00:00 [Warn] Scan failed with error for index 11655747392400790324.  Trying scan again with replica, reqId:72845722-36bc-4f60-b2c0-461d23cf3df3 :  Indexer rollback or warmup from [node_name]
  • Check whether there are any log messages indicating a rollback is occurring in ns_server.indexer.log at the relevant time:
2024-11-14T06:36:29.240+00:00 [Info] Rollback time has changed for index inst 13171324947371793852. New rollback time 1731566093984686499
2024-11-14T06:36:29.283+00:00 [Info] Rollback time has changed for index inst 5138920749074781508. New rollback time 1731566093924582046

Workarounds

The only way of preventing this is to upgrade to a fixed version.

Related to

Related articles

Comments

0 comments

Article is closed for comments.