Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[CCBC-1672] Keys with null (\u0000) prefix after online upgrade to Couchbase Server version 7.x with C SDK 3.3.x (<3.3.18)

Yuvraj 
Product: C SDK
Component: libcouchbase
Issue Link: CCBC-1672
Affects Version: 3.3.x
Fix Version: 3.3.18

Summary

Impact

  • The C SDK's pipeline could retain memory chunks in its send queue following network issues and operation retries.
  • When the SDK reconnects and attempts to resend queued operations, these leftover detached memory segments sometimes result in malformed packets, protocol errors in the KV engine, and, in rare cases, the creation of documents with unintended keys.
  • Applications may have noticed that a document stored successfully could not be retrieved immediately after the operation.

Technical Insights

  • Each SDK pipeline uses a network buffer manager that allocates and manages memory for request operations.
  • During retries (after timeouts or network disconnects), the SDK must reclaim all memory and flush all stale packets. The issue happens when part of a request (such as a large UPSERT) is sent before the interruption. The header could be flushed, but a chunk containing the body might linger in the pending queue after reconnect.
  • This scenario affects operations where segmented packet handling is frequent (large bodies, failovers, socket reconnections).
  • In extremely rare cases, a stale header left in the queue would cause the server to treat bytes from a subsequent request as a document key, resulting in the document being created and a CAS returned, but the original key may remain inaccessible for GET operations.

Verification

Approach 1

Please run the following command on each data service node. If you observe any document ID that starts with \u0000 in the output, it is most likely that the cluster is impacted. We recommend sharing this output and raising a Support ticket. 

 Also, please ensure that you update the <data-dir-path> path before executing the following commands.

 The default <data-dir-path> is: “/opt/couchbase/var/lib/couchbase/data”

for bkt in `sudo ls <data-dir-path>`;
do
for f in `sudo ls <data-dir-path>/$bkt | grep -i "couch"`;
do
   sudo /opt/couchbase/bin/couch_dbdump --json --no-body <data-dir-path>/$bkt/$f |grep -i '"id":"\\u0000\\u0000' ;
done
done

Approach 2

If you have an Index and Query service node part of the cluster, then creating a primary index and querying all the meta().id for the pattern will also help in identifying the issue.

Please repeat this process for all the buckets in the cluster and also drop the newly created primary index from the cluster.

Primary index:

Create primary index on `<bucket-name>`

N1QL Query:

Select meta().id from `<bucket-name>` where meta().id like `\u0000%`

 If you see results, we strongly recommend you open a Support ticket. Please refer to this article when opening the ticket and send us the output file containing the results from executing the commands above. 

Support will promptly engage with you and work on the next steps to remediate the issue.

Related Cases

(All fixed in version 3.3.18)

Fix

  • For fixing the impacted documents on the server-side, please reach out to the Couchbase Support Team as indicated in the verification part above.
  • To prevent the issue from happening again, all users of the C SDK (libcouchbase) prior to version 3.3.18 should upgrade immediately.

Resolution

  • As of version 3.3.18, libcouchbase now prunes all stale buffers and detached memory chunks from the pipeline during cleanup and operation wipe.
  • Only valid packets are allowed to leave the pipeline during a reconnect, eliminating these protocol violations.

Related articles

Comments

0 comments

Article is closed for comments.