Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-54274]: Couchbase should avoid immediate removal of temporary initial/deleted entries in HashTable

Jack
Updated 
Product: Couchbase Server
Component: data-service
Issue Link: MB-54274
Affects Version(s): 7.6.5
Fix Version(s): 8.0.0

Summary

  • When a GET request is made for a non-resident document, the request waits while the document is fetched from disk.
  • Once the disk fetch is successful, the document is loaded into memory as an item in the hash table, and the request completes using this in-memory data.
  • If the document does not exist on disk, a temporary item is created in the hash table. However, this temporary item is immediately deleted.
  • As a result, if multiple concurrent requests are received for the same document, each one triggers a separate disk fetch, which can cause significant performance degradation due to repeated and unnecessary disk operations.

Symptoms

  • Timeouts on GET operations for documents that don’t exist in a full-eviction bucket.

Triggers

  • The server must be using the bucket using full-eviction.
  • The server experiences high KV latency on GETs, the response to these GET requests is that the document is not found.

Verification

  • Slow operations will be seen in data service logs (memcached.log).
  • Pay special attention to slow operations that return Not found and show repeated bg.wait or bg.load calls in the trace. Normally, if this issue weren’t present, a document should require only a single disk read.

For example (21 disk fetches):

2024-09-16T10:32:39.566984+01:00 WARNING 1813: Slow operation: {"bucket":"Coupons","cid":"1CB8731800000001/000000003E0AB682/1e5990","command":"GET","duration":"3271 ms","packet":{"bodylen":56,"cas":0,"datatype":"raw","extlen":0,"key":"<ud>1a08b3d2-46ea-4698-848c-979b3fead03f</ud>","keylen":56,"magic":"ClientRequest","opaque":2421759488,"opcode":"GET","vbucket":771},"peer":{"ip":"10.122.89.211","port":43562},"response":"Not found","trace":"request=3874719048597542:3271356 bg.wait=3874719048603970:3254803 get=3874719048599876:7 execute=3874719048597542:10 bg.load=3874722303407651:80 bg.wait=3874722314442478:1177 get=3874722314441748:1 execute=3874722314441242:2 bg.load=3874722315620353:118 bg.wait=3874722315751906:1006 get=3874722315751177:1 execute=3874722315750982:2 bg.load=3874722316758240:79 bg.wait=3874722316856908:548 get=3874722316855986:2 execute=3874722316855716:3 bg.load=3874722317405459:34 bg.wait=3874722317997656:248 get=3874722317996909:2 execute=3874722317996662:13 bg.load=3874722318246524:85 bg.wait=3874722318476231:108 get=3874722318475796:17 execute=3874722318475707:18 bg.load=3874722318584387:38 bg.wait=3874722318708539:122 get=3874722318707676:44 execute=3874722318707238:46 bg.load=3874722318831130:17 bg.wait=3874722318889881:70 get=3874722318883829:26 execute=3874722318883460:26 bg.load=3874722318960855:10 bg.wait=3874722319036886:15 get=3874722319035445:2 execute=3874722319035080:3 bg.load=3874722319052447:23 bg.wait=3874722319109842:4 get=3874722319108817:3 execute=3874722319108707:3 bg.load=3874722319113978:18 bg.wait=3874722319140794:21 get=3874722319140207:1 execute=3874722319140104:1 bg.load=3874722319162777:6 bg.wait=3874722319181546:55 get=3874722319175472:23 execute=3874722319175373:25 bg.load=3874722319237015:22 bg.wait=3874722319275364:56 get=3874722319274070:2 execute=3874722319273646:3 bg.load=3874722319331926:21 bg.wait=3874722319388163:58 get=3874722319387330:10 execute=3874722319387075:10 bg.load=3874722319446731:14 bg.wait=3874722319498328:53 get=3874722319479422:20 execute=3874722319479017:21 bg.load=3874722319551690:9 bg.wait=3874722319587319:59 get=3874722319575580:44 execute=3874722319575240:45 bg.load=3874722319647122:22 bg.wait=3874722319679884:52 get=3874722319678755:43 execute=3874722319678425:51 bg.load=3874722319732166:10 bg.wait=3874722319757241:35 get=3874722319751603:28 execute=3874722319751385:29 bg.load=3874722319793045:4 bg.wait=3874722319809638:29 get=3874722319807502:10 execute=3874722319807194:11 bg.load=3874722319839512:7 bg.wait=3874722319865352:22 get=3874722319861683:10 execute=3874722319860904:11 bg.load=3874722319888110:3 bg.wait=3874722319904125:21 get=3874722319901907:8 execute=3874722319901632:10 bg.load=3874722319925956:5 get=3874722319944281:1 execute=3874722319943770:10","worker_tid":140501676164864}

Workarounds

  • Switch to a value-ejection bucket
  • Alternatively, adjust your application behavior to prevent concurrent requests for non‑existent documents. For instance, if the problem first appears with a single configuration document that multiple application instances try to access at the same time, one can resolve it by modifying applications so they avoid requesting the document until its existence is confirmed.

Related to

Related articles

Comments

0 comments

Article is closed for comments.