Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-64742] Delayed cleanup of recoveryLSS rpDataSize can lead to excessive disk usage

Callum
Updated 
Product: Couchbase Server
Component: index-service
Issue Link: MB-64742
Affects Version(s): 7.6.4
Fix Version(s): 7.2.7, 7.6.5, 8.0.0

Summary

  • An issue in the Plasma tracking statistics may incorrectly flag a stale recovery point in the recovery log as valid data. This misidentification causes the log cleaning process to run inefficiently, especially when the mutation rate is low.
  • As a result, the system is unable to effectively trim recovery point history.
  • Over time, this leads to a steady increase in disk usage.

Symptoms 

  • High disk usage on Index Service nodes (seen as a high value for the index_storage_lss_used_space(i.e. total number of bytes used by both data logs and recovery logs for the Index Service) statistic).
  • Disk usage is steadily increasing.

Triggers

  • A low mutation rate for indexes can trigger this issue, whereas at higher mutation rates, the increased fragmentation would typically activate the log cleaner and prevent the problem.

Verification

  • Look for steadily increasing disk usage / index_storage_lss_used_space, like in the following example:
ab205f0f-710a-47f6-92c6-67d00daab504.png

Workarounds

Use Plasma diag endpoints to trigger full logCleaning to temporarily reduce disk usage. Here are the steps:

  1. The following command would list all the Plasma instances along with their shardIDs present on the node:
curl -X GET -u <user>:<pass> "<indexer-addr>:9102/plasmaDiag" -d '{"Cmd":"listDBs"}'

Example output:

❯ curl -X GET -u Administrator:password "http://localhost:9102/plasmaDiag" -d '{"Cmd":"listDBs"}'
[0] travel-sample/def_primary/Mainstore#15466661961929288580:0  : 3299771691918597412:1
[1] travel-sample/def_icao/Mainstore#14607350067812831004:0  : 3299771691918597412:2
[2] travel-sample/def_city/Mainstore#16046806169494368370:0  : 3299771691918597412:3
[3] travel-sample/def_airportname/Mainstore#14148240344047422803:0  : 3299771691918597412:4
[4] travel-sample/def_type/Mainstore#17198698659954991886:0  : 3299771691918597412:5
[5] travel-sample/def_sourceairport/Mainstore#2764780246406907504:0  : 3299771691918597412:6
[6] travel-sample/def_schedule_utc/Mainstore#6577907884151080138:0  : 3299771691918597412:7
[7] travel-sample/def_route_src_dst_day/Mainstore#6806829004844818141:0  : 3299771691918597412:8
[8] travel-sample/def_inventory_airport_airportname/Mainstore#11333350465285345311:0  : 3299771691918597412:9
[9] travel-sample/def_inventory_landmark_city/Mainstore#793537448349568191:0  : 3299771691918597412:10
[10] travel-sample/def_inventory_hotel_city/Mainstore#5398324642594322639:0  : 3299771691918597412:11
[11] travel-sample/def_inventory_airport_city/Mainstore#12411821806137586766:0  : 3299771691918597412:12
[12] travel-sample/def_inventory_airport_faa/Mainstore#14123015082660928984:0  : 3299771691918597412:13
[13] travel-sample/def_inventory_route_route_src_dst_day/Mainstore#13159972829940158727:0  : 3299771691918597412:14
[14] travel-sample/def_inventory_route_schedule_utc/Mainstore#16895676477168007048:0  : 3299771691918597412:15
[15] travel-sample/def_inventory_route_sourceairport/Mainstore#3996756501928482573:0  : 3299771691918597412:16
  1. 3299771691918597412:1, 3299771691918597412:2, 3299771691918597412:3 in the above output are the shardID:instanceID pairs. Note these for the next step.
  2. The following command runs "Cmd":"logClean" 
curl -X GET -u <user>:<pass> "<indexer-addr>:9102/plasmaDiag" -d '{"Cmd":"logClean", "Args":["<<shardId:instanceId>>"]}'

Example:

For shared LSS:

❯ curl -X GET -u Administrator:password "http://localhost:9102/plasmaDiag" -d '{"Cmd":"logClean", "Args":["18379518974882921195:12"]}'
Start cleaning dataLog[/Users/testUser/ws/morpheus/ns_server/data/n_0/data/@2i/shards/shard18379518974882921195/data] status[frag 0, data: 2193169, used: 1770253, disk: 2881701, relocated: 15, rewrites: 547, retries: 6, skipped: 84, cleaned:3433638, log:(9063667 - 10833920), disk:(7952219-10833920), activeFragRatio:30, activeMaxFragRatio:80, run:2 duration:166 ms elapsed: 2108 ms paused:false]
Done cleaning dataLog[/Users/testUser/ws/morpheus/ns_server/data/n_0/data/@2i/shards/shard18379518974882921195/data] status[frag 4, data: 2193169, used: 2306018, disk: 3003041, relocated: 15, rewrites: 868, retries: 20, skipped: 162, cleaned:4502802, log:(10132831 - 13135872), disk:(10132831-13135872), activeFragRatio:30, activeMaxFragRatio:80, run:3 duration:249 ms elapsed: 3090 ms paused:false]
Start cleaning recoveryLog[/Users/testUser/ws/morpheus/ns_server/data/n_0/data/@2i/shards/shard18379518974882921195/data/recovery] status[frag 0, data: 191862, used: 174246, disk: 548864, relocated: 0, retries: 0, skipped: 817, cleaned:0, log:(0 - 548864), disk:(0-548864) aheadOff: 374618, activeFragRatio:30, activeMaxFragRatio:80, run:2 duration:0 ms elapsed: 2022 ms paused: false]
Done cleaning recoveryLog[/Users/testUser/ws/morpheus/ns_server/data/n_0/data/@2i/shards/shard18379518974882921195/data/recovery] status[frag 15, data: 142710, used: 169511, disk: 663552, relocated: 0, retries: 0, skipped: 1251, cleaned:0, log:(0 - 663552), disk:(0-663552) aheadOff: 494041, activeFragRatio:30, activeMaxFragRatio:80, run:3 duration:0 ms elapsed: 3004 ms paused: false]

For dedicated LSS:

❯ curl -X GET -u Administrator:password "http://localhost:9102/plasmaDiag" -d '{"Cmd":"logClean", "Args":["3299771691918597412:7"]}'
Start cleaning dataLog[/Users/testUser/ws/morpheus/ns_server/data/n_0/data/@2i/travel-sample_def_schedule_utc_6577907884151080138_0.index/mainIndex] status[]
Done cleaning dataLog[/Users/testUser/ws/morpheus/ns_server/data/n_0/data/@2i/travel-sample_def_schedule_utc_6577907884151080138_0.index/mainIndex] status[frag 0, data: 1425919, used: 1219817, disk: 2481696, relocated: 60, rewrites: 0, retries: 0, skipped: 72, cleaned:3019543, log:(3019543 - 4239360), disk:(1757664-4239360), activeFragRatio:30, activeMaxFragRatio:80, run:1 duration:49 ms elapsed: 49 ms paused:false]
Start cleaning recoveryLog[/Users/testUser/ws/morpheus/ns_server/data/n_0/data/@2i/travel-sample_def_schedule_utc_6577907884151080138_0.index/mainIndex/recovery] status[]
Done cleaning recoveryLog[/Users/testUser/ws/morpheus/ns_server/data/n_0/data/@2i/travel-sample_def_schedule_utc_6577907884151080138_0.index/mainIndex/recovery] status[frag 30, data: 53082, used: 75879, disk: 225280, relocated: 72, retries: 0, skipped: 73, cleaned:0, log:(0 - 225280), disk:(0-225280) aheadOff: 0, activeFragRatio:30, activeMaxFragRatio:80, run:1 duration:9 ms elapsed: 9 ms paused: false]

Plasma organizes indexes on a node using the following approach:

  • Indexes on the _default scope and collection: Each index instance has its own dedicated LSS (Log-Structured Storage).
  • Indexes on named scopes and collections: Multiple index instances share a single LSS.

Every Plasma instance is associated with a single shard. Within each shard, there can be dedicated instances (each with a dedicated LSS) and shared instances (all using one shared LSS). The Plasma log cleaner operates at the LSS level, so for shared LSS, step (3) does not need to be executed for each instance individually.

To manage disk usage, start by cleaning the largest directories first.

Note: This issue may recur. Manual log cleaning using the diagnostic endpoint will need to be performed periodically to keep disk usage under control until an upgrade with a permanent fix is available.

  • Verify the workaround was successful

Verify disk usage has decreased: There should be a significant reduction in space consumed by RP blocks in recoveryLSS after the workaround is carried out.

Before the workaround:

18972/logs/cbcollect_info_ns_1@node1_20250102-153730
❯ grep '"recovery_recoverypt_size"' couchbase.log | awk -F: '{sum += $2} END {print sum}'
1025105920000 ===> 954GB

18972/logs/cbcollect_info_ns_1@node2_20250102-153731
❯ grep '"recovery_recoverypt_size"' couchbase.log | awk -F: '{sum += $2} END {print sum}'
1007962079232 ===> 938GB

After the workaround:

18972/logsAfterWorkaround/cbcollect_info_ns_1@node1_20250107-182114
❯ grep '"recovery_recoverypt_size"' couchbase.log | awk -F: '{sum += $2} END {print sum}'
10287058944 ===> 9.58GB


18972/logsAfterWorkaround/cbcollect_info_ns_1@node2_20250107-182114
❯ grep '"recovery_recoverypt_size"' couchbase.log | awk -F: '{sum += $2} END {print sum}'
25684402176 ==> 23.92GB

Related to

Related articles

Comments

0 comments

Article is closed for comments.