Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-65347] Set maxIteration param for planner during CommandRepair

Callum
Updated 
Product: Couchbase Server
Component: index-service
Issue Link: MB-65347
Affects Version(s): 7.2.4
Fix Version(s): 7.2.8, 7.6.6, 8.0.0

Summary

  • Due to maxIteration (an internal setting within the Index Service) not being set correctly, the initial placement of indexes on nodes is used as the final placement. The initial placement does not check for equivalent Index violations, meaning the Index Service may produce violations when determining which nodes to place indexes on.

Symptoms 

  • ALTER INDEX command to change the number of index replicas fails with "Violations for index":
ALTER INDEX customer_part_fix ON entitlements WITH {"action": "replica_count", "num_replica": 1};

[
  {
    "code": 5000,
    "msg": "GSI AlterIndex() - cause: Fail to alter index: \nMemoryQuota: 53359935488\nCpuQuota: 4\n--- Violations for index <customer_part_fix 18 (replica 1), PROD, _default, entitlements> (mem 42.9899M, cpu 0) at node node1:18091 \n\tCan move to node2:18091: NoViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCannot move to node3:18091: ReplicaViolation (free mem 24.5373G, free cpu 2.9827005766474457)\n\tCannot move to node6:18091: EquivIndexViolation (free mem 25.5293G, free cpu 2.91603613212893)\n\tCan move to node5:18091: NoViolation (free mem 26.8097G, free cpu 2.831999999999999)\n\tCan move to node4:18091: NoViolation (free mem 27.5502G, free cpu 2.943666666666666)\n--- Violations for index <customer_part_fix 20 (replica 1), PROD, _default, entitlements> (mem 27.392M, cpu 0) at node node1:18091 \n\tCan move to node2:18091: NoViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCannot move to node3:18091: EquivIndexViolation (free mem 24.5373G, free cpu 2.9827005766474457)\n\tCan move to node6:18091: NoViolation (free mem 25.5293G, free cpu 2.91603613212893)\n\tCannot move to node5:18091: ServerGroupViolation (free mem 26.8097G, free cpu 2.831999999999999)\n\tCannot move to node4:18091: ReplicaViolation (free mem 27.5502G, free cpu 2.943666666666666)\n--- Violations for index <customer_part_fix 13 (replica 1), PROD, _default, entitlements> (mem 28.0159M, cpu 0) at node node1:18091 \n\tCan move to node2:18091: NoViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCannot move to node3:18091: EquivIndexViolation (free mem 24.5373G, free cpu 2.9827005766474457)\n\tCan move to node6:18091: NoViolation (free mem 25.5293G, free cpu 2.91603613212893)\n\tCannot move to node5:18091: ServerGroupViolation (free mem 26.8097G, free cpu 2.831999999999999)\n\tCannot move to node4:18091: ReplicaViolation (free mem 27.5502G, free cpu 2.943666666666666)\n--- Violations for index <customer_part_fix 5 (replica 1), PROD, _default, entitlements> (mem 27.8522M, cpu 0) at node node1:18091 \n\tCan move to node2:18091: NoViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCan move to node3:18091: NoViolation (free mem 24.5373G, free cpu 2.9827005766474457)\n\tCannot move to node6:18091: EquivIndexViolation (free mem 25.5293G, free cpu 2.91603613212893)\n\tCannot move to node5:18091: ServerGroupViolation (free mem 26.8097G, free cpu 2.831999999999999)\n\tCannot move to node4:18091: ReplicaViolation (free mem 27.5502G, free cpu 2.943666666666666)\n--- Violations for index <customer_part_fix 10 (replica 1), PROD, _default, entitlements> (mem 26.5822M, cpu 0) at node node3:18091 \n\tCan move to node1:18091: NoViolation (free mem 23.5661G, free cpu 3.047031765607814)\n\tCannot move to node2:18091: EquivIndexViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCan move to node6:18091: NoViolation (free mem 25.5293G, free cpu 2.91603613212893)\n\tCannot move to node5:18091: ReplicaViolation (free mem 26.8097G, free cpu 2.831999999999999)\n\tCannot move to node4:18091: ServerGroupViolation (free mem 27.5502G, free cpu 2.943666666666666)\n--- Violations for index <customer_part_fix 12 (replica 1), PROD, _default, entitlements> (mem 17.9903M, cpu 0) at node node6:18091 \n\tCannot move to node1:18091: EquivIndexViolation (free mem 23.5661G, free cpu 3.047031765607814)\n\tCannot move to node2:18091: ReplicaViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCan move to node3:18091: NoViolation (free mem 24.5373G, free cpu 2.9827005766474457)\n\tCan move to node5:18091: NoViolation (free mem 26.8097G, free cpu 2.831999999999999)\n\tCan move to node4:18091: NoViolation (free mem 27.5502G, free cpu 2.943666666666666)\n--- Violations for index <customer_part_fix 3 (replica 1), PROD, _default, entitlements> (mem 28.0148M, cpu 0) at node node6:18091 \n\tCannot move to node1:18091: EquivIndexViolation (free mem 23.5661G, free cpu 3.047031765607814)\n\tCan move to node2:18091: NoViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCan move to node3:18091: NoViolation (free mem 24.5373G, free cpu 2.9827005766474457)\n\tCannot move to node5:18091: ReplicaViolation (free mem 26.8097G, free cpu 2.831999999999999)\n\tCannot move to node4:18091: ServerGroupViolation (free mem 27.5502G, free cpu 2.943666666666666)\n--- Violations for index <customer_part_fix 2 (replica 1), PROD, _default, entitlements> (mem 17.9248M, cpu 0) at node node4:18091 \n\tCannot move to node1:18091: EquivIndexViolation (free mem 23.5661G, free cpu 3.047031765607814)\n\tCannot move to node2:18091: ReplicaViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCan move to node3:18091: NoViolation (free mem 24.5373G, free cpu 2.9827005766474457)\n\tCan move to node6:18091: NoViolation (free mem 25.5293G, free cpu 2.91603613212893)\n\tCan move to node5:18091: NoViolation (free mem 26.8097G, free cpu 2.831999999999999)\n--- Violations for index <customer_part_fix 7 (replica 1), PROD, _default, entitlements> (mem 42.6914M, cpu 0) at node node4:18091 \n\tCan move to node1:18091: NoViolation (free mem 23.5661G, free cpu 3.047031765607814)\n\tCannot move to node2:18091: EquivIndexViolation (free mem 33.0953G, free cpu 3.107636921230708)\n\tCannot move to node3:18091: ReplicaViolation (free mem 24.5373G, free cpu 2.9827005766474457)\n\tCannot move to node6:18091: ServerGroupViolation (free mem 25.5293G, free cpu 2.91603613212893)\n\tCan move to node5:18091: NoViolation (free mem 26.8097G, free cpu 2.831999999999999)\n"
  }
]

Triggers

  • Running an ALTER INDEX command to change the number of index replicas.
  • Having a large number of partitions increases the chance of hitting the issue since partitions are treated as independent instances, which can encounter a higher number of planning violations.
  • The issue is triggered if the first planning iteration (to determine placement of indexes) has a violation, meaning another iteration is needed, but due to maxIteration a second iteration can never occur.

Verification

  • Check that the ALTER INDEX command fails with "Violations for index".
  • Look for the following log message, which indicates an index planner iteration was stopped because the limit (which is incorrectly set to 0) is exceeded:
2025-02-13T00:13:19.092+00:00 [Info] Planner::stop planner iter per temp as maxIterPerTemp limit (0, 0) exceeded.

Workarounds

  • Re-create the index with the desired number of replicas specified in the initial index statement (eliminating the need to use ALTER INDEX).

Related to

Related articles

Comments

0 comments

Article is closed for comments.