Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-62270] Rebalances exit with reason no_quorum due to the Cluster manager granting lease to nodes that are not part of the cluster

Callum
Updated 
Product: Couchbase Server
Component: ns-server
Issue Link: MB-62270
Affects Version(s): 7.2.4
Fix Version(s): 8.0.0

Summary

  • The Cluster Manager (ns_server) may grant leases to nodes that are no longer in the cluster.

Symptoms 

  • Rebalances fail with reason no_quorum.

Triggers

  • Exact triggers remain unknown as this issue has been encountered only once, and logs were not available at the moment. Couchbase suspects this can happen if a node is failed over unsuccessfully, resulting in the node believing it is still a member of the cluster while it’s not.

Verification

  • There are Rebalance exited with reason events with no_quorum reason:
2024-06-07T09:07:30.053Z, ns_orchestrator:0:critical:message(ns_1@node-3.cluster.local) - Rebalance exited with reason {{badmatch,
                               {leader_activities_error,
                                {default,{ns_janitor,"prdoof",apply_config}},
                                {no_quorum,
                                 [{required_quorum,
                                   {all,
                                    {set,12,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],
                                       ['ns_1@node-14.cluster.local',
                                        'ns_1@node-13.cluster.local',
                                        'ns_1@node-12.cluster.local',
                                        'ns_1@node-11.cluster.local',
                                        'ns_1@node-10.cluster.local',
                                        'ns_1@node-9.cluster.local',
                                        'ns_1@node-8.cluster.local',
                                        'ns_1@node-7.cluster.local',
                                        'ns_1@node-6.cluster.local',
                                        'ns_1@node-5.cluster.local',
                                        'ns_1@node-4.cluster.local',
                                        'ns_1@node-3.cluster.local'],
                                       [],[],[],[],[],[],[],[],[],[],[]}}}}},
                                  {leases,
                                   ['ns_1@node-10.cluster.local',
                                    'ns_1@node-13.cluster.local',
                                    'ns_1@node-16.cluster.local',
                                    'ns_1@node-12.cluster.local',
                                    'ns_1@node-17.cluster.local',
                                    'ns_1@node-8.cluster.local',
                                    'ns_1@node-14.cluster.local',
                                    'ns_1@node-3.cluster.local',
                                    'ns_1@node-18.cluster.local',
                                    'ns_1@node-4.cluster.local',
                                    'ns_1@node-11.cluster.local',
                                    'ns_1@node-9.cluster.local']}]}}},
                              [{ns_janitor,cleanup_apply_config,4,
                                [{file,"src/ns_janitor.erl"},{line,211}]},
                               {ns_janitor,
                                '-cleanup_membase_bucket/4-fun-0-',4,
                                [{file,"src/ns_janitor.erl"},{line,61}]},
                               {async,'-async_init/4-fun-1-',3,
                                [{file,"src/async.erl"},{line,191}]}]}.
Rebalance Operation Id = 57fbcxse0b973218f8f3d3f70453f19c
  • There are signs that a node that is not in the cluster holds the lease for nodes within the cluster. In the below example, node node-2.cluster.local was not a member of the cluster:
[ns_server:warn,2024-06-07T08:46:37.738Z,ns_1@node-3.cluster.local:<0.11139.6062>:leader_lease_acquire_worker:handle_lease_already_acquired:226]Failed to acquire lease from 'ns_1@node-7.cluster.local' because its already taken by {'ns_1@node-2.cluster.local',
[ns_server:warn,2024-06-07T08:46:39.101Z,ns_1@node-3.cluster.local:<0.21653.6062>:leader_lease_acquire_worker:handle_lease_already_acquired:226]Failed to acquire lease from 'ns_1@node-5.cluster.local' because its already taken by {'ns_1@node-2.cluster.local',
[ns_server:warn,2024-06-07T08:46:39.372Z,ns_1@node-3.cluster.local:<0.21262.6062>:leader_lease_acquire_worker:handle_lease_already_acquired:226]Failed to acquire lease from 'ns_1@node-16.cluster.local' because its already taken by {'ns_1@node-2.cluster.local',
[ns_server:warn,2024-06-07T08:46:39.896Z,ns_1@node-3.cluster.local:<0.9889.6062>:leader_lease_acquire_worker:handle_lease_already_acquired:226]Failed to acquire lease from 'ns_1@node-15.cluster.local' because its already taken by {'ns_1@node-2.cluster.local',
[ns_server:warn,2024-06-07T08:46:51.746Z,ns_1@node-3.cluster.local:<0.11139.6062>:leader_lease_acquire_worker:handle_lease_already_acquired:226]Failed to acquire lease from 'ns_1@node-7.cluster.local' because its already taken by {'ns_1@node-2.cluster.local',
[ns_server:warn,2024-06-07T08:46:53.108Z,ns_1@node-3.cluster.local:<0.21653.6062>:leader_lease_acquire_worker:handle_lease_already_acquired:226]Failed to acquire lease from 'ns_1@node-5.cluster.local' because its already taken by {'ns_1@node-2.cluster.local',
[ns_server:warn,2024-06-07T08:46:53.379Z,ns_1@node-3.cluster.local:<0.21262.6062>:leader_lease_acquire_worker:handle_lease_already_acquired:226]Failed to acquire lease from 'ns_1@node-16.cluster.local' because its already taken by {'ns_1@node-2.cluster.local',
[ns_server:warn,2024-06-07T08:46:53.906Z,ns_1@node-3.cluster.local:<0.9889.6062>:leader_lease_acquire_worker:handle_lease_already_acquired:226]Failed to acquire lease from 'ns_1@node-15.cluster.local' because its already taken by {'ns_1@node-2.cluster.local',

Look for signs that we are receiving heartbeats from the node that is not in the cluster which holds leases:

[ns_server:warn,2024-06-07T07:09:45.166Z,ns_1@node-5.cluster.local:mb_master<0.17948.4>:mb_master:candidate:359]Candidate got master heartbeat from node 'ns_1@node-2.cluster.local' which is not in peers

Workarounds

  • Shutdown the node that is not in the cluster which is holding the lease for nodes in the cluster (node-2.cluster.local) in the above example.

Related to

Related articles

Comments

0 comments

Article is closed for comments.