Understanding Automatic Failover

As we can read in the documentation article about Automatic Failover:

Automatic Failover — or auto-failover — can be configured to fail over one or more nodes automatically. No immediate administrator intervention is required. Specifically, the Cluster Manager autonomously detects and verifies that the nodes are unresponsive, and then initiates the hard failover process. Auto-failover does not fix or identify problems that may have occurred. Once appropriate fixes have been applied to the cluster by the administrator, a rebalance is required. Auto-failover is always hard failover.

A lot of high-level details are provided in that article, and we strongly encourage you to read it.

In this KB article, we try to review some in-depth technical details to understand how exactly Cluster Manager decides if an Automatic Failover should be triggered.

The Couchbase Cluster Manager runs on all the nodes of a cluster, maintaining essential per-node processes, and coordinating cluster-wide operations.

The main Cluster-Manager process is ns-server. It consists of a few components. One of these components is Master Services. Master Services manage operations with cluster-wide impact; such as failover, rebalance, and adding and deleting buckets. Master Services is often referred to as Orchestrator.

The auto-failover logic is run by the orchestrator node, a node that runs Orchestrator. For every tick (1 second), each node sends its stats to the Orchestrator for processing. Every 5 seconds each node sends out a ping to every other node - a heartbeat. When a node records that the last heartbeat happened over 7 seconds ago (or if the last heartbeat wasn't healthy), it will report this in its stat dump. This stat dump is sent every tick (1 second) to the orchestrator node. We refer to this as a missed heartbeat.

The ns_server considers a normally functioning node to have the state "up" or "new". If a missed heartbeat is recorded ns_server (the auto-failover logic of it) starts the following steps:

• The node is transitioned to the state "half_down" and ns_server sets its half-down counter to 0. Half-down means that Couchbase Server noticed a node is down/unhealthy, but is not yet sure if it's been down for long enough to trigger an auto-failover.
• If ns_server sees a healthy heartbeat, the node state is changed from half-down to up.
• However, if ns_server does not see a healthy heartbeat, then each subsequent time it sees a missed heartbeat, the half down counter is incremented by 1.
• Once the counter reaches the specified auto failover interval, the node is transitioned to the "nearly_down" state. Nearly down means that the node has been down for long enough to trigger an Automatic Failover, but ns_server still needs to confirm it is safe to perform it.
• The ns_server ensures that there are no state changes in other nodes for 2 ticks - i.e. no other nodes enter a nearly down state for this interval. Once this has elapsed, ns_server transitions the node to the "failover" state. However, this does not necessarily mean Couchbase Server fails over the node.
• The ns_server now performs the safety checks needed before a failover. First, it checks if the node is running only one Couchbase service, or if services are co-located.
• If the node is running just one service, we will follow the auto-failover policy of that service.
• If the node has co-located services, we check if Data Service is one of those services.
• If it is, ns_server will follow the auto-failover policy of Data Service, regardless of the other service's policies, to avoid or minimize data loss.
• If not, ns_server loops through every service and checks if auto-failover is possible for that service. If a service has it disabled, this will be logged and the node will not be failed over.
• Once ns_server has confirmed the services allow failover, it makes sure the cluster has not hit the maximum number of auto-failovers.
• The ns_server now fails over the node (or not, and Couchbase Server logs why. This can often be found in the diag.log in a set of collected Couchbase Cluster logs).