Certificate related issue when adding a node to the Couchbase cluster

The content of this article applies to Couchbase Server 6.6.X and 7.X.

Issue Summary:

The issue involves the TLS handshake issue when adding a new node to the Couchbase cluster. At the end of the article, you will understand the steps involved in resolving the TLS handshake issue and some of the useful commands that help with the investigation of this issue.

Indications of the issue:

When adding a new node to the Couchbase cluster, you will likely encounter errors like the following:

…Failed to establish TLS connection to (x)  "unknown ca"

Certificates in Couchbase:

Self-Signed Certificates

Couchbase supports the use of self-signed certificates for securing communication channels. 

Cluster Protection with Root and Node Certificates documentation shows how to create a root certificate that is a trusted, self-signed authority; and how to use this to sign individual, per node certificates.

Custom/Third-Party Certificates

In enterprise environments, it's common for organizations to use custom or third-party certificates signed by the internal Certificate Authority. Couchbase seamlessly integrates with these certificates to establish secure connections. 

Externally Provided Root Certificate documentation covers the procedure to use the third-party certificate.

Resolution steps:

In Couchbase Server v6.x, before adding a new node:

• Make sure that the node certificate (contained in a file named "chain.pem") is copied to the “/opt/couchbase/var/lib/couchbase/inbox” directory.
• If intermediate certificates are being used, make sure you verify that the content of "chain.pem" contains the node certificate first, followed by the intermediate certificate(s). Note that the root CA certificate should NOT be added to "chain.pem". “Openssl” command can be used to verify this step and some of the useful “openssl” commands are at the end of this article.
• Execute the below commands to upload the cluster root CA and reload the node certificate (in that order) i.e. steps 11 & 12 in Couchbase Configure Server Certificates documentation. 

Note: the current cluster root CA can be seen in Couchbase Server Web UI: Security -> Root            Certificate.

• Once the above steps are performed, the new node can be added to the Couchbase cluster

In Couchbase Server v7.x and above, before adding a new node:

• Make sure that the node certificate (contained in a file named "chain.pem") is copied to the “/opt/couchbase/var/lib/couchbase/inbox” directory.
• Make sure that the root certificate (e.g. "ca.pem") is copied to the “/opt/couchbase/var/lib/couchbase/inbox/CA” directory. 

Note: make sure that the CA directory only contains certificate file(s).

• If intermediate certificates are being used, make sure you verify that the content of "chain.pem" contains the node certificate first, followed by the intermediate certificate(s).
• Execute the below commands to upload the cluster root CA and reload the node certificate (in that order) i.e. I.e steps 13 & 14 in Couchbase Configure Server Certificates documentation. 

• Once the above steps are performed, the new node can be added to the Couchbase cluster.

Some useful commands for further troubleshooting (if the issue still persists):

1] To verify the node certificate:

Check in the output that the node certificate contains a SAN extension that matches the node name as known by the cluster, for example:

2] To verify that the chain of trust is correct:

Using the above openssl command, check that the node certificate has been issued by the root CA, or signed by an intermediate if they are being used.

The chain of trust should look like this:

3] Another useful command to run, from a node that is part of the cluster:

From the output of the command, things that you can check:

• Look at the Subjects and Issuers fields, and check that the chain of trust is correct. 

If the issue persists, creating a support ticket and including the Couchbase Server logs and the above command outputs is advisable for a more efficient resolution.