[CBG-4478] Database is offline due to heartbeat durability impossible failure

Product: Couchbase Sync Gateway
Component: mobile
Issue Link: CBG-4478
Affects Version(s): 3.0.x, 3.1.x, 3.2.0, 3.2.1
Fix Version(s): 3.2.2, 3.3.0 

Summary

A heartbeat send durability impossible failure during Sync Gateway startup can leave it stuck in a starting/initializing state.

Symptoms 

• Sync Gateway is stuck in offline state while starting up.
• Capella App Services is stuck in initializing state.

Triggers

• A durability impossible failure during sending a heartbeat while Sync Gateway is starting up.

Verification

• Run the GET /_all_dbs API call:

curl -X GET http://Administrator:password@localhost:4985/_all_dbs\?verbose\=true                                                                                                          ✔  23:39:20 
[{"db_name":"db1","bucket":"gamesim-sample","state":"Offline","init_in_progress":true}

• Check whether the response contains "state":"Offline","init_in_progress":true indefinitely upon several retries.
• Look for the following error which indicates a durability impossible failure on sending heartbeat:

[ERR] Couldn't apply config for database "perf2db2-endpoint": couldn't reload database: durability impossible | {"status_code":161,"document_id":"_sync:heartbeat_timeout:126a53edd7fc4b29"

Workarounds

Restart Sync Gateway.