[CBG-4487] deadlock in revision cache with Get and an on-demand import

Product: Couchbase Sync Gateway
Component: mobile
Issue Link: CBG-4487
Affects Version(s): 3.2.1
Fix Version(s): 3.2.2, 3.3.0 

Summary

Sync Gateway may enter a deadlocked state if a specific sequence of events occurs:

1. A document is updated directly in the bucket on the Server (i.e. not through Sync Gateway) when it has not yet been imported by the import feed.
2. Revision cache is at capacity.
3. A client attempts to replicate the previous revision of the document, at the same time that revision is being evicted from the revision cache.

The fix for this issue can be tracked by https://jira.issues.couchbase.com/browse/CBG-4487

Symptoms 

• Slower Document Processing: The pull of documents is significantly slower than expected and some documents are not pulled to Couchbase Lite. Push of revisions from Couchbase Lite to Sync Gateway is not expected to be impacted.
• Replication Pauses: During replication, the replicator pauses midway. For example, if 400 total document changes are expected, replication pauses after syncing 23 changes, the replication state changes to BUSY and a WebSocket closed error is logged.
• Frequent Restarts: After the pause, replication restarts from 0 and takes 18-19 minutes to complete syncing all documents.

Triggers

A deadlock may occur in a shard of Sync Gateway’s revision cache in the following scenario.

• A document is updated directly in the bucket (i.e., bypassing Sync Gateway).
• The update has not yet been imported by the import feed.
• A client tries to replicate the previous revision of the document while that revision is being evicted from the revision cache as cache is at capacity.

The issue is triggered because:

• On-demand import tries to write the newly imported document to the cache at the same time as the cache eviction. This results in a race condition.
• The race condition can trigger a mutex deadlock, making that shard of the cache unavailable.

Verification

• Check whether import count (parsed_sgw_collection_import_count) rate is not increasing on one node in the cluster, but continuing to increase at a non-zero rate on other nodes, similar to the following example graph:

  

• Check whether the number of replications (parsed_sgw_database_num_replications_total) is increasing but number of document reads (parsed_sgw_collection_num_docs_reads) remains flat for any Sync Gateway node in the cluster:

  

  

Workarounds

No workaround is available. Reach out to Couchbase Technical Support Team for options if upgrade is not feasible.