Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[CBG-3947] Improved Handling for Config Polling Timeout Errors and intermittent network connectivity problems

Yuvraj
Updated 
Product: Sync Gateway
Component: sync-gateway
Issue Link: CBG-3947
Affects Versions: 3.1.x, 3.2.x
Fix Versions: 3.1.12, 3.2.1, 3.3.0

Summary

  • In affected versions of Couchbase Sync Gateway:
    • When a timeout error occurs during configuration polling (especially after the initial bootstrap startup), Sync Gateway would shut down the database.
    • During intermittent network connectivity problems, this can also lead to unnecessary transitions between offline and online states.
  • The https://jira.issues.couchbase.com/browse/CBG-3947 makes config polling more resilient:
    • Instead of shutting down the database immediately, Sync Gateway now skips certain config polling rounds if a bucket returns a timeout error.
    • Before retrying polling, Sync Gateway checks for corruptions and repairs the configurations where needed.
    • Note: This change doesn’t impact the config polling during the initial bootstrap startup.
  • This change prevents cascading failures and improves reliability, especially in environments prone to temporary network or server instability

Symptoms 

  • The database abruptly shuts down after experiencing a timeout.
  • Replication issues or database operation failures persist after a timeout without automatically resolving.
  • Frequent Sync Gateway service restarts occur on affected nodes, and they are accompanied by timeouts/panics in Sync Gateway.

Triggers

  • Timeout errors while fetching configuration from a Couchbase bucket, especially outside the initial startup window.
  • Temporary server outages or Key-Value (KV) service interruptions that delay config retrieval.

Verification

To verify if you are affected:

  • Check Sync Gateway logs for "unambiguous timeout" errors during config polling (not during initial startup).
2024-05-14T15:28:01.406Z [DBG] Config+: unable to fetch config in group "default" from bucket "bucket": unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"Get","Opaque":"673","TimeObserved":10000273097,"RetryReasons":null,"RetryAttempts":0,}
2024-05-14T15:28:01.406Z [DBG] Config+: "test" did not contain config in group "default"
2024-05-14T15:28:01.406Z [INF] Config: Error fetching config for database "bucket" to check whether we need to remove it: Not Found
2024-05-14T15:28:01.406Z [INF] Config: Database "bucket" was running on this node, but config was not found on the server - removing database
2024-05-14T15:28:01.406Z [INF] Closing db /bucket (bucket "bucket")
  • After this, check for repeated transitions to offline mode and corresponding attempts to bring the database back online.

Offline example Logs:

2024-05-14T15:28:01.406Z [DBG] Config+: unable to fetch config in group "default" from bucket "bucket": unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"Get","Opaque":"673","TimeObserved":10000273097,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"hostname:11210","LastDispatchedFrom":"127.0.0.1:33132","LastConnectionID":"3f1dfd77b5cb9630/a56678c09b70e478"}
2024-05-14T15:28:01.406Z [DBG] Config+: "testmagma" did not contain config in group "default"
2024-05-14T15:28:01.406Z [INF] Config: Error fetching config for database "bucket" to check whether we need to remove it: Not Found
2024-05-14T15:28:01.406Z [INF] Config: Database "bucket" was running on this node, but config was not found on the server - removing database
2024-05-14T15:28:01.406Z [INF] Closing db /bucket (bucket "bucket")

Online example Logs:

2024-05-14T15:28:08.651Z [DBG] Config+: Got config for group "default" from bucket "bucket" with cas 1695911698168348672
2024-05-14T15:28:08.653Z [INF] Opening db /bucket as bucket "bucket", pool "default", server <couchbase://hostname>
  • Or there may be frequent Sync Gateway restarts in systemd or similar logs, coinciding with timeouts/panics:
Jul 31 05:39:11 nodename systemd[1]: Stopped Couchbase Sync Gateway server.
Jul 31 05:42:33 nodename systemd[1]: Stopped Couchbase Sync Gateway server.
Jul 31 06:14:24 nodename systemd[1]: Stopped Couchbase Sync Gateway server.
Jul 31 06:18:20 nodename systemd[1]: Stopped Couchbase Sync Gateway server.
Jul 31 06:19:48 nodename systemd[1]: Stopped Couchbase Sync Gateway server.
Jul 31 07:06:00 nodename systemd[1]: Stopped Couchbase Sync Gateway server.
Jul 31 07:12:42 nodename systemd[1]: Stopped Couchbase Sync Gateway server.
rg -i "panic: close of closed channel" *.log  
7298:Jul 31 05:39:11 nodename bash[2140]: panic: close of closed channel
7362:Jul 31 05:42:32 nodename bash[2173]: panic: close of closed channel
7485:Jul 31 06:14:24 nodename bash[2199]: panic: close of closed channel
7555:Jul 31 06:18:20 nodename bash[2484]: panic: close of closed channel
7597:Jul 31 06:19:47 nodename bash[2514]: panic: close of closed channel
7737:Jul 31 07:05:59 nodename bash[2536]: panic: close of closed channel

Workarounds

  • Restarting Sync Gateway provides a temporary fix until the next timeout occurs.
  • Remediate the underlying cause of the timeout (network, server, or KV issues) to prevent recurrence.
  • Upgrade to a version containing the fix for automatic and permanent resolution from the Sync Gateway side.

Related to

Related articles

Comments

0 comments

Article is closed for comments.