Brief Description on Sync Gateway Sequences

In Couchbase Sync Gateway, sequences are unique identifiers assigned to each change in the database. They help track changes and ensure that clients receive updates in the correct order. These sequences play a crucial role in the synchronization process between Couchbase Lite clients and Couchbase Server. 

Types of Sequences

1. Simple Sequences: Simple sequences in Couchbase Sync Gateway are fundamental to tracking changes and ensuring efficient synchronization. Here’s a deeper look at how they work:
   • Incremental Numbers: Simple sequences are assigned as incremental numbers to each change in the database. For example, if the last sequence was 100, the next change would be assigned sequence 101. This straightforward numbering system makes it easy to track the order of changes.
   • Change Identification: Each document change, whether it's a creation, update, or deletion, gets a unique sequence number. This helps in identifying and managing each change individually.
   • Synchronization Efficiency: When a client connects to Sync Gateway, it uses the sequence number to determine which changes it has already received and which new changes it needs to sync. This ensures that clients only receive the necessary updates, reducing data transfer and improving synchronization efficiency.
   • Data Integrity: By using simple sequences, Sync Gateway ensures that all changes are accounted for and applied in the correct order. This maintains data integrity across different clients and the server.
   • Ease of Use: The simplicity of this system makes it easy to implement and manage, especially in applications where tracking the order of changes is crucial.

2. Compound Sequences: Compound sequences in Couchbase Sync Gateway are used to optimize replication and ensure efficient data synchronization. Here’s a detailed look at how they work:
   • Format: Compound sequences appear in a format like 57348::57803. This format indicates a range of sequences, where 57348 is the starting sequence and 57803 is the ending sequence.
   • Purpose: The primary purpose of compound sequences is to handle situations where the next non-compound sequence value hasn’t been received by the Sync Gateway node within a given time window. This helps avoid delays in replication by sending a compound sequence instead.
   • Optimization: By using compound sequences, Sync Gateway can continue processing changes without waiting for the next sequence number to be fully processed. This is particularly useful during spikes in load, ensuring that replication is not blocked.
   • Handling Compound Sequences: When a client receives a compound sequence, it processes all changes within the specified range. This ensures that no changes are missed and that the client stays up-to-date with the latest data.
   • Returning to Simple Sequences: Once the load returns to normal levels and the delayed mutations are processed, Sync Gateway will stop sending compound sequences and revert to simple, incremental sequences.

3. Checkpoint Sequences: These sequences are used to mark specific points in the changes feed. They help clients resume synchronization from the last known good state, ensuring no data is missed. When a client reconnects to Sync Gateway, it uses the checkpoint sequence to determine where to resume synchronization. This ensures that the client only requests and processes changes that occurred after the last checkpoint. Here’s a practical example of how Checkpoints work:
   • Initial State: A client starts synchronization and processes changes up to sequence 1000. It updates its checkpoint sequence to 1000.
   • Disconnection: The client disconnects from Sync Gateway.
   • Reconnection: Upon reconnecting, the client uses its checkpoint sequence (1000) to request changes starting from sequence 1001.
   • Continuous Updates: As the client processes new changes, it continues to update its checkpoint sequence, ensuring it can always resume from the most recent point.
4. High-Water Mark Sequences: High-water marks represent the highest sequence number that has been fully processed and acknowledged by Sync Gateway. They act as a global checkpoint, ensuring that all changes up to this sequence number have been synchronized. Unlike checkpoints, high-water marks are not client-specific; they apply to the entire Sync Gateway instance, providing a global view of the synchronization state. High water marks help maintain data integrity by ensuring that all changes up to this point are accounted for and processed. This is crucial for conflict resolution and ensuring consistent data across all clients. Additionally, high-water marks are useful for monitoring the overall synchronization status and performance of Sync Gateway, providing insights into how up-to-date the system is with processing changes.

Client Communication

Here’s a detailed look at how client communication using sequences works:

1. Changes Feed: Sync Gateway uses changes feed to track and communicate changes in the database. This feed is a sorted list of changes made to documents, identified by their sequence numbers. The changes feed API endpoint (/{db}/_changes) allows clients to request a list of changes from a specific sequence number.
2. Synchronization Methods: There are several methods to connect to the changes feed:
   • Polling: Returns the list of changes immediately. Clients need to send new requests to get subsequent changes.
   • Longpolling: Keeps the request open until a new change occurs, then returns the change and closes the connection.
   • Continuous: Keeps the connection open and sends changes as they occur, providing real-time updates.
   • WebSockets: Similar to continuous mode but uses WebSockets to avoid issues with proxy servers and gateways.
3. Synchronization: When a client connects to Sync Gateway, it uses the sequence number to determine which changes it has already received and which new changes it needs to sync. This ensures that clients only receive the necessary updates, reducing data transfer and improving efficiency.

Configuring Sequences

You can adjust certain parameters in the Sync Gateway configuration to fine-tune how sequences are handled. For example:

• rev_cache_size: This parameter controls the size of the revision cache. A larger revision cache can help improve performance by keeping more document revisions in memory.
• max_wait_skipped: Maximum wait time in milliseconds for a skipped sequence before abandoning the sequence.
• max_num_pending: Maximum number of pending sequences before skipping the sequence.
• max_wait_pending: Maximum wait time in milliseconds for a pending sequence before skipping sequences.

Compound Sequences

Compound sequences are automatically managed by Sync Gateway, but understanding their configuration can help optimize your setup. By default, Sync Gateway will use compound sequences when necessary to avoid replication delays. This typically happens during high-load scenarios where the next sequence number isn't processed in time.

Here’s an example of how compound sequences might be used in practice:

Imagine your application experiences a sudden spike in document updates. This could be due to a batch process, a surge in user activity, or an external data import. Normally, each change would get a simple, incremental sequence number. However, during this high-load period, Sync Gateway might not be able to process each sequence in real-time. To avoid delays, instead of assigning the next simple sequence number, Sync Gateway generates a compound sequence.

When a client receives a compound sequence like 57348::57803, it understands that it needs to process all changes within this range. The client will request and apply these changes, ensuring it stays up-to-date. As the client applies these changes, it updates its local database. This ensures that the client is synchronized with the server, even during high-load periods.

Once the load normalizes and the delayed sequences are processed, Sync Gateway will revert to using simple, incremental sequences. This transition happens seamlessly, ensuring continuous and efficient synchronization.

Here’s a step-by-step example:

1. The last processed sequence is 57347.
2. A batch process updates 500 documents.
3. Sync Gateway assigns the compound sequence 57348::57803 to these changes.
4. A client requests changes since sequence 57347.
5. The client receives the compound sequence 57348::57803.
6. The client requests and processes all changes within this range.
7. After the high load, Sync Gateway assigns the next simple sequence 57804 to the following change.

Pending Sequences

In Couchbase Sync Gateway, pending sequences refer to sequence numbers that have been assigned to document updates but have not yet been processed and added to the cache (channel/revision).

Example Scenario:

• A document update occurs, and Sync Gateway assigns it a sequence number.
• Due to processing delays or other operations, this sequence number is marked as pending.
• Once the processing is complete, the sequence is moved from the pending state to the channel cache.

Handling Pending Sequences:

• Monitoring: Regularly monitor the Sync Gateway logs to track pending sequences and ensure they are processed in a timely manner.
• Configuration: Adjust Sync Gateway configuration settings, such as `max_wait_pending` and max_num_pending, to optimize the handling of pending sequences.

Skipped Sequences

In Couchbase Sync Gateway, a "skipped sequence" refers to a situation where a sequence number is not processed immediately and is temporarily skipped. This can happen for several reasons, such as when a document update or channel grant has not yet been fully processed and added to the channel cache.

Key Points:

1. Channel Grants: When a document grants access to a new channel, the sequence number associated with this grant might be skipped if the document has not yet been fully processed.
2. Backfill Process: The skipped sequence will be processed later during the backfill process, ensuring that no data is missed

Example Scenario:

• A document update grants access to a new channel.
• The sequence number for this update is temporarily skipped because the document has not been fully processed.
• Once the document is processed and added to the channel cache, the skipped sequence is included in the backfill, ensuring the client receives all updates.

Handling Skipped Sequences:

• Continuous Replication: Using continuous replication can help handle skipped sequences more effectively, as Sync Gateway will wait and include the skipped sequences once they are processed.
• Monitoring and Logging: Keeping an eye on logs and monitoring skipped sequences can help identify and troubleshoot issues related to sequence processing

Unused Sequences

In Couchbase Sync Gateway, "unused sequences" refer to sequence numbers that were allocated but never used. This can happen for several reasons, such as when a document update fails, due to a conflict or an error. The operation would have been aborted before the sequence number was utilized.

For example, if Sync Gateway allocates sequence number 63 for a document update, but another update for the same document ID occurs simultaneously and succeeds, sequence 63 might never be used1.

When Sync Gateway detects these unused sequences in the changes feed, it logs them as "unused sequences." This is generally harmless and expected under normal operation1.

Example Scenario:

• Sync Gateway allocates sequence number 100 for a document update.
• The update fails due to a conflict or another issue.
• Sequence number 100 is logged as an unused sequence because it was never applied to a document.

Handling Unused Sequences:

• Monitoring: Regularly monitor logs to understand the frequency and context of unused sequences.
• Optimizations: Sync Gateway includes optimizations to handle unused sequences efficiently, minimizing their impact on performance

Benefits of Sequences

1. Efficient Change Tracking: Sequences provide a reliable way to track changes in the database. Each change gets a unique sequence number, making it easy to identify and manage updates.
2. Optimized Synchronization: Sequences help optimize the synchronization process by allowing clients to request only the changes they haven't yet received. This reduces the amount of data transferred and speeds up the sync process.
3. Conflict Resolution: Conflict resolution in Couchbase Sync Gateway involves using sequences to manage and resolve conflicts that arise when multiple clients update the same document simultaneously. Here’s a detailed look at how it works:
   • Tracking Changes: Each document change is assigned a unique sequence number. When a conflict occurs, the sequence numbers help determine the order of changes. This is crucial for deciding which version of the document should be kept.
   • Automatic Conflict Resolution: Sync Gateway can automatically resolve conflicts based on predefined policies, such as `localWins` or `remoteWins`. These policies use sequence numbers to decide which version of the document should be kept.
   • Custom Conflict Resolution: For more complex scenarios, you can implement custom conflict resolution logic using JavaScript functions. These functions can compare the conflicting document versions and decide which one to keep or merge into a new document.
   • Conflict-Free Mode: In some configurations, Sync Gateway can operate in a conflict-free mode, where it rejects conflicting updates outright, ensuring that only one version of the document is accepted.
4. Scalability: By using sequences, Sync Gateway can efficiently handle large volumes of data changes, making it suitable for large-scale applications.
5. Data Integrity: Sequences ensure that all changes are accounted for, maintaining the integrity of the data across different clients and the server.