How to interprete Couchbase Lite Logs

Basic Format

Every log level (debug, info, verbose, warning, error) has its separate log file. The first line of the log will identify the Couchbase Lite platform and version:

---- CouchbaseLite/2.8.4 (ObjC; iPadOS 17.2; iPad) Build/18 Commit/5ea5cd71+CHANGES LiteCore/2.8.4 (18) ----

• There's a distinction between Couchbase Sync Gateway and the Couchbase Lite logs files.
  • Sync Gateway has several files (debug, info, warning, error, etc.) but each one is combined with the higher log levels (i.e. the error log file contains error level logs, the warning log file contains error as well as warning level logs, and so on).
  • Couchbase Lite has only one log level per file (i.e. the error log file contains only error level logs, the warning log file contains only warning level logs and no error level logs).

Each line starts with a timestamp of the log, tagged with a "domain" keyword:

10:39:53.864354| [DB]: {DB#2} Opening database

Some Log Events/Domains you may see in the logs:

* This is a prefix on the log message, not a domain. The message will also have a domain specified in it.

It's always best to first ensure that the logs cover the time frame of the reported issue.

Also, watch the timestamps closely: be suspicious of large (>10ms) time gaps.

Sync (replication)

Connections

When the issue is about the continuous replicator being stopped, pay attention to the error message. If the error message is "the connection reset by peer" or similar, it means that the server which could be Sync Gateway or load balancer/proxy, closed the connection and the corresponding Sync Gateway log needs to be obtained and looked at.

Log Format

When the platform identifies a replicator, the string will begin with hieroglyphics like:

<*>, <!>, <o>, etc.

Document not being pushed/pulled

When the issue is that a change to a document is not being pulled or updated locally, the important thing to note is the document ID and the time when the document was supposed to be pulled. First, check the Couchbase Lite logs to look for an error that happened to the document in that period. If there is no log message indicating that the change is pulled, please check the following:

• If there is nothing mentioned about the document being pulled in the Couchbase Lite log, check Sync Gateway logs to see
  • If there is any error regarding that document in the logs.
  • If the user has access to the particular document.
• If the Couchbase Lite log indicates that the document was pulled (check if there is a revision of the document saved in the local database), try to see if the saved document conflicted with the current revision in the database. Further, check the logs to see if the conflict resolution is being done for the document or not. Please note that the document might be resolved as the local wins as well.

Tracing a Replicator

To follow a particular replicator, look for this line at the beginning, such as:

11-30 14:11:01.588 17920 17920 I CouchbaseLite/REPLICATOR: {C4Replicator#6} Starting Replicator {Repl#10} with config: {Coll#0} "_default": {"Push": disabled, "Pull": continuous

where the #X that is seen throughout the examples identifies a unique thread number specific to a task:

• C4Replicator#X
  
  • C4Replicator is the interface between the Platform and LiteCore.
• Repl#X
  • The replicator that LiteCore generates to perform the replication job. You may filter by "Repl#X" to monitor its progress. Filtering by {C4Replicator#X} tells the progress at a higher level.

After the above line, if you find the following in the logs, this means:

11-30 14:16:34.286 17920 18218 I CouchbaseLite/REPLICATOR: {C4Replicator#6} Starting Replicator {Repl#252} with config: {Coll#0} "_default": {"Push": disabled, "Pull": continuous

• After 5 minutes 33 seconds, the same C4Replicator#6 created a new worker replicator (Repl#252). This new replicator (Repl#252) is generated in the LiteCore. Usually this likely happens due to some transient error. For example, network interruption. One needs to know how to further check if there's something in the logs while encountering this. Also, if this could be the cause of the issue one's troubleshooting
• Please note that the lifecycles of "Repl#X" associated with the same C4Replicator do not overlap. Thus, each "Repl#X" is independent of each other even when associated with the same "C4Replicator". 

Tracing Collections

The "Coll#X" tag can used to find out which collection is being logged during the Replication. One can search for the "Options" log to find out which Collection corresponds to which index.

Example:

12:56:29.945588| [Sync]: {Repl#107} {Coll#-1} {{Coll#0} "flowers.roses": {"Push": one-shot, "Pull": disabled},
{Coll#1} "flowers.tulips": {"Push": disabled, "Pull": one-shot},
{Coll#2} "flowers.lavenders": {"Push": one-shot, "Pull": one-shot}}
Options={auth:{password:"********", type:"Basic", username:"sguser"}, pinnedCert:...

This log indicates that Coll#0 is "flowers.roses", Coll#1 is "flowers.tulips" and Coll#2 is "flowers.lavenders". All logs that follow during this replication will follow these indexes. There will be one of the logs for every new Replication.

Platform details can be picked up from the Replicator’s Option’s User Agent property, at the start of replication. These usually include the device, platform, platform version, Core version, and commit details. The structure might vary, depending on the platform, but should be very similar

Java/Android Log Lines

If there are two numbers after the timestamp. For Example:

2023-12-04 13:25:29.484 25123 25123

The first number represents the thread ID and the second represents the process ID. If there is only one number then that is the thread id. The process ID (pid) essentially is the name of the running application. If the log lines include a pid, you can eliminate all of the log lines that were not generated by the app by grep-ing for the app pid.