Analytics dataset (with no filters) has fewer documents than Couchbase bucket

The Problem:

Couchbase buckets can accept any valid JSON i.e. JSON objects, JSON array, and scalar values (string, boolean, numbers, and null). However, the Analytics service only accepts JSON objects. Binary documents are also ignored because you cannot query binary data.

For example, this document is ignored by Analytics since it is a JSON array:

[1,2,3]

The following document will be part of the dataset since it is a JSON object:

{
  "numbers": [1,2,3]
}

Therefore, if the bucket has documents that are not JSON objects, the Analytics dataset will have fewer documents than the bucket (assuming no filters are configured on the dataset). 

Versions Impacted:

It impacts all versions of Couchbase until this open feature request gets implemented to address this limitation.

Workaround:

Unfortunately, the only viable workaround is for the application to write JSON objects.  It might be useful to identify documents that are not JSON objects.  See below for steps to identify these documents.

How do you identify documents that aren’t JSON objects?

If you have Query/Index nodes create the following index, update "<bucket-name>" with actual bucket name:

CREATE INDEX idx_non_objects ON <bucket-name>(TYPE(SELF))
WHERE TYPE(SELF) <> 'object';

and run this query:

SELECT DISTINCT RAW META(d).id
FROM <bucket-name> d
WHERE TYPE(d) <> 'object';

If you don’t have Query/Index nodes you can use views. Here is the map function:

function (doc, meta) {
  if(doc && doc.constructor !== Object) {
  emit(meta.id, null);
  }
}

Note that the above query and map function will include keys for binary documents.  

Useful links:

https://www.json.org/json-en.html

https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/typefun.html

https://docs.couchbase.com/server/current/learn/views/views-writing.html