Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-64976] A missing index partition leads to rebalance failures when create command token is handled for a partitioned index with defer_build set to true

Callum
Updated 
Product: Couchbase Server
Component: index-service
Issue Link: MB-64824
Affects Version(s): 7.0.x, 7.1.x, 7.2.x, 7.6.x
Fix Version(s): 8.0.0

Summary

  • If an index partition is lost, the Index Service will attempt to rebuild it during the next rebalance. If the index definition contains defer_build: true, this can result in a scenario where some partitions are built(active) while the lost partitions are recreated with defer_build: true. If a create command token is present for the lost partition that the Index Service is attempting to rebuild in a deferred state while other partitions are built, the Index Service will crash.

Symptoms 

  • Index Service crashes with a panic similar to below during the rebalance:
2025-01-21T23:09:01.846-06:00 [Warn] TransferToken6:ce:ed:6a:c:3:2a:ca Detected Invalid State Change Notification. Token Id Rebalancer::checkValidNotifyStateDest: Local State TransferTokenMerge Metakv State TransferTokenInProgress
panic: Source Index Instance 9998761821617493302 and target index instance 2401552260533238979 does not have the same state (INDEX_STATE_CREATED != INDEX_STATE_ACTIVE)

goroutine 106074033510 [running]:
github.com/couchbase/indexing/secondary/common.CrashOnError(...)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:509
github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).destTokenToMergeOrReadyLOCKED.func1({_, _}, {{0xc028408240, 0x20}, {0x0, 0x0}, {0xc028408260, 0x20}, {0xc028408280, 0x20}, ...})
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:2789 +0x49a
created by github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).destTokenToMergeOrReadyLOCKED in goroutine 106074025209
  • Rebalance failures.

Triggers

  • An index partition is lost for the index with defer_build: true.
  • Index Service attempts to rebuild the lost partition while other partitions are in a built state.
  • A create command token is present for the lost partition that the Index Service is attempting to rebuild in a deferred state while other partitions are built.

Verification

  • If a rebalance failure is experienced, look for signs that this is due to an issue with the Index Service (the reason is: service_rebalance_failed,index):
2025-01-21T23:09:02.811-06:00, ns_orchestrator:0:critical:message(ns_1@node1.cluster.local) - Rebalance exited with reason {service_rebalance_failed,index,
                              {agent_died,<24703.2256.0>,
                               {lost_connection,
                                {'ns_1@node2.cluster.local',
                                 shutdown}}}}.
Rebalance Operation Id = 418b26decf3bea7145e5aa8c2521043
  • An indexer crash occurs, producing the following logs in diag.log:
2025-01-21T23:09:02.579-06:00, ns_log:0:info:message(ns_1@node-2.cluster.local) - Service 'index' exited with status 2. Restarting. Messages:
2025-01-21T23:09:01.846-06:00 [Warn] TransferToken6:ce:ed:6a:c:3:2a:af Detected Invalid State Change Notification. Token Id Rebalancer::checkValidNotifyStateDest: Local State TransferTokenMerge Metakv State TransferTokenInProgress
panic: Source Index Instance 9998761821617493302 and target index instance 2401552260533238979 does not have the same state (INDEX_STATE_CREATED != INDEX_STATE_ACTIVE)

goroutine 106074033510 [running]:
github.com/couchbase/indexing/secondary/common.CrashOnError(...)
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:509
github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).destTokenToMergeOrReadyLOCKED.func1({_, _}, {{0xc028408240, 0x20}, {0x0, 0x0}, {0xc028408260, 0x20}, {0xc028408280, 0x20}, ...})
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:2789 +0x49a
created by github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).destTokenToMergeOrReadyLOCKED in goroutine 106074025209
	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:2759 +0x2b3
  • Search for panic messages in the ns_server.indexer.log to find the instances where the indexer crashed with a panic.
  • Look for signs that the indexer failed to find create token numbers in ns_server.indexer.log (search for Failed to process create index). In the example below, the token number is 9537139879024260146:
2025-01-11T16:55:56.244-06:00 [Info] DDLServiceMgr: processing create index token /indexing/ddl/commandToken/create/9537139879024260146-5572695136392897788/5
2025-01-11T16:55:56.246-06:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CREATE_INDEX_DEFER_BUILD elapsed 66.809µs len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error Duplicate Index Instance 8761026812170097965.
2025-01-11T16:55:56.246-06:00 [Warn] DDLServiceMgr: Failed to process create index (bucket1, _default, _default, index1, 9537139879024260146, 8761026812170097965).  Error = Duplicate Index Instance 8761026812170097965..
  • Then check that there are create tokens present in metakv by searching for the following section in couchbase.log:
Indexer Metadata Tokens: 
curl -X GET -sS -k --proxy  -K- http://127.0.0.1:9102/listMetadataTokens
==============================================================================
/indexing/ddl/commandToken/create/13388360197742923453-9623504811457175456
/indexing/ddl/commandToken/create/9537139879024260146-5572695136392897788
==============================================================================

Another way to find it is to run the following REST API request:

curl -X GET -u  $CB_USERNAME:$CB_PASSWORD http://localhost:9102/listMetadataTokens

/indexing/ddl/commandToken/create/9537139879024260146-5572695136392897788 - {"DefnId":9537139879024260146,"BucketUUID":"2668e7bebcb554c1a1e84777def2dd40","ScopeId":"0","CollectionId":"0","Definitions":{"d3cdfe35fa87b54c85d0a583483d5567":[{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","instanceId":8761026812170097965,"partitions":[7,2,5],"versions":[0,0,0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"},{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":1,"instanceId":15384423129265657660,"partitions":[8,3],"versions":[0,0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"},{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":2,"instanceId":13675526913150696815,"partitions":[6,4,1],"versions":[0,0,0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"}],"d7f418b6177f961adedda7578ace6bf7":[{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":2,"instanceId":13675526913150696815,"partitions":[8,5,3,7],"versions":[0,0,0,0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"},{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":1,"instanceId":15384423129265657660,"partitions":[6,1,2],"versions":[0,0,0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"},{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","instanceId":8761026812170097965,"partitions":[4],"versions":[0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"}],"f1380ac258b61da103f8e3de4f84c1f3":[{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":1,"instanceId":15384423129265657660,"partitions":[4,5,7],"versions":[0,0,0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"},{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","instanceId":8761026812170097965,"partitions":[1,3,8,6],"versions":[0,0,0,0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"},{"defnId":9537139879024260146,"name":"idx_tran_reconciliation","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(((`documentType` = \"transaction\") and (`deletedDatetime` is missing)) and (not (`isComplete` = false)))","desc":[false,false,false,false,false,false,false,false,false],"deferred":true,"numReplica":2,"partitionKeys":["`accountCorrelationId`"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":2,"instanceId":13675526913150696815,"partitions":[2],"versions":[0],"numPartitions":8,"unexplodedSecExprs":["`accountCorrelationId`","`transactionType`","`ledgerType`","`scheduledForDate`","`amount`","`sourceSystemDescription`","`correlationId`","`transactionCode`","`checkNumber`"],"exprStmt":"`accountCorrelationId`,`transactionType`,`ledgerType`,`scheduledForDate`,`amount`,`sourceSystemDescription`,`correlationId`,`transactionCode`,`checkNumber`"}]},"RequestId":5572695136392897788}
/indexing/ddl/commandToken/create/13388360197742923453-9623504811457175456 - {"DefnId":13388360197742923453,"BucketUUID":"2668e7bebcb554c1a1e84777def2dd40","ScopeId":"0","CollectionId":"0","Definitions":{"d3cdfe35fa87b54c85d0a583483d5567":[{"defnId":13388360197742923453,"name":"idx_audit_log_status","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`httpStatus`","`createdDatetime`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(`documentType` = \"log\")","desc":[false,true],"deferred":true,"numReplica":2,"partitionKeys":["date_trunc_millis(`createdDatetime`, \"month\")"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","instanceId":2401552260567238979,"partitions":[7],"versions":[0],"numPartitions":8,"unexplodedSecExprs":["`httpStatus`","`createdDatetime`"],"exprStmt":"`httpStatus`,`createdDatetime` DESC"},{"defnId":13388360197742923453,"name":"idx_audit_log_status","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`httpStatus`","`createdDatetime`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(`documentType` = \"log\")","desc":[false,true],"deferred":true,"numReplica":2,"partitionKeys":["date_trunc_millis(`createdDatetime`, \"month\")"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":1,"instanceId":12998977347412279220,"partitions":[5,4,2,8,3,6],"versions":[0,0,0,0,0,0],"numPartitions":8,"unexplodedSecExprs":["`httpStatus`","`createdDatetime`"],"exprStmt":"`httpStatus`,`createdDatetime` DESC"},{"defnId":13388360197742923453,"name":"idx_audit_log_status","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`httpStatus`","`createdDatetime`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(`documentType` = \"log\")","desc":[false,true],"deferred":true,"numReplica":2,"partitionKeys":["date_trunc_millis(`createdDatetime`, \"month\")"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":2,"instanceId":7880710866278767885,"partitions":[1],"versions":[0],"numPartitions":8,"unexplodedSecExprs":["`httpStatus`","`createdDatetime`"],"exprStmt":"`httpStatus`,`createdDatetime` DESC"}],"d7f418b6177f961adedda7578ace6bf7":[{"defnId":13388360197742923453,"name":"idx_audit_log_status","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`httpStatus`","`createdDatetime`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(`documentType` = \"log\")","desc":[false,true],"deferred":true,"numReplica":2,"partitionKeys":["date_trunc_millis(`createdDatetime`, \"month\")"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":2,"instanceId":7880710866278767885,"partitions":[2,3,4,8,7,6],"versions":[0,0,0,0,0,0],"numPartitions":8,"unexplodedSecExprs":["`httpStatus`","`createdDatetime`"],"exprStmt":"`httpStatus`,`createdDatetime` DESC"},{"defnId":13388360197742923453,"name":"idx_audit_log_status","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`httpStatus`","`createdDatetime`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(`documentType` = \"log\")","desc":[false,true],"deferred":true,"numReplica":2,"partitionKeys":["date_trunc_millis(`createdDatetime`, \"month\")"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","instanceId":2401552260567238979,"partitions":[5,1],"versions":[0,0],"numPartitions":8,"unexplodedSecExprs":["`httpStatus`","`createdDatetime`"],"exprStmt":"`httpStatus`,`createdDatetime` DESC"}],"f1380ac258b61da103f8e3de4f84c1f3":[{"defnId":13388360197742923453,"name":"idx_audit_log_status","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`httpStatus`","`createdDatetime`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(`documentType` = \"log\")","desc":[false,true],"deferred":true,"numReplica":2,"partitionKeys":["date_trunc_millis(`createdDatetime`, \"month\")"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":1,"instanceId":12998977347412279220,"partitions":[7,1],"versions":[0,0],"numPartitions":8,"unexplodedSecExprs":["`httpStatus`","`createdDatetime`"],"exprStmt":"`httpStatus`,`createdDatetime` DESC"},{"defnId":13388360197742923453,"name":"idx_audit_log_status","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`httpStatus`","`createdDatetime`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(`documentType` = \"log\")","desc":[false,true],"deferred":true,"numReplica":2,"partitionKeys":["date_trunc_millis(`createdDatetime`, \"month\")"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","instanceId":2401552260567238979,"partitions":[4,6,2,3,8],"versions":[0,0,0,0,0],"numPartitions":8,"unexplodedSecExprs":["`httpStatus`","`createdDatetime`"],"exprStmt":"`httpStatus`,`createdDatetime` DESC"},{"defnId":13388360197742923453,"name":"idx_audit_log_status","using":"plasma","bucket":"bksmtscb","bucketUUID":"2668e7bebcb554c1a1e84777def2dd40","secExprs":["`httpStatus`","`createdDatetime`"],"exprType":"N1QL","partitionScheme":"KEY","where":"(`documentType` = \"log\")","desc":[false,true],"deferred":true,"numReplica":2,"partitionKeys":["date_trunc_millis(`createdDatetime`, \"month\")"],"NumReplica2":{"HasValue":true,"Base":2,"Incr":1,"Decr":1},"Scope":"_default","Collection":"_default","ScopeId":"0","CollectionId":"0","replicaId":2,"instanceId":7880710866278767885,"partitions":[5],"versions":[0],"numPartitions":8,"unexplodedSecExprs":["`httpStatus`","`createdDatetime`"],"exprStmt":"`httpStatus`,`createdDatetime` DESC"}]},"RequestId":9623504811457175456}

Workarounds

  • Delete the erroneous tokens on the node where the Indexer is crashing. To do so, execute the below REST API request:
curl -v -X DELETE -u Administrator:password <http://localhost:8091/_metakv/indexing/ddl/commandToken/create/<defnid>>-<instid>

For example, if you find the below tokens, and the erroneous one is 9537139879024260146-5572695136392897788:

Indexer Metadata Tokens: 
curl -X GET -sS -k --proxy  -K- http://127.0.0.1:9102/listMetadataTokens
==============================================================================
/indexing/ddl/commandToken/create/13388360197742923453-9623504811457175456
/indexing/ddl/commandToken/create/9537139879024260146-5572695136392897788
==============================================================================

The delete command to run would be the following:

curl -v -X DELETE -u Administrator:password http://localhost:8091/_metakv/indexing/ddl/commandToken/create/9537139879024260146-5572695136392897788/

Please note that the trailing slash at the end of the command is crucial.

  • Alternatively, dropping and recreating the index should resolve the issue as well, although possibly resulting in downtime or taking some time for the index to be rebuilt, depending on the index size.

Related to

Related articles

Comments

0 comments

Article is closed for comments.