Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-64604] FTS Service Panics Caused by Postings List Corruption

Callum
Updated 
Product: Couchbase Server
Component: fts-service
Issue Link: MB-64604
Affects Version(s): 7.1.x, 7.2.x, 7.6.x
Fix Version(s): 7.2.7, 7.6.5

Summary

Illegally reusing certain in-memory data structures across queries can cause the FTS service to crash repeatedly during a workload.

Symptoms 

  • FTS Service process (cbft) repeatedly crashes (crash loop) with a panic.
  • FTS operations are failing from SDK application clients (due to repeated FTS crashes).

Triggers

  • Running FTS queries on an affected version may trigger this issue.

Verification

Look for the log messages which say Service 'fts' exited with status 2 in diag.log to indicate that the FTS Service is crashing:

2024-12-11T09:09:06.565Z, ns_log:0:info:message(ns_1@10.197.32.92) - Service 'fts' exited with status 2. Restarting. Messages:
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.4-0.2024062-0.20240625181944-fdaed7be8e8a/index_alias_impl.go:476 +0xee
created by github.com/blevesearch/bleve/v2.MultiSearch in goroutine 2020318
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.4-0.20240625181944-fdaed7be8e8a/index_alias_impl.go:483 +0x171
5181944-fdaed7be8e8a/search/collector/topn.go:226 +0x4a4
github.com/blevesearch/bleve/v2.(*indexImpl).SearchInContext(0xc0002b2150, {0x161eaa8, 0xc0f00e6f30}, 0xc0cf9bfbc0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.4-0.20240625181944-fdaed7be8e8a/index_impl.go:531 +0x6e6
github.com/couchbase/cbft.(*cacheBleveIndex).SearchInContext(0xc0f00e6ed0, {0x161eaa8, 0xc0f00e6f30}, 0xc0cf9bfbc0)
        /home/couchbase/jenkins/workspace/couchbase-server-unix/cbft/cache_bleve.go:111 +0x247
github.com/blevesearch/bleve/v2.MultiSearch.func1({0x162c4d8, 0xc0f00e6ed0}, 0xc0cf9bfbc0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/bleve/v2@v2.3.4
rg -i "exited with status" diag.log
779:2024-12-11T09:09:06.565Z, ns_log:0:info:message(ns_1@10.197.32.92) - Service 'fts' exited with status 2. Restarting. Messages:
790:2024-12-11T10:11:06.465Z, ns_log:0:info:message(ns_1@10.197.32.92) - Service 'fts' exited with status 2. Restarting. Messages:
801:2024-12-11T10:11:09.052Z, ns_log:0:info:message(ns_1@10.197.32.92) - Service 'fts' exited with status 2. Restarting. Messages:
812:2024-12-11T10:11:43.481Z, ns_log:0:info:message(ns_1@10.197.32.92) - Service 'fts' exited with status 2. Restarting. Messages:
823:2024-12-11T10:13:37.140Z, ns_log:0:info:message(ns_1@10.197.32.92) - Service 'fts' exited with status 2. Restarting. Messages:

Look for panics in the ns_server.fts.log:

rg -i "panic" ns_server.fts.log
1481:panic: runtime error: integer divide by zero
2610:panic: runtime error: invalid memory address or nil pointer dereference
3486:panic: runtime error: integer divide by zero
4649:panic: runtime error: invalid memory address or nil pointer dereference
5827:panic: runtime error: invalid memory address or nil pointer dereference

Examples:

panic: runtime error: integer divide by zero

goroutine 2033519 [running]:
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc0d34b1260, 0x0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:588 +0x2d1
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc0d34b1260, 0x4879e6?)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:481 +0x25
github.com/blevesearch/zapx/v15.(*PostingsIterator).Next(0x0?)
...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x6dd42f]

goroutine 2159447 [running]:
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc0e7c548c0, 0xf8a)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:581 +0xaf
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc0e7c548c0, 0xf8a?)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:481 +0x25
github.com/blevesearch/zapx/v15.(*PostingsIterator).Advance(0xc0001d8880?, 0xf8a?)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:476 +0x13
github.com/blevesearch/bleve/v2/index/scorch.(*IndexSnapshotTermFieldReader).Advance(0xc0e083b080, {0xc0e7c4ec40, 0x8, 0x8}, 0xc0e74b1828)
panic: runtime error: integer divide by zero

goroutine 21330 [running]:
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc0575710a0, 0x0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:588 +0x2d1
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc0575710a0, 0x4879e6?)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:481 +0x25
github.com/blevesearch/zapx/v15.(*PostingsIterator).Next(0x0?)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:470 +0x15
github.com/blevesearch/bleve/v2/index/scorch.(*IndexSnapshotTermFieldReader).Next(0xc0542cde00, 0x16c7?)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x6dd42f]

goroutine 91807 [running]:
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc0a35e2c40, 0xf7e)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:581 +0xaf
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc0a35e2c40, 0xf7e?)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:481 +0x25
github.com/blevesearch/zapx/v15.(*PostingsIterator).Advance(0xc0021cca00?, 0xf7e?)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:476 +0x13
github.com/blevesearch/bleve/v2/index/scorch.(*IndexSnapshotTermFieldReader).Advance(0xc09f6abe00, {0xc0a4602508, 0x8, 0x8}, 0xc0a3ebf128)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x52731b]

goroutine 717374 [running]:
github.com/RoaringBitmap/roaring.(*intIterator).Next(0xc081d558f0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/!roaring!bitmap/roaring@v0.9.4/roaring.go:237 +0x1b
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfterClean(0xc0b1e61260, 0x0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:690 +0x38
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextDocNumAtOrAfter(0xc0b1e61260, 0x0)
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/zapx/v15@v15.3.5-0.20230516194434-edc21be64631/posting.go:576 +0x148
github.com/blevesearch/zapx/v15.(*PostingsIterator).nextAtOrAfter(0xc0b1e61260, 0x4879e6?)

This issue is characterized by repeated Full-Text Search (FTS) service crashes that generate a variety of panic messages, such as integer divide by zero, invalid memory address, or nil pointer dereference.

A key indicator of this problem is that the crashes occur at different lines of code (as seen in the goroutine dump), which prevents them from being matched to a specific pattern or source.

Therefore, if you are using an affected version and the FTS service is crashing repeatedly with these symptoms, it is strongly recommended to upgrade to a version or Maintenance Pack containing the fix as soon as possible.

Workarounds

There are no workarounds for this issue; an upgrade is necessary to resolve the crashes.

Related to

Related articles

Comments

0 comments

Article is closed for comments.