Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-64279] Don't delete indexes on the bucket if validateBucket times out due to a connection error

Callum
Updated 
Product: Couchbase Server
Component: index-service
Issue Link: MB-64279
Affects Version(s): 7.2.x, 7.6.x
Fix Version(s): 8.0.0

Summary

If an Index Service node experiences network issues that impact the ability of the Index Service to communicate with the Cluster Manager on other nodes for longer than 30 minutes, this can lead to indexes being unexpectedly and automatically dropped.

This happens when an internal call to check that a bucket is still valid times out (1x 10min timeout + 2x 10min retries), resulting in the assumption that the bucket is no longer valid. This results in the indexes being automatically dropped for that bucket on the node affected by the network issue.

If ClusterInfoCacheLiteClient 's ValidateBucket times out due to a connection error, after 2 retries (10 min each) all the indexes on the keyspace will be dropped.

Symptoms 

  • Indexes are automatically dropped.
  • The dropped indexes will not be automatically recreated, even if the indexer is restarted.

Triggers

  • Network issues that persist longer than 30 minutes when auto-failover is not possible.

Verification

Check for signs of network issues around the time the index was lost.

Check whether the node with missing indexes appears to 'go down' from the perspective of other nodes (in diag.log):

2024-11-08T21:07:39.532-07:00, ns_node_disco:5:warning:node down(node_name) - Node 'node_name' saw that node 'node_name' went down. Details: [{nodedown_reason,
2024-11-08T21:19:13.420-07:00, ns_node_disco:5:warning:node down(node_name) - Node 'node_name' saw that node 'node_name' went down. Details: [{nodedown_reason,
2024-11-08T21:19:18.940-07:00, ns_node_disco:5:warning:node down(node_name) - Node 'node_name' saw that node 'node_name' went down. Details: [{nodedown_reason,
2024-11-08T21:28:32.277-07:00, ns_node_disco:5:warning:node down(node_name) - Node 'node_name' saw that node 'node_name' went down. Details: [{nodedown_reason,

On the node with the missing indexes, look for net_tick_timeout and Connection down in ns_server.debug.log around the relevant time:

[error_logger:info,2024-11-08T21:20:27.928-07:00,babysitter_of_ns_1@cb.local:net_kernel<0.71.0>:ale_error_logger_handler:do_log:101]
=========================NOTICE REPORT=========================
{net_kernel,{'EXIT',<0.130.0>,net_tick_timeout}}
[ns_server:debug,2024-11-08T21:20:27.928-07:00,babysitter_of_ns_1@cb.local:cb_dist<0.68.0>:cb_dist:info_msg:872]cb_dist: Connection down: {con,#Ref<0.1209512049.1962934274.10465>,
                               inet_tcp_dist,<0.130.0>,
                               #Ref<0.1209512049.1962934274.10467>}
[error_logger:info,2024-11-08T21:20:27.928-07:00,babysitter_of_ns_1@cb.local:net_kernel<0.71.0>:ale_error_logger_handler:do_log:101]
=========================NOTICE REPORT=========================
{net_kernel,{'EXIT',#Port<0.9>,net_tick_timeout}}

Look for EOF errors and revrpc retries in ns_server.indexer.log around the time before the index was lost:

2024/11/08 21:28:38 revrpc: Got error (EOF) and will retry in 1s
2024/11/08 21:28:39 revrpc: Got error (dial tcp 127.0.0.1:8091: connect: connection refused) and will retry in 1s
...
2024/11/08 23:35:43 revrpc: Got error (dial tcp 127.0.0.1:8091: connect: connection refused) and will retry in 1s

Look for ConnErr in ns_server.indexer.log around the time before the index was lost:

2024-11-08T21:43:21.003-07:00 [Info] Timekeeper::handleStreamConnError RepairStream due to ConnError. StreamId MAINT_STREAM KeyspaceId bucket_name VbList [571 518 560 433 447 473 392 407 556 435 606 342 650 476 486 441 355 458 443 631 617 627 641 345 376 452 566 457 601 438 450 514 569 390 671 502 557 467 386 507 348 574 567 492 529 536 410 543 350 373 474 610 398 587 539 349 618 483 614 478 513 551 595 360 553 424 638 640 540 465 675 351 469 368 434 521 583 655 667 432 491 525 682 387 593 548 401 411 448 510 425 414 656 517 357 374 394 612 471 389 419 658 531 396 530 372 501 466 665 677 423 451 494 613 477 668 371 663 416 463 495 642 657 366 558 615 562 646 509 554 591 594 672 604 403 484 442 460 533 599 470 568 581 664 464 564 611 393 418 468 490 516 549 347 515 637 446 487 412 379 654 365 453 679 402 429 346 428 364 431 578 344 493 659 462 676 413 537 623 356 359 454 499 508 598 632 427 625 475 669 546 572 456 380 369 404 576 584 590 582 489 545 600 488 620 673 385 559 353 436 395 626 586 596 449 577 382 497 461 399 644 397 400 561 383 420 629 522 603 555 662 426 422 653 624 541 526 550 417 534 496 608 544 421 511 597 628 506 660 384 648 367 609 573 619 636 504 415 633 678 563 605 439 482 500 440 639 505 444 538 343 630 437 565 585 361 381 552 621 634 592 519 651 391 635 363 445 498 354 485 520 579 370 375 643 472 542 405 535 602 666 532 661 362 681 388 409 649 670 547 406 481 527 607 616 674 680 575 580 455 523 480 512 589 622 408 570 588 479 352 645 341 459 647 377 503 358 524 528 378 430 652]

Validate that we see Unable to GetBucketInfo due to err error event wait and handleKeyspaceNotFound Updated Index State to DELETED before the index was dropped, like in the following example:

2024-11-08T22:08:53.428-07:00 [Info] Error occurred during cluster info update (CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connect: connection refused) .. Retrying(600)
2024-11-08T22:08:55.429-07:00 [Error] KVSender::restartVbuckets MAINT_STREAM bucket_name Error in fetching cluster info CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connect: connection refused
...
2024-11-08T22:18:55.430-07:00 [Error] ValidateBucket: Unable to GetBucketInfo due to err error event wait timeout
..
2024-11-08T22:28:55.430-07:00 [Error] ValidateBucket: Unable to GetBucketInfo due to err error event wait timeout
2024-11-08T22:28:55.430-07:00 [Error] Timekeeper::sendRestartMsg Keyspace Not Found For Stream MAINT_STREAM KeyspaceId bucket_name Bucket bucket_name
2024-11-08T22:28:55.431-07:00 [Info] Indexer::handleKeyspaceNotFound StreamId MAINT_STREAM KeyspaceId bucket_name SessionId 3501
2024-11-08T22:28:55.431-07:00 [Info] ClustMgr:handleDeleteKeyspace mType: CLUST_MGR_DEL_KEYSPACE indexList: [] bucket: bucket_name scope:  collection:  streamId: MAINT_STREAM syncUpdate: false respCh: <nil> 
2024-11-08T22:28:55.431-07:00 [Info] Indexer::handleKeyspaceNotFound Updated Index State to DELETED [11829043367397496075]

If we see signs of network issues, no auto-failover (e.g., if auto-failover is disabled), and Unable to GetBucketInfo due to err error event wait, before the index is dropped, there is a high likelihood we have encountered this issue.

Workarounds

The missing indexes will not be recreated when the indexer is restarted.

Recreate the missing index with a CREATE INDEX statement.

If replicas are in use and any replica (or original) is missing, try using an ALTER INDEX command to decrease, then increase the number of replicas back to the original value, which should trigger rebuilding missing replicas.

Related to

Related articles

Comments

0 comments

Article is closed for comments.