Skip to main content
Docs Developer Videos Webinars Academy Blog Forum Contact

[MB-64227] stats_interface:latest crashes when receiving E notation numbers with no decimal place (e.g. 9e-08)

Callum
Updated 
Product: Couchbase Server
Component: ns-server
Issue Link: MB-64227
Affects Version(s): 7.6.x
Fix Version(s): 7.6.4

Summary

The ns_server may crash when processing metrics from Prometheus due to a parsing incompatibility. Erlang's float parser does not correctly handle scientific notation when a decimal point is omitted (e.g., 1e-6). Since Prometheus uses this format for very small values, ns_server receives unparseable data, which can result in a crash.

Symptoms 

  • The XDCR process (goxdcr) will repeatedly crash if the ns_server process is unreachable.

Triggers

  • When Prometheus reports a value smaller than 1e-6, it formats the number using scientific notation without a decimal point (in the format [1-9]e[+-]?[0-9]+). This format cannot be parsed by ns_server, which results in an error.

Verification

Examine the ns_server.debug.log file for an error report related to parsing. The report will reference either binary_to_float or binary_to_integer and will include a number in scientific notation that is missing a decimal point (in the format [1-9]e[+-]?[0-9]+).

Example:

=========================ERROR REPORT=========================
** Generic server guardrail_monitor terminating 
** Last message in was check
** When Server state == {state,[],#Ref<0.1726234097.39583745.115851>}
** Reason for termination ==
** {badarg,[{erlang,binary_to_integer,
                    [<<"9e-08">>],
                    [{error_info,#{module => erl_erts_errors}}]},
            {stats_interface,'-latest/3-fun-0-',2,
                             [{file,"src/stats_interface.erl"},{line,335}]},
            {lists,filtermap_1,2,[{file,"lists.erl"},{line,1417}]},
            {lists,filtermap_1,2,[{file,"lists.erl"},{line,1421}]},
            {stats_interface,for_resource_management,0,
                             [{file,"src/stats_interface.erl"},{line,200}]},
            {guardrail_monitor,check_resources,0,
                               [{file,"src/guardrail_monitor.erl"},
                                {line,692}]},
            {guardrail_monitor,handle_info,2,
                               [{file,"src/guardrail_monitor.erl"},
                                {line,640}]},
            {gen_server,try_dispatch,4,
                        [{file,"gen_server.erl"},{line,1123}]}]}

Look for the signs that the ns_server components are shutting down after the above crash report:

[ns_server:info,2024-11-10T01:15:12.914Z,ns_1@svc-xxx-node-006.xxx.cloud.couchbase.com:menelaus_cbauth<0.2888.0>:menelaus_cbauth:terminate_external_connections:159]External connections to be terminated: []
[ns_server:debug,2024-11-10T01:15:12.915Z,ns_1@svc-xxx-node-006.xxx.cloud.couchbase.com:<0.2895.0>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ssl_service_events,<0.2888.0>} exited with reason shutdown
[ns_server:debug,2024-11-10T01:15:12.915Z,ns_1@svc-xxx-node-006.xxx.cloud.couchbase.com:<0.2891.0>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_node_disco_events,<0.2888.0>} exited with reason shutdown
[ns_server:debug,2024-11-10T01:15:12.915Z,ns_1@svc-xxx-node-006.xxx.cloud.couchbase.com:<0.2893.0>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {user_storage_events,<0.2888.0>} exited with reason shutdown
[ns_server:debug,2024-11-10T01:15:12.915Z,ns_1@svc-xxx-node-006.xxx.cloud.couchbase.com:<0.2889.0>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {json_rpc_events,<0.2888.0>} exited with reason shutdown
[ns_server:debug,2024-11-10T01:15:12.915Z,ns_1@svc-xxx-node-006.xxx.cloud.couchbase.com:<0.2892.0>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {chronicle_compat_event_manager,<0.2888.0>} exited with reason shutdown
[ns_server:debug,2024-11-10T01:15:12.915Z,ns_1@svc-xxx-node-006.xxx.cloud.couchbase.com:<0.2884.0>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.2883.0>} exited with reason shutdown
[ns_server:debug,2024-11-10T01:15:12.915Z,ns_1@svc-xxx-node-006.xxx.cloud.couchbase.com:<0.14859.1>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.18119.0>} exited with reason shutdown

Workarounds

To prevent this parsing error, run the following command on any node in the cluster. This setting automatically propagates to all nodes and will stop Prometheus from sending small values in the scientific notation format that Erlang cannot parse.

curl -X POST -u <user>:<password> <addr>:<port>/internalSettings -d resourcePromQLOverride.dataSizePerNodeTB="round(kv_logical_data_size_bytes{state='active'} / 10^6) / 10^6"

Related to

Related articles

Comments

0 comments

Article is closed for comments.