Supported Open Metrics¶
The Octez node is able to produce metrics information and serve them in the Open Metrics format, an emerging standard for exposing metrics data, especially used in cloud-based systems.
The Octez node supports the following metrics, characterized by: the name of the metric, the type of the metric as in the open metrics specification, a user friendly description on the metric and a list of labels (that can be used to aggregate or query metrics).
For more information check the openmetrics specification: https://openmetrics.io/
Name |
Type |
Description |
Labels |
|---|---|---|---|
ocaml_gc_allocated_bytes |
Counter |
Total number of bytes allocated since the program was started. |
|
ocaml_gc_compactions |
Counter |
Number of heap compactions since the program was started. |
|
ocaml_gc_heap_words |
Gauge |
Total size of the major heap, in words. |
|
ocaml_gc_major_collections |
Counter |
Number of major collection cycles completed since the program was started. |
|
ocaml_gc_major_words |
Counter |
Number of words allocated in the major heap since the program was started. |
|
ocaml_gc_minor_collections |
Counter |
Number of minor collection cycles completed since the program was started. |
|
ocaml_gc_top_heap_words |
Counter |
Maximum size reached by the major heap, in words. |
|
octez_distributed_db_message_block_header_messages |
Counter |
Number of block_header messages |
action |
octez_distributed_db_message_checkpoint_messages |
Counter |
Number of checkpoint messages |
action |
octez_distributed_db_message_current_branch_messages |
Counter |
Number of current_branch messages |
action |
octez_distributed_db_message_current_head_messages |
Counter |
Number of current_head messages |
action |
octez_distributed_db_message_deactivate_messages |
Counter |
Number of deactivate messages |
action |
octez_distributed_db_message_get_block_headers_messages |
Counter |
Number of get_block_headers messages |
action |
octez_distributed_db_message_get_checkpoint_messages |
Counter |
Number of get_checkpoint messages |
action |
octez_distributed_db_message_get_current_branch_messages |
Counter |
Number of get_current_branch messages |
action |
octez_distributed_db_message_get_current_head_messages |
Counter |
Number of get_current_head messages |
action |
octez_distributed_db_message_get_operations_for_blocks_messages |
Counter |
Number of get_operations_for_blocks messages |
action |
octez_distributed_db_message_get_operations_messages |
Counter |
Number of get_operations messages |
action |
octez_distributed_db_message_get_predecessor_header_messages |
Counter |
Number of get_predecessor_header messages |
action |
octez_distributed_db_message_get_protocol_branch_messages |
Counter |
Number of get_protocol_branch messages |
action |
octez_distributed_db_message_get_protocols_messages |
Counter |
Number of get_protocols messages |
action |
octez_distributed_db_message_operation_messages |
Counter |
Number of operation messages |
action |
octez_distributed_db_message_operations_for_block_messages |
Counter |
Number of operations_for_block messages |
action |
octez_distributed_db_message_predecessor_header_messages |
Counter |
Number of predecessor_header messages |
action |
octez_distributed_db_message_protocol_branch_messages |
Counter |
Number of protocol_branch messages |
action |
octez_distributed_db_message_protocol_messages |
Counter |
Number of protocol messages |
action |
octez_distributed_db_requester_table_length |
Gauge |
Number of entries (to grab) from the network present |
requester_kind;entry_type |
octez_mempool_pending_branch_delayed |
Gauge |
Mempool pending branch delayed operations count |
|
octez_mempool_pending_branch_refused |
Gauge |
Mempool pending branch refused operations count |
|
octez_mempool_pending_outdated |
Gauge |
Mempool pending outdated operations count |
|
octez_mempool_pending_refused |
Gauge |
Mempool pending refused operations count |
|
octez_mempool_pending_unprocessed |
Gauge |
Mempool pending unprocessed operations count |
|
octez_mempool_pending_validated |
Gauge |
Mempool pending validated operations count |
|
octez_mempool_worker_completion_count |
Counter |
Number of requests completed the block validator worker |
|
octez_mempool_worker_error_count |
Counter |
Number of errors encountered by the block validator worker |
|
octez_mempool_worker_request_count |
Counter |
Number of requests received by the block validator worker |
|
octez_p2p_connections_active |
Gauge |
Number of active connections |
|
octez_p2p_connections_incoming |
Gauge |
Number of incoming connections |
|
octez_p2p_connections_outgoing |
Gauge |
Number of outgoing connections |
|
octez_p2p_connections_private |
Gauge |
Number of private connections |
|
octez_p2p_messages_advertise_received |
Counter |
Number of advertise received |
|
octez_p2p_messages_advertise_sent |
Counter |
Number of advertise sent |
|
octez_p2p_messages_bootstrap_received |
Counter |
Number of bootstrap received |
|
octez_p2p_messages_bootstrap_sent |
Counter |
Number of bootstrap sent |
|
octez_p2p_messages_broadcast_message_sent |
Counter |
Number of user message sent by broadcasting |
|
octez_p2p_messages_swap_ack_received |
Counter |
Number of swap acks received |
|
octez_p2p_messages_swap_ack_sent |
Counter |
Number of swap acks sent |
|
octez_p2p_messages_swap_requests_received |
Counter |
Number of swap received |
|
octez_p2p_messages_swap_requests_sent |
Counter |
Number of swap sent |
|
octez_p2p_messages_user_message_received |
Counter |
Number of user message received |
|
octez_p2p_messages_user_message_received_error |
Counter |
Number of user message received that resulted in error |
|
octez_p2p_messages_user_message_sent |
Counter |
Number of user message sent |
|
octez_p2p_peers_accepted |
Gauge |
Number of accepted connections |
|
octez_p2p_peers_disconnected |
Gauge |
Number of disconnected peers |
|
octez_p2p_peers_running |
Gauge |
Number of running peers |
|
octez_p2p_points_accepted |
Gauge |
Number of accepted points |
|
octez_p2p_points_disconnected |
Gauge |
Number of disconnected points |
|
octez_p2p_points_greylisted |
Gauge |
Number of greylisted points |
|
octez_p2p_points_running |
Gauge |
Number of running points |
|
octez_p2p_points_trusted |
Gauge |
Number of trusted points |
|
octez_p2p_swap_fail |
Counter |
Number of failed swap |
|
octez_p2p_swap_ignored |
Counter |
Number of ignored swap |
|
octez_p2p_swap_success |
Counter |
Number of successful swap |
|
octez_rpc_calls |
Summary |
RPC endpoint call counts and sum of execution times. |
endpoint;method |
octez_store_alternate_heads_count |
Gauge |
Current number of alternated heads known |
|
octez_store_caboose_level |
Gauge |
Current caboose level |
|
octez_store_checkpoint_level |
Gauge |
Current checkpoint level |
|
octez_store_invalid_blocks |
Gauge |
Number of blocks known to be invalid stored on disk |
|
octez_store_last_merge_time |
Gauge |
Time, in seconds, for the completion of the last store merge |
|
octez_store_last_written_block_size |
Gauge |
Size, in bytes, of the last block written in store |
|
octez_store_savepoint_level |
Gauge |
Current savepoint level |
|
octez_validator_block_already_commited_blocks_count |
Counter |
Number of requests to validate a block already handled |
|
octez_validator_block_last_finished_request_completion_timestamp |
Gauge |
Timestamp at which the latest request handled by the worker was completed |
|
octez_validator_block_last_finished_request_push_timestamp |
Gauge |
Reception timestamp of the latest request handled by the worker |
|
octez_validator_block_last_finished_request_treatment_timestamp |
Gauge |
Timestamp at which the worker started processing of the latest request it handled |
|
octez_validator_block_operations_per_pass |
Gauge |
Number of operations per pass for the last validated block |
pass_id |
octez_validator_block_outdated_blocks_count |
Counter |
Number of requests to validate a block older than the node’s checkpoint |
|
octez_validator_block_preapplication_errors_count |
Counter |
Number of refused application simulations of blocks |
|
octez_validator_block_preapplied_blocks_count |
Counter |
Number of successful application simulations of blocks |
|
octez_validator_block_precheck_failed_count |
Counter |
Number of block validation requests where the prechecking of a block failed |
|
octez_validator_block_validated_blocks_count |
Counter |
Number of requests to validate a valid block |
|
octez_validator_block_validation_errors_after_precheck_count |
Counter |
Number of requests to validate an invalid but precheckable block |
|
octez_validator_block_validation_errors_count |
Counter |
Number of requests to validate an invalid block |
|
octez_validator_block_worker_completion_count |
Counter |
Number of requests completed the block validator worker |
|
octez_validator_block_worker_error_count |
Counter |
Number of errors encountered by the block validator worker |
|
octez_validator_block_worker_request_count |
Counter |
Number of requests received by the block validator worker |
|
octez_validator_chain_branch_switch_count |
Counter |
Number of times the chain_validator switched branch |
chain_id |
octez_validator_chain_head_consumed_gas |
Gauge |
Gas consumed in the current node’s head |
chain_id |
octez_validator_chain_head_cycle |
Gauge |
Cycle of the current node’s head |
chain_id |
octez_validator_chain_head_increment_count |
Counter |
Number of times the chain_validator incremented its head for a direct successor |
chain_id |
octez_validator_chain_head_level |
Gauge |
Level of the current node’s head |
chain_id |
octez_validator_chain_head_round |
Gauge |
Round of the current node’s head |
chain_id |
octez_validator_chain_ignored_head_count |
Counter |
Number of requests where the chain validator ignored a new valid block with a lower fitness than its current head |
chain_id |
octez_validator_chain_is_bootstrapped |
Gauge |
Returns 1 if the node has bootstrapped, 0 otherwise. |
chain_id |
octez_validator_chain_last_finished_request_completion_timestamp |
Gauge |
Timestamp at which the latest request handled by the worker was completed |
chain_id |
octez_validator_chain_last_finished_request_push_timestamp |
Gauge |
Reception timestamp of the latest request handled by the worker |
chain_id |
octez_validator_chain_last_finished_request_treatment_timestamp |
Gauge |
Timestamp at which the worker started processing of the latest request it handled |
chain_id |
octez_validator_chain_synchronisation_status |
Gauge |
Returns 0 if the node is unsynchronised, 1 if the node is synchronised, 2 if the node is stuck. |
chain_id |
octez_validator_chain_worker_completion_count |
Counter |
Number of requests completed the block validator worker |
chain_id |
octez_validator_chain_worker_error_count |
Counter |
Number of errors encountered by the block validator worker |
chain_id |
octez_validator_chain_worker_request_count |
Counter |
Number of requests received by the block validator worker |
chain_id |
octez_validator_peer_connections |
Counter |
Number of time we connected to a peer. |
|
octez_validator_peer_invalid_block |
Counter |
Number of time we received an invalid block from a peer. |
|
octez_validator_peer_invalid_locator |
Counter |
Number of time we received an invalid locator from a peer. |
|
octez_validator_peer_new_branch_completed |
Counter |
Number of time we successfuly completed a new branch request from a peer. |
|
octez_validator_peer_new_head_completed |
Counter |
Number of time we successfuly completed a new head request from a peer. |
|
octez_validator_peer_on_no_request_count |
Counter |
Number of time we did no hear new messages from a peer since the last timeout. |
|
octez_validator_peer_operations_fetching_canceled_new_branch |
Counter |
Number of time we canceled the fetching of operations on a new branch request for a peer. |
|
octez_validator_peer_operations_fetching_canceled_new_known_valid_head |
Counter |
Number of time we canceled the fetching of operations on a new head request for a peer. |
|
octez_validator_peer_operations_fetching_canceled_new_unknown_head |
Counter |
Number of time we canceled the fetching of operations on a new head request or an unknown head for a peer. |
|
octez_validator_peer_system_error |
Counter |
Number of time a request trigerred a system error from a peer. |
|
octez_validator_peer_too_short_locator |
Counter |
Number of time we received a too short locator from a peer. |
|
octez_validator_peer_unavailable_protocol |
Counter |
Number of time we received an unknown protocol from a peer. |
|
octez_validator_peer_unknown_ancestor |
Counter |
Number of time we received a locator with an unknown ancestor from a peer. |
|
octez_validator_peer_unknown_error |
Counter |
Number of time an unknown error happened for a peer. |
|
octez_version |
Gauge |
Node version |
version;chain_name;distributed_db_version;p2p_version;commit_hash;commit_date |
process_cpu_seconds_total |
Counter |
Total user and system CPU time spent in seconds. |
|
process_start_time_seconds |
Counter |
Start time of the process since unix epoch in seconds. |
|
prometheus_logs_messages_total |
Counter |
Total number of messages logged |
level;src |
Usage¶
In the following, we indicate a typical monitoring setup for Octez developers. For more details on setting up the node for monitoring see Monitoring an Octez Node.
To instruct the Octez node to produce metrics, the user needs to pass the option
--metrics-addr=<ADDR>:<PORT>. The port specified on the command line is the port
where the integrated open metrics server will be available (9932 by default).
The address defaults to localhost.
When the option is not supplied at all, no metrics are produced.
Ex.:
octez-node run --metrics-addr=:9091
To query the open metrics server the user can simply query the node.
Ex.:
curl http://<node_addr>:9091/metrics
Collecting metrics¶
Different third-party tools can be used to query the Octez node and collect metrics from it. Let us illustrates this with the example of a Prometheus server.
Update the Prometheus configuration file (typically, prometheus.yml)
to add a “scrape job” - that is how Prometheus is made aware of a new data
source - using adequate values:
job_name: Use a unique name among other scrape jobs. All metrics collected through this job will have automatically a ‘job’ label with this value added to it
targets: The URL of Octez node.
- job_name: 'octez-metrics'
scheme: http
static_configs:
- targets: ['localhost:9091']