Monitoring

Explore this Page

• Overview
• Enable Monitoring
• Pool Metrics Exporter
• Stats Exporter Metrics
• CSI Metrics Exporter
• Performance Monitoring Stack
• Accessing Grafana
• I/O Performance Metrics
• Benefits of Monitoring

Overview

Effective monitoring is critical for maintaining the health and performance of storage infrastructure. This document outlines the metrics exposed by various exporters within the Replicated PV Mayastor ecosystem. These metrics enable observability of resource usage, performance trends, and operational statuses, thereby facilitating data-driven troubleshooting and capacity planning.

Depending on the situation, it performs either a full rebuild (restores the entire replica) or a partial rebuild (restores only the changed data). This flexibility ensures fast recovery with minimal disruption to applications.

Enable Monitoring

The DataCore Puls8 monitoring stack is enabled by default when you install DataCore Puls8 using Helm.

If you want to disable monitoring , use the following Helm flag:

--set monitoring.enabled=false

This disables the installation of Prometheus, Grafana, and related monitoring components that are part of the DataCore Puls8 monitoring stack.

To collect metrics such as pool usage, volume statistics, and I/O performance, ensure that monitoring is not disabled.

Pool Metrics Exporter

The Pool Metrics Exporter runs as a sidecar container alongside each I/O Engine pod. It exposes Prometheus-compatible pool metrics via the metrics HTTP endpoint on port 9502. These metrics are refreshed every five minutes to reflect recent usage and state information.

Supported Pool Metrics

Sample Pool Metrics Output

# HELP disk_pool_status Status of the disk pool (1 = healthy, 0 = unhealthy)
# TYPE disk_pool_status gauge
disk_pool_status{node="worker-0", name="mayastor-disk-pool"} 1

# HELP disk_pool_total_size_bytes Total size of the disk pool in bytes
# TYPE disk_pool_total_size_bytes gauge
disk_pool_total_size_bytes{node="worker-0", name="mayastor-disk-pool"} 5360320512

# HELP disk_pool_used_size_bytes Used size of the disk pool in bytes
# TYPE disk_pool_used_size_bytes gauge
disk_pool_used_size_bytes{node="worker-0", name="mayastor-disk-pool"} 2147483648

# HELP disk_pool_committed_size_bytes Committed size of the disk pool in bytes
# TYPE disk_pool_committed_size_bytes gauge
disk_pool_committed_size_bytes{node="worker-0", name="mayastor-disk-pool"} 9663676416

Stats Exporter Metrics

When eventing is enabled, statistics are collected by the obs-callhome-stats container within the callhome pod. These metrics are exposed on port 9090 at the /stats endpoint.

Supported Statistics Metric

CSI Metrics Exporter

The CSI metrics exporter provides insights into volume-level statistics. These metrics are collected by kubelet and exported for Prometheus monitoring.

Supported Volume Metrics

Performance Monitoring Stack

Initially, metrics exporters cached data which might not reflect real-time usage during Prometheus polls. This has been improved by directly querying the IO Engine in sync with the Prometheus polling cycle.

It is recommended to set the Prometheus poll interval to at least 5 minutes.

Accessing Grafana

Grafana provides a visual interface to monitor metrics collected by Prometheus. To access Grafana in your environment, follow these steps:

1. Verify the Grafana Pod is running.

   Copy

   Verify Grafana Pod

   kubectl get pods -n [NAMESPACE] | grep -i grafana

2. Check the Grafana service IP and port.

   Copy

   Find External Port Exposed by Grafana Service

   kubectl get svc -n [NAMESPACE] | grep -i grafana

3. Access Grafana via Port-Forwarding.

   • Use port-forwarding to connect to Grafana locally if external access is not available:
   Copy

   Connect to Grafana Locally

   kubectl port-forward --namespace [NAMESPACE] pods/[grafana-pod-name] [grafana-forward-port]:[grafana-cluster-port]

   Copy

   Example: Port Forward the Grafana Service from the Puls8 Namespace to Your Local Port 8080

   kubectl port-forward svc/puls8-grafana -n puls8 8080:80

   • Once port-forwarding is established, Open a browser and visit http://127.0.0.1:[grafana-forward-port] (Example: http://127.0.0.1:8080).
   • Use the default login credentials: Username: admin and Password: admin.
   • After logging in, the Home page is displayed.
   
   • To view the Puls8 dashboards, click Dashboards on the left-hand panel. For example, if you select Puls8/Replicated PV/Mayastor/Diskpool, you can view:
     • Diskpool Status
     • Diskpool Total Size
     • Diskpool Used Size
     • Diskpool Available Size
     • IOPS
     • Throughput
     • Latency
   

I/O Performance Metrics

DiskPool I/O Statistics

Replica I/O Statistics

Volume Target I/O Statistics

Benefits of Monitoring

• Enables real-time visibility into storage usage, performance, and system health.
• Assists in proactive detection and resolution of issues before they impact workloads.
• Provides historical data for capacity planning and trend analysis.
• Facilitates compliance with SLAs and performance benchmarks.

Learn More

• Supportability
• Call-Home Metrics
• Replication