DiskPool I/O Monitoring and Failure Handling

Explore this Page

• Overview
• DiskPool States and Alerts
• Hot-Removal Behavior
• DiskPool I/O Error Handling
• I/O Stall Detection and Intermittent Backend Failure Handling
• Configure DiskPool I/O Alert Thresholds
• Clear DiskPool I/O Errors
• View DiskPool Status
• Benefits of DiskPool I/O Monitoring and Failure Handling

Overview

DataCore Puls8 Replicated PV Mayastor continuously monitors DiskPools and their underlying storage devices to detect disk I/O errors, device removal events, and I/O stalls. This capability improves storage fault visibility by detecting device hot-removal, stalled I/O conditions, and I/O errors in real time.

DiskPools automatically report updated pool states, alerts, and diagnostic information to help administrators identify unhealthy storage devices, understand workload impact during disk-related failures, and prevent unhealthy pools from being selected for future storage operations.

Typical scenarios addressed by this feature include:

• A backing disk is physically detached or removed from a node.
• A disk starts returning I/O errors or experiencing stalled I/O operations.

Disk I/O failures and device removal events may result in degraded volumes, unavailable replicas, or failed storage operations. Review DiskPool state and alert information promptly to prevent data availability impact.

DiskPool States and Alerts

DataCore Puls8 Replicated PV Mayastor reports DiskPool health using pool states, alert statuses, and diagnostic information. These indicators provide a complete picture of pool and device health to support timely investigation and remediation.

Pool States

Alert Status

Hot-Removal Behavior

When a backing disk is physically detached or becomes unavailable, DataCore Puls8 Replicated PV Mayastor automatically unloads the affected pool. As part of this process:

• The DiskPool transitions to an Offline state.
• The DiskPool Custom Resource (CR) PoolReady condition is set to False.
• Diagnostic information is updated to report the detected failure reason.

After the pool is unloaded, the control plane probes the io-engine to determine whether the backing device has been reattached to the node. If the device becomes available again, Replicated PV Mayastor automatically imports the pool and restores normal operations.

DiskPool I/O Error Handling

DataCore Puls8 Replicated PV Mayastor tracks runtime disk I/O errors and exposes alert information for affected DiskPools.

If the number of I/O errors reported by a DiskPool exceeds the configured threshold, the DiskPool transitions to the Suspected state. Pools in the Suspected state are less likely to be selected for future volume placement operations.

The ioErrorThreshold parameter defines the number of allowed I/O errors before a DiskPool transitions to the Suspected state. By default, ioErrorThreshold is set to 64. This parameter can be configured using Helm and applies globally to all DiskPools in the cluster.

The current DiskPool I/O error count is exposed through the status.error_info.io_error_count field in the DiskPool custom resource and can be used to monitor error accumulation over time.

I/O Stall Detection and Intermittent Backend Failure Handling

DataCore Puls8 Replicated PV Mayastor monitors DiskPools for stalled I/O operations. When an I/O request submitted to the pool backend remains incomplete beyond the user-configured stall detection timeout (stallDeadline), the following actions are taken:

• The DiskPool is marked as stalled.
• The DiskPool state transitions to Suspected.
• The DiskPool alert status transitions to Critical.
• The pool is avoided for future replica placement operations.

I/O stalls can result from storage device failures or unresponsive disks, backend storage path instability, or network disruptions affecting storage access. The current stall status of a DiskPool is exposed through the status.error_info.io_stalled field in the DiskPool custom resource. The status.error_info.io_stall_transition_count field reports the number of stall-to-resume and resume-to-stall transitions detected within the configured stallTransitionWindow.

Intermittent Stall Detection

DataCore Puls8 Replicated PV Mayastor monitors repeated I/O stall and recovery transitions to detect unstable or intermittently failing storage backends. Such conditions can occur when storage paths become unreliable, causing I/O operations to alternate between stalled and completed states.

To help identify these scenarios, Replicated PV Mayastor tracks the frequency of stall transitions within a configurable time window. Administrators can define the maximum number of allowable stall transitions by configuring the stallTransitionThreshold and stallTransitionWindow parameters.

For example, if stallTransitionWindow is configured as 30 minutes and stallTransitionThreshold is set to 10, a DiskPool is marked as Suspected and a Warning alert is raised when 10 stall transitions occur within any 30-minute period. DiskPools in the Suspected state become less preferred for future replica placement operations.

If one or more stall transitions occur within the configured time window but do not exceed the configured threshold, the DiskPool alert status transitions to Attention, indicating a potential storage path or backend stability issue that should be monitored.

When intermittent stalls are detected:

• The DiskPool alert status transitions to Attention.
• Repeated stall transitions within the configured time window may raise the alert level to Warning.
• Pools experiencing frequent intermittent stalls become less preferred for replica scheduling.

This behavior helps identify unstable storage devices or paths that repeatedly recover and fail over time, allowing administrators to act before a complete failure occurs.

Configure DiskPool I/O Alert Thresholds

DataCore Puls8 Replicated PV Mayastor allows you to configure DiskPool I/O monitoring thresholds using Helm values. The following parameters control disk error and stall detection behavior:

Clear DiskPool I/O Errors

DataCore Puls8 Replicated PV Mayastor allows you to clear recorded disk I/O errors for a DiskPool after the underlying issue has been investigated and resolved. Clearing errors resets the ioErrorCount for the pool and removes any associated pool alerts that were generated as a result of the recorded I/O errors.

kubectl-puls8 mayastor clear-errors pool <pool-id> -n <namespace>

Clearing errors resets the recorded I/O error count and associated alerts for the specified pool. This operation does not resolve underlying disk, device, or infrastructure issues. Ensure that the root cause has been fully addressed before clearing errors to avoid masking an active or recurring failure condition.

View DiskPool Status

Use one of the following commands to view DiskPool state, alerts, and conditions. The output includes pool health indicators, alert statuses, capacity metrics, and diagnostic information.

View DiskPools Using Kubernetes Resources

Use the following command to list all DiskPool resources across all namespaces and review their current state and alert status.

kubectl get diskpools -A

View Pools Using the kubectl-puls8 Plugin

Use the following command to view detailed pool status, including state, alert level, capacity, and allocation, using the kubectl-puls8 plugin.

kubectl-puls8 mayastor get pool <pool-name> -n <namespace>

ID                DISKS                         MANAGED  NODE         STATUS  ALERTS               CAPACITY  ALLOCATED  AVAILABLE  COMMITTED  ENCRYPTED  DISK-CAPACITY  MAX-EXPANDABLE-SIZE  VOLUMES  SNAPSHOTS
pool-kind-worker  aio:///dev/kind-worker/lvol0  true     kind-worker  Online  Attention (IoError)  508 MiB   0 B        508 MiB    0 B        false      512 MiB        127.8 GiB            0        0

View DiskPool Details

Use the following command to inspect DiskPool conditions, alerts, and diagnostic information in detail, including the PoolReady condition and error fields.

kubectl get dsp <pool-name> -n <namespace>

NAME               NODE          STATE     STATUS   ERROR   ALERTS                ENCRYPTED   CAPACITY   USED   AVAILABLE   DISK-CAPACITY   MAX-EXPANDABLE-SIZE
pool-kind-worker   kind-worker   Created   Online           Attention (IoError)   false       508 MiB    0 B    508 MiB     512 MiB         127.8 GiB

Benefits of DiskPool I/O Monitoring and Failure Handling

• Proactive Fault Detection: Continuously monitors DiskPools for I/O errors, stalls, and device removal events, enabling early identification of storage issues before they impact workloads.
• Improved Storage Reliability: Automatically transitions unhealthy pools to Suspected or Faulted states and excludes them from future volume placement, reducing the risk of data unavailability.
• Automatic Recovery: Detects when a removed or offline backing device is reattached to a node and automatically re-imports the pool to restore normal operations without manual intervention.
• Granular Alert Visibility: Provides four-level alert statuses (Healthy, Attention, Warning, Critical) so administrators can prioritize and respond to storage events based on severity.
• Intermittent Failure Identification: Tracks repeated stall-to-resume transitions within a configurable time window to surface unstable storage paths that may not trigger standard fault thresholds.
• Configurable Thresholds: Administrators can tune I/O error and stall detection parameters through Helm to match the specific reliability and sensitivity requirements of their environment.
• Workload Protection: Pools in degraded states are deprioritized for replica and volume scheduling, preventing new workloads from being placed on unhealthy storage devices.

Learn More

• Cordon and Uncordon Pools
• Creating a DiskPool
• Node Lifecycle Operations