Node Lifecycle Operations

Explore this Page

• Overview
• Cordon a Node
• Uncordon a Node
• Drain a Node
• Delete a Node (Offline)
• Benefits of Node Cordon, Drain, and Delete Operations

Overview

Efficient management of cluster nodes is crucial for maintaining the stability and performance of a Kubernetes environment. The following operations assist in node lifecycle management:

• Cordoning a node marks it as unschedulable, preventing the deployment of new resources while allowing existing workloads to continue operating.
• Draining a node not only marks the node as unschedulable but also safely migrates existing volume targets to other nodes.
• Deleting a node (offline) removes an irrecoverable node from the Puls8 Replicated PV Mayastor control plane when the node can no longer be reached or recovered.

This document describes how to use the kubectl puls8 mayastor CLI to perform cordon, uncordon, drain, and offline node deletion operations.

Cordon a Node

Cordoning a node marks it as unschedulable, preventing the Kubernetes scheduler from placing new workloads on the node while allowing existing workloads and storage resources to continue running undisturbed. This operation is commonly used when a node requires maintenance, is under investigation, or needs to be taken out of rotation without disrupting active workloads.

To add a label and cordon a node:

kubectl puls8 mayastor cordon node <node_name> <label>

To retrieve a list of all cordoned nodes:

kubectl puls8 mayastor get cordon nodes

 ID  GRPC ENDPOINT  STATUS  VERSION  CORDON LABELS
(no rows - no nodes are currently cordoned)

To view the labels associated with a specific cordoned node:

kubectl puls8 mayastor get cordon node <node_name>

 ID                  GRPC ENDPOINT      STATUS  VERSION  CORDON LABELS
 node-0-puls8-20851  5.223.65.31:10124  Online  v2.12.2

Uncordon a Node

Uncordoning a node restores a cordoned node to schedulable status, allowing the Kubernetes scheduler to place new workloads on the node again. This operation is typically performed once the maintenance is completed and the node is ready to resume normal operations.

To uncordon a node:

kubectl puls8 mayastor uncordon node <node_name> <label>

Drain a Node

Draining a node safely relocates all volume targets from the node and prevents new workloads from being scheduled on it. This operation is commonly performed before taking a node offline for planned maintenance or upgrades to ensure workloads remain highly available throughout the process.

To initiate the drain operation on a node:

kubectl puls8 mayastor drain node <node_name> <label>

To retrieve a list of nodes that have undergone a drain operation:

kubectl puls8 mayastor get drain nodes

 ID  GRPC ENDPOINT  STATUS  VERSION  DRAIN STATE  DRAIN LABELS
(no rows - no nodes are currently draining)

To stop an ongoing drain operation or make a drained node schedulable again:

kubectl puls8 mayastor uncordon node <node_name> <label>

Delete a Node (Offline)

Offline node deletion, referred to as purge, allows you to remove an unreachable Replicated PV Mayastor node from the control plane when the node is no longer reachable and cannot be recovered. Purge removes the node's representation from the control plane along with associated resources that are no longer accessible on that node, without requiring communication with the underlying host.

This operation is intended for failure scenarios where the node is considered irrecoverably lost and cannot be deleted through normal means that require access to the node.

Offline node deletion is irreversible and may result in permanent loss of access to volume or snapshot data. Purge is not part of the normal node lifecycle and should be used only when you are certain that the node cannot be recovered.

Before performing purge, ensure that:

• The node is offline.
• The node is cordoned to prevent new workloads from being scheduled.
• The openebs.io/engine label has been removed from the Kubernetes node.

Before purge, cordon the node and remove the openebs.io/engine label.

kubectl puls8 mayastor cordon node <node name> puls8-offline=true -n puls8
kubectl label node <node_name> openebs.io/engine-

Node node-0-469923 cordoned successfully
node/node-0-469923 unlabeled

When to Use Purge

Use purge only when:

• The node is no longer accessible and cannot be recovered.
• The host running the node is permanently unavailable.
• Storage resources on the node are lost or inaccessible.

Do not use purge for temporary failures or conditions where the node may become available again.

Verify the Node State

Before proceeding, confirm that the target node is in an Offline state.

kubectl puls8 mayastor get nodes -n puls8

ID             GRPC ENDPOINT       STATUS   VERSION    POOLS  VOLUMES  SNAPSHOTS
node-0-469923  5.223.46.235:10124  Offline  v2.11.0    1      2        0
node-1-469923  5.223.44.23:10124   Online   v2.11.0    1      1        0
node-2-469923  5.223.46.36:10124   Online   v2.11.0    1      1        0

Review Purge Impact

Before performing purge, review the affected replicas and workloads.

kubectl puls8 mayastor delete node <node name> -n puls8 --purge --show-impact

NODE               STATUS      REPLICAS    VOLUMES     READY
node-0-469923      Unknown     1           1           true

When the node holds pools, a further table follows the summary, breaking the impact down for each pool.

To review the replica topology before purge:

kubectl puls8 mayastor get volume-replica-topologies -n puls8

VOLUME-ID                             REPLICA-ID                            NODE           POOL                STATUS   ENCRYPTED  CAPACITY  ALLOCATED  SNAPSHOTS  CHILD-STATUS  REASON  REBUILD  HEALTHY 
f373835e-4ada-4eec-a4f3-b35f1be7817b  b63e1f76-22ce-4c9f-8d3c-196b7ecb503e  node-2-469923  pool-node-2-469923  Online   false      1 GiB     1 GiB      0 B        Online        <none>  <none>   true 
├─                                    ca275563-e044-46e3-ad60-22b5701223cb  node-0-469923  pool-node-0-469923  Unknown  false      <none>    <none>     <none>     Faulted       <none>  <none>   false 
└─                                    ff544d71-9fa6-461a-85e6-a9e3e922d339  node-1-469923  pool-node-1-469923  Online   false      1 GiB     1 GiB      0 B        Online        <none>  <none>   true 
ec307c1d-3311-4768-9a60-eb457b84277d  2961803b-01c2-49bf-b8bf-c6014f336f96  node-0-469923  pool-node-0-469923  Unknown  false      <none>    <none>     <none>     <none>        <none>  <none>   true

Running --show-impact is a pre-check and is not required. A purge that would cause data loss is rejected, and the error identifies the affected volumes and snapshots in the same detail. Refer to Data Loss Reported by a Rejected Purge for the reported columns.

Data Loss Reported by a Rejected Purge

A purge that would destroy the last healthy copy of a volume or a snapshot is rejected unless the corresponding confirmation flag is supplied. The error identifies exactly what would be lost, under two headings, each shown only when it has entries.

Volumes That Would Lose Their Last Healthy Replica

Listed when purging the node would leave a volume with no healthy replica. Accepting this loss requires --accept-volume-loss.

Snapshots That Would Lose Their Last Replica Snapshot

Listed when purging the node would leave a snapshot with no replica snapshot. Accepting this loss requires --accept-snapshot-loss, which can only be used together with --accept-volume-loss.

Review both listings before supplying a confirmation flag. Once the purge proceeds, any volume reporting 0 under HEALTHY-AFTER is unrecoverable.

Delete/Purge an Offline Node

Use the following command to permanently remove an irrecoverable node from the control plane.

kubectl puls8 mayastor delete node <node name> -n puls8 --purge --yes

To purge the node and remove the DiskPoolClaim and DiskPool custom resources (CRs) of the pools on it in a single operation, add the --cleanup-dsp option.

kubectl puls8 mayastor delete node <node name> -n puls8 --purge --yes --cleanup-dsp

Purge Options

If the node contains critical resources, additional confirmation flags may be required.

What the Cleanup Removes

With --cleanup-dsp, the Kubernetes resources that describe the pools of the node are removed after the node is deleted from the control plane.

• Every DiskPoolClaim placed on the node is deleted, which in turn removes the DiskPool of each claim.
• This includes a claim whose DiskPool was never created. Such a claim would otherwise remain, reconciling against a node that no longer exists.
• Any DiskPool that the claims do not account for is then deleted directly.

Each deletion waits for the pool operator to process the finalizer before the command returns, up to --cleanup-dsp-timeout.

A run that includes --show-impact only reports what a purge would do, and never performs the cleanup. Use --ignore-not-found to suppress the error when the node has already been removed; the cleanup of any remaining claims or DiskPools still runs.

Data Loss Confirmation

If the purge operation impacts the last healthy replica of a volume, the following message is displayed:

Volumes would lose their last healthy replica. Use --accept-volume-loss to proceed,
or --accept-data-loss to also accept snapshot loss in a single flag.

To continue the purge operation with data loss acknowledged:

kubectl puls8 mayastor delete node <node name> -n puls8 --purge --yes --accept-data-loss

NODE               VOLUME-LOSS     SNAPSHOT-LOSS 
node-0-469923      1 volume(s)     <none>

Impact on Workloads

Verify Purge Completion

After purge, verify that the node has been removed and that affected workloads reflect the updated state.

kubectl puls8 mayastor get nodes

 ID                  GRPC ENDPOINT       STATUS  VERSION
 node-2-puls8-20851  5.223.68.243:10124  Online  v2.12.2
 node-1-puls8-20851  5.223.67.79:10124   Online  v2.12.2
 node-0-puls8-20851  5.223.65.31:10124   Online  v2.12.2

Expected Results

• The purged node is no longer listed.
• Associated pools are no longer listed.
• Affected workloads reflect updated states.

Benefits of Node Cordon, Drain, and Delete Operations

• Prevents new workload deployments on unhealthy or under-maintenance nodes, ensuring stability.
• Safely migrates workloads off a node during maintenance or upgrades, avoiding downtime.
• Helps in node maintenance and upgrades without disrupting existing workloads.
• Removes irrecoverable or orphaned nodes from the control plane after host, storage, or infrastructure failures.
• Provides a managed method to remove nodes when standard deletion is no longer possible.

Learn More

• DataCore Puls8 Kubectl Plugin
• Volume Snapshots