Node Lifecycle Operations

Explore this Page

• Overview
• Cordon a Node
• Uncordon a Node
• Drain a Node
• Delete a Node (Offline)
• Benefits of Node Cordon, Drain, and Delete Operations

Overview

Efficient management of cluster nodes is crucial for maintaining the stability and performance of a Kubernetes environment. The following operations assist in node lifecycle management:

• Cordoning a node marks it as unschedulable, preventing the deployment of new resources while allowing existing workloads to continue operating.
• Draining a node not only marks the node as unschedulable but also safely migrates existing volume targets to other nodes.
• Deleting a node (offline) removes an irrecoverable node from the Puls8 Replicated PV Mayastor control plane when the node can no longer be reached or recovered.

This document describes how to use the kubectl puls8 mayastor CLI to perform cordon, uncordon, drain, and offline node deletion operations.

Cordon a Node

Cordoning a node marks it as unschedulable, preventing the Kubernetes scheduler from placing new workloads on the node while allowing existing workloads and storage resources to continue running undisturbed. This operation is commonly used when a node requires maintenance, is under investigation, or needs to be taken out of rotation without disrupting active workloads.

To add a label and cordon a node:

kubectl puls8 mayastor cordon node <node_name> <label>

To retrieve a list of all cordoned nodes:

kubectl puls8 mayastor get cordon nodes

To view the labels associated with a specific cordoned node:

kubectl puls8 mayastor get cordon node <node_name>

Uncordon a Node

Uncordoning a node restores a cordoned node to schedulable status, allowing the Kubernetes scheduler to place new workloads on the node again. This operation is typically performed once the maintenance is completed and the node is ready to resume normal operations.

To uncordon a node:

kubectl puls8 mayastor uncordon node <node_name> <label>

Drain a Node

Draining a node safely relocates all volume targets from the node and prevents new workloads from being scheduled on it. This operation is commonly performed before taking a node offline for planned maintenance or upgrades to ensure workloads remain highly available throughout the process.

To initiate the drain operation on a node:

kubectl puls8 mayastor drain node <node_name> <label>

To retrieve a list of nodes that have undergone a drain operation:

kubectl puls8 mayastor get drain nodes

To stop an ongoing drain operation or make a drained node schedulable again:

kubectl puls8 mayastor uncordon node <node_name> <label>

Delete a Node (Offline)

Offline node deletion, referred to as purge, allows you to remove an unreachable Replicated PV Mayastor node from the control plane when the node is no longer reachable and cannot be recovered. Purge removes the node's representation from the control plane along with associated resources that are no longer accessible on that node, without requiring communication with the underlying host.

This operation is intended for failure scenarios where the node is considered irrecoverably lost and cannot be deleted through normal means that require access to the node.

Offline node deletion is irreversible and may result in permanent loss of access to volume or snapshot data. Purge is not part of the normal node lifecycle and should be used only when you are certain that the node cannot be recovered.

Before performing purge, ensure that:

• The node is offline.
• The node is cordoned to prevent new workloads from being scheduled.
• The openebs.io/engine label has been removed from the Kubernetes node.

Before purge, cordon the node and remove the openebs.io/engine label.

kubectl puls8 mayastor cordon node <node name> puls8-offline=true -n puls8
kubectl label node <node_name> openebs.io/engine-

Node node-0-469923 cordoned successfully
node/node-0-469923 unlabeled

When to Use Purge

Use purge only when:

• The node is no longer accessible and cannot be recovered.
• The host running the node is permanently unavailable.
• Storage resources on the node are lost or inaccessible.

Do not use purge for temporary failures or conditions where the node may become available again.

Verify the Node State

Before proceeding, confirm that the target node is in an Offline state.

kubectl puls8 mayastor get nodes -n puls8

ID             GRPC ENDPOINT       STATUS   VERSION    POOLS  VOLUMES  SNAPSHOTS
node-0-469923  5.223.46.235:10124  Offline  v2.11.0    1      2        0
node-1-469923  5.223.44.23:10124   Online   v2.11.0    1      1        0
node-2-469923  5.223.46.36:10124   Online   v2.11.0    1      1        0

Review Purge Impact

Before performing purge, review the affected replicas and workloads.

kubectl puls8 mayastor delete node <node name> -n puls8 --purge --show-impact

NODE               STATUS      REPLICAS    VOLUMES     READY
node-0-469923      Unknown     1           1           true

To review the replica topology before purge:

kubectl puls8 mayastor get volume-replica-topologies -n puls8

VOLUME-ID                             REPLICA-ID                            NODE           POOL                STATUS   ENCRYPTED  CAPACITY  ALLOCATED  SNAPSHOTS  CHILD-STATUS  REASON  REBUILD  HEALTHY 
f373835e-4ada-4eec-a4f3-b35f1be7817b  b63e1f76-22ce-4c9f-8d3c-196b7ecb503e  node-2-469923  pool-node-2-469923  Online   false      1 GiB     1 GiB      0 B        Online        <none>  <none>   true 
├─                                    ca275563-e044-46e3-ad60-22b5701223cb  node-0-469923  pool-node-0-469923  Unknown  false      <none>    <none>     <none>     Faulted       <none>  <none>   false 
└─                                    ff544d71-9fa6-461a-85e6-a9e3e922d339  node-1-469923  pool-node-1-469923  Online   false      1 GiB     1 GiB      0 B        Online        <none>  <none>   true 
ec307c1d-3311-4768-9a60-eb457b84277d  2961803b-01c2-49bf-b8bf-c6014f336f96  node-0-469923  pool-node-0-469923  Unknown  false      <none>    <none>     <none>     <none>        <none>  <none>   true

Delete/Purge an Offline Node

Use the following command to permanently remove an irrecoverable node from the control plane.

kubectl puls8 mayastor delete node <node name> -n puls8 --purge --yes

Use the following commands to remove the DiskPoolClaim/DiskPool custom resources (CRs) for the pools on the node.

NS=puls8
NODE=<node-name>

# 1. List every DiskPool on the node BEFORE purging.
POOLS=$(kubectl -n "$NS" get diskpools.openebs.io \
  -o jsonpath="{range .items[?(@.spec.node=='$NODE')]}{.metadata.name} {end}")
echo "Pools on $NODE:" $POOLS

# 2. For every DPC-managed pool, delete its DiskPoolClaim (cascades to the DiskPool).
#    Standalone pools (no DPC) are skipped here and handled in step 3.
for p in $POOLS; do
  if kubectl -n "$NS" get dpc "$p" >/dev/null 2>&1; then
    kubectl -n "$NS" delete dpc "$p" --wait=false
    # Dropping the error here as, if left long enough alone,
    #  the finalizer might be removed by itself.
    kubectl -n "$NS" patch dpc "$p" --type=merge -p '{"metadata":{"finalizers":null}}' 2> /dev/null || true
  fi
done

# 3. Mop up any DiskPool CRs still present — standalone pools, plus any left
#    orphaned by the finalizer patch. Safe now: the DPCs are gone, so nothing
#    will recreate them.
kubectl puls8 -n "$NS" mayastor delete node "$NODE" --purge --yes --cleanup-dsp

Purge Options

If the node contains critical resources, additional confirmation flags may be required.

Data Loss Confirmation

If the purge operation impacts the last healthy replica of a volume, the following message is displayed:

Volumes would lose their last healthy replica. Use --accept-volume-loss to proceed,
or --accept-data-loss to also accept snapshot loss in a single flag.

To continue the purge operation with data loss acknowledged:

kubectl puls8 mayastor delete node <node name> -n puls8 --purge --yes --accept-data-loss

NODE               VOLUME-LOSS     SNAPSHOT-LOSS 
node-0-469923      1 volume(s)     <none>

Impact on Workloads

Verify Purge Completion

After purge, verify that the node has been removed and that affected workloads reflect the updated state.

kubectl puls8 mayastor get nodes

Expected Results

• The purged node is no longer listed.
• Associated pools are no longer listed.
• Affected workloads reflect updated states.

Benefits of Node Cordon, Drain, and Delete Operations

• Prevents new workload deployments on unhealthy or under-maintenance nodes, ensuring stability.
• Safely migrates workloads off a node during maintenance or upgrades, avoiding downtime.
• Helps in node maintenance and upgrades without disrupting existing workloads.
• Removes irrecoverable or orphaned nodes from the control plane after host, storage, or infrastructure failures.
• Provides a managed method to remove nodes when standard deletion is no longer possible.

Learn More

• DataCore Puls8 Kubectl Plugin
• Volume Snapshots