StorageClass Parameters

Explore this Page

• Overview
• Common StorageClass Parameters
• StorageClass Parameters for Replicated PV Mayastor
• StorageClass Parameters for Local PV Hostpath
• StorageClass Parameters for Local PV LVM
• StorageClass Parameters for Local PV ZFS

Overview

This document provides a comprehensive guide to configuring StorageClass parameters for provisioning replicated Persistent Volumes (PVs) in DataCore Puls8. These parameters play a crucial role in defining how storage volumes behave, including aspects like filesystem type, replication levels, provisioning strategy (thick/thin), and support for volume expansion. In addition, it details topology-aware configurations to optimize the placement of volume replicas across nodes and pools in a Kubernetes cluster.

Beyond Replicated PV Mayastor, the document also outlines configuration parameters for Local PVs, enabling customized local storage provisioning with options like mount control, file system selection, and storage type enforcement.

Common StorageClass Parameters

This section describes the common StorageClass parameters supported for both Local Storage and Replicated Storage. These parameters are essential to determine the behavior of volume provisioning and binding.

Provisioner (Required)

Specifies the external Container Storage Interface or Container Native Storage driver responsible for provisioning volumes. The provisioner value must match the storage being used. This field is required for the StorageClass to function properly.

provisioner: io.openebs.csi-mayastor

provisioner: openebs.io/local

The provisioner is responsible for managing the lifecycle of persistent volumes, including creation, attachment, detachment, and deletion. Always ensure the specified provisioner matches the installed storage engine in your cluster.

Reclaim Policy (Optional)

Reclaim Policy defines what happens to a PV after its associated PVC is deleted. If not specified, the default policy is Delete.

• Delete: Automatically deletes the persistent volume when the associated PVC is deleted.
• Retain: Retains the persistent volume and its data for manual recovery or reattachment.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-mayastor-1
provisioner: io.openebs.csi-mayastor
parameters:
  protocol: nvmf
  repl: "3"
  fsType: "ext4"
reclaimPolicy: Delete          ## Reclaim policy can be specified here. It also accepts Retain

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  vgpattern: "lvmvg.*"
reclaimPolicy: Delete          ## Reclaim policy can be specified here. It also accepts Retain

StorageClass Parameters for Replicated PV Mayastor

The following parameters are commonly used to control the behavior of volumes provisioned through a StorageClass in DataCore Puls8. These define core attributes such as the file system type, replication, provisioning, and expansion capabilities.

repl (Optional)

Indicates the desired replication factor. This value must be a number greater than zero. A value of 1 means no fault tolerance, 2 tolerates one node failure, 3 tolerates two node failures, and so on. The default repl value is 1.

protocol (Optional)

Defines the protocol for mounting the volume. Currently supports nvmf (NVMe over TCP protocol).

thin (Optional)

Enables thin provisioning of volumes when set to "true". Thin provisioning allows dynamic allocation of storage. Monitoring is required to prevent degradation or faults due to space exhaustion. Additional configurations can be set using the openebs.mayastor.agents.core.capacity.thin spec in the Helm chart:

• poolCommitment: Maximum allowed pool commitment (%). The default value is 250%.
• volumeCommitment: Minimum free space (%) required in each replica pool for new replicas of an existing volume. The default value is 40%.
• volumeCommitmentInitial: Minimum free space (%) required in each replica pool for new volume creation. The default value is 40%.

The volumes can either be thick or thin provisioned. Use thin: "true" in environments with limited capacity or when using snapshots/cloning.

encrypted (Optional)

Enables encryption of volumes when set to "true". Encrypted volumes are provisioned only if a sufficient number of encrypted pools are available, as required by the repl (replication factor) setting in the StorageClass.

snapshotRestorePolicy (Optional)

Controls how a volume restore from a snapshot behaves when not every replica pool can host a clone of the source snapshot. The supported values are strict and bestEffort. If the parameter is not specified, strict is used.

• strict: Every requested replica must be cloned from the snapshot. If any of the replica pools of the snapshot cannot host a clone, for example because a source pool has run out of space, the restore fails.
• bestEffort: The restore proceeds as long as at least one clone succeeds. The volume comes up under-replicated, and the remaining replicas are created through a normal rebuild.

A volume restored using bestEffort does not tolerate the number of node failures implied by its repl value until the outstanding rebuilds are complete. Refer to the Restore a Volume from a Snapshot documentation for more information.

fsType (Optional)

Specifies the file system to use when mounting the volume. Supported file systems are ext4 (Default), xfs, and btrfs.

It is recommended to use xfs for better performance. Ensure the required filesystem driver is installed on all worker nodes in the cluster before use.

formatOptions (Optional)

Allows you to specify additional formatting options when initializing the device with a file system. By default, Replicated PV Mayastor uses ext4 to format the devices. Based on fsType parameter (Example: xfs, btrfs), refer to the Linux Documentation for supported formatting options.

overrideGlobalFormatOpts (Optional)

Overrides the global XFS formatting options defined via Helm values. In certain environments, Helm charts may configure global xfs format options which get applied to all volumes using the XFS file system.

To override these global options for a specific volume, set overrideGlobalFormatOpts: true in the StorageClass and define the custom options via formatOptions. This ensures the provided formatOptions are used instead of the global settings.

If both global Helm options and per-volume formatOptions are specified, Replicated PV Mayastor applies both sets of options together unless overrideGlobalFormatOpts is explicitly set to true.

Example

If the global Helm configuration sets -m bigtime=0 -m inobtcount=0 and you wish to override these settings for a specific volume with -m bigtime=1 -m inobtcount=1, then the following parameters should be specified in the StorageClass:

overrideGlobalFormatOpts: true
formatOptions: "-m bigtime=1 -m inobtcount=1"

allowVolumeExpansion (Optional)

Enables expansion of PVs through PVCs. Set this parameter to true in the StorageClass. To expand, edit the PVC size. Refer to the Volume Resize Documentation for more information.

nodeAffinityTopologyLabel (Optional)

Places replicas only on nodes with labels that exactly match those defined in the StorageClass.

StorageClass Definition

cat <<EOF | kubectl create -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: puls8-mayastor-1
parameters:
  protocol: nvmf
  repl: "2"
  nodeAffinityTopologyLabel: |
    zone: us-west-1
provisioner: io.openebs.csi-mayastor
volumeBindingMode: Immediate
EOF

Apply Node Labels

kubectl puls8 mayastor label node worker-node-1 zone=us-west-1
kubectl puls8 mayastor label node worker-node-2 zone=eu-east-1
kubectl puls8 mayastor label node worker-node-3 zone=us-west-1

kubectl puls8 mayastor get nodes -n puls8 --show-labels

ID                  GRPC ENDPOINT       STATUS  VERSION  POOLS  VOLUMES  SNAPSHOTS  LABELS
node-0-puls8-20741  5.223.46.14:10124   Online  v2.12.2  1      0        0
node-1-puls8-20741  5.223.45.255:10124  Online  v2.12.2  1      0        0
node-2-puls8-20741  5.223.45.56:10124   Online  v2.12.2  1      0        0

nodeHasTopologyKey (Optional)

Places replicas on nodes that have label keys matching the provided key, regardless of their values.

StorageClass Definition

cat <<EOF | kubectl create -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: puls8-mayastor-1
parameters:
  protocol: nvmf
  repl: "2"
  nodeHasTopologykey: |
    rack
provisioner: io.openebs.csi-mayastor
volumeBindingMode: Immediate
EOF

Apply Node Labels

kubectl puls8 mayastor label node worker-node-1 rack=1
kubectl puls8 mayastor label node worker-node-2 rack=2
kubectl puls8 mayastor label node worker-node-3 rack=2

 kubectl puls8 mayastor get nodes -n puls8 --show-labels
 ID             GRPC ENDPOINT        STATUS  LABELS
 worker-node-1  65.108.91.181:10124  Online  rack=1
 worker-node-3  65.21.4.103:10124    Online  rack=2
 worker-node-3  37.27.13.10:10124    Online  rack=2

nodeSpreadTopologyKey (Optional)

Ensures that replicas are distributed across nodes sharing a common key but having different values.

StorageClass Definition

cat <<EOF | kubectl create -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: mayastor-1
parameters:
  protocol: nvmf
  repl: "2"
  nodeSpreadTopologyKey: |
    zone
provisioner: io.openebs.csi-mayastor
volumeBindingMode: Immediate
EOF

Apply Node Labels

kubectl puls8 mayastor label node worker-node-1 zone=us-west-1
kubectl puls8 mayastor label node worker-node-2 zone=eu-east-1
kubectl puls8 mayastor label node worker-node-3 zone=us-west-1

kubectl puls8 mayastor get nodes -n puls8 --show-labels
ID             GRPC ENDPOINT        STATUS  LABELS
worker-node-1  65.108.91.181:10124  Online  zone=eu-west-1
worker-node-3  65.21.4.103:10124    Online  zone=eu-east-1
worker-node-3  37.27.13.10:10124    Online  zone=us-west-1

poolAffinityTopologyLabel (Optional)

Places replicas only on pools with labels that exactly match the values provided.

StorageClass Definition

cat <<EOF | kubectl create -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: puls8-mayastor-1
parameters:
  protocol: nvmf
  repl: "2"
  poolAffinityTopologyLabel: |
    zone: us-west-1
provisioner: io.openebs.csi-mayastor
volumeBindingMode: Immediate
EOF

Label Pools via DiskPool Definitions

cat <<EOF | kubectl create -f -
apiVersion: "openebs.io/v1beta2"
kind: DiskPool
metadata:
  name: pool-on-node-0
  namespace: mayastor
spec:
  node: worker-node-0
  disks: ["/dev/sdb"]
  topology:
    labelled:
        zone: us-west-1
---
apiVersion: "openebs.io/v1beta2"
kind: DiskPool
metadata:
  name: pool-on-node-1
  namespace: mayastor
spec:
  node: worker-node-1
  disks: ["/dev/sdb"]
  topology:
    labelled:
        zone: us-east-1
---
apiVersion: "openebs.io/v1beta2"
kind: DiskPool
metadata:
  name: pool-on-node-2
  namespace: mayastor
spec:
  node: worker-node-2
  disks: ["/dev/sdb"]
  topology:
    labelled:
        zone: us-west-1
EOF

poolHasTopologyKey (Optional)

Selects pools with label keys that match the key specified in the StorageClass.

StorageClass Definition

cat <<EOF | kubectl create -f -
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: puls8-mayastor-1
parameters:
  protocol: nvmf
  repl: "2"
  poolHasTopologykey: |
    zone
provisioner: io.openebs.csi-mayastor
volumeBindingMode: Immediate
EOF

Filter Pools Based on Labels

kubectl puls8 mayastor get pools -n puls8 --selector zone=eu-west-1
ID             GRPC ENDPOINT        STATUS  LABELS
ID              DISKS                                                     MANAGED  NODE           STATUS  CAPACITY  ALLOCATED  AVAILABLE  COMMITTED
pool-on-node-0  aio:///dev/sdb?uuid=b7779970-793c-4dfa-b8d7-03d5b50a45b8  true     worker-node-0  Online  10GiB     0 B        10GiB      0 B
pool-on-node-2  aio:///dev/sdb?uuid=b7779970-793c-4dfa-b8d7-03d5b50a45b8  true     worker-node-2  Online  10GiB     0 B        10GiB      0 B

kubectl puls8 mayastor get pools -n puls8 --selector zone=eu-east-1
ID             GRPC ENDPOINT        STATUS  LABELS
ID              DISKS                                                     MANAGED  NODE           STATUS  CAPACITY  ALLOCATED  AVAILABLE  COMMITTED
pool-on-node-1  aio:///dev/sdb?uuid=b7779970-793c-4dfa-b8d7-03d5b50a45b8  true     worker-node-1  Online  10GiB     0 B        10GiB      0 B

stsAffinityGroup (Optional)

Groups volumes associated with StatefulSet pods to prevent single points of failure. The following rules are enforced:

• Anti-affinity among single-replica volumes
• Optimized distribution for multi-replica volumes
• Anti-affinity for volume targets

To enable, set stsAffinityGroup: true in the StorageClass YAML.

For multi-replica volumes that are part of a stsAffinityGroup, scaling down is permitted only up to two replicas. Reducing the replica count below two is not supported.

Volume Affinity Group Scale-Down Restrictions

When using stsAffinityGroup, replicas of volumes belonging to the same StatefulSet are distributed across different nodes to avoid a single point of failure. Because of these anti-affinity rules, scaling a volume down to 1 replica may be restricted if doing so would place the last remaining replica on a node that already hosts another single-replica volume from the same affinity group.

A scale-down to 1 replica is allowed only when the current replicas are already placed on different nodes. If the replicas end up on the same node, for example, after scaling from 3 replicas to 2, the system may block the scale-down until the placement is improved.

If a scale-down is blocked, you can resolve it by temporarily scaling the volume up to add a replica whilst the volume is published and then scaling down again. This reshuffles the replicas to meet the affinity group’s placement rules.

These restrictions ensure that a single node failure does not impact multiple StatefulSet instances, preserving fault isolation and reliability for applications using affinity-grouped volumes.

cloneFsIdAsVolumeId (Optional)

Controls how the UUID of a cloned/restored volume is handled:

• true: The clone/restore receives a new UUID.
• false (default): The clone retains the original UUID and is mounted using the nouuid flag.

Set to true when using btrfs and concurrent mounts on the same node are expected.

StorageClass Parameters for Local PV Hostpath

These parameters allow customization of HostPath volumes, enabling configurations such as storage type, custom base paths, node affinity control, volume directory permissions, topology-constrained placement, and XFS or EXT4 quota management. The default StorageClass is called local-hostpath and its BasePath is configured as /var/openebs/local.

StorageType (Required)

Defines the type of backend storage used by the Local PV. For HostPath volumes, this must be set to hostpath.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-local-hostpath
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: hostpath
      - name: BasePath
        value: /var/local-hostpath
provisioner: openebs.io/local
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

BasePath (Optional)

The BasePath parameter defines the root directory on each node where DataCore Puls8 provisions local volumes. By default, the Local PV Hostpath provisioner uses /var/openebs/local as the base path. You can customize this path in your StorageClass definition to meet your storage architecture or policy requirements.

If BasePath does not exist on the node, Dynamic Local PV Provisioner will attempt to create the directory, when the first local volume is scheduled on to that node. You must ensure that the value provided for BasePath is a valid absolute path.

A BasePath supplied through a cas.openebs.io/config annotation on a PersistentVolumeClaim is ignored. This prevents a user who is able to create PersistentVolumeClaims from choosing the directory on the node where the volume is created. Set BasePath on the StorageClass instead. To allow a PersistentVolumeClaim to override it, set the openebs.localpv-provisioner.localpv.allowInsecurePvcBasePathOverride Helm value to true. It is disabled by default.

NodeAffinityLabels (Optional)

Specifies custom node labels to restrict volume provisioning to specific nodes matching these labels.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-local-hostpath
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: "hostpath"
      - name: NodeAffinityLabels
        list:
          - "openebs.io/custom-node-unique-id"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer

Using NodeAffinityLabels does not influence the scheduling of the application pod. Use allowedTopologies to constrain the scheduling of the pod as well.

FilePermissions (Optional)

By default, Local PV Hostpath creates the volume directory with 0777 permissions. For some workloads these permissions are wider than necessary. Use the FilePermissions config to set the permissions that the volume directory is created with, through the mode key.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-local-hostpath
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: "hostpath"
      - name: BasePath
        value: "/var/local-hostpath"
      - name: FilePermissions
        data:
          mode: "0770"
provisioner: openebs.io/local
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

With the above StorageClass, the directory of every volume provisioned by it is created with 0770 permissions.

The permissions are applied when the volume directory is created, so changing FilePermissions later does not affect volumes that already exist. FilePermissions cannot be set through the Helm chart values for the default StorageClass. To use it, create a custom StorageClass as shown above.

XFSQuota (Optional)

Enables support for XFS project quotas on XFS-formatted HostPath volumes. Useful for enforcing space usage limits per volume. Refer to the XFS Quota documentation for the full procedure.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-hostpath-xfs
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: "hostpath"
      - name: BasePath
        value: "/var/openebs/local/"
      - name: XFSQuota
        enabled: "true"
        data:
          softLimitGrace: "0%"
          hardLimitGrace: "0%"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

EXT4Quota (Optional)

Enforces a project quota on the volume directory in the same way as XFSQuota, for a BasePath that is held on an ext4 file system. It accepts the same softLimitGrace and hardLimitGrace keys.

EXT4 Quota requires the project quota feature to be enabled on the ext4 file system, and that file system to be mounted with the prjquota mount option. Refer to the EXT4 Quota documentation for the full procedure.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-hostpath-ext4
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: "hostpath"
      - name: BasePath
        value: "/var/openebs/local/"
      - name: EXT4Quota
        enabled: "true"
        data:
          softLimitGrace: "0%"
          hardLimitGrace: "0%"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

allowedTopologies (Optional)

By default, a Local PV Hostpath volume can be provisioned on any node in the cluster. If the BasePath is present on certain nodes only, use the standard Kubernetes allowedTopologies field to list the nodes where the path is available. Volumes of that StorageClass are then provisioned and scheduled on those nodes only.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-local-hostpath
  annotations:
    openebs.io/cas-type: local
    cas.openebs.io/config: |
      - name: StorageType
        value: "hostpath"
      - name: BasePath
        value: "/var/local-hostpath"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: kubernetes.io/hostname
    values:
      - worker-2
      - worker-3

The above StorageClass declares that the BasePath is available on nodes worker-2 and worker-3 only.

Unlike NodeAffinityLabels, allowedTopologies also influences the scheduling of the application pod. To set allowedTopologies on the StorageClass that the Helm chart creates, use the openebs.localpv-provisioner.hostpathClass.allowedTopologies value.

StorageClass Parameters for Local PV LVM

These parameters allows customization of features such as volume expansion, mount options, file systems, volume sharing, and more.

IOPS and bandwidth limits are not set through the StorageClass. They are defined in a Kubernetes VolumeAttributesClass and attached to a PVC, which allows one StorageClass to serve workloads with different performance profiles. Refer to Volume Quality of Service (QoS) for more information.

volgroup or vgpattern (Required)

Either volgroup or vgpattern must be provided to specify volume group identification.

• volgroup: Required if vgpattern is not provided.
• vgpattern: Required if volgroup is not provided.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  volgroup: "lvmvg"       ## volgroup specifies name of lvm volume group

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  vgpattern: "lvmvg.*"     ## vgpattern specifies pattern of lvm volume group name

It is recommended to use vgpattern since volumegroup will be deprecated in future.

AllowVolumeExpansion (Optional)

To enable volume expansion, set the allowVolumeExpansion field to true in the StorageClass definition. If not specified, volume expansion is not supported.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
allowVolumeExpansion: true  # If set to true then dynamically it allows expansion of volume
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  vgpattern: "lvmvg.*"

MountOptions (Optional)

Volumes provisioned via Local PV LVM can be mounted with specified options in the StorageClass. If not specified, the -o default option is used. Invalid mount options may cause volume mount failures.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  vgpattern: "lvmvg.*"
mountOptions:
  - debug  # Various mount options of volume can be specified here

A raw block volume is attached to the pod as a block device and is not mounted with a file system. Mount options are therefore not applied to raw block volumes.

FsType (Optional)

Defines the file system type for the volume. If not specified, it defaults to ext4.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
allowVolumeExpansion: true
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  vgpattern: "lvmvg.*"
  fsType: xfs               ## Supported filesystems are ext2, ext3, ext4, xfs & btrfs

FormatOptions (Optional)

Use the formatOptions parameter to pass additional options to the mkfs command that formats the volume with the file system specified by fsType. Provide the options as a single space-separated string. Refer to the documentation of the file system in use for the format options that it supports.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  volgroup: "lvmvg"
  formatOptions: "-b 4096 -N 5000000"      ## Extra mkfs options for filesystem volumes

The options are applied only while the volume is being formatted, which happens the first time the volume is mounted. Changing formatOptions in the StorageClass has no effect on volumes that are already formatted.

Format options are not validated by the driver. If the options are not valid for the chosen file system, formatting fails and the volume does not mount. A raw block volume is not formatted with a file system. Format options are therefore not applied to raw block volumes.

Node Agent Default Format Options

The node agent can hold a default set of mkfs options for each file system, applied when a StorageClass does not set formatOptions. Configure it through the Helm values:

lvmNode:
  defaultFormatOptions:
    xfs: "-i nrext64=0"

A value set in the StorageClass replaces the node default for that file system; the two are not merged. An entry naming an unknown file system causes the node agent to fail at startup rather than format volumes that the node cannot mount.

mkfs.xfs 6.5 and later enable the nrext64 feature by default, and only Linux kernel 5.19 and later can mount a file system that has it. On a cluster where any node runs an older kernel - including RHEL 8 and 9, Ubuntu 20.04 and 22.04, and SLES 15 - an XFS volume formatted by a newer mkfs.xfs fails to mount on that node with Superblock has unknown incompatible features (0x20) enabled. Set the XFS node agent default so that volumes are formatted without the feature. The default applies only when a volume is formatted, so volumes that already exist are unaffected. A cluster where every node runs kernel 5.19 or later requires no change.

Shared (Optional)

To allow multiple pods on the same node to share a volume, set shared to yes. The parameter accepts yes and no, and the default value is no.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-shared-lvmsc
allowVolumeExpansion: true
provisioner: local.csi.openebs.io
parameters:
  volgroup: "lvmvg"
  shared: "yes"             ## Parameter that states volume can be shared among multiple pods

ThinProvision (Optional)

To enable thin provisioning, set thinProvision to yes (default is no). Ensure the dm_thin_pool kernel module is loaded before using thin provisioning.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  volgroup: "lvmvg"
  thinProvision: "yes"      ## Parameter that enables thinprovisioning

Verify if Thin Provisioning Module is Loaded:

lsmod | grep dm_thin_pool

dm_thin_pool           73728  0
dm_persistent_data     90112  1 dm_thin_pool
dm_bio_prison          20480  1 dm_thin_pool

If not loaded, execute:

modprobe dm_thin_pool

dm_thin_pool           73728  0
dm_persistent_data     90112  1 dm_thin_pool
dm_bio_prison          20480  1 dm_thin_pool

Scheduler (Optional)

The scheduler parameter selects the algorithm that the Local PV LVM driver uses to choose the node on which a volume is provisioned. Only the volume groups matching the volgroup or vgpattern parameter are considered. If the parameter is not specified, SpaceWeighted is used.

• SpaceWeighted: Selects the node that has a volume group with the highest free space.
• CapacityWeighted: Selects the node containing a volume group that has the least allocated storage in terms of capacity.
• VolumeWeighted: Selects the node containing a volume group that has the least number of volumes provisioned on it.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  volgroup: "lvmvg"
  scheduler: "CapacityWeighted"      ## SpaceWeighted (default), CapacityWeighted, or VolumeWeighted

The scheduling algorithm accounts only for the volume groups, and does not consider other factors such as available CPU or memory. If the application pod has node selector or affinity rules, or CPU and memory constraints, use the Kubernetes scheduler instead by setting volumeBindingMode to WaitForFirstConsumer.

VolumeBindingMode (Optional)

The volumeBindingMode determines when and how a PV is bound to a PersistentVolumeClaim (PVC).

• Immediate: Volume binding and dynamic provisioning occur as soon as the PVC is created.
• WaitForFirstConsumer (Late Binding): Binding and provisioning of the PVC are delayed until a pod requesting the PVC is created.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  vgpattern: "lvmvg.*"
volumeBindingMode: WaitForFirstConsumer     ## It can also replaced by Immediate volume binding mode depending on the use case.

VolumeBindingMode "Immediate" is not supported for Local PV Hostpath.

StorageClass with Custom Node Labels

To assign volumes to specific nodes based on available volume groups, use allowedTopologies.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: puls8-lvm-sc
allowVolumeExpansion: true
parameters:
  volgroup: "lvmvg"
provisioner: local.csi.openebs.io
allowedTopologies:
  - matchLabelExpressions:
    - key: openebs.io/nodename
      values:
        - node-1
        - node-2

VolumeGroup Availability

If the LVM volume group is available only on certain nodes, use allowedTopologies to specify those nodes.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-lvmpv
allowVolumeExpansion: true
parameters:
  storage: "lvm"
  volgroup: "lvmvg"
provisioner: local.csi.openebs.io
allowedTopologies:
- matchLabelExpressions:
  - key: kubernetes.io/hostname
    values:
      - lvmpv-node1
      - lvmpv-node2

The provisioner name for the LVM driver is "local.csi.openebs.io"; use this while creating the StorageClass to ensure volume provisioning requests are correctly routed.

StorageClass Parameters for Local PV ZFS

These parameters define essential aspects of the storage configuration, such as volume creation and operation while additional optional parameters such as FsType, recordsize, compression, and deduplication - enable further customization.

Poolname (Required)

The poolname parameter specifies the name of the storage pool where the volume is created. This parameter is required and can either refer to the root dataset or a child dataset. The dataset provided under poolname must exist on all nodes with the same name specified in the StorageClass.

poolname: "zfspv-pool"
poolname: "zfspv-pool/child"

FsType (Optional)

Defines the file system type for the volume. If FsType is set to zfs, the driver creates a ZFS dataset, and no additional formatting is required. If set to ext2, ext3, ext4, btrfs, or xfs, the driver creates a ZVOL and formats the volume accordingly. This parameter cannot be modified once the volume has been provisioned. If omitted, Kubernetes defaults to ext4.

"zfs", "ext2", "ext3", "ext4", "xfs", "btrfs"

FormatOptions (Optional)

Use the formatOptions parameter to pass additional options to the mkfs command that formats the volume with the file system specified by FsType. Provide the options as a single space-separated string. Refer to the documentation of the file system in use for the format options that it supports.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: puls8-zfspv
provisioner: zfs.csi.openebs.io
parameters:
  poolname: "zfspv-pool"
  fstype: "xfs"
  formatOptions: "-i nrext64=0"      ## Extra mkfs options for filesystem volumes

The options are applied only while the volume is being formatted, which happens the first time the volume is mounted. Changing formatOptions in the StorageClass has no effect on volumes that are already formatted.

Format options are not validated by the driver. If the options are not valid for the chosen file system, formatting fails and the volume does not mount.

This parameter applies only to file system volumes. When FsType is zfs, the driver creates a ZFS dataset and no formatting takes place, so the parameter is ignored.

Node Agent Default Format Options

The node agent can hold a default set of mkfs options for each file system, applied when a StorageClass does not set formatOptions. Configure it through the Helm values:

zfsNode:
  defaultFormatOptions:
    xfs: "-i nrext64=0"

A value set in the StorageClass replaces the node default for that file system; the two are not merged. An entry naming an unknown file system causes the node agent to fail at startup rather than format volumes that the node cannot mount.

mkfs.xfs 6.5 and later enable the nrext64 feature by default, and only Linux kernel 5.19 and later can mount a file system that has it. On a cluster where any node runs an older kernel - including RHEL 8 and 9, Ubuntu 20.04 and 22.04, and SLES 15 - an XFS volume formatted by a newer mkfs.xfs fails to mount on that node with Superblock has unknown incompatible features (0x20) enabled. Set the XFS node agent default so that volumes are formatted without the feature. The default applies only when a volume is formatted, so volumes that already exist are unaffected. A cluster where every node runs kernel 5.19 or later requires no change.

Recordsize (Optional)

Applicable only when FsType is set to zfs. This parameter specifies the suggested block size for files stored in the filesystem.

Any power of 2 from 512 bytes to 128 KB

Volblocksize (Optional)

When FsType is anything other than zfs, a ZVOL (a raw block device carved from the ZFS pool) is created. The volblocksize parameter defines the block size for the ZVOL. The volume size must be a multiple of volblocksize and cannot be zero.

Any power of 2 from 512 bytes to 128 KB

Compression (Optional)

The compression parameter specifies the block-level compression algorithm to be applied to the ZFS volume and datasets. Setting it to on enables ZFS to use the default compression algorithm.

"on", "off", "lzjb", "zstd", "zstd-fast", "zstd-1" to "zstd-19", "gzip", "gzip-1" to "gzip-9", "zle", "lz4"

Deduplication/Dedup (Optional)

The dedup parameter enables block-level deduplication, reducing redundant data storage.

"on", "off"

Atime (Optional)

The atime parameter controls whether the access time of a file is updated when the file is read. Setting it to off avoids the write traffic that is otherwise generated by reading files, which can improve performance for read-heavy workloads.

This parameter is applicable only when fsType is set to zfs. For any other fsType the driver creates a ZVOL, where atime does not apply, and the value is ignored. If atime is not specified, the volume inherits the value from the parent ZFS pool or dataset.

"on", "off"

Logbias (Optional)

The logbias parameter provides a hint to ZFS about how to handle synchronous requests for the volume. With latency, ZFS uses the separate log devices (SLOG) of the pool, if any, to handle these requests at low latency. With throughput, ZFS does not use the separate log devices and instead optimizes synchronous operations for overall pool throughput.

This parameter applies to both ZFS datasets and ZVOLs. If logbias is not specified, the volume inherits the value from the parent ZFS pool or dataset.

"latency", "throughput"

Thin Provisioning/Thinprovision (Optional)

The thinProvision parameter determines whether space reservation is required for the source volume. If set to yes, the volume is thin-provisioned and can be created even if the ZPOOL lacks sufficient capacity. If set to no, the volume is thick-provisioned, requiring adequate reserved space.

"yes", "no"

Quotatype (Optional)

The quotatype parameter selects the ZFS property that is used to enforce the size of the volume. With quota, the limit applies to the dataset together with everything that it contains, including its snapshots and clones. With refquota, the limit applies only to the data that the dataset itself references, so snapshots and clones are not counted against it.

quotatype also determines the property that is used to reserve space when thinProvision is set to no. With quota, the space is reserved using reservation, and with refquota, it is reserved using refreservation.

This parameter is applicable only when fsType is set to zfs; otherwise it is ignored. quotatype cannot be modified after the volume is provisioned. If it is not specified, the driver uses quota. When a dataset volume is resized, the driver updates the property that is selected here, and for a thick-provisioned dataset it updates the corresponding reservation or refreservation property as well.

"quota", "refquota"

Shared Volume Access/Shared (Optional)

The shared parameter specifies whether the volume can be accessed by multiple pods simultaneously. If not explicitly set to yes, the ZFS-LocalPV Driver restricts the volume to a single pod. The default value is no.

"yes", "no"

Standard StorageClass Fields

In addition to the parameters above, which are set under parameters, Local PV ZFS honors the following standard Kubernetes StorageClass fields. These are set at the top level of the StorageClass, alongside provisioner.

• allowVolumeExpansion: Volumes can be expanded only when this field is set to true. If it is not specified, volume expansion is not supported. Local PV ZFS supports online volume expansion, so the application does not need to be scaled down for the volume to be resized.
• mountOptions: Applied to volumes that are mounted with a file system, which covers both ZFS datasets and formatted ZVOLs. Mount options are not validated; if they are invalid, the volume mount fails.
• volumeBindingMode: Accepts Immediate and WaitForFirstConsumer. Use WaitForFirstConsumer when the application pod has node selector or affinity rules, or CPU and memory constraints, so that Kubernetes schedules the pod first and the driver then provisions the volume on the selected node. Refer to Creating a StorageClass for an example.
• reclaimPolicy: Accepts Delete and Retain, and defaults to Delete. Refer to Common StorageClass Parameters for more information.
• allowedTopologies: If the ZFS pool is available on certain nodes only, use this field to list those nodes. The driver then creates volumes on those nodes only. Refer to Creating a StorageClass for an example.

btrfs does not support online volume resize, so a btrfs volume cannot be resized. A raw block volume is attached to the pod as a block device and is not mounted with a file system. Mount options are therefore not applied to raw block volumes.

Learn More

• Creating a PVC
• High Availability
• Volume Resize