Provisioning NFS PVCs for RWX

Explore this Page

Overview

In Kubernetes environments, shared access to persistent storage is often essential for workloads that need concurrent read/write access across multiple pods or nodes. While block storage is ideal for single-node access, it does not natively support multi-node sharing.

This document outlines the process of provisioning Network File System (NFS)-based Persistent Volume Claims (PVCs) using Replicated PV Mayastor as the backend and exposing it via an NFS server. The setup uses the NFS CSI driver to enable dynamic provisioning and ReadWriteMany (RWX) access modes, supporting scalable and resilient multi-node data sharing.

Concept

Replicated PV Mayastor enables RWX access by serving volumes through an in-cluster NFSv4 server. These volumes are managed using the NFS CSI driver and backed by Replicated PV Mayastor storage. This setup allows multiple applications to share the same volume seamlessly, offering a Kubernetes-native, scalable shared storage solution.

Requirements

NFS Server as a Pod

Running an NFS server within the Kubernetes cluster enables the creation of a shared file system accessible by multiple pods. The NFS server mounts a Persistent Volume created by Replicated PV Mayastor.

NFS CSI Driver

The NFS CSI driver (nfs.csi.k8s.io) integrates with Kubernetes to dynamically provision subdirectories under a specified NFS server path. This enables users to request RWX PVCs in a declarative and automated manner.

Ensure an NFS server (v3 or v4) is already deployed before installing the driver.

Install Replicated PV Mayastor

Refer to the Installing DataCore Puls8 documentation for installation steps using Helm.

Setting Up a Single NFS Server

Create a Replicated PV Mayastor Pool.

Copy
View Existing DiskPools
kubectl get dsp -n puls8
Copy
Sample Output
NAME             NODE            STATE     POOL_STATUS   ENCRYPTED   CAPACITY   USED   AVAILABLE
pool-on-node-0   node-0-352384   Created   Online        false       15 GiB     0 B    15 GiB
pool-on-node-1   node-1-352384   Created   Online        false       15 GiB     0 B    15 GiB
pool-on-node-2   node-2-352384   Created   Online        false       15 GiB     0 B    15 GiB

Create a Replicated PV Mayastor StorageClass.

Copy
View Existing StorageClasses
kubectl get sc
Copy
Sample Output
mayastor-2                io.openebs.csi-mayastor   Delete          Immediate              false                  24h
mayastor-3                io.openebs.csi-mayastor   Delete          Immediate              false                  27h
mayastor-etcd-localpv     openebs.io/local          Delete          WaitForFirstConsumer   false                  27h
mayastor-loki-localpv     openebs.io/local          Delete          WaitForFirstConsumer   false                  27h
mayastor-single-replica   io.openebs.csi-mayastor   Delete          Immediate              true                   27h

Create a Namespace for the NFS Server.

Copy
Create Namespace for NFS Server
kubectl create ns nfs-server
Copy
Sample Output
namespace/nfs-server created

Create a PVC for the NFS Server.

Copy
nfs-pvc.yaml
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-server-claim
  namespace: nfs-server
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: mayastor-3
EOF
Copy
Sample Output
persistentvolumeclaim/nfs-server-claim created

Verify the PVC Creation.

Copy
Verify PVC Status
kubectl get pvc -n nfs-server
Copy
Sample Output
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
nfs-server-claim   Bound    pvc-3763a93c-dfcd-4be3-8255-038b44bd5432   5Gi        RWO            mayastor-3     <unset>                 8s

Deploy the NFS Server.

Copy
nfs-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-server
  namespace: nfs-server
spec:
  replicas: 1
  selector:
    matchLabels:
      role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      volumes:
        - name: nfs-vol
          persistentVolumeClaim:
            claimName: nfs-server-claim
      containers:
        - name: nfs-server
          image: itsthenetwork/nfs-server-alpine
          env:
            - name: SHARED_DIRECTORY
              value: /nfsshare
          ports:
            - name: nfs
              containerPort: 2049
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /nfsshare
              name: nfs-vol
Copy
Deploy the NFS Server
kubectl apply -f nfs-deploy.yaml

Verify the NFS Server Deployment.

Copy
Confirm NFS Server is Running
kubectl get pods -n nfs-server
Copy
Sample Output
NAME                          READY   STATUS    RESTARTS   AGE
nfs-server-779b64df65-5pqmk   1/1     Running   0          49s

Create a ClusterIP Service for the NFS Server.

Copy
nfs-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: nfs-server
  namespace: nfs-server
spec:
  ports:
    - name: nfs
      port: 2049
  selector:
    role: nfs-server
Copy
Create Service for NFS Server
kubectl apply -f nfs-svc.yaml
Copy
Sample Output
service/nfs-server created

Verify the NFS Server Service.

Copy
Check NFS Server Service Status
kubectl get svc -n nfs-server
Copy
Sample Output
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
nfs-server   ClusterIP   34.118.238.249   <none>        2049/TCP   8s

Setting Up the NFS CSI Driver

Create a Namespace for the NFS CSI Driver.

Copy
Create Namespace for NFS CSI Driver
kubectl create ns csi-nfs
Copy
Sample Output
namespace/csi-nfs created

Create a StorageClass for the NFS CSI Driver. Update the values.yaml with the following configuration:

Copy
NFS CSI Driver StorageClass Configuration
storageClass:
  create: true
  name: nfs-csi
  parameters:
    server: nfs-server.nfs-server.svc.cluster.local
    share: /
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  mountOptions:
    - nfsvers=4.1

Add and update the Helm repository for the NFS CSI Driver to prepare for installation.

Copy
Add New Helm Chart Repo
helm repo add csi-driver https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm repo update

Install the NFS CSI Driver Using Helm.

Copy
Install NFS CSI Driver
helm install csi-nfs csi-driver/csi-driver-nfs --namespace csi-nfs -f values.yaml

Verify the CSI Driver Pods.

Copy
Check CSI Driver Pod Status
kubectl get pods -n csi-nfs
Copy
Sample Output
NAME                                 READY   STATUS    RESTARTS   AGE
csi-nfs-controller-7fff64574-cnwdp   4/4     Running   0          53s
csi-nfs-node-j27w7                   3/3     Running   0          53s
csi-nfs-node-slp95                   3/3     Running   0          53s
csi-nfs-node-srsbm                   3/3     Running   0          53s

Verify the StorageClass.

Copy
Validate NFS CSI StorageClass
kubectl get sc | grep nfs-csi
kubectl describe sc nfs-csi
Copy
Sample Output
Name:                  nfs-csi
IsDefaultClass:        No
Annotations:           meta.helm.sh/release-name=csi-nfs,meta.helm.sh/release-namespace=csi-nfs
Provisioner:           nfs.csi.k8s.io
Parameters:            mountPermissions=0,server=nfs-server.nfs-server.svc.cluster.local,share=/
AllowVolumeExpansion:  <unset>
MountOptions:
  nfsvers=4.1
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>

Creating NFS PVC

Create a ReadWriteMany PVC.

Copy
nfs-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-claim-1
spec:
  storageClassName: nfs-csi
  accessModes: [ "ReadWriteMany" ]
  resources:
    requests:
      storage: 5Gi
Copy
Create RWX PVC Using NFS CSI Driver
kubectl apply -f nfs-pvc.yaml

Verify the NFS PVC.

Copy
Verify PVC Binding and Status
kubectl get pvc
Copy
Sample Output
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
nfs-claim-1   Bound    pvc-f795f8bc-58b3-40d7-b9b2-2d16c17f3d47   5Gi        RWX            nfs-csi        <unset>                 10s

Deploy an application using the shared NFS PVC.

Copy
nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: "/volume"
              name: data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: nfs-claim-1

Verify the application deployment.

Copy
Confirm Multi-Replica Pod Access to Shared Volume
kubectl get pods -o wide
Copy
Sample Output
NAME                               READY   STATUS    RESTARTS   AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
nginx-deployment-6c664cc4c-5kjc8   1/1     Running   0          22s   10.32.1.32   gke-gke-ssd-1-default-pool-d802bc13-67bv   <none>           <none>
nginx-deployment-6c664cc4c-xzlwq   1/1     Running   0          22s   10.32.2.17   gke-gke-ssd-1-default-pool-d802bc13-z9hf   <none>           <none>
nginx-deployment-6c664cc4c-z64h4   1/1     Running   0          22s   10.32.0.14   gke-gke-ssd-1-default-pool-d802bc13-3j4s   <none>           <none>

Learn More