Performance Optimization

Explore this Page

• Overview
• SPDK Blobstore Cluster Size
• RDMA Enablement
• NVMe-oF Transport Tuning
• CPU Isolation

Overview

To ensure optimal performance of the Replicated PV Mayastor, you can optionally fine-tune four key configuration areas that influence efficiency, throughput, and operational consistency:

• Storage Performance Development Kit (SPDK) Blobstore Cluster Size: Adjusts the allocation unit for the SPDK blobstore. Choosing the appropriate cluster size (per pool or globally) balances storage efficiency with metadata overhead and directly impacts pool creation, import times, and large sequential I/O performance.
• Remote Direct Memory Access (RDMA) Enablement: Enables NVMe-over-Fabrics (NVMe-oF) with RDMA to deliver low-latency, high-throughput data paths. This section covers hardware prerequisites, interface validation, and TCP fallback behavior when RDMA is unavailable.
• NVMe-oF Transport Tuning: Adjusts the NVMe-oF target and initiator transport parameters, such as queue depth, buffer pools, and I/O sizing. Separate sets of values apply to the TCP and RDMA transports.
• Central Processing Unit (CPU) Isolation: Isolates CPU cores for Replicated PV Mayastor’s reactor threads to minimize scheduling interruptions and maximize I/O responsiveness. It explains kernel parameter configuration and Helm settings for dedicated core allocation.

This document explains how to configure the blobstore cluster size, enable and validate RDMA, tune the NVMe-oF transports, and set up CPU isolation to achieve consistent performance and simplified management across your Kubernetes storage environment.

SPDK Blobstore Cluster Size

The SPDK Blobstore Cluster Size configuration feature helps you fine-tune storage performance and efficiency for Replicated PV Mayastor DiskPools in Kubernetes environments. By selecting an appropriate cluster size during pool creation, you can optimize on-disk layout, reduce metadata overhead, and accelerate pool import and rebuild operations, especially on large-capacity storage devices.

Blobstore Cluster Size Considerations

The blobstore cluster size determines the allocation unit size of data in blobstore backend of a Replicated PV Mayastor DiskPool.

• Smaller cluster sizes (default: 4 MiB) provide higher storage efficiency but generate more metadata overhead.
• Larger cluster sizes (for example, 16 MiB or 32 MiB) reduce metadata overhead, accelerate pool creation and import operations, and facilitate better performance for large sequential I/O workloads.

Before modifying the default setting, evaluate application I/O patterns and device capacities. Maintaining a consistent cluster size across pools simplifies replica scheduling and ongoing management.

The cluster size also determines how much the data area of each replica is aligned up, which contributes to the pool space a replica consumes beyond the volume's requested size. A larger cluster size increases this overhead on every replica, which is most noticeable on pools hosting many small volumes. Refer to Creating a DiskPool.

Configuring Blobstore Cluster Size

1. Per-Pool Configuration (DiskPool Custom Resource)

   Specify the cluster_size field in the DiskPool custom resource manifest to configure cluster size for an individual pool. This provides granular control for specific storage devices.

   Copy

   Example: DiskPool Custom Resource with Cluster Size

   apiVersion: "openebs.io/v1beta3"
   kind: DiskPool
   metadata:
     name: <pool_name>
     namespace: <namespace>
   spec:
     node: <node_name>
     disks: ["/disk/path"]
     cluster_size: 32MiB

2. Global Configuration (Helm Chart)

   Set a global cluster size for all new pools that do not specify it in their custom resource. Provide the size in bytes.

   Copy

   Example: Helm Chart Variable for Global Cluster Size

   --set openebs.engines.replicated.mayastor.agents.core.poolClusterSize=33554432
   (The value above sets the global cluster size to 32 MiB.)

3. Volume Provisioning

   A new StorageClass parameter, poolClusterSize, ensures that only pools matching the specified cluster size are used when scheduling replicas.

   • If sufficient matching pools are unavailable, volume provisioning will fail.
   • Replica rebuilds for existing volumes may also fail if matching pools cannot be located.
   Copy

   Example: StorageClass with poolClusterSize Parameter

   parameters:
     poolClusterSize: "32MiB"

Best Practices

• Advanced Configuration: Changing the default cluster size (4 MiB) is intended for advanced configurations. Perform a thorough assessment of application I/O patterns and storage capacity before making adjustments.
• Validated Scale: Internal testing has verified a 32 MiB cluster size on devices up to 20 TiB, with pool import times averaging about three minutes on high-performance cloud disks.
• Operational Consistency: For simplified management and predictable replica scheduling, minimize the number of cluster sizes in your deployment. As a best practice for large-capacity environments, configure a global blobstore cluster size of 16 MiB or 32 MiB to achieve an optimal balance of performance and efficiency.

Benefits of Larger Blobstore Cluster Size

• Faster Pool Creation: When a pool is created, the device is formatted by writing metadata for every cluster. Fewer clusters mean significantly less metadata to write, leading to a significant reduction in the time it takes to create a pool on a large device.
• Quicker Pool Imports: During startup or recovery, Replicated PV Mayastor imports existing pools by reading their metadata from disk. A more compact metadata layout (due to larger clusters) requires fewer I/O operations, making the import process much quicker.
• Reduced Metadata Overhead: Larger clusters decrease the amount of metadata that SPDK must maintain.

RDMA Enablement

RDMA support in Replicated PV Mayastor enables significant improvements in storage performance by reducing latency and increasing throughput for workloads using NVMe-over-Fabrics (NVMe-oF). This feature utilizes RDMA-capable network interfaces (RNICs) to achieve high-speed, low-latency communication across nodes.

Requirements

Interface Validation

Ensure the interface specified by the io_engine.target.nvmf.iface Helm parameter exists on all io-engine nodes and is RDMA-capable. If not, those nodes will default to TCP communication.

Application Node Requirements

Application nodes must also have RDMA capable devices to establish RDMA connections. This requirement is independent of the iface parameter and specific to where the application is scheduled.

Kernel Module Requirement

RDMA-capable hardware alone is not sufficient. The nvme_rdma kernel module must also be loaded on the node, because the initiator negotiates the RDMA transport at connection time. A node is treated as RDMA-capable only when both conditions are met, and a node that has an RNIC but no loaded module is treated as though it has no RDMA hardware at all.

Load the module and make it persistent on every node that participates in RDMA connections. Refer to the Manual Prerequisites Setup documentation for the commands.

ls /sys/module/nvme_rdma

The two signals are reported separately to the control plane when the node registers, as rdmaHcaPresent and nvmeRdmaModuleLoaded, so a missing module can be distinguished from missing hardware.

Enabling RDMA via Helm

To enable the RDMA feature via Helm:

1. Set openebs.mayastor.io_engine.target.nvmf.rdma.enabled to true.
2. Set openebs.mayastor.io_engine.target.nvmf.iface to a valid network interface name that exists on an RNIC.
3. Verify that all nodes are properly configured with RDMA-capable hardware and that network interfaces are correctly identified and accessible.

openebs.mayastor.csi.node.nvme.tcpFallback

rdma link add rxe0 type rxe netdev eth0

Benefits of RDMA Enablement

• Lower Latency Data Path: RDMA eliminates kernel network stacks for storage traffic, enabling direct memory-to-memory transfers and significantly reducing I/O latency.
• Higher Throughput: By offloading network processing from CPUs to RDMA-capable NICs, Replicated PV Mayastor can sustain higher bandwidth and handle more concurrent operations.

NVMe-oF Transport Tuning

Replicated PV Mayastor exposes the NVMe-oF transport parameters of the io-engine as Helm values. Separate sets of values apply to the TCP and the RDMA transports, so each transport can be sized for the fabric it runs on. A further set applies to the initiator on the application nodes.

The default values suit the majority of deployments. Change them only when you have a specific reason to, ideally with guidance from DataCore Support, as an unsuitable value can reduce throughput or prevent volumes from connecting. Every parameter in this section takes effect when the io-engine starts, so the io-engine pods restart when you apply a change.

Target Parameters

The following parameters apply to the NVMe-oF target regardless of the transport in use.

TCP Transport Parameters

The following parameters apply to the NVMe-oF TCP transport.

RDMA Transport Parameters

The following parameters apply to the NVMe-oF RDMA transport. They are supplied to the io-engine only when RDMA is enabled through openebs.mayastor.io_engine.target.nvmf.rdma.enabled, and are otherwise ignored.

The parameters that have no default are left unset deliberately, so that the transport applies its own sizing. Set them only when tuning for a specific fabric.

RDMA Traffic Classification

On a RoCEv2 fabric, storage traffic can be marked with a Differentiated Services Code Point (DSCP) value so that switches and network adapters classify it into a queue on which Priority Flow Control (PFC) is enabled, giving lossless transport. Use the transportTos parameter to set the Type of Service value applied to target-side RDMA traffic. When it is not set, traffic is left unmarked and receives best-effort treatment.

--set openebs.mayastor.io_engine.nvme.transportTos=104

Application Node Parameters

The following parameters apply to the NVMe-oF initiator on the nodes where your applications run, and are configured on the CSI node plugin.

CPU Isolation

The Replicated PV Mayastor fully utilizes each CPU core assigned to it by spawning a dedicated thread (reactor) on each. These reactor threads execute continuously, serving I/O operations without sleeping or blocking. Other threads within the I/O engine, which are not bound to specific CPUs, may block or sleep as needed.

For optimal performance, it is important that these bound reactor threads experience minimal interruptions. Ideally, they should only be interrupted by essential kernel-based time accounting processes. In practice, this is difficult to achieve, but improvements can be made using the isolcpus kernel parameter.

The isolcpus boot parameter does not prevent kernel threads or other Kubernetes pods from running on the isolated CPUs. However, it does prevent system services such as kubelet from interfering with the I/O engine's dedicated cores.

Configure Kernel Boot Parameters

Add the isolcpus kernel parameter to instruct the Linux scheduler to isolate specific CPU cores from general scheduling.

The location of the GRUB configuration file may vary depending on your Linux distribution. For example:

• Standard Linux: /etc/default/grub
• Ubuntu 20.04 on AWS EC2: /etc/default/grub.d/50-cloudimg-settings.cfg

In this example, we isolate CPU cores 2 and 3 (on a 4-core system).

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=2,3"

Update GRUB Configuration

After modifying the GRUB configuration file, update the bootloader to apply changes.

sudo update-grub

Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/40-force-partuuid.cfg'
Sourcing file `/etc/default/grub.d/50-cloudimg-settings.cfg'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.8.0-29-generic
Found initrd image: /boot/microcode.cpio /boot/initrd.img-5.8.0-29-generic
Found linux image: /boot/vmlinuz-5.4.0-1037-aws
Found initrd image: /boot/microcode.cpio /boot/initrd.img-5.4.0-1037-aws
Found Ubuntu 20.04.2 LTS (20.04) on /dev/xvda1
done

Reboot the System

Reboot the system to enable the new kernel parameters.

sudo reboot

Verify Isolated CPU Cores

Once the system is back online, confirm that the isolcpus parameter is active and functioning as expected.

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-5.8.0-29-generic root=PARTUUID=7213a253-01 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 isolcpus=2,3 panic=-1

cat /sys/devices/system/cpu/isolated

2-3

Update Helm Configuration

To ensure Replicated PV Mayastor utilizes the isolated cores, update its configuration using the kubectl puls8 mayastor plugin.

Ensure that the kubectl puls8 mayastor plugin is installed and matches the Helm chart version of your deployment.

kubectl puls8 mayastor upgrade -n <namespace> --set 'openebs.mayastor.io_engine.coreList={2,3}'

CPU core indexing begins at 0. Therefore, coreList={2,3} corresponds to the third and fourth cores.

Benefits of CPU Isolation

• Consistent I/O Performance: Dedicating CPU cores to Replicated PV Mayastor’s reactor threads minimizes context switching and scheduling delays, reducing latency spikes.
• Predictable Resource Allocation: Explicitly reserving cores prevents unexpected contention from other workloads, simplifying capacity planning and performance tuning.
• Better Real-Time Responsiveness: Reactor threads can run uninterrupted, improving stability and predictability for latency-sensitive storage operations.

Learn More

• Disk Emulation
• Monitoring
• High Availability