DataCore Puls8 Architecture

Explore this Page

Understanding DataCore Puls8 Storage Architecture

DataCore Puls8 leverages containerization to dynamically provision volumes and deliver essential data services such as high availability, replication, and storage orchestration. It seamlessly integrates with Kubernetes to manage storage services efficiently, ensuring reliability and scalability for stateful workloads.

DataCore Puls8 consists of three primary components:

  • Management Plane: Enhances the OpenEBS Control Plane with additional simplification and supplemental functions.
  • Control Plane: Manages storage configuration, provisioning, and integration with Kubernetes.
  • Data Engines: Handle data persistence, replication, and volume management.

Management Plane

The Management Plane in DataCore Puls8 enhances the OpenEBS Control Plane by providing additional functionalities that simplify operations, improve visibility, and streamline storage lifecycle management. It acts as a central point of coordination, ensuring seamless interaction between the Control Plane and Data Engines while offering enterprise-grade enhancements for efficiency, automation, and governance.

Control Plane

The control plane is responsible for managing storage resources across Kubernetes clusters. It provides essential functions such as provisioning, monitoring, and policy enforcement.

Helm Chart

DataCore Puls8 can be installed using a highly configurable Helm chart. It is also available through Kubernetes distributions like OpenShift, EKS, and Rancher as a marketplace application, add-on, or plugin. The control plane components are all stateless. It depends on the Custom Resources (CRs) and etcd server for managing their internal configuration state and reporting the status of the various components.

Declarative API

DataCore Puls8 follows a declarative API model, using Kubernetes CRs to manage storage operations. This API enables:

  • Storage pool and volume configuration
  • Volume life-cycle management (creation, snapshots, cloning, backup, and restore)
  • Upgrades for storage pools and volumes

Data Engine Operators

Data Engine Operators manage storage resources and orchestrate storage-related tasks at both the cluster and node levels. These operators ensure efficient volume management by handling:

  • Storage discovery
  • Volume creation and deletion
  • Replica management and rebuilding
  • Snapshot and backup operations

Data Engine Operators are commonly referred as the Control Plane of Data Engines, as they play a crucial role in managing volumes and data services associated with them. The number of operators a Data Engine requires varies based on its features and functionality. Some may utilize multiple operators, while local volume operations can be directly integrated into the respective CSI controller or provisioner.

CSI Driver

The CSI driver (Dynamic Volume Provisioner) acts as an interface between Kubernetes and the storage components. The behavior of a CSI driver is controlled or customized using parameters defined in the StorageClass. It facilitates:

  • Volume provisioning and deletion
  • Volume attachment/detachment to application nodes
  • File system formatting and mounting

Plugins and Integrations

The core storage functionalities are extended through integrations with industry-standard tools:

  • Backup and Recovery: Integration with Velero for application-consistent backups
  • Monitoring and Alerting: Integration with Prometheus, Grafana, and Alert Manager
  • Security and Compliance: Enforcement of security policies via PodSecurityPolicies or Kyverno
  • Logging: Integration with logging stacks like ELK, Loki, and Logstash
  • Visualizations: Integration with standard Kubernetes Dashboards or custom Grafana dashboards

Data Engines

Data Engines are responsible for performing read and write operations to the underlying persistent storage on behalf of stateful workloads. These engines play a crucial role in aggregating available block storage capacity, provisioning volumes for applications, and ensuring data consistency and integrity. They provide standard system and network transport interfaces (Example: NVMe) and support essential volume services such as synchronous replication, compression, encryption, and snapshot management.

The Data Engines perform core storage operations and consist of the following layers:

Volume Access Layer

This layer enables applications to perform read and write operations using standard POSIX-compliant mechanisms. Applications may access storage via raw block devices or through file systems such as XFS and Ext4. The Kubernetes CSI driver manages volume attachment and formatting as needed.

You have the flexibility to configure mount options and define filesystem permissions at this layer, which will then be executed by the CSI node driver or Kubelet.

Volume Services Layer

Volume Services Layer also known as the Volume Target or Volume Controller Layer as it is responsible for providing a Logical Volume. This layer is responsible for managing application read and write operations. It achieves this through dedicated Volume Targets that regulate access to storage volumes, ensure synchronous data replication across nodes, and coordinate master replica designation and recovery processes.

This architecture enhances high availability by reducing metadata overhead and minimizing the impact of node failures. This components:

  • Manages logical volume access
  • Handles synchronous data replication
  • Determines the master replica
  • Facilitates data rebuilding after failures

Unlike traditional multi-metadata controller approaches,DataCore Puls8's dedicated storage controllers simplify replication management and limit the disruption caused by node failures, improving overall system resilience.

Volume Data Layer

DataCore Puls8 creates Volume Replicas that persist application data. Replicas are pinned to specific nodes and can be stored as:

  • Sub-directories
  • Full or Partitioned device
  • Logical volumes

For applications requiring local storage, the persistent volume will be created using one of the above. If high availability is needed, DataCore Puls8 synchronously replicates data across multiple nodes.

DataCore Puls8 Volume Replicas transition through the following states:

  • Initializing: Registration of a new replica
  • Healthy: Actively participating in read/write operations
  • Offline: Inaccessible due to node/storage failure
  • Rebuilding: Restoring data from healthy replicas after failure recovery
  • Terminating: Removal of the replica when the volume is deleted

Storage Layer

The storage layer forms the foundation for data persistence. It consists of block devices that may be:

  • Locally attached (via PCIe, SAS, NVMe)
  • Remotely accessed through SAN or cloud storage
  • Sub-directories on mounted filesystems

DataCore Puls8 Data Engines interact with storage as devices, device pools, or directories, ensuring compatibility with standard Kubernetes storage constructs.

Dynamic Persistent Volume Provisioning

Kubernetes CSI layer intercepts persistent volume requests and forwards them to DataCore Puls8’s control plane for provisioning. Based on the defined StorageClass and PersistentVolumeClaim (PVC), DataCore Puls8 determines the appropriate storage backend (Local or Replicated) and provisions the requested volume.

DataCore Puls8 dynamically deploys storage services as Kubernetes applications, ensuring storage scalability alongside containerized workloads. Upon provisioning, volumes are available under the DataCore Puls8 namespace and are ready for use by applications.

Learn More