FAQs

In this topic:

How does Bolt help to keep my data safe?

Bolt's storage engine supports synchronous mirroring to enhance the durability of data at rest within the physical persistence layer in use. When volumes are provisioned which are configured for replication (a user can control the count of active replicas which should be maintained, on a per StorageClass basis), write operations issued by an application to that volume are amplified by its controller and dispatched to all its active replicas. Only when every replica completes the write successfully on its own underlying block device will the operation be acknowledged by the controller as complete. Otherwise, the operation is failed and the caller must make its own decision as to whether it should be retried. If a replica is determined to have faulted (operations cannot be serviced within the configured timeout period, or not without error), the control plane will automatically take corrective action and remove it from the volume. If spare capacity is available within a DiskPool, a new replica will be created as a replacement and automatically brought into synchronization with the existing replicas.

How is Bolt configured?

DataCore Bolt can be configured and managed using the command line. DataCore documentation provides detailed steps on installing and configuring DataCore Bolt.

What is the on-disk format used by Bolt DiskPools? Is it also open source?

Bolt makes use of parts of the open source Storage Performance Development Kit (SPDK) project, contributed by Intel. Bolt's Storage Pools use the SPDK's Blobstore structure as their on-disk persistence layer. Blobstore structures and layout are documented.

Since the replicas (data copies) of Bolt volumes are held entirely within Blobstores, it is not possible to directly access the data held on pool's block devices from outside of Bolt. Equally, Bolt cannot directly 'import' and use existing volumes which aren't of Bolt origin. The project's maintainers are considering alternative options for the persistence layer which may support such data migration goals.

Can the size / capacity of a Bolt DiskPool be changed?

The size of a Bolt DiskPool is fixed at the time of creation and is immutable. A single pool may have only one block device as a member. These constraints may be removed in later versions.

How can I ensure that replicas aren't scheduled onto the same node? How about onto nodes in the same rack / availability zone?

The replica placement logic of Bolt's control plane doesn't permit replicas of the same volume to be placed onto the same node, even if it were to be within different DiskPools. For example, if a volume with replication factor 3 is to be provisioned, then there must be three healthy DiskPools available, each with sufficient free capacity and each located on its own Bolt node. Further enhancements to topology awareness are under consideration by the maintainers.

Is there a way to automatically re-balance data across available nodes? Can data be manually re-distributed?

No. This may be a feature of future releases.

Can Bolt do a sync replication to a node in the same cluster? How about a different cluster?

Yes, Bolt does synchronous replication. For a different cluster, sync replication is not possible.

Does Bolt support RAID?

DiskPools do not implement any form of RAID, erasure coding or striping. If higher levels of data redundancy are required, Bolt volumes can be provisioned with a replication factor of greater than 1, which will result in synchronously mirrored copies of their data being stored in multiple DiskPools across multiple storage nodes. If the block device on which a DiskPool is created is actually a logical unit backed by its own RAID implementation (e.g. a fibre channel attached LUN from an external SAN) it can still be used within a DiskPool whilst providing protection against physical disk device failures.

Does Bolt perform compression and/or deduplication?

No.

Does Bolt support encryption?

No. This may be a feature of future releases.

Does Bolt support snapshot and clones?

No. These may be features of future releases.

Which CPU architectures are supported? What are the minimum hardware requirements?

Bolt builds and releases are compiled and tested on x86. Some effort has been made to allow compilation on ARM platforms but this is currently considered experimental and is not subject to integration or end-to-end testing by Bolt's maintainers.

Minimum hardware requirements are discussed in the quickstart section of this documentation.

Bolt does not run on Raspbery Pi as the version of the SPDK used by Bolt requires ARMv8 Crypto extensions which are not currently available for Pi.

Does Bolt suffer from TCP congestion when using NVMe/TCP?

Bolt, as any other solution leveraging TCP for network transport, may suffer from network congestion as TCP will try to slow down transfer speeds. It is important to keep an eye on networking and fine-tune TCP/IP stack as appropriate. This tuning can include (but is not limited to) send and receive buffers, MSS, congestion control algorithms (e.g. you may try, DCTCP) etc.

Why do Bolt pods show high levels of CPU utilization when there is little or no I/O being processed?

Bolt is designed to leverage the performance capabilities of contemporary high-end solid-state storage devices. This includes a polling based I/O service queue, rather than an interrupt driven one. This minimizes the latency introduced into the data path but at the cost of additional CPU utilization by the "reactor" - the poller operating at the heart of the io-engine pod. When io-engine pods have been deployed to a cluster, it is expected that these DaemonSet instances will make full utilization of their CPU allocation, even when there is no I/O load on the cluster. This is simply the poller continuing to operate at full speed, waiting for I/O. For the same reason, it is recommended that when configuring the CPU resource limits for the Bolt DaemonSet, only full, not fractional, CPU limits are set; fractional allocations will also incur additional latency, resulting in a reduction in overall performance potential. The extent to which this performance degradation is noticeable in practice will depend on the performance of the underlying storage in use, as well as whatever other bottlenecks or constraints may be present in the system, as configured.

What are the storage requirements for etcd?

etcd is the distributed reliable key-value store for the most critical data of a distributed system. It should be compacted at regular intervals to avoid performance degradation and space exhaustion.

Click on the links below to view the table that characterizes the storage requirements for different cluster sizes. The results are collated assuming the worst-case compaction does not occur for a whole year.

Cluster specifications Results

Tiny cluster

replicas per volume: 2

volumes created or deleted per day: 1

volumes mounted by a pod per day: 5

 

 Days Volume Turnover Volume Attach / Detach Disk Usage
36 36 180 1 MiB 742 KiB
72 72 360 3 MiB 461 KiB
108 108 540 5 MiB 180 KiB
144 144 720 6 MiB 923 KiB
180 180 900 8 MiB 642 KiB
216 216 1080 11 MiB
252 252 1260 12 MiB
288 288 1440 14 MiB
324 324 1620 16 MiB
365 325 1825 18 MiB

Cluster specifications Results

Small cluster

replicas per volume: 2

volumes created or deleted per day: 10

volumes mounted by a pod per day: 15

 

 Days Volume Turnover Volume Attach / Detach Disk Usage
36 360 540 6 MiB 436 KiB
72 720 1080 13 MiB
108 1080 1620 20 MiB
144 1440 2160 26 MiB
180 1800 2700 33 MiB
216 2160 3240 39 MiB
252 2520 3780 45 MiB
288 2880 4320 52 MiB
324 3240 4860 58 MiB
365 3650 5475 66 MiB

Cluster specifications Results

Medium cluster

replicas per volume: 3

volumes created or deleted per day: 50

volumes mounted by a pod per day: 100

 

 Days Volume Turnover Volume Attach / Detach Disk Usage
36 1800 3600 55 MiB
72 3600 7200 109 MiB
108 5400 10800 163 MiB
144 7200 14400 218 MiB
180 9000 18000 272 MiB
216 10800 21600 326 MiB
252 12600 25200 381 MiB
288 14400 28800 435 MiB
324 16200 32400 489 MiB
365 18250 36500 551 MiB

Cluster specifications Results

Large cluster

replicas per volume: 3

volumes created or deleted per day: 100

volumes mounted by a pod per day: 200

 

 Days Volume Turnover Volume Attach / Detach Disk Usage
36 3600 7200 109 MiB
72 7200 14400 218 MiB
108 10800 21600 326 MiB
144 14400 28800 435 MiB
180 18000 36000 544 MiB
216 21600 43200 652 MiB
252 25200 50400 761 MiB
288 28800 57600 869 MiB
324 32400 64800 978 MiB
365 36500 73000 1102 MiB

Does the supportability tool expose sensitive data?

The supportability tool generates support bundles which are used for debugging purposes. These bundles are created in response to user invocation of the tool and can be transmitted only by the user.

Below is the information collected by the supportability tool that might be identified as 'sensitive' based on the organization's data protection/privacy commitments and security policies.

Logs:

The default installation of DataCore Bolt includes the deployment of a log aggregation subsystem based on Grafana Loki. All the pods deployed in the same namespace as DataCore Bolt and labeled with datacore.com/logging=true will have their logs incorporated within this centralized collector. These logs may include the following information:

  • Kubernetes node hostnames

  • IP addresses

    • container addresses

    • API endpoints

      • Bolt

      • K8s

  • Container names

  • Kubernetes Persistent Volume names (provisioned by DataCore Bolt)

  • Bolt DiskPool names

  • Block device details (except the content)

  • Pod names of the user applications using Bolt-provisioned persistent volumes

Kubernetes Definition Files

The support bundle includes definition files for all the DataCore Bolt components. Some of which are listed below:

  • Deployments

  • DaemonSets

  • StatefulSets

K8s Events

The archive files generated by the supportability tool contain information on all the active events of the Kubernetes cluster present in the same namespace as DataCore Bolt.

etcd Dump

The default installation of DataCore Bolt deploys an etcd instance for its exclusive use. This key-value pair is used to persist state information for DataCore Bolt-managed objects. These key-value pairs are required for diagnostic and troubleshooting purposes. The etcd dump archive file consists of the following information:

  • Kubernetes node hostnames

  • IP addresses

  • PVC/PV names

  • Container names

    • Bolt

    • User applications within the bolt namespace

  • Block device details (except data content)

The list provided above is frequently reviewed and updated by the DataCore team. However, it might not be fully exhaustive, given that "sensitive information" is a subjective term.