Proxmox Hyper-Converged Deployment

Installing Guest Agent and Services

Microsoft Windows does not have native support for VirtIO devices included. There is external support through open-source drivers, which are available compiled and signed for Windows.

Download the latest build of the ISO. Click here to download.

Without this agent, the network interface cards (NIC) and discs are not detected in Windows. This is why the ISO file must be embedded in the system as a virtual CD device. Refer to Windows VirtIO Drivers for more information.

Adding a Disk for the Pool

The preferred way to present storage to a DataCore virtual machine (VM) would be raw device mapping.

Identifying the Disk

1. Identify the disks/devices that are available to the Promox (PVE) host and which of them are to be connected to DataCore VM.
2. Display the existing devices using the following CLI command:
Copy

Command

find /dev/disk/by-id/ -type l|xargs -I{} ls -l {}|grep -v -E '[0-9]$' |sort -k11|cut -d' ' -f9,10,11,12

Copy

Sample Output

13:11 /dev/disk/by-id/scsi-3600605b00b30d1501fa061fe0c634f7 -> ../../sda
13:11 /dev/disk/by-id/wwn-0x600605b00b30d1501fa061fe0c634f7 -> ../../sda
13:11 /dev/disk/by-id/scsi-3600605b00b040f02c67295c0c3313c -> ../../sdb
13:11 /dev/disk/by-id/wwn-0x600605b00b040f02c67295c0c3313c -> ../../sdb
13:11 /dev/disk/by-id/scsi-3600605b00b040f02c6729f3c995b67e -> ../../sdc
13:11 /dev/disk/by-id/wwn-0x600605b00b040f02c6729f3c995b67e -> ../../sdc

3. Use the SCSI string with the corresponding serial number of the disk for the mapping.

Adding a Physical Device as a New Virtual SCSI Disk

Establish the new connection using the following command:

qm set <VM-ID> -scsi<No> /dev/disk/by-id/scsi-<serial-no>

qm set 400 -scsi1 /dev/disk/by-id/scsi-3600605b000b040f02c67295fc0c3313c

Sample Output

Verifying the Configuration File

Use the following command to check the configuration file:

grep fc0c3313c /etc/pve/qemu-server/400.conf

scsi1: /dev/disk/by-id/scsi-3600605b000b040f02c67295fc0c3313c,size=5719182M

Preparing the SANsymphony iSCSI

The IQN names of the iSCSI ports must be given a speaking name to easily configure the iSCSI setup and later administration.

It is recommended to replace the number after the dash with the function.

Original: iqn.2000-08.com.datacore:sds1-5

modified: iqn.2000-08.com.datacore:sds1-fe1

Proxmox iSCSI

Installing iSCSI Daemon

iSCSI is a widely employed technology that is used to connect to storage servers. Almost all storage vendors support iSCSI. There are also open-source iSCSI target solutions available, which is based on Debian.

To use Debian, install the Open-iSCSI (open-iscsi) package. This is a standard Debian package, however, not installed by default to save the resources. Refer to Open-iSCSI documentation for installation instructions.

apt-get install open-iscsi

iSCSI Connection

Changing the iSCSI-Initiator Name on each PVE

The Initiator Name must be unique for each iSCSI initiator. Do NOT duplicate the iSCSI-Initiator Names.

1. Edit the iSCSI initiator name in the /etc/iscsi/initiatorname.iscsi file to assign a unique name in a way that the IQN refers to the server and the function. This change makes administration and troubleshooting easier.
• Original: InitiatorName=iqn.1993-08.org.debian:01:bb88f6a25285
• Modified: InitiatorName=iqn.1993-08.org.debian:01:<Servername + No.>
2. Restart iSCSI to take effect using the following command:
   Copy

   Command

   systemctl restart iscsid

Discovering the iSCSI Target to PVE

1. Before attaching the SANsymphony iSCSI target, discover all the iqn-portname using the following command:
   Copy

   Command

   iscsiadm -m discovery -t sendtargets -p <IP-address>:3260

Copy

Example

# iscsiadm -m discovery -t sendtargets -p 172.16.41.21:3260

2. Attach all the SANsymphony iSCSI targets on each PVE using the following command:

Copy

Command

iscsiadm --mode node --targetname <IQN> -p <IP-address> --login

Refer to Wiki - iSCSI Multipath for more information.
Copy

Example

iscsiadm --mode node --targetname iqn.2000-08.com.datacore:pve-sds11-fe1 -p 172.16.41.21 --login
iscsiadm --mode node --targetname iqn.2000-08.com.datacore:pve-sds11-fe2 -p 172.16.42.21 --login
iscsiadm --mode node --targetname iqn.2000-08.com.datacore:pve-sds10-fe1 -p 172.16.41.20 --login
iscsiadm --mode node --targetname iqn.2000-08.com.datacore:pve-sds10-fe2 -p 172.16.42.20 --login

Detailed information about the iSCSI connections and the hardware can be displayed with this command:

iscsiadm --mode session --print=1

iSCSI Settings

The /etc/iscsi/iscsid.conf file must change the line so that the initiator starts automatically.

node.startup = manual 
node.leading_login = No

The default 'node.session.timeo.replacement_timeout' is 120 seconds. It is recommended to use a smaller value of 15 seconds instead.

node.session.timeo.replacement_timeout = 15

If a port reinitialize is done, the port may be unable to login on its own. In this case, the attempts must be increased here:

node.session.initial_login_retry_max = 64

Restart the service using the following command:

systemctl start iscsid

Logging in to the iSCSI Targets on Boot

For each connected iSCSI target, set the node.startup parameter in the target to automatic. The target is specified in the following file.

/etc/iscsi/nodes/<TARGET>/<PORTAL>/default

SCSI Disk Timeout

Set the timeout to 80 seconds for all the SCSI devices created from the SANsymphony virtual disks.

For Example: cat /sys/block/sda/device/timeout

Two methods may be used to change the SCSI disk timeout for a given device.

• Use the ‘echo’ command – this is temporary and will not survive the next reboot of the Linux host server.
• Create a custom ‘udev rule’ – this is permanent, however, will require a reboot for the setting to take effect.

Using the ‘echo’ Command (will not survive a reboot)

Set the SCSI Disk timeout value to 80 seconds using the following command:

echo 80 > /sys/block/[disk_device]/device/timeout

Creating a Custom ‘udev’ Rule

Create a file called /etc/udev/rules.d/99-datacore.rules with the following settings:

SUBSYSTEM=="block", ACTION=="add", ATTRS{vendor}=="DataCore", ATTRS{model}=="Virtual Disk ", RUN+="/bin/sh -c 'echo 80 > /sys/block/%k/device/timeout'"

Refer to Linux Host Configuration Guide FAQ for more information.

iSCSI Multipath

Installing Multipath Tools

The default installation does not include the 'multipath-tools' package. Use the following commands to install the package:

apt-get update
apt-get install multipath-tools

Refer to iSCSI Multipath for more information.

Creating a Multipath.conf File

After installing the package, create the following multipath configuration file: /etc/multipath.conf.

Refer to the Linux Configuration Guide for the relevant settings and the adjustments for the PVE in the Wiki iSCSI Multipath.

    defaults {
        user_friendly_names     yes
        polling_interval           60
        find_multipaths          "smart"
        }

    blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
               devnode "^hd[a-z]"
        }

    devices {
            device {
                    vendor                  "DataCore"
                    product                 "Virtual Disk"
                    path_checker         tur
                    prio                         alua
                    failback                   10
                    no_path_retry         fail
                    dev_loss_tmo         60
                    fast_io_fail_tmo      5
                    rr_min_io_rq            100
                    path_grouping_policy    group_by_prio
               }
    }

Restarting the Service

Restart the multipath service using the following command to reload the configuration:

multipath -r

Refer to ISCSI Multipath for more information.

Serving a SANsymphony Virtual Disk to the Proxmox Node

Following the necessary configurations in the SANsymphony graphical user interface (GUI) such as the assignment of the iSCSI port to the relevant host and the virtual disk, the virtual disk must now be integrated into the host.

To make the virtual disk visible in the system, the iSCSI connection must be scanned again using the following command:

iscsiadm -m session --rescan

The list of block devices and whether the virtual disk is correctly detected with its paths may be identified and with the multipath name “mpathX” the dives may also identified.

lsblk

sdd       8:48   0   100G   0   disk
 └─mpathb 252:8   0   100G   0   mpath
sde       8:64   0   100G   0   disk
 └─mpathb 252:8   0   100G   0   mpath
sdf       8:80   0   1.5T   0   disk
 └─mpatha 252:7   0   1.5T   0   mpath
sdg       8:96   0   1.5T   0   disk
 └─mpatha 252:7   0   1.5T   0   mpath

Use the 'multipath' command to determine which and whether all necessary paths are now available from the SANsymphony server for the virtual disk:

multipath -ll

root@de-prox02:~# multipath -ll  
mpatha (360030d9095f0370627edc4fd4d0fa4a9) dm-7 DataCore, Virtual Disk  
size=1.5T features="0" hwhandler="1 alua" wp=rw  
  |- 7:0:0:1 sdg 8:96 active ready running  
  `- 6:0:0:1 sdf 8:80 active ready running  
mpathb (360030d9012bb36060faaaf8355276e9b) dm-8 DataCore, Virtual Disk  
size=100G features="0" hwhandler="1 alua" wp=rw  
  |- 7:0:0:0 sde 8:64 active ready running  
  `- 6:0:0:0 sdd 8:48 active ready running 

Creating a Proxmox File-system (LVM)

Creating a Physical Volume for the Logical Volume

“pvcreate” initializes the specified physical volume for later use by the Logical Volume Manager (LVM).

pvcreate /dev/mapper/mpathX

Creating a Volume Group

vgcreate vol_grp<No> /dev/mapper/mpathX

Displaying a Volume Group

vgscan

root@de-prox02:/# vgscan
Found volume group "vol_grp2" using metadata type lvm2
Found volume group "vol_grp1" using metadata type lvm2
Found volume group "pve" using metadata type lvm2

Adding LVM at the Datacenter Level

To create a new LVM storage, follow the steps below:

1. Access the PVE GUI at the Datacenter level and click Storage in the left-hand menu.
2. Click Add and select LVM from the list.
3. Select the Shared checkbox (mandatory) to allow the storage to be used across all nodes.
4. Click ADD to complete the storage creation.

Finding out the UUID and the Partition Type

The blkid command is used to query information from the connected storage devices and their partitions.

blkid /dev/mapper/mpath*

root@prox1:/# blkid /dev/mapper/mpath*
/dev/mapper/mpatha: UUID="X3bM3Dy-YyYf-3m8B-eUaA-n5li-Idbt-soaI0x" TYPE="LVM2_member"
/dev/mapper/mpathb: UUID="7zVp2P-58Ea-Atpp-hoHu-dzJQ-Gnd8-YRjWE2" TYPE="LVM2_member"

Proxmox LVM storage does not support SANsymphony advanced features such as snapshots and Continuous Data Protection (CDP) at the virtual machine disk level. To use SANsymphony snapshot or CDP functionality, present the SANsymphony virtual disk directly to the virtual machine using RAW device mapping instead of configuring the disk as Proxmox LVM storage. In this configuration, snapshot and CDP operations must be managed manually from the SANsymphony DataCore Management Console.

Disk Raw Device Mapping to Virtual Machine

Follow the steps below if the prerequisites of an operating system or an application should be that a RAW device mapping into the virtual machine is necessary:

1. After successfully serving the virtual disk (single or mirror) to the Proxmox (PVE) node, run a rescan to make the virtual disk visible in the system using the following command:

Copy

Command

iscsiadm -m session --rescan

2. Identify the virtual disk to use as a RAW device and identified multipath name “mpathX” using the following command:
Copy

Command

lsblk

3. Navigate to the “/dev/mapper” directory and run the ls -la command to verify which dm-X the required device is linked to.
Copy

Sample Output

lrwxrwxrwx 1 root root 7 May 28 17:05 mpatha -> ../dm-5
lrwxrwxrwx 1 root root 8 May 28 17:05 mpathb -> ../dm-8

4. Hot-Plug/Add physical device as new virtual SCSI disk using the following command:
Copy

Command

qm set VM-ID -scsi>No< /dev/dm->No<

Copy

Example

qm set 100 -scsi2 /dev/dm-11

Restarting the Connection after Rebooting the Proxmox Server

Use the following commands to restart the Proxmox (PVE) Server:

systemctl restart open-iscsi.service
iscsiadm -m node --login

If an error occurs, then attempt the following command first:

iscsiadm -m node –logout

The connection will restart.

Deployment Examples

Minimum Design

Best Practices Design