Proxmox: Host Configuration Guide

PVE iSCSI

The iSCSI function was implemented based on the Proxmox Wiki article, which reflects the current recommendations of the manufacturer. The steps below were useful at the time of creation.

Before setting up iSCSI, it is recommended to read the Proxmox ISCSI installation - Proxmox VE wiki article.

Installing iSCSI Daemon

iSCSI is a widely employed technology that is used to connect to storage servers. Almost all storage vendors support iSCSI. There are also open-source iSCSI target solutions available, which is based on Debian.

To use Debian, you need to install the Open-iSCSI (open-iscsi) package. This is a standard Debian package, however, not installed by default to save the resources.

apt-get install open-iscsi

Changing the iSCSI-Initiator Name on each PVE

The Initiator Name must be unique for each iSCSI initiator. Do NOT duplicate the iSCSI-Initiator Names.

1. Edit the iSCSI initiator name in the /etc/iscsi/initiatorname.iscsi file to assign a unique name in a way that the IQN refers to the server and the function. This change makes administration and troubleshooting easier.

Original: InitiatorName=iqn.1993-08.org.debian:01:bb88f6a25285

Modified: InitiatorName=iqn.1993-08.org.debian:01:<Servername + No.>

2. Restart iSCSI to take effect using the following command:

systemctl restart iscsid

Discovering the iSCSI Target to PVE

1. Before attaching the SANsymphony iSCSI target, you must discover all the iqn-portname using the following command:
Copy

Command

iscsiadm -m discovery -t sendtargets -p <IP-address>:3260

Copy

Example

# iscsiadm -m discovery -t sendtargets -p 172.16.41.21:3260

2. Attach all the SANsymphony iSCSI targets on each PVE using the following command:

iscsiadm --mode node --targetname <IQN> -p <IP-address> --login

Displaying the Active Session

iscsiadm -m session --rescan

Detailed information about the iSCSI connections and the hardware can be displayed with this command:

iscsiadm --mode session --print=1

iSCSI Settings

The iSCSI service does not start automatically by default when the PVE-node boots. Refer to the iSCSI Multipath document for more information.

The /etc/iscsi/iscsid.conf file must change the line so that the initiator starts automatically.

node.startup = automatic
node.leading_login = No

The default 'node.session.timeo.replacement_timeout' is 120 seconds. It is recommended to use a smaller value of 15 seconds instead.

node.session.timeo.replacement_timeout = 15

If a port reinitialization is done, it can be that the port is unable to login on its own. In this case, the attempts must be increased:

node.session.initial_login_retry_max = 64

Restart the service using the following command:

systemctl start iscsid

Logging in to the iSCSI Targets on Reboot

For each connected iSCSI target, you need to modify the node.startup parameter in the target to automatic. The target is specified in the

/etc/iscsi/nodes/<TARGET>/<PORTAL>/default

If the connection after reboot the PVE-Server fails

The services must be started with the command:

systemctl restart iscsid

or

systemctl restart open-iscsi.service

Re-connect all iSCSI targets:

iscsiadm -m node --login

If an error occurs, then first:

iscsiadm -m node --logout

PVE-Fiber Channel

Installing Required Packages

Packages for FC-SAN support

apt-get update
apt-get install scsitools
apt-get install -y sysfsutils
apt-get install lvm2

This command updates your package lists and installs the required tools. LVM2 is required for the management of logical disks.

Checking HBA Info and Drivers

Determine the Manufacturer and Model of the HBAs

lspci | grep "Fibre Channel"

Get the Vendor and Device IDs for the HBAs installed

lspci -nn | grep "Fibre Channel"

Example:

1077 = vendor ID from Qlogic

2031 = device ID

Check if the driver modules are installed

This can be done by searching the list of available modules. (Replace 6.8.12-4-pve with your kernel version in the command below).

grep 1077 /lib/modules/6.8.12-4-pve/modules.* | grep 2031

Find Kernel version / System information

hostnamectl

Check drivers for these HBAs are loaded in the kernel

lsmod | grep qla2xxx

Reload Module

modprobe -v qla2xxx

Getting HBA Information

Simple

You can find more detailed information about the Fiber Channel adapters in the directory /sys/class/fc_host/.

ls -la /sys/class/fc_host/

The output returns the device path with PCI address and port number for each port (host).

Detailed

The listed directories contain specific information for each adapter, such as node name (WWN), port name (WWN), type, speed, status, etc.

The easiest way to get detailed information is to use the systool command with the option -v.

systool -c fc_host -v host0

Rescan SCSI-Bus

rescan-scsi-bus.sh

This identifies all SCSI devices, including new LUNs, and integrates them into your environment.

SCSI Disk Timeout

Set the timeout to 80 seconds for all the SCSI devices created from the SANsymphony virtual disks.

For example:

cat /sys/block/sda/device/timeout

There are two methods that can be used to change the SCSI disk timeout for a given device.

• Use the ‘echo’ command – this is temporary and will not survive the next reboot of the Linux host server.

• Create a custom ‘udev rule’ – this is permanent but will require a reboot for the setting to take effect.

Using the ‘echo’ Command (will not survive a reboot)

Set the SCSI Disk timeout value to 80 seconds using the following command:

echo 80 > /sys/block/[disk_device]/device/timeout

Creating a Custom ‘udev’ Rule (permanent but requires a reboot)

Create a file called /etc/udev/rules.d/99-datacore.rules with the following settings:

SUBSYSTEM=="block", ACTION=="add", ATTRS{vendor}=="DataCore", ATTRS{model}=="Virtual Disk    ", RUN+="/bin/sh -c 'echo 80 > /sys/block/%k/device/timeout' "

Command:

touch /etc/udev/rules.d/99-datacore.rules

Refer to the Linux Host Configuration Guide for more information.

apt-get update
apt-get install multipath-tools

defaults {
    user_friendly_names    yes
    polling_interval       60
    find_multipaths        "smart"
}

blacklist {
    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
    devnode "^hd[a-z]"
}

devices {
    device {
        vendor               "DataCore"
        product              "Virtual Disk"
        path_checker          tur
        prio                  alua
        failback              10
        no_path_retry         fail
        dev_loss_tmo          60
        fast_io_fail_tmo      5
        rr_min_io_rq          100
        path_grouping_policy  group_by_prio
    }
}

systemctl restart multipath-tools

iscsiadm -m session --rescan

lsblk

multipath -ll

pvcreate /dev/mapper/mpathX

vgcreate vol_grp<No> /dev/mapper/mpathX

vgscan

blkid /dev/mapper/mpath*

iscsiadm -m session --rescan

lsblk

qm set VM-ID -scsi>No< /dev/dm->No<

qm set 100 -scsi2 /dev/dm-11

systemctl restart open-iscsi.service
iscsiadm -m node --login

iscsiadm -m node –logout