VMware ESXi Configuration Guide (Formerly Known as FAQ 1556)

Added

ESXi 8.x has been added.

VMware ESXi compatibility lists – All

Known Issues - All

Removed

All references to ESXi 5.x and ESXi 6.x up to and including ESXi 6.5 have been removed as some of these are now considered 'End of General Support' and are also soon to be 'End of Technical Guidance' from VMware.

If there is still a requirement to use an specific older ESXi version with SANsymphony then contact DataCore Technical Support to give advice on any relevant information that may have now been removed from this document.

Updated

Most of the tables have been updated to indicate a minimum SANsymphony requirement where applicable – please check the document for more information.

Updated

The example showing how to modify and remove an existing SATP rule has been updated.

Added
FC-NVME (NVME over FC) capable HBAs may report their storage adapters twice in ESXi and SANsymphony vDisks cannot be detected on the Host using them.

FC-NVME is not currently supported for SANsymphony Virtual Disk mappings. To turn off the FC-NVME feature on the ESXi host’s HBA driver use one of the following commands:

For Qlogic HBAs:

esxcfg-module -s 'ql2xnvmesupport=0' qlnativefc

For Emulex HBAs:

esxcli system module parameters set -m lpfc -p lpfc_enable_fc4_type=1

A reboot of the ESXi host is required in either case.

Updated

The command syntax for setting the Round Robin PSP has been amended.

Previously, it was:

esxcli storage nmp satp rule add -V DataCore -M 'Virtual Disk' -s VMW_SATP_ALUA -P VMW_PSP_RR -O iops=10 -o disable_action_OnRetryErrors -c tpgs_on

It had been reported that using the command as stated above was not applying the ‘IOPs’ value consistently, resulting in the default (1000) being used instead. The command switches have now been amended to be more explicit which will resolve this problem.

The new command is now:

esxcli storage nmp satp rule add --vendor="DataCore" --model="Virtual Disk"

--satp="VMW_SATP_ALUA" --psp="VMW_PSP_RR" --psp-option="iops=10"

--option="disable_action_OnRetryErrors" --claim-option="tpgs_on"

--description="DataCore SANsymphony ALUA RR"

Updated

Affects ESXi 7.0 Updates 2 & 2a and ESXi 6.7 Update 3

Using DataCore’s recommended DiskMaxIOSize of 512KB causes unexpected IO timeouts and latency

ESXi 7.0 - PR 2751564 and ESXi 6.7 - PR 2752542

If you lower the value of the DiskMaxIOSize advanced config option, ESXi hosts I/O operations might fail If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

This can result in unexpected IO timeouts on the host, leading to increased latency.

For ESXi 7.0 Update 2 and 2a:

A fix is available in either ESXi 7.0 Update 2c or Update 3.

A description of the fix can be found in the Release Notes.

For ESXi 6.7 Update 3:

A fix is available in Patch Release ESXi670-202111001

A description of the fix can be found in the Release Notes.

Users that are on ESXi 7.0 Updates 2 & 2a and ESXi 6.7 Update 3 should increase the DiskMaxIOSize setting from the DataCore recommends 512KB to 1024KB as a workaround. Refer to VMware ESXi Host Settings on how to change the ESXi DiskMaxIOSize setting.

Updated

Affects ESXi 7.0 Updates 2 & 2a and ESXi 6.7

Using DataCore’s recommended DiskMaxIOSize of 512KB causes unexpected IO timeouts and latency.

The DataCore-recommended DiskMaxIOSize setting of 512KB is being ignored by ESXi which can result in unexpected IO timeouts on the host, leading to increased latency.

For ESXi 7.0:

The problem only occurs in ESXi 7.0 Update 2 and 2a. A fix is available in VMware ESXi 7.0 Update 2c and 3. A description of the fix can be found in the VMware ESXi 7.0 Update 2c Release Notes.

PR 2751564: If you lower the value of the DiskMaxIOSize advanced config option, ESXi hosts I/O operations might fail. If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

ESXi 7.0 Update 2 and 2a users should increase the DataCore recommend DiskMaxIOSize setting from 512KB to 1024KB as a workaround. Refer to VMware ESXi Host Settings for an explanation on changing the DiskMaxIOSize setting.

For ESXi 6.7:

The problem occurs in ESXi 6.7 Update 3 (earlier versions of ESXi 6.7 appear to not have the issue). Currently, there is no fix available from VMware. Please contact VMware if more information is required. DataCore suggests mentioning the ESXi 7.0 PR number above as a reference.

ESXi 6.7 users should increase the DataCore recommend DiskMaxIOSize setting from 512KB to 1024KB as a workaround. Refer to VMware ESXi Host Settings for an explanation on changing the DiskMaxIOSize setting.

Updated

Affects ESXi 7.0 and 6.7

Using DataCore’s recommended DiskMaxIOSize of 512KB causes unexpected IO timeouts and latency.

Using the DataCore-recommended DiskMaxIOSize of 512KB is being ignored by ESXi which results in unexpected IO timeouts on the host, leading to increased latency.

For ESXi 7.0:

A fix is now available in VMware ESXi 7.0 Update 2c - ESXi_7.0.2-0.20.18426014

PR 2751564: If you lower the value of the DiskMaxIOSize advanced config option, ESXi hosts I/O operations might fail. If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

See also VMware ESXi 7.0 Update 2c Release Notes

For ESXi 6.7:

The problem occurs in ESXi 6.7 Update 3 (earlier versions of ESXi 6.7 appear to not have the issue). Currently, a fix is not available from VMware. In this case DataCore recommends that the DiskMaxIOSize setting be increased from 512KB to 1024KB until a fix is available. Refer to VMware ESXi Host Settings for more information on how to change the DiskMaxIOSize setting.

Updated

Affects ESXi 7.0 and 6.7

Using DataCore’s recommended DiskMaxIOSize of 512KB causes unexpected IO timeouts and latency.

There is a problem for ESXi 7.0 Updates 2 and 2a as well as ESXi 6.7 Update 3 when using the DataCore-recommended DiskMaxIOSize of 512KB. DataCore had discovered that this setting was being ignored by ESXi, resulting in unexpected IO timeouts on the host, leading to increased latency.

For ESXi 7.0 Update 2:

A fix is now available – refer to VMware ESXi 7.0 Update 2c Release Notes.

PR 2751564: If you lower the value of the DiskMaxIOSize advanced config option, ESXi hosts I/O operations might fail If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

For ESXi 6.7 Update 3:

DataCore recommends that the DiskMaxIOSize setting be increased from 512KB to 1024KB until a fix is available. Refer to VMware ESXi Host Settings for more information on how to change the DiskMaxIOSize setting.

Added

Affects ESXi 7.0 and 6.7

Using DataCore’s recommended DiskMaxIOSize of 512KB causes unexpected IO timeouts and latency.

DataCore have identified a problem with VMware for ESXi 7.0 Updates 2, 2a & 3 and ESXi 6.7 Update 3 when using DataCore’s recommended DiskMaxIOSize of 512KB. In some cases, this setting is being ignored which causes an IO timeout on the ESXi host which leads to increased latency.

VMware are aware of the problem but until a fix is released (possibly in ESXi 7.0 P03 and ESXi 6.7 P06) DataCore recommends that this setting be increased from 512KB to 1024KB. Users who wish to discuss this with VMware directly should quote SR21234904207 as there is yet no official VMware KB article that DataCore are aware of.

Added

Affects ESXi 7.0 Update 2

Read latency of QLogic 16Gb Fibre Channel adapters supported by the qlnativefc driver increases in certain conditions

Refer to the "Networking Issues" section in Known Issues for more information.

Affects ESXi 7.x and 6.7

Host ports do not log back into DataCore front-end (FE) ports but remain shown as ‘logged out’.

When using qlnative fc driver versions between 4.1.34.0 and 4.1.35.0 (ESXi 7.0) or 3.1.36.0 (ESXi 6.7) and an event occurs that causes the DataCore FE port to log out (e.g., stopping and starting a DataCore Server), the host port will may log back into the DataCore FE port once it is available and at the same time the ESXi host logs are continuously flooded with ‘ADISC failure’ messages. Manually re-initializing the DataCore FE port may workaround the problem. DataCore recommends using a minimum qlnative fc driver version of 4.1.35.0 (ESXi 7) or 3.1.65.0 (ESXi 6.7).

Updated

The command to add the new SATP rule was missing a ‘-o’ switch before the disable_action_OnRetryError setting.

Before (incorrect)

esxcli storage nmp satp rule add -V DataCore -M 'Virtual Disk' -s VMW_SATP_ALUA -P VMW_PSP_RR -O iops=10 disable_action_OnRetryErrors -c tpgs_on

After (corrected)

esxcli storage nmp satp rule add -V DataCore -M 'Virtual Disk' -s VMW_SATP_ALUA -P VMW_PSP_RR -O iops=10 -o disable_action_OnRetryErrors -c tpgs_on

Updated

This document has been reviewed for SANsymphony 10.0 PSP 11.

No additional settings or configurations are required.

Added

ESXi 7.0 is now qualified when using ALUA enabled connections to a DataCore Server.

All other compatibility lists in this document that apply to ESXi 7.0 have been updated.

Added

DataCore recommends that port binding is not used for any ESXi Host connections to DataCore Servers. Refer to ISCSI Settings for more details.

VMware Path Selection Policies – Round Robin/Fixed/Most Recently Used

Updated

Minor edits have been made to the notes to simplify and consolidate repeated information over these three sections. There are no settings changes required or any new information.

Known Issues

Updated

Failover/Failback

The information below has been simplified for clarity. No new changes have been added.

Affects all ESXi versions

ESXi hosts may see premature APD events during fail-over when a DataCore Server is stopped or shutdown if the SATP ‘action_OnRetryErrors’ default setting is used.

The default ‘action_OnRetryErrors’ setting switched between disabled and enabled and back to disabled again for subsequent releases of ESXi 6.x. Therefore, to guarantee expected failover behavior when a DataCore Server is stopped or shutdown on any ESXi 6.x Host, DataCore require that the SATP ‘action_OnRetryErrors’ setting is always explicitly set to ‘disabled’ (i.e., off). Refer to the VMware Path Selection Policies – Round Robin section for more information.

iSCSI connections

The information below has been simplified for clarity. No new changes have been added.

Affects all ESXi versions connected to SANsymphony 10.0 PSP 6 Update 5 and earlier

Multiple iSCSI sessions from the same IQN to a DataCore Front End port is not supported.

I.e. iSCSI Initiators that have different IP addresses, but which all share the same IQN when trying to connect to the same DataCore FE ports will not work as expected in SANsymphony versions 10.0 PSP 6 Update 5 and earlier. Only the first IP address to create an iSCSI session login will be accepted. All other iSCSI session login attempts from the same IQN regardless of the IP address will be rejected.

Added

Affects ESXi 6.0 and 6.5

ESXi 5.5, 6.0 and 6.5 do not support 4K enabled virtual disks

Refer to the Knowledge base article kB2091600.

Also see the section ‘Device Sector Formats’ in VMware’s own vSphere Storage guide for ESXi 6.7 for more limitations of 4K enabled storage devices.

It is possible to create 512B virtual disks from a 4KB configured Disk Pool.
Refer to 4 KB Sector Support for more information.

Updated

Affects ESXi 6.x and 5.5

When using QLogic's Dual-Port, 10Gbps Ethernet-to-PCIe Converged Network Adaptor

When used in CISCO UCS solutions, disable both the adaptor's ‘BIOS’ settings and 'Select a LUN to Boot from' option to avoid misleading and unexpected ‘disconnection-type’ messages reported by the DataCore Server Front End ports during a reboot of Hosts sharing these adaptors.

Updated

The ‘notes’ section has been updated.

VMware Path Selection Policies - All (RR, Fixed and MRU PSPs)

Information about how to remove an already configured SATP rule (to update/change the existing SATP rule) had been inadvertently removed in previous editions of this document and has now been re-instated.

Affects ESXi 6.x

ESXi hosts using ALUA may see premature APD events during fail-over scenarios when a DataCore Server is stopped/shutdown.

A new feature called ‘action_OnRetryErrors’ was introduced in ESXi 6.0. In ESXi 6.0 and then in ESXi 6.5 the setting was by default always set to ‘disable’ (off).

However in ESXi 6.7 VMware changed the default setting to ‘enable’ (on) and this caused unexpected failover/failback behaviour when a DataCore Server is stopped/shutdown. So a code change was made in SANsymphony 10.0 PSP 9 (to allow for this ‘enable’ setting to work as expected). Then in ESXi 6.7 Update3 VMware reverted the default setting back ‘disable’ once more.

As DataCore can never guarantee which versions of ESXi are being used in any SANsymphony Server Group at any time, this ‘action_OnRetryErrors’ SATP setting needs to be explicitly set (on your ESXi Host) according to the SANsymphony version that is installed.

For any ESXi 6.x Hosts the ‘action_OnRetryErrors’ SATP setting should be explicitly set to disabled (off).

Refer to the VMware Path Selection Policies – Round Robin section about how to configure the SATP rule for this setting and see the Knowledge base article.

Removed

The note about being able to use the ‘default’ SATP rule has been removed as this contradicts the statement that a custom SATP rule must be used for MRU.

Updated

Added an ‘Also see’ link.

Updated

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi-vcenter-server-67-performance-best-practices.pdf

Added

Affects ESXi 6.x

ESXi hosts using ALUA may see premature APD events during fail-over scenarios when a DataCore Server is stopped/shutdown.

A feature called ‘action_OnRetryErrors’ introduced in ESXi 6.0 was set to ‘disable’ (off) by default, this was also the default setting in ESXi 6.5. However, in ESXi 6.7 VMware changed this default value to ‘enable’ (on) which then required a SANsymphony code change (in 10.0 PSP 9). As DataCore cannot guarantee which versions of ESXi are being used in a SANsymphony Server Group, this SATP setting now needs to be explicitly set according to your SANsymphony version to guarantee expected behavior. Refer to the Knowledge Base article KB67006.

• For SANsymphony 10.0 PSP 9 and later ‘action_OnRetryErrors’ should be enabled (on) for all ESXi 6.x versions.
• For SANsymphony 10.0 PSP 8 and earlier ‘action_OnRetryErrors’ should be disabled (off) for all ESXi 6.x versions.

Refer to the VMware Path Selection Policies – Round Robin section about how to configure the SATP rule for this setting and see the Knowledge Base article KBKB67006.

VMware ESXi Path Selection Policies

Updated

Round Robin

Information updated based on the Known Issue – Failover/failback section regarding “ESXi hosts using ALUA may see premature APD events during fail-over scenarios when a DataCore Server is stopped/shutdown.” (See added Known Issue above).

Round Robin, Fixed and MRU

Updated

Previously it was stated that DataCore ‘recommends’ that users create custom SATP rules. This has been changed to ‘required’, following the settings listed in the appropriate sections.

Added

Affects ESXi 6.5

Virtual machine hangs and issues ESXi host based VSCSI reset

Virtual machines with disk size larger than 256GB in VMFS5 and larger than 2.5TB in VMFS6.

• Host becomes unresponsive, cannot restart specific hung virtual machines.
• Host requires a reboot to restart the virtual machines.

See the Knowledge Base article KB2152008.

Updated

This document has been reviewed for SANsymphony 10.0 PSP 9.

No additional settings or configurations are required.

Removed

Any information that is specific to ESXi versions 5.0 and 5.1 have been removed as these are considered ‘End of General Support’. See: Knowledge Base article KB2145103.

Added

Affects ESXi 6.x and 5.x

Thin Provisioning Block Space Reclamation (VAAI UNMAP) does not work if the volume is not native VMFS-5 (i.e. it is converted from VMFS-3) or the partition table of the LUN was created manually.

See the Knowledge Base article KB2048466.

The DataCore Server’s settings – Port Roles

Updated

The DataCore Server’s settings – Port Roles

Removed

All information regarding SANsymphony-V 9.x as this version is end of life (EOL).

Refer to End of life notifications for DataCore Software products for more information.

Appendix B - Reclaiming Storage from Disk Pools

Updated

Defragmenting data on virtual disks

For ESXi a VMFS volume cannot be defragmented.

Refer to VMware’s own Knowledgebase article: Does fragmentation affect VMFS datastores?

The VMware ESXi Host's settings

Updated

Advanced settings – DiskMaxIOSize

An explanation has been added on why DataCore recommends that the default value for an ESXi host is changed.

Appendix B - Reclaiming Storage from Disk Pools

Reclaiming storage on the Host manually

This section now has vSphere-specific references for the manual method of creating a new VMDK using ‘Thick Provisioning Eager Zero’.

VMware Path Selection Policies – Round Robin PSP

Updated

Creating a custom SATP rule

A minor update to the explanation when changing the RR IOPs value. It is now clearer when a change to the rule would or would not be expected to be persistent over reboot of the ESXi Host – see the ‘Notes’ section under the example.

Updated

ESXi 6.5 is now supported using DataCore’s SANsymphony Storage Replication Adaptor 2.0. ESXi 6.7 is currently not qualified. Please see DataCore’s SANsymphony Storage Replication Adaptor 2.0 release notes from https://datacore.custhelp.com/app/downloads.

Added

Affects ESXi 6.7 only

Failover/Failback takes significantly longer than expected.

Users have reported to DataCore that before applying ESXi 6.7, Patch Release ESXi-6.7.0-20180804001 (or later) failover could take in excess of 5 minutes. DataCore is recommending (as always) to apply the most up-to-date patches to your ESXi operating system. Refer to the Knowledge Base article KB56535.

VMware ESXi compatibility lists - ESXi operating system versions

Added

Added to the ‘Notes’ section:

iSER (iSCSI Extensions for RDMA) is not supported.

VMware ESXi compatibility lists

Updated

VMware VVOL VASA API 2.0

Previously ESXi 5.x was incorrectly listed as VVOL/VASA compatible with 10.0 PSP 4 and later.