Fibre Channel and iSCSI Connections

Multi-ported vs Single-ported Adaptors

Fibre Channel host bus adapters (HBA) and network interface cards (NICs) are often available in single, dual, and quad-port configurations but often there can be very little difference in performance capabilities when comparing two single-ported adaptors with one dual-port adaptor, or two dual-port HBAs with one quad-port HBA1.

There is however a significant implication for high availability when using a single adaptor - even if it has multiple ports in it - as most types of adaptor failures will usually affect all ports on it rather than just one (or some) of them.

Using many adaptors that have a small number of ports on them will reduce the risk of multiple port failures happening at the same time.

iSCSI Connections

Link Aggregation and SCSI Load Balancing

For better performance use faster, separate network adaptors and links instead of teaming multiple, slower adaptors. For high availability use more individual network connections and multipath I/O software than rely on either teaming or spanning tree protocols to manage redundancy. Just like Fibre Channel environments, that use independent switches for redundant fabrics, this also prevents ‘networkloops’ making spanning tree protocols obsolete and simplifies the overall iSCSI implementation.

Fundamentally, SCSI load-balancing and failover functions are managed by Multipath I/O protocols2; TCP/IP uses a completely different set of protocols for its own load-balancing and failover functions. When SCSI commands, managed by Multipath I/O protocols but ‘carried’ by TCP/IP protocols are combined (i.e., iSCSI), then the interaction between the two protocols for the same function can lead to unexpected disconnections or even complete connection loss.

NIC Teaming

NIC teaming is not recommended for iSCSI connections as it adds more complexity (without any real gain in performance); and although teaming iSCSI Targets - i.e. Front-end or Mirror ports - would increase the available bandwidth to that target, it still only allows a single target I/O queue rather than, for example, two, separate NICs which would allow two, independent target queues with the same overall bandwidth.

Secondary/Multiple IP-Addresses per NIC

SANsymphony’s iSCSI Target implementation binds to the IP Address rather than to the MAC Address of a NIC. This means multiple IP-Addresses per NIC can bind multiple iSCSI Targets to the same NIC Hardware. This allows the system to scale iSCSI IO performance to match the capabilities of the NIC. Each iSCSI target instance can process up to 80k IO/s. A 10 GbE NIC Port would support up to 160k IO/s from the physical interface specification. This means it makes sense to apply up to 3 IP-Addresses per 10 GbE NIC for maximum saturation of a NIC.

Additional CPU cores may be required for high iSCSI target counts.

iSCSI Loopback Connections

Do not connect iSCSI Initiator sessions to iSCSI Targets at the identical IP- and MAC-Address. Always make sure to have sessions between different physical MAC-Addresses, otherwise, the IP stack will compete for identical transmission buffers, leading to very high latency and low IO performance.

Unwanted iSCSI Targets

SANsymphony’s iSCSI Target automatically binds to all available IP-Addresses found on regular network interfaces. Sometimes users want to have no iSCSI bound to certain network ports and do not want to see unwanted iSCSI targets in the configuration. To remove unwanted iSCSI Targets permanently the appropriate DataCore Software iScsi Adapter instance in Device Manager/DataCore Fibre-Channel Adapters must be disabled. The instance can be identified by matching the MAC Address of the port in the Management Console with one of the instances of Device Manager. After disabling the instance in Device Manager, the iSCSI Port in SANsymphony can be safely deleted and will not re-appear unless the disabled adapter instance in Device Manager is re-enabled.

Spanning Tree Protocols (STP/RSTP/MSTP)

None of the Spanning Tree Protocols - STP, RSTP, or MSTP – are recommended on networks used for iSCSI as they may cause unnecessary interruptions to I/O; for example, other, unrelated devices generating unexpected network topology causing STP to re-route iSCSI commands inappropriately or even blocking them completely from their intended target.

Also see:

The DataCore and Host Servers - Qualified Hardware

Known Issues - Third-Party Hardware and Software

SANsymphony - iSCSI Best Practices

A Summary of Recommendations for Fibre Channel and iSCSI Connections

Fibre Channel and iSCSI Connections

  • Use many adapters with a smaller numbers of ports on them as opposed to fewer adapters with larger numbers of ports.

ISCSI Connections

  • Use faster, separate network adaptors instead of NIC teaming.
  • Do not use NIC teaming or STP protocols with iSCSI connections. Use more individual network connections (with Multipath I/O software) to manage redundancy.
  • Use independent network switches for redundant iSCSI networks.
  • Add multiple IP-Addresses to NICs to scale iSCSI Targets.
  • Do not connect Loopback iSCSI sessions between the same IP and MAC.
  • Unwanted iSCSI Targets can be removed by disabling the DataCore Software iSCSI Adapter instance in Device Manager/DataCore Fibre-Channel Adapters.