SANsymphony supports increased resiliency of Metro Cluster configurations involving VMware ESX. This support is intended for those who know how to implement a greater level of SDS customization to match their precise networking and computing choices.
The solution outlined here should not be confused with the Witness feature and has no relationship with the witness workflow. The Witness feature is invoked when all means of communications are cut-off between 2 SDS nodes involved in a mirrored virtual disk. Here, the resilient option kicks in only when inter-SDS mirror communication is cut-off but the out-of-band (LAN) communication is still active and able to carry signaling CAP messages across mirror partners.
In some installations, all virtual machines might be configured to run specifically on one of the SDS pairs in metro-clusters, as in Site A in this picture:
In such configurations, it is possible for a SANsymphony virtual disk to become inaccessible at Site A due to an action from Site B, even though Site A still contains up-to-date data. This can cause virtual machines at Site A to fail even though the local virtual disk data is up-to-date.
SANsymphony's high-availability failover logic includes the notion of a “resilient server”. This provides a way for the SANsymphony administrator to express a priority for Site A that can enable SANsymphony to preferably remain I/O active and turn around a “go inaccessible” notification back to its originator at Site B.
The following sequence of drawings illustrates how Site B can affect Site A production, even when Site B has no virtual machines running in production. In the following drawing, all virtual machines are running happily at Site A.
Next, assume a particular failure mode of the Inter-site Links (ISLs) as shown. This failure mode will prevent the DataCore Server at Site B from sending mirroring messages to Site A. At this point, virtual machines are still running at Site A.
Notice in this failure mode that a switch has failed. As a result of this switch failure, the ESX node at Site B will send a Compare and Write SCSI command to the virtual disk as served from the DataCore Server at Site B.
The DataCore Server at Site B will be unable to mirror the content of the Compare and Write command to the DataCore Server at Site A, because the switch connecting DataCore Server B to ISL2 has failed, and because ISL1 itself has failed. Because it has failed to mirror the content of the CAW command, Server at Site B will tell server at Site A “go inaccessible because you do not have the latest data”.
SANsymphony's resiliency setting allows you to declare that the side primarily selected to be running the storage becomes the resilient side of the pair, if at all possible depending on states.
In SANsymphony, you can select a "resilient" server individually per virtual disk. That selection is made through a Powershell cmdlet. Each resiliency setting applies to a specific virtual disk and must be set up per virtual disk. Powershell makes it easy to set this up, even for a large number of virtual disks.
The resiliency setting cannot be derived purely from the Host "preferred server" configuration, because typically - in metro-cluster configurations especially - one Host will be configured to be preferred on side A while another Host in the cluster preferred on side B. It's when those two Hosts compete for exclusive access (via the inaccessible state) that the configured "resilient" server will be the tie breaker and win.
The basic principle is that when a SDS cannot mirror I/O it sends a message to block the partner's client I/Os (the Out-Of-Sync message, aka. OOS) to prevent the partner diverging. If this OOS message is received by a "resilient" side it would then be replied back to the message originator with a specific error code that would instead instruct the originator to go OOS.
The Resilient Server setting is not an unconditional override to SANsymphony’s High Availability failover/failback logic. Rather, it is an extension of the logic that can help an administrator control certain contention scenarios that would otherwise resolve less deterministically, resulting in an undesired outcome as shown above.
Tip: Do not use the resiliency setting on virtual disks when hosts accessing that virtual disk are configured in “Preferred All”. Using resiliency on a virtual disk that has attached Hosts configured to use their paths as "preferred all" will result in inaccessible patterns new to the user. For example, setting side A to be resilient and losing the mirror link from B → A will actually make B become inaccessible whereas in previous versions it was side A that would become inaccessible. For data consistency that will be perfectly fine as either side may go inaccessible in such case. There will be specific trace messages in CAP to highlight this. In the end one of the 2 sides will end up being OOS and that is the safeguard that we need to know we have.
Before enabling resiliency, ensure that your environment meets the following requirements:
o Each host must have a different preferred server. For example, Host1 preferred on SDS1 and Host2 preferred on SDS2.
o Serve a mirrored virtual disk to each host. Continuing with the example, this would be Host1 and Host2.
o Assume all virtual machines are configured to run on Host1 which is talking to SDS1.
To enable resiliency, use PowerShell to set the virtual disk’s properties. The following example enables resiliency for SDS1:
Set-DcsVirtualDiskProperties -VirtualDisk vdisk1 -PreferredServer SDS1 -ResiliencyEnabled $true
For details on the resiliency-related options in Set-DcsVirtualDiskProperties, see the DataCore Cmdlet Reference Guide.