Maintenance Mode
In this topic:
Redistributing to a DataCore Server
Using the Dynamic Data Resiliency Method
Also see:
Auto-evacuating Pools on Offline Servers
Replacing/Moving a Storage Source in a Virtual Disk
Shared Multi-port Array Support
About Maintenance Mode
The Maintenance Mode feature allows users to perform maintenance operations while maintaining high availability to hosts.
Uses of the Maintenance Mode include:
- Evacuating storage sources from a server that will be unavailable for a period of time, such as during maintenance. After maintenance is performed, the virtual disk storage sources can be moved back to the original server or redistributed among other servers in the server group.
- Moving storage sources before a server is permanently removed from the server group configuration.
- Rebalancing the virtual disk load among other servers in the server group.
The Maintenance Mode feature consists of two operations which provide a way to evacuate and redistribute virtual disk storage sources created from pools from one server to another while maintaining uninterrupted host access to the virtual disks. A minimum of three DataCore Servers are required in the server group to perform the operations.
Evacuating and redistributing virtual disk storage sources may be accomplished by either the use of a shared disk pool or 3-copy virtual disks. These operations are handled in slightly different ways depending on the method used, see Using the Shared Disk Pool Method and Using the Dynamic Data Resiliency Mirroring Method for more information. Read this topic in its entirety before performing the evacuation or redistributing operations.
Evacuating a DataCore Server
The Evacuate DataCore Server operation moves or "evacuates" all eligible selected virtual disks from the server that initiated the evacuation command to another server in the same server group. The destination server may be another server that shares the same pool or another server that has a storage source in a 3-copy virtual disk.
The virtual disks must meet the requirements to be moved.
As part of the evacuation operation, an evacuation plan is generated for the server where the operation was initiated.
The evacuation plan initially consists of an inventory of each virtual disk on the server:
- Warnings are generated for virtual disks that are not eligible to be moved with the reason they cannot be moved. It may be possible to take actions in order to make the virtual disk eligible to be evacuated. After all desired actions have been taken to remove warnings from the plan, the evacuation operation may be re-run to create an updated evacuation plan.
- Eligible virtual disks can be selectively excluded from the plan, in which case they will not be moved when the operation is performed. This is useful when rebalancing virtual disk load on servers.
- Eligible virtual disks that have been selected to move have the most suitable destination server automatically selected. By default, the software selects the best candidate based on the lightest virtual disk load and number of available paths. Servers with redundant mirror paths to the storage source will be preferred over single paths. When a sufficient number of candidates exist, the destination server may be selected from the list of available servers. Candidate destination servers are ranked starting with the most suitable at the top of the list.
Only eligible virtual disks that are selected in the evacuation plan are moved during the evacuation operation. See Performing the Operations for instructions.
Redistributing to a DataCore Server
The Redistribute to DataCore Server operation moves or "redistributes" all selected eligible virtual disks in the server group to the server that initiated the redistribute command.
The virtual disks must meet the requirements to be moved.
As part of the redistribution operation, a redistribution plan is generated.
The redistribution plan initially consists of all virtual disks in the server group that are capable of being moved to the server that initiated the operation:
- Virtual disks that are capable of being moved include either virtual disks from a shared pool that do not already have a storage source from the server that initiated the operation, or are 3-copy virtual disks with a storage source from the server that initiated the operation.
- Warnings are generated for virtual disks that are not eligible to be moved with the reason they cannot be moved. Actions may be taken in order to make the virtual disk eligible to be redistributed. After all desired actions have been taken to remove warnings from the plan, the redistribution operation may be re-run to update it.
- Eligible virtual disks can be selectively excluded from the plan by clearing the associated check boxes, in which case they will not be moved when the operation is performed. This is useful when rebalancing virtual disk load on servers.
- The origin server, where storage sources are currently located, can be selected in the plan when virtual disk storage sources exist on a sufficient number of servers.
Only eligible virtual disks that are selected in the redistribution plan are moved during the redistribute operation. See Performing the Operations for instructions.
Using the Shared Disk Pool Method
Virtual disks are moved from one server to another that are sharing the same pool:
- Virtual disks that are eligible to be moved have storage sources created from a pool that is shared or is capable of being temporarily shared in order to perform the operation. The storage sources will be moved from the originating server to another server (destination server) that shares that same pool. The destination server cannot already have a storage source in the virtual disk being moved.
This applies to both evacuate and redistribute.
- In order to move a storage source there must be at least one additional server (destination server) that shares the same pool, but does not already have a storage source in the virtual disks being moved.
Examples:
- Assume that a dual virtual disk is created from the same Shared Pool 1 on Server 1 and Server 2. The storage source on Server 1 could be moved to Server 3 that also shares Shared Pool 1.
In this example, moving a storage source in a dual virtual disk requires a minimum of three servers that share the same pool.
- Assume that a mirrored virtual disk is created from Non-shared Pool 1 on Server 4 and Shared Pool 1 on Server 1. The storage source from Shared Pool 1 on Server 1 could be moved to another server that shares the same Shared Pool 1.
In this case, the storage source from Non-shared Pool 1 on Server 4 could not be moved because that pool was not shared with any other server. Although three servers were used in this operation, two servers that share the same pool were required.
- Assume that a dual virtual disk is created from the same Shared Pool 1 on Server 1 and Server 2. The storage source on Server 1 could be moved to Server 3 that also shares Shared Pool 1.
- Single, dual, or mirrored virtual disks can be moved.
Hosts will temporarily lose access to single virtual disks during the operation.
- The shared disk pool can be an authorized Shared Multi-port Array, although this is not a requirement. See Shared Multi-port Array Support for requirements for shared pools.
- Servers do not need to be licensed for Shared Multi-port Array (SMPA), but in that case some preparation is required to make the pool capable of being temporarily shared in order to perform the operation.
Limitations apply. See Important Notes on Using Shared Disk Pools.
Using the Dynamic Data Resiliency Mirroring Method
Virtual disks with dynamic data resiliency have copies of data on three different DataCore Servers, and are referred to as 3-copy virtual disks. Host access is enabled from two servers at the same time. (See Dynamic Data Resiliency for more information.)
During an evacuation operation, host access to the virtual disk storage source being evacuated is switched to another storage server in the 3-copy virtual disk to maintain high availability during the operation and maintenance procedure. The redistribute operation works in the same manner. During these operations, the three storage sources remain associated with the virtual disk; only paths are adjusted for the new storage source. This method allows uninterrupted access to the data and mirroring capabilities during the entire process.
- For example, a server group contains DataCore Servers named Server1, Server2, and Server3 and 3-copy virtual disks exist from these servers. The administrator plans on performing maintenance on Server1 which requires considerable down-time. Data is being actively transferred from the host using front-end paths to Server1 and Server2. The administrator evacuates Server1 to Server3 which switches the active front-end paths from Server1 to Server3. High availability and data redundancy is maintained through Server2 and Server3. Once maintenance procedures are complete, redistribution to Server1 can restore the original front-end paths.
Inactive storage sources (with host access disabled) are not evacuated from the server to be evacuated. When the virtual disks are redistributed after maintenance is performed, data on the inactive storage source is resynchronized. If the server is to be permanently removed from the configuration, the inactive storage sources can be split from the 3-copy virtual disk.
General Maintenance Mode Notes
When the evacuation operation is being performed during a software upgrade, do not stop or restart the server that was upgraded until an additional server has been upgraded to the same version. Always leave one upgraded server running until all servers in the server group are upgraded. This applies to both evacuate and redistribute.
- When storage sources are moved, mirror and front-end paths are adjusted to use the new storage source.
Ensure that path connections exist that can be used for creating mirror paths to the new storage source, as well as path connections between new storage sources and hosts for creating front-end paths between the hosts and server. The multi-copy virtual disks will not be moved unless mirror paths can be recreated.
- If the virtual disk had redundant paths (two paths in each direction), we attempt to re-establish redundant paths, but if there is an insufficient number of paths, then single paths will be created and a warning message is logged.
- If the virtual disk has more than redundant paths, the software will create a maximum of redundant (two) paths. Additional paths must be added manually.
- The software will attempt to match LUNs. If LUNs cannot be matched and the host requires LUN matching, an alert will be generated; otherwise an available LUN will be used.
- If the virtual disk is served and front-end paths or connections do not exist, the storage source will be moved and an alert will be generated.
- Moving virtual disks from the specified server will cause the host to lose access to this server for the selected virtual disks. To ensure continued host access verify there are available front-end paths to another server.
Hosts will temporarily lose access to single virtual disks during the operation.
- Virtual disk storage sources can be moved when the server is offline or unavailable. However, the software cannot determine if the server was shutdown cleanly and the cache was flushed before the server was shutdown.
- Due to the risk of data mismatch, mirrored storage sources from an originating server that is unavailable or failed will not be moved unless those mirrored virtual disks were previously placed in write-through mode.
- Due to the risk of data mismatch, mirrored storage sources from a failed local disk on the originating server that are in recovery or needing recovery will not be moved.
Alert warnings are posted for each virtual disk that cannot be moved.
-
The following virtual disks cannot be moved:
- Snapshots and virtual disks that are the source of snapshots cannot be moved. Snapshots must be deleted in order to move the source virtual disks.
- Rollbacks and virtual disks that are the source of rollbacks cannot be moved. Rollbacks must be deleted in order to move the source virtual disks.
- Virtual disks with sequential storage enabled cannot be moved. Sequential storage must be disabled in order to move the virtual disks.
- Storage sources used in replication cannot be moved. Split the replications in order to move the virtual disks.
- Inactive storage sources in 3-copy virtual disks are not moved (host access disabled).
- Before beginning an evacuation or distribution operation, there is a 30-second wait period, during which time the paths are off-lined and the cache is flushed (if possible), then the operation begins. If virtual disks are in write-through mode, the 30-second wait period is not required and the process starts immediately. The wait period does not apply to servers that are offline.
Important Notes on Using Shared Disk Pools
Using Shared Pools without an SMPA License
- Servers do not need to be licensed for Shared Multi-port Array (SMPA) to use the Evacuate DataCore Server and Redistribute to DataCore Server operations.
- In order to perform the Evacuate DataCore Server and Redistribute to DataCore Server operations without being licensed for SMPA:
- All storage sources (virtual disks) from a pool must be evacuated or redistributed. This means performing a full evacuation or redistribution of virtual disks from the pool; no partial evacuation or redistribution.
If any storage sources are not healthy at the time of evacuation or redistribution and therefore cannot be moved, the operation will fail.
- All storage sources must be moved to the same DataCore Server.
- Storage sources from pools configured with mirrored pool disks can be moved.
- All storage sources (virtual disks) from a pool must be evacuated or redistributed. This means performing a full evacuation or redistribution of virtual disks from the pool; no partial evacuation or redistribution.
- To prepare to move storage sources without an SMPA license:
- A minimum of three servers must exist in the same server group. Originating server and destination server must be in the same server group.
- A common pool of managed storage must be created between the originating server (where the storage sources exist) and the destination server (where the storage sources will be moved). The storage array must be capable of presenting disks to more than one DataCore Server simultaneously.
- Back-end paths must be created between each disk in the pool to be evacuated on the originating server and the destination server.
- Mirror paths must be available between the originating server and the destination server.
- Front-end ports must exist to allow path connections from the host to both originating and destination servers.
For example, assume that virtual disk storage sources are created from Non-shared Disk Pool 1 on Server 1 and Non-shared Disk Pool 2 on Server 2, and you wish to evacuate the storage sources on Non-shared Disk Pool 1 to Server 3 in the same server group. Server 1 is the originating server and Server 3 is the destination server.
- On the storage array, temporarily create back-end paths from each pool disk in Non-shared Disk Pool 1 to Server 3. This will create a temporary environment where storage sources can be moved from Server 1 to Server 3. Then the evacuate operation from Server 1 to Server 3 can begin with the license limitations described in this section.
- In order to perform the Evacuate DataCore Server and Redistribute to DataCore Server operations without being licensed for SMPA:
- In order to (1) perform a partial evacuation or redistribution, (only some of the storage sources are evacuated or redistributed), or (2) evacuate or redistribute storage sources to more than one DataCore Server, the SMPA license is required and the pool must be an authorized SMPA. See Shared Multi-port Array Support for more information about authorized SMPA pools.
If not licensed for SMPA, operations for either case will be cancelled.
Performing the Operations
Read this topic in its entirety before performing these operations.
Perform this operation to move virtual disks from the server that initiated the command.
- In the DataCore Servers Panel, right-click the server and select Evacuate DataCore Server. This operation can also be performed from the ribbon when the Server Details page is open.
- The evacuation plan appears listing all virtual disks created from the server.
- In the check box column, virtual disks that are eligible to be moved are automatically selected. Clear the check boxes for eligible virtual disks that should not be moved.
The check box in the column header can be clicked to clear all check boxes (useful when there are numerous eligible virtual disks, but only a few should be selected).
- In the Destination Server column, the best destination server choice, based on virtual disk load and available paths, is automatically selected. This selection can be changed if another eligible server exists. To change the server selection, click the down arrow in the Destination Server column and select another server from the list.
- In the Warning column, warnings are provided for virtual disks that cannot be evacuated. Only selected virtual disks without warnings can be evacuated.
It may be possible to perform actions which can make the virtual disks eligible to be moved. If that is the case, to make those virtual disks eligible to be moved, perform the actions and initiate the evacuation operation again to create a new evacuation plan.
- In the check box column, virtual disks that are eligible to be moved are automatically selected. Clear the check boxes for eligible virtual disks that should not be moved.
- Click Finish to evacuate the eligible virtual disks that are selected; otherwise click Cancel to cancel the operation.
- After evacuation has been completed, hosts will have to be rescanned (or other similar operation) in order to recognize the new front-end paths.
To redistribute virtual disks:
Perform this operation to move virtual disks to the server that initiated the command.
- In the DataCore Servers Panel, right-click the server and select Redistribute to DataCore Server. This operation can also be performed from the ribbon when the Server Details page is open.
- The redistribute plan appears listing all virtual disks that are capable of being moved to the server.
- In the check box column, virtual disks that are eligible to be moved to the server are automatically selected. Clear the check boxes for eligible virtual disks that should not be moved to the server.
The check box in the column header can be clicked to clear all check boxes (useful when there are numerous eligible virtual disks, but only a few should be selected).
- In the Origin Server column, select the server where the virtual disk storage source should be moved from if multiple server options exist The storage source selected will be moved to the server initiating the command.
- In the Warnings column, warnings are provided for virtual disks that cannot be redistributed. Only selected virtual disks without warnings can be redistributed.
It may be possible to perform actions which can make the virtual disks eligible to be moved. If that is the case, to make those virtual disks eligible to be moved, perform the actions and initiate the redistribute operation again to create a new redistribution plan.
- In the check box column, virtual disks that are eligible to be moved to the server are automatically selected. Clear the check boxes for eligible virtual disks that should not be moved to the server.
- Click Finish to redistribute the eligible virtual disks that are selected; otherwise click Cancel to cancel the operation.
- After redistribution has been completed, hosts will have to be rescanned (or other similar operation) in order to recognize the new front-end paths.