Situation

#

We encountered an unrecoverable issue where the secondary server (HOSTEX-DR) had the databases mounted but became irrecoverable. As a result, the databases could not be mounted on the primary server (BHM-ITOPS-EX16).

To resolve this, we had to:

1. Stop the Database Availability Group (DAG) and remove the DAG members.
2. Remove the database copies from the secondary server (HOSTEX-DR).

Once the copies were deleted, the databases successfully began mounting on the primary server (BHM-ITOPS-EX16).

Steps for Data Center Recovery

#

1. Stop the DAG on the Secondary Site

#

Use the following command to stop the DAG on the secondary site, which may not be responsive due to failures:

Stop-DatabaseAvailabilityGroup -Identity Host-DAG -ActiveDirectorySite DR-Site  

You may encounter cluster operation errors. In this case, proceed to stop the DAG on individual servers.

2. Stop the DAG on Specific Mailbox Servers

#

Run the following command to stop the DAG on the both servers:

Stop-DatabaseAvailabilityGroup -Identity Host-DAG -MailboxServer HOSTEX-DR  

Stop-DatabaseAvailabilityGroup -Identity Host-DAG -MailboxServer BHM-ITOPS-EX16

  
If errors persist, use the `-ConfigurationOnly` flag to stop the DAG:  
```powershell  
Stop-DatabaseAvailabilityGroup -Identity Host-DAG -MailboxServer HOSTEX-DR -ConfigurationOnly  

Stop-DatabaseAvailabilityGroup -Identity Host-DAG -MailboxServer BHM-ITOPS-EX16 -ConfigurationOnly

## 3. Check Mailbox Database Copy Status

Use the following command to check the status of the database copies across the DAG:  
```powershell  
Get-MailboxDatabaseCopyStatus *  

4. Stop the Cluster Service

#

On the affected node, stop the cluster service to ensure that the failed node is no longer part of the cluster:

net stop clussvc  

5. Clear Cluster Node

#

To remove the failed node from the cluster:

Clear-ClusterNode  

6. Remove Database Copies from the Failed Server

#

For each mailbox database that is on the failed server, remove the copy using the following command:

Remove-MailboxDatabaseCopy -Identity <DatabaseName>\<ServerName>  

For example:

Remove-MailboxDatabaseCopy -Identity BarronMachineFAB\HOSTEX-DR  

7. Remove the Failed Server from the DAG

#

After all database copies are removed, remove the server from the DAG configuration:

Remove-DatabaseAvailabilityGroupServer -Identity Host-DAG -MailboxServer "HOSTEX-DR" -ConfigurationOnly  

8. Check Database Mount Status

#

Verify that databases on the active node are mounted correctly:

Get-MailboxDatabaseCopyStatus *  

9. Remove the DAG

#

Once the failed node has been removed and all databases are successfully mounted on the active node, remove the DAG:

Remove-DatabaseAvailabilityGroup -Identity Host-DAG