You are browsing the archive for Azure Backup Server.

Azure Backup Server: Unprotected servers still showing up in the Azure portal even though their protection was stopped 3 months ago

9:49 am in Azure, Azure Backup, Azure Backup Server, Azure portal, Cloud, Cloud backup, MABS, MABS v2 by Wim Matthyssen

 

To help protect your hybrid backup setup with an Azure Backup Server (MABS), Microsoft introduced some security features built on three principals – Prevention, Alerting and Recovery. These features are enabled by default for newly create Recovery Services vaults, for existing vaults this link will show you how you can enable them. One of these features related to recovery will ensure you that Azure backup will retain all deleted backup data for 14 days, which ensures you can recover data using any old or recent recovery point(s).

clip_image002

Sometime ago I reconfigured a Protection Group which protected some Hyper-V VMs. Two Domain Controllers (DCs) were taken out of the Group and setup to only backup the C drive and the System State. On the MABS server all configuration went well and did not cause any specific issues or errors. However last week when I was checking the Recovery Services vault used to store the cloud backups,I noticed those two DCs were still showing up in the Backup items overview.

clip_image004

Like you can see, those two VMs are still there with no Disk or Cloud Recovery Points created after the protection was disabled.

clip_image006

To get the issue fixed, I followed some standard steps I always follow when having issues with a MABS. The first one is checking the current Azure Backup Agent version installed on the MABS, which was version 2.0.9109.0. Because there is a newer version available (at the time of writing version 2.0.9118.0), step one was getting that one in place.

clip_image008

To download the latest agent go to your Recovery Services vault blade in the Azure portal. Select Backup and on the Getting Started with Backup blade, select Backup goal. In the drop-down menu(s), select On-premises and Files and folders, click OK. In the Prepare Infrastructure blade, click Download Agent for Windows Server or Windows Client. Save MARSAgentInstaller.exe.

clip_image010

clip_image012

Install the latest agent on the MABS server. After the agent installation completes restart the following service:

Microsoft Azure Recovery Services Management Agent

clip_image014

clip_image016

clip_image018

Although the agent is now at the latest version it still did not fix the protection status of the deleted servers in the Azure portal.

After doing a little more troubleshooting (reading the logs, etc.) , I decided to open an Azure support ticket. The support agent who assisted me, told me, just like I already suspected, that this was currently  the default behavior from the azure backup service in some Azure regions (current backend design behavior like they say). The product team was already aware of this issue and they definitively will fix it in some later update.

If you cannot wait for the update, there is a quicker fix for the issue, you just need to delete the whole MABS server from the Azure portal and reconnect the server all over again. However, for me and even more for the customer this was a no go. So, we will wait for the proper backend update which will hopefully not take that long anymore.

Hope this helps whenever you face the same backup behavior in the Azure portal with your deleted MABS backups.

Wim Matthyssen (@wmatthyssen)

Azure Backup Server: SQL 2016 AlwaysOn protection fails with Internal error code 0x80990F75

3:54 pm in 0x80990F75, Azure, Azure Backup, Azure Backup Server, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, SQL Always On Availability Groups, SQL Server 2016 by Wim Matthyssen

While setting up a new backup job to protect a SQL Server 2016 AlwaysOn Availability Group (AG) using Azure Backup Server (MABS), the job failed and ended up with the Protection Status – Replica is inconsistent.

clip_image002

Because this status does not say a lot about what exactly went wrong, I looked up the Critical alert under the Monitoring tab. There the following more detailed message was shown:

The DPM job failed for SQL Server 2016 database <DBname> on <serverName > because the SQL Server instance refused a connection to the protection agent. (ID 30172 Details: Internal error code: 0x80990F75)

clip_image004

This issue occurs because when you create a new Availability Group by default, the location where backups should occur is set to Prefer Secondary and the setting Make Readable Secondary is set to No, which always results in MABS getting the above error.

clip_image006

To resolve the issue, open SQL Management Studio and connect to server instance that hosts the primary replica. Expand the Always On High Availability node and the Availability Groups node. Click the availability group whose replica you want to change and expand Availability Replicas.

clip_image008

Right-click the Availability Replica, and click Properties (be sure to repeat this steps for all Availability Replicas you want to backup with MABS).

clip_image010

In the Availability Replica Properties dialog box, set the value of the field Readable secondary to Yes. Click OK to save the new setting.

clip_image012

When you now run the Perform consistency check … job on the failed Protected member in the MABS console, the status should end up in OK.

If not, and the status still ends up in Replica is inconsistent, be sure to check out my previous blog post http://scug.be/wim/2017/06/19/microsoft-azure-backup-server-unable-to-configure-protection-for-a-sql-database-id-3170-and-33424/ to see if the user NT Authority\SYSTEM has sysadmin rights on the SQL Server instance(s).

If on the other hand the status ends up in Online recovery point creation failed, just right-click the Protected member again and select Resume azure backups…

I hope this helps and if you have any questions feel free to contact my through my Twitter handle.

Wim Matthyssen (@wmatthyssen)

Azure Backup Server: Remove Unprotected computers with protection agent installed

8:36 am in Azure Backup Server, MABS, MABS v2, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, PowerShell by Wim Matthyssen

While doing maintenance on a customer’s Azure Backup Server (MABS), I was unable to remove some unprotected computers in the MABS console. The Remove resulted in a fail and the error page didn’t show a direct reason why this occurred.

clip_image002

clip_image004

clip_image006

clip_image008

But no worries, this is where PowerShell came into the rescue to force the removal of the broken agent from the DPMDB database. Take notice that this solution will not uninstall the protection agent from the (un)protected computer. When required, you still need to uninstall that agent manually.

Open the DPM Management Shell and run the following command, it will prompt you for the rest of the parameters one at a time (always use the FQDN name for both parameters).

You can also run this command with all paramaters already filled in. Just replace [DPMServerName] with the name of the MABS server and [ProtectedComputerName] with the name of the (un)protected computer that must be removed.

image

Like you can see the agent(s) are now removed from the MABS console.

clip_image012

Hope it helps.

Wim Matthyssen (@wmatthyssen)

MABS v2: Unable to install DPM Remote Administration console on a W2K8 R2 SP1 server because mi.dll is missing

9:32 am in Azure, Azure Backup, Azure Backup Server, DPM Remote Administration, mi.dll, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, PowerShell, WMF 5.1 by Wim Matthyssen

While installing the DPM Remote Administrator console on a Windows Server 2008 R2 SP1 (W2K8 R2 SP1) for remote administration of a customers Microsoft Azure Backup Server (MABS) v2, I stumbled upon the below error message, which resulted in the setup being aborted:

The Program can’t start because mi.dll is missing from your computer. Try reinstalling the program to fix this problem.

clip_image002

This error shows up because one of the following requirements is not installed: Windows Management Framework 4.0, .NET Framework 4.0 or Visual C++ Redistributable for Visual Studio 2012 Update 4

needs to be installed to be able to deploy the DPM Remote Administration console on a W2K8 R2 server.

To fix the issue, I checked if all latest Windows Updates were installed. Afterwards I installed the Windows Management Framework 5.1 (WMF 5.1), .NET Framework 4.0 and the Visual C++ Redistributable for Visual Studio 2012 Update 4 on the W2K8 R2 SP1 server, which can be downloaded from the link above. To ease up and to automate the installation, you can use the below PowerShell script (copy and/or save as .ps1) to get things downloaded somewhat faster.

clip_image004

When the C:\Temp folder opens after the downloads, run Install-WMF5.1.ps1. (PowerShell window with Administrator privileges) to install WMF 5.1

clip_image006

clip_image008

clip_image010

clip_image012

Before rebooting, also run the two other packages, dotNetFx40_Full_setup.exe and vcredist_x64.exe (if required). When done reboot the server.

clip_image014

When the server is rebooted, check if mi.dll exists under C:\Windows\System32.

clip_image016

You can now start Setup.exe (Microsoft Azure Backup Server folder) and start the DPM Remote Administration installation.

clip_image018

clip_image020

Hope this post helps whenever you face the same problem.

Wim Matthyssen (@wmatthyssen)

MABS v2: Error [0x8007007b] when performing a System State Backup on a DC running on a VMware VM

8:33 am in Azure, Azure Backup, Azure Backup Server, Cloud, Error [0x8007007b], MABS, MABS v2, Power, PowerShell, VMware by Wim Matthyssen

While configuring a Microsoft Azure Backup Server (MABS) v2 at a customer site, I encountered a problem while performing a System State Backup of their domain controllers (DC’s). The Protection Status showed Replica is inconsistent.

clip_image002

When looking in the Monitoring tab, following detailed message is show:

DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code: 0x605A140).(ID 30229 Details: Internal error code: 0x8099ED0)

clip_image002[6]

Because the first part of making a System State Backup is done by the local Windows Server Backup (WSB) feature, logon to the protected server and open Windows Server Backup (Server Manager – Tools – Windows Server Backup). There a message was shown indicating that the last backup has Failed.

clip_image006

To view the error message a bit more in detail, open the Windows Server backup log file (with the exact date and timestamp) located in C:\Windows\Logs\WindowsServerBackup.

clip_image008

In the log file the following error message was shown:

Error in backup of C:\windows\\systemroot\ during enumerate: Error [0x8007007b] The filename, directory name, or volume label syntax is incorrect.

clip_image010

When looking in the Event Viewer (Application log) I could also find the following errors (CAPI2 – 513, Backup – 517):

Event ID 513

Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.

Details:

AddLegacyDriverFiles: Unable to back up image of binary Microsoft Link-Layer Discovery Protocol.

System Error:

Access is denied.

Event ID 517

The backup operation that started at ‘‎2017‎-‎11‎-‎16T15:16:22.000076700Z’ has failed with following error code ‘0x80780049′ (None of the items included in backup were backed up.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

clip_image012

clip_image014

Because all those errors descriptions do not really tell you what exactly is going wrong causing the backup to fail, you need to use the Diskshadow command-line tool to determine if there is an issue with the functionality of the VSS service or any of the application independent VSS writers.

To open the Diskshadow tool interface start PowerShell with elevated privileges and enter the below commands to write the output to a logfile.

clip_image016

When the logfile (c:\out.txt) is created open it with notepad and search for \\.

clip_image018

clip_image020

In my case, I found out there was an issue with the vsock.sys driver, which is part of the VMware vSockets Service and which is usually located in the C:\Windows\system32\drivers folder.

To fix the issue open the Registry Editor and go to the following location, HKLM\system\controlset001\services\vsock and changed the Start value to 1.

clip_image022

clip_image024

clip_image026

Also change the ImagePath entry from \SystemRoot\system32\DRIVERS\vsock.sys to system32\DRIVERS\vsock.sys.

clip_image028

clip_image030

When you have changed all those registry keys, logon to your MABS server and right click the failed System State backup and Perform a consistency check… (be aware that this could take a while). If the fix also solved your issue it would show OK when completed.

clip_image032

clip_image034

Hope this helps whenever you face the same error in your MABS environment. If you have any questions feel free to contact me trough my Twitter handle.

Wim Matthyssen (@wmatthyssen)