You are browsing the archive for Microsoft Azure Backup Server.

MABS v2: Creation of a Protection Group fails with error 33507 related to running Trend Micro Antivirus

9:39 am in Azure, Azure Backup, Error 33057, MABS, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, PowerShell, Trend Micro by Wim Matthyssen

Last week while assisting a customer with the deployment of a new Microsoft Azure Backup Server (MABS), I ran run across error 33507 when creating a new Protection Group.

clip_image002

In the Event Viewer on the MABS error 370 was shown several times.

clip_image004

After troubleshooting for some time, I finally found out the problem was caused by the Trend Micro antivirus (version 12.0.1708) running on the MABS. The issue is caused by an incompatibility between Trend Micro and the DPM filter driver.

clip_image006

When the antivirus was disabled MABS was able to create the Protection Group. But because running a server without an antivirus solution is not an option you can resolve the issue by adding the following registry key on any MABS server running with Trend Micro.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TmFilter\Parameters
Add DWORD (32 bit)
BypassDPMfilter, vaule of 1

You can add the registry key quite easy by running the following PowerShell one-liner (run PowerShell as an administrator).

clip_image008

After adding the registry key, reboot the server and try the process of creating a Protection Group all over again. I should work now.

Lesson learned, sometimes it’s the easy stuff. Disabling antivirus during a backup problem is often step 1. Now it was more like step 10 but at least in the end it helped fixing the issue. Till next time.

Wim Matthyssen (@wmatthyssen)

MABS: DPM database size has exceeded the threshold limit (ID 3168)

10:18 am in Azure, Azure Backup, DPM database, ID 3168, MABS, Microsoft Azure Backup Server by Wim Matthyssen

Last week I saw the below Warning message pop up at a customer’s Microsoft Azure Backup Server (MABS). The description of this Warning message described the following:

“DPM database size has exceeded the threshold limit.

DPM database size: 1.15 GB

DPM database location: e:\DPMDB\ on “servername” (ID 3168)”

clip_image002

I could also find the same Warning message in the Event Viewer.

clip_image004

The message itself was confusing because the E: drive where the DPM database is located had plenty of free disk space and the DPM Database size alert was unmarked as you can see in the below screenshots.

clip_image006

clip_image008

Another important point is that the only way to open the Tape Catalog Retention box (on a MABS server) is by clicking the Modify Catalog Alert Threshold size … link which is only show in the Recommended action field of the warning message itself.

clip_image010

However, in the end I was able to solve this warning, by marking the setting Alert me when DPM database size reaches: and changing the size to 10 GB. Afterwards I unmarked this setting again and pressed OK. Almost directly after that, the Warning message disappeared.

clip_image012

clip_image014

Conclusion

Probably this warning message is some sort of bug in the MABS software or some kind of leftover from DPM 2012 (tape backups) on which the MABS v1 code is based. For the moment, the only way to get rid of this Warning message is by using the above workaround.

If you have any questions, feel free to contact me through my twitter handle.

Wim Matthyssen (@wmatthyssen)

Microsoft Azure Backup Server: Unable to configure protection for a SQL database (ID 3170 and 33424)

2:45 pm in Azure, Azure Backup, ID 3170, ID 33424, MABS, Microsoft Azure Backup Server, SQL Server by Wim Matthyssen

Last week while configuring backup for some SQL databases for a customer with the Microsoft Azure Backup Server (MABS), I received the following Protection Status error: Unable to configure protection.

clip_image002

When opening the Monitoring tab on the MABS server, to investigate the problem, I found the following description for the error:

DPM could not start a recovery, consistency check, or initial replica creation job for SQL Server 2012 database “SQLServername\model” on “SQL Server” for following reason: (ID 3170)

The DPM job failed for SQL Server 2012 database “SQLServername\model” on “SQL Server” because the protection agent did not have sysadmin privileges on the SQL Server instance. (ID 33424 Details:)

clip_image004[6]

You can also find the similar error description in the Event Viewer on the MABS server, by opening the Application and Services LogsDPM Alerts.

clip_image006[6]

As the error suggests, the problem is that the built-in NT Authority\SYSTEM does not have sysadmin rights on that SQL Server instance. So to resolve this issue, perform the following steps. Open Microsoft SQL Server Management Studio on the SQL server. Open Security, open Logins, select the NT\AUTHORITY SYSTEM user and click Properties. In the Server Roles screen sysadmin should be checked, what for this specific database was not the case. So check sysadmin and press OK to save. You need to repeat this step for all instances having this problem.

clip_image008[5]

clip_image010[6]

After fixing this, you need to perform a consistency check on the MABS for all those databases with status Unable to configure protection. To do so right-click the unprotected database and select Perform consistency check …, which will retry the protection and solve the problem.

clip_image012

After completion, the Protection Status should be showing OK.

clip_image014

Hope this helps you fixing this problem when it occurs. If you have, any questions feel free to contact me through my twitter handle.

Wim Matthyssen (@wmatthyssen)

How to install Microsoft Azure Backup Server v2 on Windows Server 2016

7:53 pm in Azure, Azure Backup, Hybrid backup, MABS, MABS v2, Microsoft Azure Backup, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, Modern Backup Storage, PowerShell, Windows Server 2016 by Wim Matthyssen

Last week Microsoft released the second version (v2) of their Microsoft Azure Backup Server (MABS v2). As a hybrid backup solution, this new release based on System Center Data Protection Manager 2016 (SCDPM 2016) enables you to store data onto disk (low RTO) and in Azure (long retention, up to 99 years). MABS v2 uses RCT-based change tracking by using Windows Server 2016. This makes backups more reliable and scalable, but also improves backup performance (backup jobs could be up to 70 percent faster). MABS v2, which is included with the Azure Backup service and currently has version number 12.0.332.0., now not only supports Windows Server 2016 (W2K16) but also vSphere 6.5 (Preview mode). Beside those, you can also use it now to backup business critical Microsoft workloads such as SQL 2016, SharePoint 2016 and Exchange 2016. Those can be running on premise (physical servers, Hyper-V or VMware) or in the Azure cloud. As a nice extra, you can also back up Windows 10 client workloads.

clip_image002

In a previous blog post, I already told you all about MABS v1 on how to install it on a Windows Server 2012 R2. In this blog post, I will show you how you can deploy MABS v2 on a W2K16 server.

MABS v2 server requirements

  • MABS v2 can be installed as an on premise standalone physical server or VM, but also as an Azure IaaS VM (minimum size A2 Standard).
  • MABS v2 will run on following supported Operating Systems: Windows Server 2012 R2 and Windows Server 2016 (is required if you want to use the Modern Backup Storage feature).
  • MABS v2 must be domain joined. Be sure to add the server to the domain before the MABS installation. Microsoft does not support adding this server to the domain after the MABS installation.
  • The processor minimum requirements for a MABS v2 server are 1GHz dual-core CPU, recommended 2.33 GHz quad-core CPU.
  • The minimum RAM needed by a MABS v2 server is 4GB, recommended is 8 GB.
  • The recommended hard drive space is 3 GB.
  • MABS v2 must have .NET 3.5 SP1, .NET 4.6.1 features installed as a prerequisite.
  • MABS v2 should also have Hyper-V PowerShell installed.
  • MABS v2 should be running a dedicated, single-purpose server. Either it cannot be running on the same server, which has SCDPM or a SCDPM agent installed.
  • A validate Windows Server license is needed for the MABS v2 server.
  • The MABS v2 server needs to have access to the Internet because Microsoft Azure should be accessible from the MABS server.
  • To temporarily store, the largest restore from the Azure cloud, some scratch space is required when needed. So keep approximately 5 % of the total amount of data that needs to be backed-up to the cloud free on the C: drive.
  • A separate data disk for the backup storage pool is required. Like every other backup product the recommendation for the size of this disk is 1.5 times the size of the data you are going to protect.

MABS v2 prerequisites installation

Before we start the prerequisites installation, be shore to have a Recovery Services vault in place (create a new one, or use an existing) and download the vault credentials. When downloaded, place this file on the C:\Temp folder of the MABS server.

clip_image004

clip_image006

To install all required prerequisites, logon to the server you wish to use for your MABS v2 installation, open PowerShell and administrator and run the following commands to install .NET 3.5 SP1 and Hyper-V PowerShell (be shore to have the Windows Server 2016 installation ISO mounted – in my example to the D: drive). Be aware the server will reboot when the installation is completed. You can also download the complete script (.ps1) from the Microsoft TechNet Gallery.

clip_image008

MABS v2 software download

To download the MABS v2 software open PowerShell as an administrator and run the following PowerShell script. You can download the complete script (.ps1) from the Microsoft TechNet gallery. The script will download all the necessary files (8 files), extract them and start the setup.

MABS v2 installation

Click Microsoft Azure Backup Server to launch the setup wizard.

clip_image010

Setup will start copying some temporary files.

clip_image012

On the Welcome screen, click the Next.
clip_image014

This opens up the Prerequisite Check section. On this screen, click on the Check button to determine if the hardware and software prerequisites for Azure Backup Server have been met. If all of is OK, you will see a message indicating that the machine meets the requirements. Click Next.

clip_image016

On the SQL Settings page select, Install new Instance of SQL Server with this Setup, to install SQL 2016 SP1. Click Check and Install. You could encounter some error messages. If so follow the instructions and most likely, you should reboot the server and start the MABS installation all over again.

clip_image018

If the computer meets, the software and hardware requirements click Next.

clip_image020

Provide a location for the installation of all the files and click Next. In my example, I changed all locations to my E: drive.
clip_image022

Provide a strong password for restricted local user accounts (this password will not expire) and click Next.
clip_image024

It is strongly recommended to use Microsoft update when you check for updates because this will offer all security and important updates for MABS. Select whether to use Microsoft Update or not and click Next.

clip_image026

Review all settings and if all are OK click Install.

clip_image028

clip_image030

Click Next to start the Microsoft Azure Recovery Service Agent installation.

clip_image032

Click Install.

clip_image034

clip_image036

When the agent installation is completed, click Next.

clip_image038

Provide your vault credentials to register the machine to the Azure backup vault. Click Next.
clip_image040

Provide a passphrase to encrypt/decrypt the data sent between Azure and your premises. You can automatically generate a passphrase or provide your own minimum 16-character passphrase. Also, enter a location to save the passphrase. If all is done click Next.

clip_image042

Once registration succeeded the wizard proceeds with the installation and configuration of SQL Server 2016 SP1. This could take some time.

clip_image044

clip_image046

It is possible that you receive the following error message, if so just click OK (you can change the staging area after the MABS setups completes).
clip_image048

When setup completes successfully, click Close.
clip_image050

Double click the Microsoft Azure Backup server icon on your desktop to launch MABS.

clip_image052

clip_image054

You can also verify if the MABS server connection to the Recovery Services vault. To do so go to your Recovery Services vault, click Overview and click Backup management servers. There you should see the newly installed MABS server.

clip_image056

As a final step, do not forget to run Windows update to install all necessary updates after the MABS installation.

clip_image058

Now you are ready to start working with this brand new product. Have fun and till next time!

Wim Matthyssen (@wmatthyssen)

Microsoft Azure Backup Server: Anti-Virus Exclusions

1:05 pm in Anti-Virus Exclusions, Azure, Azure Backup, Cloud, hybrid cloud, MABS, Microsoft Azure Backup Server by Wim Matthyssen

Running a solid, constantly updated antivirus product on your servers is a necessity to keep a healthy and secure server environment. However, with installing an antivirus product, you also risk having issues with certain workloads and services on those severs. Just like System Center Data Protection Manager (SCDPM), the Microsoft Azure Backup Server (MABS) is compatible with most antivirus software products. Though, the implemented antivirus product can also affect MABS performance and, if not configured properly, can cause data corruption of replicas and recovery points.

clip_image002

So, to avoid file conflicts and to minimize performance degradation between your MABS server and the antivirus software running on top of it, you should disable real-time monitoring by the antivirus software for all of the following processes and directories, which are listed below.

MABS processes to exclude from antivirus real-time monitoring

For information about configuring real-time monitoring based on process name or folder name, check the documentation of your antivirus vendor.

  • DPMRA.exe (*full path: C:\Program Files\Microsoft Azure Backup\DPM\DPM\bin\DPMRA.exe)
  • csc.exe  (*full path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe -> you can also exclude csc.exe in all the other Microsoft.NET Framework folders)
  • cbengine.exe (*full path: C:\Program Files\Microsoft Azure Backup\DPM\MARS\Microsoft Azure Recovery Services Agent\bin\cbengine.exe)

 

clip_image004

clip_image006

clip_image008

 

MABS directories in the MABS Program Files folder to exclude from antivirus real-time monitoring

Be aware that when you installed MABS on another drive then “C:”, like in the example below, look under the correct drive for the folders to exclude.

  • C:\Program Files\Microsoft Azure Backup\DPM\DPM\Temp\MTA\*
  • C:\Program Files\Microsoft Azure Backup\DPM\DPM\XSD\*
  • C:\Program Files\Microsoft Azure Backup\DPM\DPM\bin
  • C:\Program Files\Microsoft Azure Backup\DPM\MARS\Microsoft Azure Recovery Services Agent\bin
  • C:\Program Files\Microsoft Azure Backup\DPM\DPM\Cache (*MABS scratch folder)

 

clip_image010

clip_image012

clip_image014

clip_image016

clip_image018

 

Delete infected files on the MABS server

As a final remark, I would also advise to configure to delete infected files by default on the MABS server rather than automatically cleaning or quarantining them. Automatic cleaning and quarantining can result in data corruption because these processes cause the antivirus software to modify files, making changes MABS cannot detect.

 

In summary, there are a lot of antivirus settings you should keep track of when running MABS. I’ve tried to list all of the exclusions, so hopefully it will help you with getting the most out of your MABS setup. If you have any questions, feel free to contact me through my Twitter handle.

Wim Matthyssen (@wmatthyssen)

Microsoft Azure Backup Server: Error when installing on Windows Server 2016 – The Single Instance Store (SIS) component is not installed

3:36 pm in Azure, Azure Backup, hybrid cloud, MABS, Microsoft Azure Backup, Microsoft Azure Backup Server, PowerShell, Public Cloud, SIS, SIS-Limited, Windows Server 2016, WS2016 by Wim Matthyssen

 
Hi All,

Last week I was contacted by a customer who tried to install Microsoft Azure Backup Server (MABS) on an on-premise Windows Server 2016. However, when he started the installation he always received an error because a prerequisite was not installed, namely the Single Instance Store (SIS) component.

 
clip_image002

When opening the DpmSetup.log with PowerShell (as Administrator), you could see the following error:

 

clip_image004

However, when you try to install this missing component through PowerShell it gives you an Error: 0x800f080cFeature name SIS-Limited is unknown.

clip_image006

The reason for this is that because from Windows Server 2016 the SIS-Limited component is replaced by Microsoft’s deduplication or data footprint reduction (DFR) technology, like you can read in the following article from MVP Greg Schulz: http://storageioblog.com/rip-windows-sis-single-instance-storage-or-at-least-in-server-2016/

Also, when you go to the Microsoft Azure Backup Server download page and you expand System Requirements you can see that Windows Server 2016 at the present time is not listed as a supported Operating System (OS) to deploy MABS, probably because it does not have this SIS component.

clip_image008

Conclusion

Currently you’re not able to use Windows Server 2016 as OS for you MABS server. Probably in the near future Microsoft will release a new version of MABS which will allow it, but until then you need to stick with Windows Server 2012 (R2) or Windows Server 2008 R2 to install your MABS on.

Hope this helps you with this error.

Wim Matthyssen (@wmatthyssen)

Microsoft Azure Backup Server: Install a new version of the Microsoft Azure Recovery Services Agent

3:06 pm in Azure, Azure Backup, Cloud, hybrid cloud, Microsoft Azure Backup Server by Wim Matthyssen

Hi all,

Some time ago a client received following alert on his Microsoft Azure Backup Server (MABS):

“Azure Backup raised the following alert for the subscription in use: (ID 33406). A new version of Windows Azure Backup Agent is available. You can review details about the new version and download it from http://go.microsoft.com/fwlink/p/?LinkId=229525. (ID 100083) More information

Like you all can read this alert was raised because there is a new version of the Azure Backup Agent available. In the Event Viewer on the MABS server you can also find following Warning message under the Application and Services Logs, CloudBackup, Operational:

“A newer version of Microsoft Azure Recovery Services Agent is required.”

clip_image002

To install this new agent the following steps were taken:

1) Check the current Azure Backup Agent Version. To do so open the MABS console and click Management. Under Online you can find the Azure Backup agent version. Like you can see in the screenshot below for the moment version 2.0.9032.0 is installed

clip_image003

2) Download the latest version of the agent via following link: https://support.microsoft.com/en-us/kb/3162327 . Go to the section Update information were you can find the update package for agent version 2.0.9037.0 In my example I saved it under the Temp folder

clip_image005

clip_image007

3) Go to the Temp folder and Run the MARSAgentInstaller as administrator

clip_image009

4) If the UAC screen pops up, click Yes

clip_image010

5) To continue installing the update, click Next

clip_image012

6) If all required software is in place, click Upgrade. This will start the upgrade process

clip_image014

clip_image016

7) When the Upgrade is successful click Finish

clip_image018

8) If you check the Azure Backup Agent version again, you can see version 2.0.9037.0 is installed

clip_image019

9) You can also verify the current version by opening Run and typing appwiz.cpl to open Programs and Features. Look for the Microsoft Azure Recovery Services Agent which should have version 2.0.9037.0

clip_image020

clip_image022

10) Be aware that by default the Warning message will stay there for 30 days. To clear this message after the update you can inactivate it by right clicking the message and selecting Inactivate alert or by rebooting the MABS server several times (3 times at least)

clip_image023

This concludes this blog post, hope it helps!

Wim Matthyssen (@wmatthyssen)

Microsoft Azure Backup Server: System State backup fails with WSB Event ID: 546

7:07 am in Azure, Azure Backup, DC, IaaS, Microsoft Azure Backup Server, PowerShell, Public Cloud by Wim Matthyssen

After setting up a Microsoft Azure Backup Server (MABS) running on an IaaS v1 VM for a customer, I encountered a problem with a System State backup. The server having the problem was a Domain Controller (DC) also running as an Azure IaaS v1 VM and a member of the Domain Controller Protection Group on the MABS server. You can see the error message found on the MABS server in the screenshot below

clip_image002

“DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 546, WSB Error Code: 0x1751870). (ID 30229 Details: Internal error code: 0x80990ED0)”

On the DC itself following error message was found:

clip_image004

“The backup operation attempted at ‘‎2016‎-‎05‎-‎10T14:00:28.139491000Z’ has failed to start, error code ‘2155348032’ (The backup storage location is invalid. You cannot use a volume that is included in the backup as a storage location. ). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.”

To fix this problem I followed the steps below:

1) On the server with the problem, verify if no other backup or recovery operation is running by opening a PowerShell window (as administrator) and typing:

clip_image006

If the command output indicates that no operation is running, then you can proceed to step 2. Otherwise wait until the current job is completed and retry the failed System State backup job

2) In the same PowerShell windows type the following command:

clip_image008

Verify of the writer named System Writer is running Stable and without any errors. If so proceed to step 3. Otherwise manually restart the VSS Writer, if it fails again it’s necessary to reboot the server

3) Open Services and verify if the Cryptographic Services is running and set to startup Automatically.

clip_image009

clip_image011

4) The check if Windows Server Backup (WSB) is able to take a local System State Backup, open up PowerShell (as an administrator) and run the below command. When asked press (Y). Be aware it could take a while before the job completes:

clip_image013

clip_image015

clip_image017

If the backup is successfully proceed to step 5. Otherwise check the Windows event viewer for other errors

5) Open the registry editor and go to the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wbengine. Create a key called SystemStateBackup and set the values of this entry as follows:

Name: AllowSSBToAnyVolume
Data type: DWORD
Value data: 1

clip_image018

clip_image020

You also can use PowerShell (as an administrator) to create the registry key, to do so run following command:

6) When done, logon to the MABS server and retry the failed System State backup job by right-clicking and selecting Perform consistency check … Verification will start and at the end the job will complete with success

clip_image022

image

This concludes this blog post, hope it’s useful. Till next time!

Wim Matthyssen (@wmatthyssen)

Microsoft Azure Backup Server: System State backup fails with the message replica is inconsistent

8:00 pm in Azure, Azure Backup, Cloud, IaaS, Microsoft Azure Backup Server, PowerShell, Public Cloud, Windows Server Backup by Wim Matthyssen

Last week I was setting up a Microsoft Azure Backup Server (MABS) for a customer running on an Azure IaaS v1 VM for taking disk and System State backups which are stored in an Azure Backup vault for long-term retention. After done all of the configuration, like deploying the agents and creating the Protection Group(s), most of the backup jobs ran without any issues, only the System State backups were failing. This is shown in the screenshot below.

clip_image002

For those who not know, I will first list some extra information to clarify what a System State backup really is:

  • A System State backup uses the Windows Server Backup (WSB) feature to take a snapshot of the system state and saves it locally to the server before moving it to the MABS server storage pool
  • It contains the following system components: System Registry, SYSVOL directory, Boot files including the system files, Active Directory, COM + Class Registration Database, Certificate Services, Cluster service information, IIS Metadirectory and System files that are under Windows File Protection (some of these items are only included if the specified service is installed – AD, ISS, Certificates)
  • §System state backup and restore operations include all system state data: you cannot choose to backup or restore individual components due to dependencies among the system state components. However, you can restore system state data to an alternate location in which only the registry files, Sysvol directory files, and system boot files are restored.

Troubleshooting a System State backup failure can be frustrating as there are a lot of underlying components that come into play during the process. To figure out what was causing the System State backup(s) to fail, I followed the steps below:

1) Check out the Critical alerts for a specific server having the issue in the Monitoring pane on the MABS server. Here I found the following error:

clip_image004

2) After reading the description carefully, you can see there should be enough free space on the target server and also WSB should be installed. So to verify,  first logon to the specific server trough RDP

3) Check out the free disk space by opening Disk Management

clip_image005

4) Like you can see, there is plenty enough of free disk space to store a local System State backup

clip_image007

5) To check if WSB is installed, run following cmdlet in PowerShell (as an Administrator)

clip_image009

6) Like you can see WSB is not installed and probably causing the failure of the System State backup

7) Install WSB trough PowerShell (as an Administrator)

clip_image011

clip_image013

8) Check if this fixes the issue by opening the Microsft Azure Backup MMC on to the MABS server. Go to Protection and select Perform consistency check …

clip_image015

9) Click Yes to perform the consistency check

clip_image016

10) You will see the Protection Status changing to “Replica is inconsistent, verification in progress”

clip_image018

11) To be sure everything is working like it should, you can logon to the specific server and open WSB locally by opening the Run box and typing wbadmin.msc

clip_image019

12) Select Local Backup. If all is running fine, their should be a local backup job running at the moment

clip_image021

13) After a while the job should complete Successful

clip_image023

14) You should also verify the successful System State backup in the MABS console. Like you can see the Protection Status of the job is OK now (also for all other System State jobs, after WSB was installed locally on all those servers)

clip_image025

This concludes this blog post, hope it helps!

Wim Matthyssen (@wmatthyssen)