MABS v2: Error [0x8007007b] when performing a System State Backup on a DC running on a VMware VM

November 23, 2017 at 8:33 am in Azure, Azure Backup, Azure Backup Server, Cloud, Error [0x8007007b], MABS, MABS v2, Power, PowerShell, VMware by Wim Matthyssen

While configuring a Microsoft Azure Backup Server (MABS) v2 at a customer site, I encountered a problem while performing a System State Backup of their domain controllers (DC’s). The Protection Status showed Replica is inconsistent.

clip_image002

When looking in the Monitoring tab, following detailed message is show:

DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code: 0x605A140).(ID 30229 Details: Internal error code: 0x8099ED0)

clip_image002[6]

Because the first part of making a System State Backup is done by the local Windows Server Backup (WSB) feature, logon to the protected server and open Windows Server Backup (Server Manager – Tools – Windows Server Backup). There a message was shown indicating that the last backup has Failed.

clip_image006

To view the error message a bit more in detail, open the Windows Server backup log file (with the exact date and timestamp) located in C:\Windows\Logs\WindowsServerBackup.

clip_image008

In the log file the following error message was shown:

Error in backup of C:\windows\\systemroot\ during enumerate: Error [0x8007007b] The filename, directory name, or volume label syntax is incorrect.

clip_image010

When looking in the Event Viewer (Application log) I could also find the following errors (CAPI2 – 513, Backup – 517):

Event ID 513

Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.

Details:

AddLegacyDriverFiles: Unable to back up image of binary Microsoft Link-Layer Discovery Protocol.

System Error:

Access is denied.

Event ID 517

The backup operation that started at ‘‎2017‎-‎11‎-‎16T15:16:22.000076700Z’ has failed with following error code ‘0x80780049′ (None of the items included in backup were backed up.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

clip_image012

clip_image014

Because all those errors descriptions do not really tell you what exactly is going wrong causing the backup to fail, you need to use the Diskshadow command-line tool to determine if there is an issue with the functionality of the VSS service or any of the application independent VSS writers.

To open the Diskshadow tool interface start PowerShell with elevated privileges and enter the below commands to write the output to a logfile.

clip_image016

When the logfile (c:\out.txt) is created open it with notepad and search for \\.

clip_image018

clip_image020

In my case, I found out there was an issue with the vsock.sys driver, which is part of the VMware vSockets Service and which is usually located in the C:\Windows\system32\drivers folder.

To fix the issue open the Registry Editor and go to the following location, HKLM\system\controlset001\services\vsock and changed the Start value to 1.

clip_image022

clip_image024

clip_image026

Also change the ImagePath entry from \SystemRoot\system32\DRIVERS\vsock.sys to system32\DRIVERS\vsock.sys.

clip_image028

clip_image030

When you have changed all those registry keys, logon to your MABS server and right click the failed System State backup and Perform a consistency check… (be aware that this could take a while). If the fix also solved your issue it would show OK when completed.

clip_image032

clip_image034

Hope this helps whenever you face the same error in your MABS environment. If you have any questions feel free to contact me trough my Twitter handle.

Wim Matthyssen (@wmatthyssen)

How to run the Hyper-V role on a VMware VM

November 21, 2017 at 10:16 am in Hyper-V, MABS v2, Nested Virtualization, PowerShell, VMware, Windows Server 2016 by Wim Matthyssen

When you install Microsoft Azure Backup Server (MABS) v2 on a Windows Server 2016, one of the prerequisites (MABS v2 prerequisites installation script) is that you install the Hyper-V role and the Hyper-V PowerShell feature.

However, while I was installing a new MABS v2 for a customer on a VMware VM (vSphere 6.5), I encountered following errors in the Hyper-V event log (41, 15350, 15340) after the Hyper-V role was installed.

Event 41 showed the following error message:

Hypervisor launch failed, Either VMX not present or not enabled in BIOS.

clip_image002

When I ran the Get-WindowsFeature in PowerShell it seemed Hyper-V was installed correctly. But this was not the case.

clip_image004

To fix the errors and get Hyper-V running like it should you need to enable Nested Virtualization for the VMware VM. To do so, shut down the VM and open the Virtual Machine Settings. Then go the Virtual Hardware tab and open the CPU options. There you need to check the box Expose hardware assisted virtualization to the guest OS. Also set CPU/MMU Virtualization to Hardware CPU and MMU.

clip_image006

When you now start the VM all Hyper-V related errors should be gone and all necessary Hyper-V services should be running.

clip_image008

clip_image010

Hope this blog post will help you whenever you need to setup Hyper-V on a VMware VM.

Wim Matthyssen (@wmatthyssen)

MABS v2: Creation of a Protection Group fails with error 33507 related to running Trend Micro Antivirus

November 16, 2017 at 9:39 am in Azure, Azure Backup, Error 33057, MABS, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, PowerShell, Trend Micro by Wim Matthyssen

Last week while assisting a customer with the deployment of a new Microsoft Azure Backup Server (MABS), I ran run across error 33507 when creating a new Protection Group.

clip_image002

In the Event Viewer on the MABS error 370 was shown several times.

clip_image004

After troubleshooting for some time, I finally found out the problem was caused by the Trend Micro antivirus (version 12.0.1708) running on the MABS. The issue is caused by an incompatibility between Trend Micro and the DPM filter driver.

clip_image006

When the antivirus was disabled MABS was able to create the Protection Group. But because running a server without an antivirus solution is not an option you can resolve the issue by adding the following registry key on any MABS server running with Trend Micro.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TmFilter\Parameters
Add DWORD (32 bit)
BypassDPMfilter, vaule of 1

You can add the registry key quite easy by running the following PowerShell one-liner (run PowerShell as an administrator).

clip_image008

After adding the registry key, reboot the server and try the process of creating a Protection Group all over again. I should work now.

Lesson learned, sometimes it’s the easy stuff. Disabling antivirus during a backup problem is often step 1. Now it was more like step 10 but at least in the end it helped fixing the issue. Till next time.

Wim Matthyssen (@wmatthyssen)

Azure PowerShell: Your Azure credentials have not been set up or have expired (classic – ASM)

October 3, 2017 at 8:56 am in ASM, Azure, Azure PowerShell, PowerShell by Wim Matthyssen

Last week, after a long time I was cleaning up an Azure ASM environment with Azure PowerShell for a customer when I stumbled upon the following error:

“Your Azure credentials have not been set up or have expired, please run Add-AzureAccount to set up your Azure credentials.”

clip_image002

After checking the account settings, everything seemed valid. So probably for some reason, some old settings were cached causing the problem.

clip_image004

Solution:

After trying and testing out some cmdlets, the following finally solved the issue:

clip_image005

When you have run the above cmdlet, try to run the Add-AzureAccount cmdlet and afterwards any other cmdlet to see if this also solves the issue for you.

clip_image007

Hope this helps you when you are facing the same issue.

Wim Matthyssen (@wmatthyssen)

Azure IaaS: Deploying B-series VMs

September 22, 2017 at 10:18 am in ARM, Azure, Azure PowerShell, B-Series VMs, IaaS, PowerShell, Public Cloud by Wim Matthyssen

Last week Microsoft introduced the B-Series VM size in preview. These B-Series VMs can run workloads that burst in their performance, but do not need continuous full performance of the CPU. Servers that would be eligible for this new “burstable” VM size are servers with small databases, webservers, development servers, quality assurance (QA) and test servers. But also servers with other workloads that do not utilize the full located vCPUs are grate candidates to benefit and lower costs by using a B-Series VM.

clip_image002

Herby a list of the 6 B-Series VM sizes, which are currently available in Preview in the following Azure Regions (Europe-West, US-West 2, US-East, Asia Pacific-Southeast):

Instance Size vCPU vMemory: GiB Tempory Storage / Local SSD: GiB Max data disks Max NICs Credits banked / hour Max Banked Credits
Standard_B1s 1 1 2 2 2 6 144
Standard_B2s 2 4 8 4 3 24 576
Standard_B1ms 1 2 4 2 2 12 288
Standard_B2ms 2 8 16 4 3 36 864
Standard_B4ms 4 16 32 8 4 54 1296
Standard_B8ms 8 32 64 16 4 81 1944

You can also read more about this new VM size here and find all pricing info

Now I am going to show you how you can deploy a new B-Series VM trough Azure PowerShell.

First, you need to request quota to be able to deploy this B-Series VMs. To do so you should logon to the Azure portal and go to Help + support. To request an increase or to be able to deploy B-Series VMs, select New support request.

clip_image004

You need to create a Quota support case for Cores. So, on the NEW SUPPORT REQUEST page, select Issue type as “Quota” and Quota type as “Cores”. Also, select the Subscription and the correct Support plan. Click Next.

clip_image006

Select Severity “C – Minimal Impact”, Deployment model “Resource Manager” and the correct Location, which in my case is West Europe. Select as SKU Families that requires an increase “BS Series” and set the NEW LIMIT higher than before, for example 15 instead of 10. Click Next.

clip_image008

In the Contact Information blade, select your Preferred contact method, provide a Response time, select your preferred Language and fill in the Contact Information. Click Create to create the new support ticket.

clip_image010

clip_image012

In my case I received an email after the quota was been approved, which normally does not take that much time. So from here we can go further with the deployment.

clip_image014

If you open Azure PowerShell, and run following commands, you can now built a new B-Series VM. You can copy the commands or save them to as a PowerShell script (.ps1). Do not forget to adjust all variables were needed.

clip_image016

clip_image018

Hope this helps you getting started with this new B-Series VMs.

Wim Matthyssen (@wmatthyssen)

Azure Backup: Create a Recovery Services vault with Azure PowerShell

July 12, 2017 at 9:46 am in Azure, Azure Backup, Azure PowerShell, PowerShell, Public Cloud, Recovery Services vault by Wim Matthyssen

A Recovery Services vault is an online storage entity used to backup workloads to the Azure cloud. You can use it to hold backup data for various Azure services such as IaaS VMs (Linux or Windows) and Azure SQL databases, but it can also be used by System Center Data Protection Manager (SCDPM) or Azure Backup Server (MABS v1 and MABS v2) to enable cloud backups.

clip_image002[5]

These days it is quite easy to create or manage a Recovery Services vault through the Azure portal, but it is even faster when you make use of a scripting language like Azure PowerShell to automate the setup. Therefore, below you can find the PowerShell script I mostly use to do all the work for me. You can just copy and paste or you can download the complete script (.ps1) from the Microsoft TechNet gallery.

To use the script, first adjust all variables to your use. Afterwards login into an Azure PowerShell window as an administrator and when asked login with the credentials for your Azure Subscription.

The script will first create a Resource Group and then the Recovery Services vault in your Azure Subscription. At the end, it will also set the storage redundancy for the newly created vault. Keep in mind that you can only use Locally Redundant Storage (LRS) or Geo Redundant Storage (GRS).

clip_image004[6]

clip_image006[6]

clip_image008[5]

Useful Azure PowerShell cmdlets for Azure Backup

List all available Azure Backup PowerShell cmdlets

clip_image010[7]

List all available Recovery Services vaults in your subscription

clip_image012[4]

Hope this post helps you when you start using Azure Backup.

Wim Matthyssen (@wmatthyssen)

Azure IaaS: Troubelshooting Windows Update error 8024402F

July 6, 2017 at 3:31 pm in 8024402F, ARM, ASM, Azure, hybrid cloud, PowerShell, Windows Server 2012 R2, Windows Update, WSUS by Wim Matthyssen

 
Last week I was troubleshooting a Windows Update issue at several Azure IaaS VMs for a customer. All those Windows Server 2012 R2 servers were workgroup members and had no Network Security Group (NSG) attached which could block the connection to the Microsoft Update servers. But whenever starting Windows Update the below error was shown after a few minutes.

clip_image002

To get this error fixed I followed the below steps. Be aware that you can retry running Windows Update again after each step because it could be already working again.

 

Step 1

If the server has been configured to use WSUS to get its updates, first wipe out those registry keys by running the below command in a PowerShell window (with admin privileges). Press Y to delete all registry keys when asked:

clip_image004

clip_image006

This also may reset some Windows Update settings, for instance, the one that decides if updates should install automatically or after asking permission.  Therefore, you need to set your preferred settings afterwards.

Check for updates using Windows Update and see if the issue has been resolved, if not proceed to step 2.

 

Step 2

If you still receive the same error, run the following PowerShell Script to rename the SoftwareDistribution and catroot2 folder. These folders, which are maintained by the WUAgent (Windows Update Agent), are essential components for Windows Update. However, sometimes the content of these folders could prevent Windows Update from applying new updates to the server. When having trouble with Windows Update, it is safe to delete this folder. The server will always re-download all the necessary files, or re-create the folder and re-download all the components, if removed.

clip_image008

Now please check for updates using Windows Update to see if the issue has been resolved.

 

Step 3

If step 2 also does not fix the problem, you could try running the below command from an elevated PowerShell window. This command will import proxy information used by Internet Explorer in the Windows HTTP Services (WinHTTP). Several server roles, like the Microsoft Windows Update client, rely on WinHTTP to manage all HTTP and HTTPS traffic. Windows Update uses it mainly to scan for available updates.

clip_image010

 

Step 4

As a last solution, you could try running the Windows Update Troubleshooter tool. To download and startup this tool run the below PowerShell commands.

clip_image012
clip_image014

When the tool opens, go through all steps to get Windows Update fixed.

If all goes well, Windows Update should be working again by the use of one of the above steps. Hope it helps and if you have any questions feel free to contact me through my twitter handle.

Wim Matthyssen (@wmatthyssen)

MABS: DPM database size has exceeded the threshold limit (ID 3168)

June 26, 2017 at 10:18 am in Azure, Azure Backup, DPM database, ID 3168, MABS, Microsoft Azure Backup Server by Wim Matthyssen

Last week I saw the below Warning message pop up at a customer’s Microsoft Azure Backup Server (MABS). The description of this Warning message described the following:

“DPM database size has exceeded the threshold limit.

DPM database size: 1.15 GB

DPM database location: e:\DPMDB\ on “servername” (ID 3168)”

clip_image002

I could also find the same Warning message in the Event Viewer.

clip_image004

The message itself was confusing because the E: drive where the DPM database is located had plenty of free disk space and the DPM Database size alert was unmarked as you can see in the below screenshots.

clip_image006

clip_image008

Another important point is that the only way to open the Tape Catalog Retention box (on a MABS server) is by clicking the Modify Catalog Alert Threshold size … link which is only show in the Recommended action field of the warning message itself.

clip_image010

However, in the end I was able to solve this warning, by marking the setting Alert me when DPM database size reaches: and changing the size to 10 GB. Afterwards I unmarked this setting again and pressed OK. Almost directly after that, the Warning message disappeared.

clip_image012

clip_image014

Conclusion

Probably this warning message is some sort of bug in the MABS software or some kind of leftover from DPM 2012 (tape backups) on which the MABS v1 code is based. For the moment, the only way to get rid of this Warning message is by using the above workaround.

If you have any questions, feel free to contact me through my twitter handle.

Wim Matthyssen (@wmatthyssen)

Microsoft Azure Backup Server: Unable to configure protection for a SQL database (ID 3170 and 33424)

June 19, 2017 at 2:45 pm in Azure, Azure Backup, ID 3170, ID 33424, MABS, Microsoft Azure Backup Server, SQL Server by Wim Matthyssen

Last week while configuring backup for some SQL databases for a customer with the Microsoft Azure Backup Server (MABS), I received the following Protection Status error: Unable to configure protection.

clip_image002

When opening the Monitoring tab on the MABS server, to investigate the problem, I found the following description for the error:

DPM could not start a recovery, consistency check, or initial replica creation job for SQL Server 2012 database “SQLServername\model” on “SQL Server” for following reason: (ID 3170)

The DPM job failed for SQL Server 2012 database “SQLServername\model” on “SQL Server” because the protection agent did not have sysadmin privileges on the SQL Server instance. (ID 33424 Details:)

clip_image004[6]

You can also find the similar error description in the Event Viewer on the MABS server, by opening the Application and Services LogsDPM Alerts.

clip_image006[6]

As the error suggests, the problem is that the built-in NT Authority\SYSTEM does not have sysadmin rights on that SQL Server instance. So to resolve this issue, perform the following steps. Open Microsoft SQL Server Management Studio on the SQL server. Open Security, open Logins, select the NT\AUTHORITY SYSTEM user and click Properties. In the Server Roles screen sysadmin should be checked, what for this specific database was not the case. So check sysadmin and press OK to save. You need to repeat this step for all instances having this problem.

clip_image008[5]

clip_image010[6]

After fixing this, you need to perform a consistency check on the MABS for all those databases with status Unable to configure protection. To do so right-click the unprotected database and select Perform consistency check …, which will retry the protection and solve the problem.

clip_image012

After completion, the Protection Status should be showing OK.

clip_image014

Hope this helps you fixing this problem when it occurs. If you have, any questions feel free to contact me through my twitter handle.

Wim Matthyssen (@wmatthyssen)

How to install Microsoft Azure Backup Server v2 on Windows Server 2016

June 8, 2017 at 7:53 pm in Azure, Azure Backup, Hybrid backup, MABS, MABS v2, Microsoft Azure Backup, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, Modern Backup Storage, PowerShell, Windows Server 2016 by Wim Matthyssen

Last week Microsoft released the second version (v2) of their Microsoft Azure Backup Server (MABS v2). As a hybrid backup solution, this new release based on System Center Data Protection Manager 2016 (SCDPM 2016) enables you to store data onto disk (low RTO) and in Azure (long retention, up to 99 years). MABS v2 uses RCT-based change tracking by using Windows Server 2016. This makes backups more reliable and scalable, but also improves backup performance (backup jobs could be up to 70 percent faster). MABS v2, which is included with the Azure Backup service and currently has version number 12.0.332.0., now not only supports Windows Server 2016 (W2K16) but also vSphere 6.5 (Preview mode). Beside those, you can also use it now to backup business critical Microsoft workloads such as SQL 2016, SharePoint 2016 and Exchange 2016. Those can be running on premise (physical servers, Hyper-V or VMware) or in the Azure cloud. As a nice extra, you can also back up Windows 10 client workloads.

clip_image002

In a previous blog post, I already told you all about MABS v1 on how to install it on a Windows Server 2012 R2. In this blog post, I will show you how you can deploy MABS v2 on a W2K16 server.

MABS v2 server requirements

  • MABS v2 can be installed as an on premise standalone physical server or VM, but also as an Azure IaaS VM (minimum size A2 Standard).
  • MABS v2 will run on following supported Operating Systems: Windows Server 2012 R2 and Windows Server 2016 (is required if you want to use the Modern Backup Storage feature).
  • MABS v2 must be domain joined. Be sure to add the server to the domain before the MABS installation. Microsoft does not support adding this server to the domain after the MABS installation.
  • The processor minimum requirements for a MABS v2 server are 1GHz dual-core CPU, recommended 2.33 GHz quad-core CPU.
  • The minimum RAM needed by a MABS v2 server is 4GB, recommended is 8 GB.
  • The recommended hard drive space is 3 GB.
  • MABS v2 must have .NET 3.5 SP1, .NET 4.6.1 features installed as a prerequisite.
  • MABS v2 should also have Hyper-V PowerShell installed.
  • MABS v2 should be running a dedicated, single-purpose server. Either it cannot be running on the same server, which has SCDPM or a SCDPM agent installed.
  • A validate Windows Server license is needed for the MABS v2 server.
  • The MABS v2 server needs to have access to the Internet because Microsoft Azure should be accessible from the MABS server.
  • To temporarily store, the largest restore from the Azure cloud, some scratch space is required when needed. So keep approximately 5 % of the total amount of data that needs to be backed-up to the cloud free on the C: drive.
  • A separate data disk for the backup storage pool is required. Like every other backup product the recommendation for the size of this disk is 1.5 times the size of the data you are going to protect.

MABS v2 prerequisites installation

Before we start the prerequisites installation, be shore to have a Recovery Services vault in place (create a new one, or use an existing) and download the vault credentials. When downloaded, place this file on the C:\Temp folder of the MABS server.

clip_image004

clip_image006

To install all required prerequisites, logon to the server you wish to use for your MABS v2 installation, open PowerShell and administrator and run the following commands to install .NET 3.5 SP1 and Hyper-V PowerShell (be shore to have the Windows Server 2016 installation ISO mounted – in my example to the D: drive). Be aware the server will reboot when the installation is completed. You can also download the complete script (.ps1) from the Microsoft TechNet Gallery.

clip_image008

MABS v2 software download

To download the MABS v2 software open PowerShell as an administrator and run the following PowerShell script. You can download the complete script (.ps1) from the Microsoft TechNet gallery. The script will download all the necessary files (8 files), extract them and start the setup.

MABS v2 installation

Click Microsoft Azure Backup Server to launch the setup wizard.

clip_image010

Setup will start copying some temporary files.

clip_image012

On the Welcome screen, click the Next.
clip_image014

This opens up the Prerequisite Check section. On this screen, click on the Check button to determine if the hardware and software prerequisites for Azure Backup Server have been met. If all of is OK, you will see a message indicating that the machine meets the requirements. Click Next.

clip_image016

On the SQL Settings page select, Install new Instance of SQL Server with this Setup, to install SQL 2016 SP1. Click Check and Install. You could encounter some error messages. If so follow the instructions and most likely, you should reboot the server and start the MABS installation all over again.

clip_image018

If the computer meets, the software and hardware requirements click Next.

clip_image020

Provide a location for the installation of all the files and click Next. In my example, I changed all locations to my E: drive.
clip_image022

Provide a strong password for restricted local user accounts (this password will not expire) and click Next.
clip_image024

It is strongly recommended to use Microsoft update when you check for updates because this will offer all security and important updates for MABS. Select whether to use Microsoft Update or not and click Next.

clip_image026

Review all settings and if all are OK click Install.

clip_image028

clip_image030

Click Next to start the Microsoft Azure Recovery Service Agent installation.

clip_image032

Click Install.

clip_image034

clip_image036

When the agent installation is completed, click Next.

clip_image038

Provide your vault credentials to register the machine to the Azure backup vault. Click Next.
clip_image040

Provide a passphrase to encrypt/decrypt the data sent between Azure and your premises. You can automatically generate a passphrase or provide your own minimum 16-character passphrase. Also, enter a location to save the passphrase. If all is done click Next.

clip_image042

Once registration succeeded the wizard proceeds with the installation and configuration of SQL Server 2016 SP1. This could take some time.

clip_image044

clip_image046

It is possible that you receive the following error message, if so just click OK (you can change the staging area after the MABS setups completes).
clip_image048

When setup completes successfully, click Close.
clip_image050

Double click the Microsoft Azure Backup server icon on your desktop to launch MABS.

clip_image052

clip_image054

You can also verify if the MABS server connection to the Recovery Services vault. To do so go to your Recovery Services vault, click Overview and click Backup management servers. There you should see the newly installed MABS server.

clip_image056

As a final step, do not forget to run Windows update to install all necessary updates after the MABS installation.

clip_image058

Now you are ready to start working with this brand new product. Have fun and till next time!

Wim Matthyssen (@wmatthyssen)