Azure Backup Server: SQL 2016 AlwaysOn protection fails with Internal error code 0x80990F75

January 23, 2018 at 3:54 pm in 0x80990F75, Azure, Azure Backup, Azure Backup Server, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, SQL Always On Availability Groups, SQL Server 2016 by Wim Matthyssen

While setting up a new backup job to protect a SQL Server 2016 AlwaysOn Availability Group (AG) using Azure Backup Server (MABS), the job failed and ended up with the Protection Status – Replica is inconsistent.

clip_image002

Because this status does not say a lot about what exactly went wrong, I looked up the Critical alert under the Monitoring tab. There the following more detailed message was shown:

The DPM job failed for SQL Server 2016 database <DBname> on <serverName > because the SQL Server instance refused a connection to the protection agent. (ID 30172 Details: Internal error code: 0x80990F75)

clip_image004

This issue occurs because when you create a new Availability Group by default, the location where backups should occur is set to Prefer Secondary and the setting Make Readable Secondary is set to No, which always results in MABS getting the above error.

clip_image006

To resolve the issue, open SQL Management Studio and connect to server instance that hosts the primary replica. Expand the Always On High Availability node and the Availability Groups node. Click the availability group whose replica you want to change and expand Availability Replicas.

clip_image008

Right-click the Availability Replica, and click Properties (be sure to repeat this steps for all Availability Replicas you want to backup with MABS).

clip_image010

In the Availability Replica Properties dialog box, set the value of the field Readable secondary to Yes. Click OK to save the new setting.

clip_image012

When you now run the Perform consistency check … job on the failed Protected member in the MABS console, the status should end up in OK.

If not, and the status still ends up in Replica is inconsistent, be sure to check out my previous blog post http://scug.be/wim/2017/06/19/microsoft-azure-backup-server-unable-to-configure-protection-for-a-sql-database-id-3170-and-33424/ to see if the user NT Authority\SYSTEM has sysadmin rights on the SQL Server instance(s).

If on the other hand the status ends up in Online recovery point creation failed, just right-click the Protected member again and select Resume azure backups…

I hope this helps and if you have any questions feel free to contact my through my Twitter handle.

Wim Matthyssen (@wmatthyssen)

Azure Backup Server: Remove Unprotected computers with protection agent installed

January 4, 2018 at 8:36 am in Azure Backup Server, MABS, MABS v2, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, PowerShell by Wim Matthyssen

While doing maintenance on a customer’s Azure Backup Server (MABS), I was unable to remove some unprotected computers in the MABS console. The Remove resulted in a fail and the error page didn’t show a direct reason why this occurred.

clip_image002

clip_image004

clip_image006

clip_image008

But no worries, this is where PowerShell came into the rescue to force the removal of the broken agent from the DPMDB database. Take notice that this solution will not uninstall the protection agent from the (un)protected computer. When required, you still need to uninstall that agent manually.

Open the DPM Management Shell and run the following command, it will prompt you for the rest of the parameters one at a time (always use the FQDN name for both parameters).

You can also run this command with all paramaters already filled in. Just replace [DPMServerName] with the name of the MABS server and [ProtectedComputerName] with the name of the (un)protected computer that must be removed.

image

Like you can see the agent(s) are now removed from the MABS console.

clip_image012

Hope it helps.

Wim Matthyssen (@wmatthyssen)

Azure Interactives

December 27, 2017 at 10:46 am in Azure, Azure Interactives, Cloud by Wim Matthyssen

 
Azure is a growing collection of integrated services that IT professionals and developers can use to build, deploy or manage applications in the cloud. With so many services now available, most of the time it is somewhat of a challenge to find the right information. To help you have a clear overview, Microsoft has released a new beta webpage, the Azure Interactives, which will guide and introduce you to all the various services that are available on Azure.

clip_image002

From this interactive page, you can easily navigate to the three experiences listed below, to display and find all information you want.
 

Azure Products

From here, you can easily select a specific Azure product and find all information, documentation and pricing links about it.

clip_image004

Cloud design patterns

This page will give you architecture guidance and lists common problems and patterns for your cloud applications.

clip_image006

Azure security + operations management

This page will give you a guide on how to efficiently manage and protect your Azure and on-premises resources.

clip_image008

Have fun checking this all out.

Wim Matthyssen (@wmatthyssen)

MABS v2: Unable to install DPM Remote Administration console on a W2K8 R2 SP1 server because mi.dll is missing

December 12, 2017 at 9:32 am in Azure, Azure Backup, Azure Backup Server, DPM Remote Administration, mi.dll, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, PowerShell, WMF 5.1 by Wim Matthyssen

While installing the DPM Remote Administrator console on a Windows Server 2008 R2 SP1 (W2K8 R2 SP1) for remote administration of a customers Microsoft Azure Backup Server (MABS) v2, I stumbled upon the below error message, which resulted in the setup being aborted:

The Program can’t start because mi.dll is missing from your computer. Try reinstalling the program to fix this problem.

clip_image002

This error shows up because one of the following requirements is not installed: Windows Management Framework 4.0, .NET Framework 4.0 or Visual C++ Redistributable for Visual Studio 2012 Update 4

needs to be installed to be able to deploy the DPM Remote Administration console on a W2K8 R2 server.

To fix the issue, I checked if all latest Windows Updates were installed. Afterwards I installed the Windows Management Framework 5.1 (WMF 5.1), .NET Framework 4.0 and the Visual C++ Redistributable for Visual Studio 2012 Update 4 on the W2K8 R2 SP1 server, which can be downloaded from the link above. To ease up and to automate the installation, you can use the below PowerShell script (copy and/or save as .ps1) to get things downloaded somewhat faster.

clip_image004

When the C:\Temp folder opens after the downloads, run Install-WMF5.1.ps1. (PowerShell window with Administrator privileges) to install WMF 5.1

clip_image006

clip_image008

clip_image010

clip_image012

Before rebooting, also run the two other packages, dotNetFx40_Full_setup.exe and vcredist_x64.exe (if required). When done reboot the server.

clip_image014

When the server is rebooted, check if mi.dll exists under C:\Windows\System32.

clip_image016

You can now start Setup.exe (Microsoft Azure Backup Server folder) and start the DPM Remote Administration installation.

clip_image018

clip_image020

Hope this post helps whenever you face the same problem.

Wim Matthyssen (@wmatthyssen)

MABS v2: Error [0x8007007b] when performing a System State Backup on a DC running on a VMware VM

November 23, 2017 at 8:33 am in Azure, Azure Backup, Azure Backup Server, Cloud, Error [0x8007007b], MABS, MABS v2, Power, PowerShell, VMware by Wim Matthyssen

While configuring a Microsoft Azure Backup Server (MABS) v2 at a customer site, I encountered a problem while performing a System State Backup of their domain controllers (DC’s). The Protection Status showed Replica is inconsistent.

clip_image002

When looking in the Monitoring tab, following detailed message is show:

DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code: 0x605A140).(ID 30229 Details: Internal error code: 0x8099ED0)

clip_image002[6]

Because the first part of making a System State Backup is done by the local Windows Server Backup (WSB) feature, logon to the protected server and open Windows Server Backup (Server Manager – Tools – Windows Server Backup). There a message was shown indicating that the last backup has Failed.

clip_image006

To view the error message a bit more in detail, open the Windows Server backup log file (with the exact date and timestamp) located in C:\Windows\Logs\WindowsServerBackup.

clip_image008

In the log file the following error message was shown:

Error in backup of C:\windows\\systemroot\ during enumerate: Error [0x8007007b] The filename, directory name, or volume label syntax is incorrect.

clip_image010

When looking in the Event Viewer (Application log) I could also find the following errors (CAPI2 – 513, Backup – 517):

Event ID 513

Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.

Details:

AddLegacyDriverFiles: Unable to back up image of binary Microsoft Link-Layer Discovery Protocol.

System Error:

Access is denied.

Event ID 517

The backup operation that started at ‘‎2017‎-‎11‎-‎16T15:16:22.000076700Z’ has failed with following error code ‘0x80780049′ (None of the items included in backup were backed up.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

clip_image012

clip_image014

Because all those errors descriptions do not really tell you what exactly is going wrong causing the backup to fail, you need to use the Diskshadow command-line tool to determine if there is an issue with the functionality of the VSS service or any of the application independent VSS writers.

To open the Diskshadow tool interface start PowerShell with elevated privileges and enter the below commands to write the output to a logfile.

clip_image016

When the logfile (c:\out.txt) is created open it with notepad and search for \\.

clip_image018

clip_image020

In my case, I found out there was an issue with the vsock.sys driver, which is part of the VMware vSockets Service and which is usually located in the C:\Windows\system32\drivers folder.

To fix the issue open the Registry Editor and go to the following location, HKLM\system\controlset001\services\vsock and changed the Start value to 1.

clip_image022

clip_image024

clip_image026

Also change the ImagePath entry from \SystemRoot\system32\DRIVERS\vsock.sys to system32\DRIVERS\vsock.sys.

clip_image028

clip_image030

When you have changed all those registry keys, logon to your MABS server and right click the failed System State backup and Perform a consistency check… (be aware that this could take a while). If the fix also solved your issue it would show OK when completed.

clip_image032

clip_image034

Hope this helps whenever you face the same error in your MABS environment. If you have any questions feel free to contact me trough my Twitter handle.

Wim Matthyssen (@wmatthyssen)

How to run the Hyper-V role on a VMware VM

November 21, 2017 at 10:16 am in Hyper-V, MABS v2, Nested Virtualization, PowerShell, VMware, Windows Server 2016 by Wim Matthyssen

When you install Microsoft Azure Backup Server (MABS) v2 on a Windows Server 2016, one of the prerequisites (MABS v2 prerequisites installation script) is that you install the Hyper-V role and the Hyper-V PowerShell feature.

However, while I was installing a new MABS v2 for a customer on a VMware VM (vSphere 6.5), I encountered following errors in the Hyper-V event log (41, 15350, 15340) after the Hyper-V role was installed.

Event 41 showed the following error message:

Hypervisor launch failed, Either VMX not present or not enabled in BIOS.

clip_image002

When I ran the Get-WindowsFeature in PowerShell it seemed Hyper-V was installed correctly. But this was not the case.

clip_image004

To fix the errors and get Hyper-V running like it should you need to enable Nested Virtualization for the VMware VM. To do so, shut down the VM and open the Virtual Machine Settings. Then go the Virtual Hardware tab and open the CPU options. There you need to check the box Expose hardware assisted virtualization to the guest OS. Also set CPU/MMU Virtualization to Hardware CPU and MMU.

clip_image006

When you now start the VM all Hyper-V related errors should be gone and all necessary Hyper-V services should be running.

clip_image008

clip_image010

Hope this blog post will help you whenever you need to setup Hyper-V on a VMware VM.

Wim Matthyssen (@wmatthyssen)

MABS v2: Creation of a Protection Group fails with error 33507 related to running Trend Micro Antivirus

November 16, 2017 at 9:39 am in Azure, Azure Backup, Error 33057, MABS, Microsoft Azure Backup Server, Microsoft Azure Backup Server v2, PowerShell, Trend Micro by Wim Matthyssen

Last week while assisting a customer with the deployment of a new Microsoft Azure Backup Server (MABS), I ran run across error 33507 when creating a new Protection Group.

clip_image002

In the Event Viewer on the MABS error 370 was shown several times.

clip_image004

After troubleshooting for some time, I finally found out the problem was caused by the Trend Micro antivirus (version 12.0.1708) running on the MABS. The issue is caused by an incompatibility between Trend Micro and the DPM filter driver.

clip_image006

When the antivirus was disabled MABS was able to create the Protection Group. But because running a server without an antivirus solution is not an option you can resolve the issue by adding the following registry key on any MABS server running with Trend Micro.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TmFilter\Parameters
Add DWORD (32 bit)
BypassDPMfilter, vaule of 1

You can add the registry key quite easy by running the following PowerShell one-liner (run PowerShell as an administrator).

clip_image008

After adding the registry key, reboot the server and try the process of creating a Protection Group all over again. I should work now.

Lesson learned, sometimes it’s the easy stuff. Disabling antivirus during a backup problem is often step 1. Now it was more like step 10 but at least in the end it helped fixing the issue. Till next time.

Wim Matthyssen (@wmatthyssen)

Azure PowerShell: Your Azure credentials have not been set up or have expired (classic – ASM)

October 3, 2017 at 8:56 am in ASM, Azure, Azure PowerShell, PowerShell by Wim Matthyssen

Last week, after a long time I was cleaning up an Azure ASM environment with Azure PowerShell for a customer when I stumbled upon the following error:

“Your Azure credentials have not been set up or have expired, please run Add-AzureAccount to set up your Azure credentials.”

clip_image002

After checking the account settings, everything seemed valid. So probably for some reason, some old settings were cached causing the problem.

clip_image004

Solution:

After trying and testing out some cmdlets, the following finally solved the issue:

clip_image005

When you have run the above cmdlet, try to run the Add-AzureAccount cmdlet and afterwards any other cmdlet to see if this also solves the issue for you.

clip_image007

Hope this helps you when you are facing the same issue.

Wim Matthyssen (@wmatthyssen)

Azure IaaS: Deploying B-series VMs

September 22, 2017 at 10:18 am in ARM, Azure, Azure PowerShell, B-Series VMs, IaaS, PowerShell, Public Cloud by Wim Matthyssen

Last week Microsoft introduced the B-Series VM size in preview. These B-Series VMs can run workloads that burst in their performance, but do not need continuous full performance of the CPU. Servers that would be eligible for this new “burstable” VM size are servers with small databases, webservers, development servers, quality assurance (QA) and test servers. But also servers with other workloads that do not utilize the full located vCPUs are grate candidates to benefit and lower costs by using a B-Series VM.

clip_image002

Herby a list of the 6 B-Series VM sizes, which are currently available in Preview in the following Azure Regions (Europe-West, US-West 2, US-East, Asia Pacific-Southeast):

Instance Size vCPU vMemory: GiB Tempory Storage / Local SSD: GiB Max data disks Max NICs Credits banked / hour Max Banked Credits
Standard_B1s 1 1 2 2 2 6 144
Standard_B2s 2 4 8 4 3 24 576
Standard_B1ms 1 2 4 2 2 12 288
Standard_B2ms 2 8 16 4 3 36 864
Standard_B4ms 4 16 32 8 4 54 1296
Standard_B8ms 8 32 64 16 4 81 1944

You can also read more about this new VM size here and find all pricing info

Now I am going to show you how you can deploy a new B-Series VM trough Azure PowerShell.

First, you need to request quota to be able to deploy this B-Series VMs. To do so you should logon to the Azure portal and go to Help + support. To request an increase or to be able to deploy B-Series VMs, select New support request.

clip_image004

You need to create a Quota support case for Cores. So, on the NEW SUPPORT REQUEST page, select Issue type as “Quota” and Quota type as “Cores”. Also, select the Subscription and the correct Support plan. Click Next.

clip_image006

Select Severity “C – Minimal Impact”, Deployment model “Resource Manager” and the correct Location, which in my case is West Europe. Select as SKU Families that requires an increase “BS Series” and set the NEW LIMIT higher than before, for example 15 instead of 10. Click Next.

clip_image008

In the Contact Information blade, select your Preferred contact method, provide a Response time, select your preferred Language and fill in the Contact Information. Click Create to create the new support ticket.

clip_image010

clip_image012

In my case I received an email after the quota was been approved, which normally does not take that much time. So from here we can go further with the deployment.

clip_image014

If you open Azure PowerShell, and run following commands, you can now built a new B-Series VM. You can copy the commands or save them to as a PowerShell script (.ps1). Do not forget to adjust all variables were needed.

clip_image016

clip_image018

Hope this helps you getting started with this new B-Series VMs.

Wim Matthyssen (@wmatthyssen)

Azure Backup: Create a Recovery Services vault with Azure PowerShell

July 12, 2017 at 9:46 am in Azure, Azure Backup, Azure PowerShell, PowerShell, Public Cloud, Recovery Services vault by Wim Matthyssen

A Recovery Services vault is an online storage entity used to backup workloads to the Azure cloud. You can use it to hold backup data for various Azure services such as IaaS VMs (Linux or Windows) and Azure SQL databases, but it can also be used by System Center Data Protection Manager (SCDPM) or Azure Backup Server (MABS v1 and MABS v2) to enable cloud backups.

clip_image002[5]

These days it is quite easy to create or manage a Recovery Services vault through the Azure portal, but it is even faster when you make use of a scripting language like Azure PowerShell to automate the setup. Therefore, below you can find the PowerShell script I mostly use to do all the work for me. You can just copy and paste or you can download the complete script (.ps1) from the Microsoft TechNet gallery.

To use the script, first adjust all variables to your use. Afterwards login into an Azure PowerShell window as an administrator and when asked login with the credentials for your Azure Subscription.

The script will first create a Resource Group and then the Recovery Services vault in your Azure Subscription. At the end, it will also set the storage redundancy for the newly created vault. Keep in mind that you can only use Locally Redundant Storage (LRS) or Geo Redundant Storage (GRS).

clip_image004[6]

clip_image006[6]

clip_image008[5]

Useful Azure PowerShell cmdlets for Azure Backup

List all available Azure Backup PowerShell cmdlets

clip_image010[7]

List all available Recovery Services vaults in your subscription

clip_image012[4]

Hope this post helps you when you start using Azure Backup.

Wim Matthyssen (@wmatthyssen)