You are browsing the archive for Cloud.

Unable to RDP to an Azure VM due to a CredSSP Encryption Oracle Remediation error

7:22 pm in Azure, Cloud, CredSSP, Encryption Oracle Remediation, RDP, Remote Desktop Connection, VM, Windows 10 by Wim Matthyssen

After applying some Windows updates on my Windows 10 Version 1803 home pc I was unable to make a Remote Desktop Connection (RDP) connection to some Microsoft Azure virtual machine(s) (VM).

When I made an RDP connection, I received the following error message:

An authentication error has occurred. The function requested is not supported. Remote computer: <computer name or IP>. This could be due to CredSSP encryption oracle remediation. For more information, see https://go.microsoft.com/fwlink/?linkid=866660.

clip_image002

What is CredSSP and why did it cause the error

The Credential Security Support Provider protocol (CredSSP) is a security protocol utilized to process authentication requests for separate applications like RDP. It allows you to securely forward credentials encrypted from the Windows client to the target servers for remote authentication.

Because of a critical vulnerability that has been discovered in CredSSP, which affects all versions of Windows and could allow remote attackers to exploit RDP and WinRM to steal data and run malicious code, Microsoft has released security update(s).
You can find the list of the corresponding KB number(s) for each operating system here: https://portal.msrc.microsoft.com/en-us/security-guidance/advisory/CVE-2018-0886

In my case my recently updated Windows 10 pc could not communicate with a non-updated server (not allowed to setup an insecure RDP connection).

Workaround

To solve the error, first of all, I needed to temporarily change the policy settings on my Windows 10 to gain RDP access to the server.

To do so, open Run and execute gpedit.msc to change the settings in the Local Group Policy Editor. Browse to Computer Configuration / Administrative Templates / System. Open Credentials Delegation in the left pane.

clip_image004

clip_image006

Change the Encryption Oracle Remediation policy to Enabled, and Protection Level to Vulnerable.

clip_image008

You can also use the following PowerShell script to do it in an more automated way: https://gallery.technet.microsoft.com/PowerShell-Workaround-956e0d7e.

Or you can simply use this command line one-liner which can also be run in PowerShell (run as admin):

After this change, I was able to setup an unsecure RDP connection to the server(s) where I installed the missing security update.

clip_image010

After deploying the specific update on the server(s), I was able to connect to it without the error and with the Encryption Oracle Remediation settings reset to the default.

clip_image012

Of course you can also use PowerShell to set everything back to the default (copy and save as .ps1).

Or like before you can simply use a command line one-liner in PowerShell (run as admin):

This concludes this blog post, hope it helps if you face this error.

Wim Matthyssen (@wmatthyssen)

Azure: Clean up unused, inactive or old directories from your Azure subscription

9:31 am in AAD, Azure, Azure Active Directory, Azure AD, Azure tenant, B2B, Cloud, GDPR, MyApps by Wim Matthyssen

I am already working as an Azure Consultant/Architect for almost 5 years. In those 5 years I setup a lot of Azure IaaS/PaaS environments for different customers. To do all the necessary work involved in such setup, I mostly was invited to their Azure tenant as admin with my Microsoft account (personal account) or my work account (B2B user) to do all the necessary work. When all the work was done a thing mostly forgotten is to clean up that specific user in Azure Active Directory (AAD), causing that tenant still showing up or even starting as the default directory when logging on to the Azure portal. After a while you could even be unable to be invited to a new tenant because the maximum of 20 AAD’s is reached for that specific account.

clip_image002

Until some time ago, May 14 2018 to be specific, to unlink those lingering directories you had to contact another global admin of the inviting organization to have that account removed from their AAD tenant. Even as an admin you were not able to delete your own guest account. Sometimes, when a lot of time was passed since you last worked for that customer, finding a global admin for that tenant to delete that user could be a lot of work.

Luckily, thanks to Europe’s General Data Protection Regulation (GDPR), this can now be done in a much easier way. A B2B user can now easily leave an organization on their own (self-service leaving), to which he or she has been invited at any time, without having to contact an administrator.

Keep in mind that when a user leaves an organization, the user account is soft deleted in the directory. By default, the user object moves to the Deleted users state in AAD but is not permanently deleted for 30 days. This soft deletion enables the administrator to restore the user account (including groups and permissions), if the user makes a request to restore the account within that 30-day period.

To leave an organization you can follow the below steps:

Log in with your B2B account at https://myapps.microsoft.com/

When logged in select your name on the access panel in the upper-right corner.

clip_image004

Under Organizations, select the organization you want to leave.

clip_image006

Select your name again in the upper-right corner.

clip_image008

Select Leave organization next to the correct organization.

clip_image010

When asked to confirm, select Leave.

clip_image012

clip_image014

After a while you should receive an email at that specific account, telling you that you left the organization.

clip_image016

Repeat these steps if you need to leave any other organization you are associated with.

Hope this helps and thanks to my colleague Guido (@ggibens) for pinpointing me to this new simplified capability.

Wim Matthyssen (@wmatthyssen)

Azure Backup Server: Unprotected servers still showing up in the Azure portal even though their protection was stopped 3 months ago

9:49 am in Azure, Azure Backup, Azure Backup Server, Azure portal, Cloud, Cloud backup, MABS, MABS v2 by Wim Matthyssen

 

To help protect your hybrid backup setup with an Azure Backup Server (MABS), Microsoft introduced some security features built on three principals – Prevention, Alerting and Recovery. These features are enabled by default for newly create Recovery Services vaults, for existing vaults this link will show you how you can enable them. One of these features related to recovery will ensure you that Azure backup will retain all deleted backup data for 14 days, which ensures you can recover data using any old or recent recovery point(s).

clip_image002

Sometime ago I reconfigured a Protection Group which protected some Hyper-V VMs. Two Domain Controllers (DCs) were taken out of the Group and setup to only backup the C drive and the System State. On the MABS server all configuration went well and did not cause any specific issues or errors. However last week when I was checking the Recovery Services vault used to store the cloud backups,I noticed those two DCs were still showing up in the Backup items overview.

clip_image004

Like you can see, those two VMs are still there with no Disk or Cloud Recovery Points created after the protection was disabled.

clip_image006

To get the issue fixed, I followed some standard steps I always follow when having issues with a MABS. The first one is checking the current Azure Backup Agent version installed on the MABS, which was version 2.0.9109.0. Because there is a newer version available (at the time of writing version 2.0.9118.0), step one was getting that one in place.

clip_image008

To download the latest agent go to your Recovery Services vault blade in the Azure portal. Select Backup and on the Getting Started with Backup blade, select Backup goal. In the drop-down menu(s), select On-premises and Files and folders, click OK. In the Prepare Infrastructure blade, click Download Agent for Windows Server or Windows Client. Save MARSAgentInstaller.exe.

clip_image010

clip_image012

Install the latest agent on the MABS server. After the agent installation completes restart the following service:

Microsoft Azure Recovery Services Management Agent

clip_image014

clip_image016

clip_image018

Although the agent is now at the latest version it still did not fix the protection status of the deleted servers in the Azure portal.

After doing a little more troubleshooting (reading the logs, etc.) , I decided to open an Azure support ticket. The support agent who assisted me, told me, just like I already suspected, that this was currently  the default behavior from the azure backup service in some Azure regions (current backend design behavior like they say). The product team was already aware of this issue and they definitively will fix it in some later update.

If you cannot wait for the update, there is a quicker fix for the issue, you just need to delete the whole MABS server from the Azure portal and reconnect the server all over again. However, for me and even more for the customer this was a no go. So, we will wait for the proper backend update which will hopefully not take that long anymore.

Hope this helps whenever you face the same backup behavior in the Azure portal with your deleted MABS backups.

Wim Matthyssen (@wmatthyssen)

Azure Tip: Use Ctrl+Alt+D to check Azure Portal load times

6:55 pm in Azure, Azure portal, Azure Tip, Cloud, Keyboard shortcut by Wim Matthyssen

 

The Azure Portal is the go-to place to manage all of your Azure services in one hub. I myself spend a lot of time in the portal to build, deploy, modify and manage customers cloud resources. I am sure a lot of you do the same.

But sometimes this portal feels slow without any specific reason and then it is really difficult to find out why. Whenever that is the case there is a keyboard shortcut you can use to check the portal load time of all opened blades.

If you press the keyboard shortcut CTRL + ALT + D you can see the load time and other useful information for every title.

clip_image002

clip_image004

clip_image006

clip_image008

clip_image010

Pressing CTRL + ALT + D again will remove the portal load information.

Beside this useful keyboard shortcut there are some others you can use specifically for the Azure portal. You can open the Keyboard shortcut help item in the Help Menu on the top-right of the portal to see all of these shortcuts.

clip_image012

Hope it helps!

Wim Matthyssen (@wmatthyssen)

Creation of an Azure VPN gateway failed due to associated NSG

8:53 am in Azure, Cloud, GatewaySubnet, NSG, VNet, VPN gateway by Wim Matthyssen

 

A VPN gateway is a specific type of virtual network gateway that sends encrypted traffic between your virtual network (VNet) and your on-premises location across a public connection. You can also use a VPN gateway to send traffic between virtual networks across the Azure backbone.

While deploying such a gateway trough the Azure portal, the creation took a very long time and in the end the deployment Failed.

clip_image002

In the Activity log the following Error Code was showed.

OnlyTagsSupportedForPatch

clip_image004

clip_image006

After some troubleshooting and reviewing the complete VNet deployment, which was done through Azure PowerShell, I finally found out what caused the gateway deployment to fail.

An important remark is mentioned in the Microsoft technical documentation for creating a Site-to-Site connection in the Azure portal. It states that you may not associate a network security group (NSG) to the gateway subnet, which in my case was causing the issue.

clip_image008

The Azure PowerShell script used to setup the VNet and all of its Subnets also created NSGs for all subnets, the GatewaySubnet included.

To resolve the issue, I deleted the Failed gateway and set the Network security group for the GatewaySubnet to None.

clip_image010

clip_image012

Afterwards the creation of the gateway succeeded without any issues.

 

Conclusion

When you create a gateway subnet for your VNet you should never associated a NSG to it. This is not supported and the gateway will stop functioning as expected or completely fail. Always set the NSG to ‘None’. The gateway subnet also needs to be named ‘GatewaySubnet’ to work properly and never deploy any VMs or anything else to it.

Wim Matthyssen (@wmatthysen)

Azure Backup: Upgrade your Recovery Services Vault to enable support for large disk backups

6:59 am in Azure, Azure Backup, Cloud, Recovery Services vault by Wim Matthyssen

 

On March 13, 2018 the Azure Backup team announced the general availability for backup of Azure IaaS Virtual Machines (VMs) with large disks (1 to 4 TB), both managed and unmanaged. At the same time they released a set of other improvements to speed up the overall backup and restore process.

To enable these new features a one-time, one-directional upgrade must be done for every Azure Subscription where you wish to use these enhancements. Good to know is that this VM backup stack upgrade, can be started from any vault in your Subscription and will retain all your existing policies and recovery points.

 

Upgrade procedure

 

Open the Azure portal and login with you Azure credentials.

Go to your Recovery Services vault dashboard, on the top of the blade you will need to click the banner which says Support for > 1 TB disk VMs and improvements to backup and restore speed ->. If you do need see a banner, you can open Properties, go to VM backup stack and click Upgrade.

clip_image002

image

The Upgrade to new VM backup stack blade will open. Click on Upgrade.

clip_image004

The upgrade procedure will start, be aware that this process could take up to two hours.

clip_image006

Have fun backing up Azure VMs with these new enhancements. Till next time!

Wim Matthyssen (@wmatthyssen)

Azure Interactives

10:46 am in Azure, Azure Interactives, Cloud by Wim Matthyssen

 
Azure is a growing collection of integrated services that IT professionals and developers can use to build, deploy or manage applications in the cloud. With so many services now available, most of the time it is somewhat of a challenge to find the right information. To help you have a clear overview, Microsoft has released a new beta webpage, the Azure Interactives, which will guide and introduce you to all the various services that are available on Azure.

clip_image002

From this interactive page, you can easily navigate to the three experiences listed below, to display and find all information you want.
 

Azure Products

From here, you can easily select a specific Azure product and find all information, documentation and pricing links about it.

clip_image004

Cloud design patterns

This page will give you architecture guidance and lists common problems and patterns for your cloud applications.

clip_image006

Azure security + operations management

This page will give you a guide on how to efficiently manage and protect your Azure and on-premises resources.

clip_image008

Have fun checking this all out.

Wim Matthyssen (@wmatthyssen)

MABS v2: Error [0x8007007b] when performing a System State Backup on a DC running on a VMware VM

8:33 am in Azure, Azure Backup, Azure Backup Server, Cloud, Error [0x8007007b], MABS, MABS v2, Power, PowerShell, VMware by Wim Matthyssen

While configuring a Microsoft Azure Backup Server (MABS) v2 at a customer site, I encountered a problem while performing a System State Backup of their domain controllers (DC’s). The Protection Status showed Replica is inconsistent.

clip_image002

When looking in the Monitoring tab, following detailed message is show:

DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code: 0x605A140).(ID 30229 Details: Internal error code: 0x8099ED0)

clip_image002[6]

Because the first part of making a System State Backup is done by the local Windows Server Backup (WSB) feature, logon to the protected server and open Windows Server Backup (Server Manager – Tools – Windows Server Backup). There a message was shown indicating that the last backup has Failed.

clip_image006

To view the error message a bit more in detail, open the Windows Server backup log file (with the exact date and timestamp) located in C:\Windows\Logs\WindowsServerBackup.

clip_image008

In the log file the following error message was shown:

Error in backup of C:\windows\\systemroot\ during enumerate: Error [0x8007007b] The filename, directory name, or volume label syntax is incorrect.

clip_image010

When looking in the Event Viewer (Application log) I could also find the following errors (CAPI2 – 513, Backup – 517):

Event ID 513

Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.

Details:

AddLegacyDriverFiles: Unable to back up image of binary Microsoft Link-Layer Discovery Protocol.

System Error:

Access is denied.

Event ID 517

The backup operation that started at ‘‎2017‎-‎11‎-‎16T15:16:22.000076700Z’ has failed with following error code ‘0x80780049′ (None of the items included in backup were backed up.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

clip_image012

clip_image014

Because all those errors descriptions do not really tell you what exactly is going wrong causing the backup to fail, you need to use the Diskshadow command-line tool to determine if there is an issue with the functionality of the VSS service or any of the application independent VSS writers.

To open the Diskshadow tool interface start PowerShell with elevated privileges and enter the below commands to write the output to a logfile.

clip_image016

When the logfile (c:\out.txt) is created open it with notepad and search for \\.

clip_image018

clip_image020

In my case, I found out there was an issue with the vsock.sys driver, which is part of the VMware vSockets Service and which is usually located in the C:\Windows\system32\drivers folder.

To fix the issue open the Registry Editor and go to the following location, HKLM\system\controlset001\services\vsock and changed the Start value to 1.

clip_image022

clip_image024

clip_image026

Also change the ImagePath entry from \SystemRoot\system32\DRIVERS\vsock.sys to system32\DRIVERS\vsock.sys.

clip_image028

clip_image030

When you have changed all those registry keys, logon to your MABS server and right click the failed System State backup and Perform a consistency check… (be aware that this could take a while). If the fix also solved your issue it would show OK when completed.

clip_image032

clip_image034

Hope this helps whenever you face the same error in your MABS environment. If you have any questions feel free to contact me trough my Twitter handle.

Wim Matthyssen (@wmatthyssen)

New features for Azure Backup and Azure Site Recovery released

10:02 am in Azure, Azure Backup, Azure Site Recovery, Cloud, Modern Backup Storage, Windows Server 2016 by Wim Matthyssen

Microsoft was very busy on the last day of May, because yesterday they launched many new features, not only for Azure Backup but also for Azure Site Recovery. I tried to list some of them below.

Azure Backup

  • Windows Server System State backups with Azure Backup now in public preview

This new extension allows the Azure Backup agent (MARS Agent) to integrate with the Windows Server Backup feature that is available natively on every Windows Server. It allows and provides seamless and secure backups of your Windows Server System State directly to Azure without the need to provision any on-premises infrastructure.

You can read more about it here

 

clip_image002

  • Microsoft Azure Backup Server v2 released which allows Windows Server 2016 and vCenter/ESXi 6.5 protection

This week Microsoft also released the second version (v2) of their Microsoft Azure Backup Server (MABS v2), which supports Windows Server 2016, vSphere 6.5 and the latest business critical applications such as SQL 2016, SharePoint 2016 and Exchange 2016. This new version is available for download from a Recovery Services vault in the Azure Portal or directly from here.

 

clip_image004

If you are interested to read more about MABS v2 you can do so over here

An important remark to make is that when you install MABS v2 on a Windows Server 2016 the VMware protection will be in preview mode, because VMware first needs to release support for VDDK 6.5.

In addition, the UserVoice I opened to address this issue to the Azure Team will be closed, so everyone who voted will get some votes back.

  • Introducing Modern Backup Storage with Azure Backup Server on Windows Server 2016

With the latest release of Azure Backup Server (MABS v2), which is based on System Center Data Protection Manager 2016 (SCDPM 2016), Modern Backup Storage can be used. This technology will improve performance and reduces consumption (50 % disk storage savings and 3x faster backups) by leveraging ReFS block cloning and deduplication.

 

clip_image006

You can read more about it here

 

Azure Site Recovery

  • Disaster recovery for Azure IaaS virtual machines with Azure Site Recovery is now in public preview

This will allow you to use Azure Site Recovery (ASR) to easily replicate and protect IaaS based applications running on Azure to a different Azure region of your choice without deploying any additional infrastructure components or software appliances in your subscription.

 

clip_image007

You can read more about it here

 

Enjoy reading about these nice new features and have fun testing them out.

Wim Matthyssen (@wmatthyssen)

Azure Security Center: Endpoint Protection installation failed with “Permission denied”

2:56 pm in Azure, Azure Security Center, Cloud, Endpoint Protection, Microsoft Antimalware by Wim Matthyssen

Most of you who are already familiar with Azure Security Center (ASC), know that it periodically analyzes the security state of your Azure resources. Whenever Security Center identifies a potential security vulnerability, it creates a recommendation. Last week when trying to apply the solution for such a Recommendation, namely Install Endpoint Protection, the Endpoint Protection installation failed with “Permission denied”.

clip_image002

The error showed that the installation failed because of an RBAC issue (permission error), However, the user used was a Subscription co-admin (Role Owner), so that could not cause the problem because he has all permissions needed.

Because Endpoint Protection is deployed as an extension and deployments of extensions are handled by the VM agent, my next troubleshoot step was to check the log of Azure VM agent on that particular VM.

The path to access this log is: “C:\WindowsAzure\Logs\WaAppAgent.log

clip_image004

clip_image006

But also no issue over here.

Therefore, after troubleshooting for some time, I finally opened a support request to Microsoft. As a response to this request, Microsoft confirmed that this error is under investigation of the product team and that there currently is a design change request in the making to get this problem fixed. For the moment, the problem only occurs in some Azure Regions. In the meantime as a temporary workaround in wait for the real fix, they suggest to install the Azure Antimalware extension from Compute or Azure PowerShell instead of with ASC.

To deploy the Azure Antimalware extension using the Azure Portal you can follow these steps:

Log in to the Azure portal

Select the VM, select Extensions and click Add on the Extensions blade

clip_image008

Select Microsoft Antimalware and click Create on the Microsoft Antimalware blade

clip_image010

To enable Antimalware with the default settings just click OK without putting in any configuration values. If you prefer you can also configure it with your own settings and values

clip_image012

Once the extension successfully installs, it reflects in ASC and the recommendation for that specific VM is gone. Hope this helps!

Wim Matthyssen (@wmatthyssen)