Early this morning, a widespread fault with Microsoft Windows machines running the CrowdStrike Falcon agent caused chaos around the globe – grounding flights, taking banks, hospital systems, and media offline, and causing a massive global disruption to companies and services around the world.
What Happened?
Cybersecurity firm CrowdStrike said that the issue believed to be behind the outage was not a security incident or cyberattack — the problem occurred when it deployed a faulty update to computers running Microsoft Windows.
Microsoft stated, “We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state.”
Steps to Resolve the Issue
Microsoft Azure released a fix for this issue. For detailed instructions, visit: https://azure.status.microsoft/en-gb/status
1. Restart Your Virtual Machines
Many users have reported success by repeatedly restarting their VMs. Although it may take multiple attempts (as many as 15 in some cases), this has proven to be an effective troubleshooting step. You can restart your VMs through the Azure Portal or using the Azure CLI:
- Using the Azure Portal: Navigate to your affected VMs and click on ‘Restart.’
- Using the Azure CLI or Azure Shell: Follow the instructions here to restart your VMs: Azure CLI Documentation
2. Restore from a Backup
If you have backups from before 19:00 UTC on July 18th, restoring from these backups is a reliable solution. Here’s how you can do it if you are using Azure Backup:
- Follow the instructions in this guide: How to Restore Azure VM Data
3. Repair the OS Disk
Another option is to repair the OS disk by attaching it to a repair VM. This allows you to delete the problematic file directly. Here are the steps:
- Attach the OS disk to a repair VM through the Azure Portal.
- Navigate to the disk and delete the file located at Windows/System32/Drivers/CrowdStrike/C00000291*.sys.
- Detach the disk and reattach it to the original VM.
For detailed instructions on repairing the OS disk, refer to: Troubleshoot a Windows VM
.sys Removal Script
This script automatically finds and removes the problematic .sys file on the host. This script can be put on a USB drive and executed with administrative privileges for ease of use across multiple systems.
Ongoing Support
The affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance.
Microsoft is continuing to investigate additional mitigation options for customers and will share more information as it becomes known. For current updates, visit: https://azure.status.microsoft/en-gb/status