Massive Microsoft Windows Crash – 8.5 Million Businesses Left Paralysed
The Hidden Risks of Automated Updates
On July 19, 2024, the world witnessed a stark reminder of our vulnerability in the digital age. A seemingly routine software update disabled 8.5 million business computers running Microsoft Windows, bringing operations to a grinding halt. This incident serves as a warning: our increasing reliance on computers and networks comes with hidden risks that can have far-reaching consequences.
One possible scenario is the distribution of software updates containing deliberate or accidental coding errors through automatic update mechanisms built into most PCs and online appliances. In January this year, I explored some of the various ways online systems could be compromised in a global manner, including the risks of remotely altering device and PC functionality using “software update” mechanisms.
The recent incident has made it clear that these risks are not just theoretical. The consequences of such an outage can be devastating:
- Widespread disruptions to supply chains and logistics
- Loss of critical infrastructure, such as healthcare or financial systems
- Economic instability, with potential losses in the billions
- Increased vulnerability to cyber attacks and data breaches
As a system administrator, I’ve been advocating for businesses to move away from connected online services and deploy privately self-managed computers using industry-standard open-source software. Fortunately, my clients were not affected by this incident, thanks to their understanding of the weaknesses inherent in telecommunications services, including phone and internet connectivity. This has given them a competitive edge during challenging times.
Key Takeaways:
- Maintain hard currency (cash) as a means to trade and purchase goods and services, ensuring business continuity in the face of digital disruptions.
- Exercise caution with Central Bank Digital Currencies, recognising their potential risks and vulnerabilities.
- Approach Digital IDs with skepticism, considering the implications of linking access to goods and services to these digital identifiers.
- Automatic Updates, in software systems are good to stay ahead of security exploits but pose their own risks when entrusted to large companies.
- Running a Proprietary Operating System, on all of your business infrastructure is poor disaster management. It is a single point of failure. Microsoft, Google, Apple, etc. are not immune to making mistakes.
- Mistakes happen, but what if cybercriminals were responsible for instigating the “mistake”. Your business plans should anticipate more incidents like this in the future.
Remember, while these measures can help mitigate risks, they do not guarantee complete immunity. However, by taking proactive steps, businesses can better protect their production capabilities in an increasingly interconnected world.
To those affected, I felt that it would help if I summarised the technical aspects of the cause and scope of the affected systems to help in getting their systems back online.
Microsoft Windows – System Crash
(19 July 2024 – Falcon security update)
The problem specifically affected MS Windows devices running CrowdStrike’s Falcon sensor versions 7.11 and above. This means that the issue was not tied to a particular Windows version, but rather to the version of the CrowdStrike Falcon security software installed on the devices.
Key points about the affected configurations:
- Falcon sensor version: The issue impacted systems running Falcon sensor versions 7.11 and higher.
- Windows versions: The problem affected various Windows versions, including both workstations and servers. There’s no mention of specific Windows versions being immune to the issue.
- Update mechanism: The problem was caused by a faulty configuration file (Channel File 291) that was pushed to systems as part of a routine content update.
- Kernel-level operation: The issue occurred because CrowdStrike’s software operates at the kernel level, which allowed the faulty update to cause system-wide crashes.
- Windows devices only: The problem was specific to Windows hosts. MacOS and Linux users were not impacted by this issue.
- Enterprise focus: While the issue affected less than 1% of all Windows machines globally, it had a significant impact because it primarily affected enterprise systems running critical services.
It’s important to note that the configuration that triggered the problem was not related to the Windows version itself, but rather to the presence of the CrowdStrike Falcon security software (version 7.11 or higher) and the reception of the faulty update. This explains why some of your company’s Windows devices were affected while others were not – the determining factor was likely the presence and version of the CrowdStrike Falcon software on each device.
If you are affected, then I recommend that you read the following article on CrowdStrike blog.