CrowdStrike Response to Windows Issue

Datacom is closely monitoring the widespread IT issue impacting organisations around the world, including several in Australia and New Zealand. CrowdStrike has posted advice on reports of crashes on Windows hosts related to their Falcon Sensor. We are working with CrowdStrike and will continue to support our customers who have been impacted.

All Datacom Support Services are back online, with the service desk and support services fully operational.

To our customers - please follow your Datacom escalation path, and our service desk will work with your teams to remediate. Your Datacom lead can be a point of contact if required. Please be aware our service desk is expecting large volumes of calls Monday morning. Our Service Desk agents will do their best to support everyone as soon as possible.

We are asking everyone to please remain vigilant to the potential for threat actors and phishing attempts during this time, as we see an increase in cybercriminal activity.

All client service desk IVRs have been setup for our customers for instructions on how to attempt to remediate the issue.

Some users have been able to successfully log in to their machines after multiple reboots. We are recommending that users reboot their devices and where available, use a wired connection instead of Wi-Fi. The device may require multiple reboots to enable the patch to download and for the device to restart. Please avoid using any workarounds from public forums.

For System Administrators, should this not be successful, the following steps can be taken

Workaround Steps:
- Reboot the host. If it crashes again, then:   
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys” and delete it
- Boot the host normally
Note: BitLocker-encrypted hosts may require a recovery key.

For the CrowdStrike Tech Alert please refer to the below information:

Summary

• CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
  
  

Details

• Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
  
  
• Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
  
  
• Windows hosts which are brought online after 0527 UTC will also not be impacted.
  
  
• This issue is not impacting Mac- or Linux-based hosts.
  
  
• Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
  
  
• Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.
  
  

Current Action

• CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
  
• If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
  Workaround Steps for individual hosts:
  - Reboot the host to give it an opportunity to download the reverted channel file.  If the host crashes again, then:
  - Boot Windows into Safe Mode or the Windows Recovery Environment
  - Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
  - Locate the file matching “C-00000291*.sys”, and delete it.
  - Boot the host normally.
  Note:  Bitlocker-encrypted hosts may require a recovery key.
  
  

Workaround Steps for public cloud or similar environment including virtual:

• Option 1:
  - Detach the operating system disk volume from the impacted virtual server
  - Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
  - Attach/mount the volume to to a new virtual server
  - Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory
  - Locate the file matching “C-00000291*.sys”, and delete it.
  - Detach the volume from the new virtual server
  - Reattach the fixed volume to the impacted virtual server

• Option 2:

              - Roll back to a snapshot before 0409 UTC. 

Workaround Steps for Azure via serial

  1.  Login to Azure console --> Go to Virtual Machines --> Select the VM
  2.  Upper left on console --> Click : "Connect" --> Click --> Connect --> Click "More ways to Connect"  --> Click : "Serial Console"
  3.  Step 3 : Once SAC has loaded, type in 'cmd' and press enter.
  1.  type in 'cmd' command
  2.  type in : ch -si 1
  4.  Press any key (space bar). Enter Administrator credentials
  5.  Type the following:
  1.  bcdedit /set {current} safeboot minimal
  2.  bcdedit /set {current} safeboot network
  6.  Restart VM
  7.  Optional: How to confirm the boot state? Run command:
  ◦  wmic COMPUTERSYSTEM GET BootupState
For additional information please see this Microsoft article.

Latest Updates

• 2024-07-19 05:30 AM UTC | Tech Alert Published.
  
  
• 2024-07-19 06:30 AM UTC | Updated and added workaround details.
  
  
• 2024-07-19 08:08 AM UTC | Updated
  
  
• 2024-07-19 09:45 AM UTC | Updated

If customers have critical systems impacted, Datacom technical teams and SDM will maintain a direct dialogue with customers.