I thought this world-wide trouble might be due to some widely-used
cloud service, but it appears the common factor may be something else,
namely their dependence on a security service called “CrowdStrike”.

<https://www.nzherald.co.nz/nz/bank-problems-reports-bnz-asb-kiwibank-anz-visa-paywave-services-down/R2EY42QKQBALXNF33G5PA6U3TQ/>

In comp.misc Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
> I thought this world-wide trouble might be due to some widely-used
> cloud service, but it appears the common factor may be something else,
> namely their dependence on a security service called ?CrowdStrike?.

Yes, though it has in turn taken many VMs in Microsoft's Azure
cloud service offline, and amusingly Microsoft's official advice to
affected users is to turn their VMs off and on again, repeatedly:

https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/

--
__ __
#_ < |\| |< _#

Computer Nerd Kev <not@telling.you.invalid> wrote at 13:40 this Friday (GMT):
> In comp.misc Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>> I thought this world-wide trouble might be due to some widely-used
>> cloud service, but it appears the common factor may be something else,
>> namely their dependence on a security service called ?CrowdStrike?.
>
> Yes, though it has in turn taken many VMs in Microsoft's Azure
> cloud service offline, and amusingly Microsoft's official advice to
> affected users is to turn their VMs off and on again, repeatedly:
>
> https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/

So what was actually affected?
--
user <candycane> is generated from /dev/urandom

In article <slrnv9ksmb.47d.candycanearter07@candydeb.host.invalid>,
candycanearter07 <candycanearter07@candycanearter07.nomail.afraid> wrote:
>So what was actually affected?

I first heard about it here:

https://acecomments.mu.nu/?post=410680

which in turn contains these links:

https://www.abc.net.au/news/2024-07-19/global-it-outage-crowdstrike-microsoft-banks-airlines-australia/104119960
https://x.com/_JohnHammond/status/1814178288220479565
https://x.com/disclosetv/status/1814192537348833699
https://x.com/senadaruc/status/1814182862146429367

Where I work, the cloudy time-and-attendance software we use (which lives on
Azure IIRC) is affected, so people can't clock in and out and HR and
accounting can't do their usual things within it.

--
_/_
/ v \ Scott Alfter (remove the obvious to send mail)
(IIGS( https://alfter.us/ Top-posting!
\_^_/ >What's the most annoying thing on Usenet?

On 19/07/2024 15:10, candycanearter07 wrote:
> Computer Nerd Kev <not@telling.you.invalid> wrote at 13:40 this Friday (GMT):
>> In comp.misc Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>>> I thought this world-wide trouble might be due to some widely-used
>>> cloud service, but it appears the common factor may be something else,
>>> namely their dependence on a security service called ?CrowdStrike?.
>>
>> Yes, though it has in turn taken many VMs in Microsoft's Azure
>> cloud service offline, and amusingly Microsoft's official advice to
>> affected users is to turn their VMs off and on again, repeatedly:
>>
>> https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/
>
>
> So what was actually affected?

Crowdstrike is an endpoint monitoring system that is meant to detect bad
stuff on your computers and stop intrusions and shit. We have it on
Windows laptops and Windows servers. Never seems to do anything apart
from once when it decided the software my employer pays me to write was
a threat when I was testing it ;-)

There was an update today that like many security updates, can get
rolled out automatically. My employers don't do this... they do do some
testing of all updates. The update caused Windows (maybe just Win10)
machines to BSOD when booting.

There is a manual fix, do a safe boot (doesn't load lots of extra
drivers and utilities), go and find the erroneous file in one of
Windows' folders and delete it. Reboot, job done. Repeat on all affected
machines.

My place has about 17000 Windows laptops and servers (and almost as much
Linux stuff) so I think IT will be quite pleased they didn't push the
update out automatically.

On 2024-07-19 15:30:08 +0000, mm0fmf said:

> On 19/07/2024 15:10, candycanearter07 wrote:
>> Computer Nerd Kev <not@telling.you.invalid> wrote at 13:40 this Friday (GMT):
>>> In comp.misc Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>>>> I thought this world-wide trouble might be due to some widely-used
>>>> cloud service, but it appears the common factor may be something else,
>>>> namely their dependence on a security service called ?CrowdStrike?.
>>>
>>> Yes, though it has in turn taken many VMs in Microsoft's Azure
>>> cloud service offline, and amusingly Microsoft's official advice to
>>> affected users is to turn their VMs off and on again, repeatedly:
>>>
>>> https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/
>>
>>
>> So what was actually affected?
>
> Crowdstrike is an endpoint monitoring system that is meant to detect
> bad stuff on your computers and stop intrusions and shit. We have it on
> Windows laptops and Windows servers. Never seems to do anything apart
> from once when it decided the software my employer pays me to write was
> a threat when I was testing it ;-)
>
> There was an update today that like many security updates, can get
> rolled out automatically. My employers don't do this... they do do some
> testing of all updates. The update caused Windows (maybe just Win10)
> machines to BSOD when booting.
>
> There is a manual fix, do a safe boot (doesn't load lots of extra
> drivers and utilities), go and find the erroneous file in one of
> Windows' folders and delete it. Reboot, job done. Repeat on all
> affected machines.
>
> My place has about 17000 Windows laptops and servers (and almost as
> much Linux stuff) so I think IT will be quite pleased they didn't push
> the update out automatically.

An issue caused by a combination of two or three IT stupidities:

1. Using and relying on silly "cloud" services for everything.

2. Using and relying on Microsloth Windoze.
(Linux and MacOS computers were not affected)

3. Having your computer system / software set to automatically
install updates.

The only way to fix it is to manually boot into Safe Mode and delete /
rename the updated file from computers. A long and labourious process
for many IT departments and a lot of money for businesses. (No doubt
Crowdstrike is going to be facing a lot of lawsuits, especially in
America, land of the "sue 'em all" mentality and little taking
responsibility for your own decisions!)

In comp.misc BungleBob <bunglebob@thejungle.com> wrote:
>
> The only way to fix it is to manually boot into Safe Mode and delete /
> rename the updated file from computers.

It seems the anti-virus software checks for updates when it starts
up, so if the internet connection is available early enough after
booting it may fix itself automatically before it has a chance to
break the system:

https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/

--
__ __
#_ < |\| |< _#

On Fri, 19 Jul 2024 16:30:08 +0100, mm0fmf wrote:

> My place has about 17000 Windows laptops and servers (and almost as much
> Linux stuff) ...

The problem was only on the Windows machines.

On 2024-07-19 22:32:55 +0000, Computer Nerd Kev said:
> In comp.misc BungleBob <bunglebob@thejungle.com> wrote:
>>
>> The only way to fix it is to manually boot into Safe Mode and delete /
>> rename the updated file from computers.
>
> It seems the anti-virus software checks for updates when it starts
> up, so if the internet connection is available early enough after
> booting it may fix itself automatically before it has a chance to
> break the system:
>
> https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
>

Possibly, *if* a lot of things happen perfectly, but even that link
still says to do it manually if the computer keeps crashing.

On 19/07/2024 23:41, Lawrence D'Oliveiro wrote:
> On Fri, 19 Jul 2024 16:30:08 +0100, mm0fmf wrote:
>
>> My place has about 17000 Windows laptops and servers (and almost as much
>> Linux stuff) ...
>
> The problem was only on the Windows machines.

I am aware. The comment was to give a scale of the number of machines
and hence likely size of IT dept.

On Sat, 20 Jul 2024 09:01:31 +0100, mm0fmf wrote:

> On 19/07/2024 23:41, Lawrence D'Oliveiro wrote:
>
>> On Fri, 19 Jul 2024 16:30:08 +0100, mm0fmf wrote:
>>
>>> My place has about 17000 Windows laptops and servers (and almost as
>>> much Linux stuff) ...
>>
>> The problem was only on the Windows machines.
>
> I am aware. The comment was to give a scale of the number of machines
> and hence likely size of IT dept.

Interesting, though that you have so much Linux-based gear. Could that be
typical nowadays?

trimmed followups to comp.misc

In comp.misc BungleBob <bunglebob@thejungle.com> wrote:
> On 2024-07-19 15:30:08 +0000, mm0fmf said:

<snip>

> An issue caused by a combination of two or three IT stupidities:
>
> 1. Using and relying on silly "cloud" services for everything.

We are going back to pre 199x days.

> 2. Using and relying on Microsloth Windoze.
> (Linux and MacOS computers were not affected)

Same is true for the BSDs :)

> 3. Having your computer system / software set to automatically
> install updates.

Yes, I was surprised this was still a thing in Windows.
I would think people should be prompted say once per day,
after 5 declines, the update(s) are forced in.

<snip>

--
csh(1) - "An elegant shell, for a more... civilized age."
- Paraphrasing Star Wars

In comp.misc Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>
> I thought this world-wide trouble might be due to some widely-used
> cloud service, but it appears the common factor may be something else,
> namely their dependence on a security service called ?CrowdStrike?.

Microsloth as usual starts to blame everyone else rather than it's own
craptastic bug-ridden software.

Microsoft Blames European Commission for Major Worldwide Outage
<https://www.macrumors.com/2024/07/22/microsoft-blames-european-commission-for-outage/>

On Tue, 23 Jul 2024, BungleBob wrote:

> In comp.misc Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
>>
>> I thought this world-wide trouble might be due to some widely-used
>> cloud service, but it appears the common factor may be something else,
>> namely their dependence on a security service called ?CrowdStrike?.
>
> Microsloth as usual starts to blame everyone else rather than it's own
> craptastic bug-ridden software.
>
>
> Microsoft Blames European Commission for Major Worldwide Outage
> <https://www.macrumors.com/2024/07/22/microsoft-blames-european-commission-for-outage/>

Wow! But I shouldn't be surprised. All people I know run linux and were
completely unaffected. There is a link floating around that crowdstrike
did crash debian and rocky linux, but on linux there is no need for
crowdstrikes products in the first place so I imagine that there were very
few debian and rocky installations that were affected.

Now Crowdstrike are offering their client companies a bribe ...
insanely it is a measley US$10 Uber Eats gift card!!
<https://futurism.com/the-byte/crowdstrike-10-gift-card-apology>

Crowdstrike is a company that is about to be made bankrupt by numerous
court claims for millions, if not billions, of dollars in compensation.
(US$5 billion is esitmated to have been lost by the US Fortune 500
companies alone.)