That seems like enough of a line of bullshit to steer me away from ever using wyze.
Given their openness in the rest of the communications, I don't see why they would make this part up.
Edit: Of course, I'm also curious what the actual bug was. A discussion below is suggesting several plausible ways (e.g. concurrency issues, insufficient entropy in some key) how a problem could happen under load (although many of these would also lead to the problem happening with less load, just much less often).
Hardware. Rowhammer-type effects occurring accidentally under sudden load spikes. The hardware has just got too dense.
(I should clarify this is speculation, but reading the recent article included here on sudo using special maximum-distance bitfields to hold state internally (https://news.ycombinator.com/item?id=39165342)... it must be a problem that's being observed in the wild)
It just seems like the most obvious root cause to me, a single bit-flip in a hashed value is going to give you the wrong result data without any other error because the hash value is already essentially heavily compressed, meanwhile the hash table is almost certain to be 100% stored in memory and very heavily accessed from multiple directions in a read/write manner.
No, I'm not questioning whether or not it was a caching issue. I'm taking exception to the lack of accountability. They chose the library. They (probably) chose to ignore a documented or common failure mode of caching systems through either poor choice of key or lack of synchronization. They've obviously designed their infrastructure in a way that isn't resilient to its current level of usage (cold start is a normal part of software's lifecycle).
They could have chosen to own that, but instead they blamed everyone else. That's not a sign of a trustworthy service provider.
"As a result of increased demand, it mixed up device ID" - no, it mixed up IDs as a result of some sort of a concurrency bug. I don't understand the point of deflecting this far.
I'll bet money that their statement was run through legal and stripped of all possible blamey statements.
> They could have chosen to own that, but instead they blamed everyone else. That's not a sign of a trustworthy service provider.
I agree. Companies need to own up to their fuckups, even with legal tells them that it can hurt. Because all companies will fuck up; how they handle it is the differentiator.
Caching is normally read heavy, not write heavy, so it's plausible it wouldn't be something you'd see much under typical operation. After an outage, they'd be dealing with a thundering herd level of traffic as everything tries to reconnect, that'd be very different from normal write loads, even different than the write load they'd have seen when they first enabled caching.
It sounds like they're trying to shift blame for the incident but then they try to pat themselves on the back for all the effort they put into security. It comes across as dishonest.
Technical details are appreciated but they should've emphasized that this is their own fault. Bonus points if they commit to at least consider E2EE which would sidestep the issue.
Very little ownership on Wyze's side.
-- Phil Karlton
The collision is not in the insertion into the hash map but rather in the look up.