Wyze security incident update
forums.wyze.com
forums.wyze.com
Kudos to Wyze for doing the things noted in the thread like being honest and prompt with notification etc, but "thumbnails from your Events were visible... and that thumbnail was tapped" is a pretty mealy-mouthed way to say "another person saw your private pictures and videos taken with your Wyze cameras"
I actually appreciate the more specific details on how the private pictures and videos were actually viewed using terminology from the Wyze app.
On the Wyze subreddit, people have been griping about the fact that it took them days to even acknowledge something happened. When I read the Wyze email about it, there was no new info in it, it had pretty much all already been discussed online.
https://support.wyze.com/hc/en-us/articles/360026245231-Wyze...
A good response to this might be to put it back, and to extend other devices to be dual-use (Wyze Cloud or HA).
I've use the official RTSP firmware in the past on some v2 cams, but I remember it having some problems and not being as good as this solution.
I continue to be amazed that there is not a reasonably priced, open source, audited, local-first solution, which doesn’t require a significant personal investment of time to install and maintain.
Local-first cameras are super cheap and easy to find. Most just run their own local RTSP server, which you can connect to live with VLC, homeassistant, whatever. OpenIPC is like ddwrt for ip cameras, here is the supported devices page: https://github.com/OpenIPC/wiki/blob/master/en/guide-support...
But unless you're Richard Stallman, go with the closed source Reolink RTSP camera, which is about $100 and used by big corporate installs. It can integrate with the cloud, but you can set it up to just have each camera run a RTSP server with user/pass auth. You have to secure your own network.
But there isn't a really great open source platform for the kind of multi cam security that businesses might need. You have to do your own storage. But grab four reolinks, send the feeds to homeassistant, and most homeowners will be fine.
I have spent days searching for an affordable solution that meets my requirements. I know there are a lot of people that want the same thing, have done their own research, and have reached a similar conclusion. What we want simply does not exist.
I will stop posting this kind of comment when someone finally gives me a straight answer, but no one has done that yet.
Edit: Another comment mentioned https://gitlab.com/Shinobi-Systems/Shinobi which I haven’t explored but am definitely going to also try out now.
I don't have time to mess around with hacking a camera to do RTSP and then figure out how to set up and use something unfriendly like ZoneMinder.
The margins on these devices are already pretty high because they're enterprise and business focused - I "buy once cry once"'d when I bought my <vendor>* drive for the home. I would have bought a different vendor if I was forced to pay an additional premium for a render capable CPU.
(* - I'm probably tin foil hating here, but realized from a security posture perspective I don't want to publicly state the vendors I use in my network.)
The V3 models need to be downgraded to a specific firmware first and patching it exposes RSTP streams using https://github.com/AlexxIT/go2rtc. Everything doable without ever installing Wyze app on an environment air gapped environment with no internet.
1. all cameras (firmware v4.36.9.139) have 64gb+ micro SD cards and record to local storage -- many people seem to have issues with anything greater than 32gb in v3's but I've found that this Verbatim tool [0] formats FAT32 at high capacity with no problems
2. all cameras have wz_mini_hacks [1] on the SD card with RTSP enabled
3. all cameras are connected via ethernet instead of wifi using this adapter [2] and wz_mini_hacks config
4. network blocks all outgoing internet connections for all cameras to keep them LAN-only -- this means I have to connect to VPN to review video when outside the house, but I'm cool with that
5. all RTSP streams are also recorded over the network via Agent DVR [3] to a NAS
6. the Wyze app (free tier, not paid) works normally with all of the above in place -- I find it much more intuitive to review recent videos in-app (streamed off the SD card), and then review the very occasional older video from a computer off the NAS (scrubbing through in VLC on a computer)
For what it's worth, I don't use them like a Ring camera where you're responding to realtime video events / talking through the camera to a delivery person -- this is mostly just for 24/7 recording. I have all object/motion detection events turned off, just a straight uninterrupted feed recording local and on the network.
Links:
0. https://www.verbatim.com/index/search.php?words=fat32+tool
1. https://github.com/gtxaspec/wz_mini_hacks
If you don't and IP addresses cycle, your RTSP stream URLs will change and your recording software will just consider the camera disconnected. The Wyze app sometimes will get confused too (with ethernet in particular, if IP addresses change), and rebooting the cameras sometimes means physically cycling power if you don't know where it's at in your network. Better to just reserve a specific IP up front, write it down, and never think about it again.
I also use a lot of these types of crappy smart plugs [0] throughout the house (as well as starting to replace some with actual wifi outlets [1] for aesthetics) and it's actually really useful to be able to force a power cycle on certain things from my phone.
Links:
0. https://www.amazon.com/gp/product/B09LXGHR5X
1. https://www.amazon.com/KP200-Outlet-Required-Control-Certifi...
Do you have any Wyze cams outdoors, but under cover of a porch? Or are they indoor only?
That said, all the modifications and tinkering is definitely less turn-key than some people prefer, but I’m a glutton for punishment haha
As time and motivation permit, I've been converting the cameras I care about over to POE. But having to run cable across the house for each one means I haven't done them all.
Even a bad implementation with cloud-synced encryption keys (which defeats most of the benefits of e2e) would have stopped this.
The response in this case (notifying customers and specifically stating whether they were affected or not) is excellent, but this seems to be a repeat of a previous incident from September 2023: https://www.theverge.com/2023/9/8/23865255/wyze-security-cam...
https://www.consumerreports.org/home-garden/home-security-ca...
Responses typically range from "I'm not that interesting" to "I really don't care". I think it's too abstract of a threat for most people to take seriously before it happens to them.
I totally agree with you, but then I put my phone on a qi charger on my nightstand and go to sleep. It's a device with both quality cameras and microphones, so I feel a little hypocritical given that there is a non-zero chance that someone could be listening or watching through my phone.
With IoT there often aren't any security patches and your audio & video are just being live streamed to the OEM's cloud waiting for someone to listen in, it doesn't even require a security exploit.
It's easily abused by employees, it even happened at Tesla where they watched their customers through the onboard cameras, taking screenshots of them walking around naked, and sharing them on company Slack channel for laughs.
That's why I find it so mind boggling, the company could incidentally hire a pervert and now you find yourself being watched in your own home by someone who knows your home address. I find this scary because it doesn't require a security exploit, just a deranged mind and those are dime a dozen.
I agree with everything you're saying, but you may be overstating security patches. Until recently, most Android phones only had a few years of security updates.
I guess what I'm getting at is that if I truly believed in keeping Internet connected cameras outside of private areas I wouldn't have a smart phone at all.
The problem with Teslas wasn't the firmware on the cameras, but rather the infrastructure behind it. Ideally the data would be encrypted on servers and decrypted locally when needed. This doesn't pair nicely with services that perform analytics on video streams, of course, but it's a better option for privacy.
At the end of the day I share your concerns, and I want only devices which are controlled locally. I have been making efforts to make this a reality.
I'm just making a distinction between "connected to the internet" and "streaming private data to the cloud 24/7".
Most of us use a smartphone under the assumption that nobody else has access to it, and that it's not going to send all of our data to some cloud. If someone gains that kind of access to my device, I'll have bigger problems to worry about than someone listening to my conversations, like locking down bank accounts, investment accounts and changing dozens of passwords.
> Until recently, most Android phones only had a few years of security updates.
Tell me about it, I begrudgingly buy a new device when the old one runs out of security updates. I'm not a fan of Samsung or Pixel line (which now offer longer support) so I was planning to switch to an iPhone after my current Android device is made obsolete, but I changed my mind with Apple's latest EU meltdown.
I’d go with “don’t put internet-connected cameras in your house if you don’t want those images on the internet”. I’ve got a Wyze in my garage looking over my mountain bikes, and for $35 I don’t really care if somebody else sees that image. But I’d never put one in my living space, regardless of their security track record.
Unless the camera has lasers, it isn't very good at stopping said theft. ;)
Often you use a police report. My local police department is on an unofficial slowdown strike because they don't feel appreciated post-BLM. Possibly also because if they "forget" to file police reports or they "get lost in the system" then the official crime rate goes down. It took me over a month to get a police report I could send to insurance for a simple break in, and spent more of my labor by hourly wage trying to get that report than the cost insurance reimbursed.
The only thing that will is to stop being trigger happy menaces to public safety.
I wish I could say it was still worth it because filing the report and providing the information about the theft gave the police additional info to use if they ever caught the thief, but nope. They didn't even want a copy of the video from the security cameras at the store in whose parking lot it happened. They made it acutely obvious that I was wasting their time by being there and that they had no interest whatsoever in doing what I naïvely thought was their job.
What an absolute waste of time.
How could it cost more? Are you counting the time spent compiling the documentation?
Generally filing a police report will suffice, if I’m not mistaken. Could you lie? Of course, but (1) it’s generally not in your best interest because the insurer might raise your rates and/or discontinue coverage, and (2) you’d be making a decision to make false statements to the police for purpose of fraud, which most reasonable people won’t do.
Other than perhaps using cameras as a means to deter thieves, I’m not sure that low-value (under USD 5000) items like bicycles are worth the time and effort for insurers to launch full investigations over.
I would agree, yet I have seen it happen. It is a somewhat difficult to predict path. I have seen some smaller claims (~$2000) take significantly longer and with more investigation than other very large (>$50,000) claims. I would assume there are certain metrics and algorithmic methods that effect how this plays out, so it is possible that someone who has had multiple small claims my get more attention than a single large claim.
Either way - The concept of the camera is simple in providing evidence in cases where that evidence can help you.
You've seen insurance claims denied because someone didn't have an unverified video that allegedly showed a theft taking place, even with a police report in hand? (Seriously: How would an insurer know that you hadn't arranged to have your brother-in-law pretend to steal your bike for the camera?)
and I said:
"I would agree, yet I have seen it happen."
I was precise in answering your precise statement.
I have seen claims for less than $5000 items create significant investigative and delay-inducing efforts from an insurer. A video of the theft would have, in some of those cases, reduced that effort. Having video evidence is in almost no circumstances going to increase the time it will take to get an insurance claim paid. By you having the video evidence, you have the choice if you want to disclose it. It is just upside that you control.
If you do not want to put a camera in your garage pointing at your mountain bike, by all means do not.
However it is always good to recognize the primary goals of the insurance parties - which is to not pay. They have strong incentives in that direction, and you providing proof of theft reduces the opportunities for those incentives to slow or reduce the payment.
It is especially true in cases of undeclared specific items, like an expensive mountain bike that you do not have a dedicated policy on.
We quickly realized that cache is fast but not infallible. Use proper security on all your resources. Don’t rely on UUIDs to obfuscate your data as security.
That seems like enough of a line of bullshit to steer me away from ever using wyze.
Given their openness in the rest of the communications, I don't see why they would make this part up.
Edit: Of course, I'm also curious what the actual bug was. A discussion below is suggesting several plausible ways (e.g. concurrency issues, insufficient entropy in some key) how a problem could happen under load (although many of these would also lead to the problem happening with less load, just much less often).
No, I'm not questioning whether or not it was a caching issue. I'm taking exception to the lack of accountability. They chose the library. They (probably) chose to ignore a documented or common failure mode of caching systems through either poor choice of key or lack of synchronization. They've obviously designed their infrastructure in a way that isn't resilient to its current level of usage (cold start is a normal part of software's lifecycle).
They could have chosen to own that, but instead they blamed everyone else. That's not a sign of a trustworthy service provider.
"As a result of increased demand, it mixed up device ID" - no, it mixed up IDs as a result of some sort of a concurrency bug. I don't understand the point of deflecting this far.
I'll bet money that their statement was run through legal and stripped of all possible blamey statements.
> They could have chosen to own that, but instead they blamed everyone else. That's not a sign of a trustworthy service provider.
I agree. Companies need to own up to their fuckups, even with legal tells them that it can hurt. Because all companies will fuck up; how they handle it is the differentiator.
Hardware. Rowhammer-type effects occurring accidentally under sudden load spikes. The hardware has just got too dense.
(I should clarify this is speculation, but reading the recent article included here on sudo using special maximum-distance bitfields to hold state internally (https://news.ycombinator.com/item?id=39165342)... it must be a problem that's being observed in the wild)
It just seems like the most obvious root cause to me, a single bit-flip in a hashed value is going to give you the wrong result data without any other error because the hash value is already essentially heavily compressed, meanwhile the hash table is almost certain to be 100% stored in memory and very heavily accessed from multiple directions in a read/write manner.
Caching is normally read heavy, not write heavy, so it's plausible it wouldn't be something you'd see much under typical operation. After an outage, they'd be dealing with a thundering herd level of traffic as everything tries to reconnect, that'd be very different from normal write loads, even different than the write load they'd have seen when they first enabled caching.
It sounds like they're trying to shift blame for the incident but then they try to pat themselves on the back for all the effort they put into security. It comes across as dishonest.
Technical details are appreciated but they should've emphasized that this is their own fault. Bonus points if they commit to at least consider E2EE which would sidestep the issue.
Very little ownership on Wyze's side.
-- Phil Karlton
The collision is not in the insertion into the hash map but rather in the look up.
> The incident was caused by a third-party caching client library that was recently integrated into our system. This client library received unprecedented load conditions caused by devices coming back online all at once. As a result of increased demand, it mixed up device ID and user ID mapping and connected some data to incorrect accounts.
What? How does load on the system affect correctness?
> The outage originated from our partner AWS
What does this mean? Was there an AWS outage for a service they use, or was this just a normal loss of an instance?
It's interesting that they blame external entities for the root causes of the incident and don't take responsibility for what is ultimately on them.
Seen this happen quite often with code that is not multi-thread safe, especially in languages like c# and java, such as using a static class property for data that should be request-scoped, or not using the appropriate concurrent collection classes etc.
Instead of blaming the users, we must hold the companies responsible. Data privacy laws must be stricter and these incidents must be taken more seriously.
You might be able to cause consequences after the fact, but your data will still get leaked first. You can't undo a privacy violation with tort law, and there won't ever be criminal penalties.
If you want your information to stay private, don't store it on other people's computers. IDGAF what "the default today" is. (Also, that's wrong - everyone that is serious and actually wants their data to remain private doesn't store it in the cloud. This is why the CIA got Amazon to build a custom airgapped on-prem AWS region at Langley, for instance.)
> Also, that's wrong - everyone that is serious and actually wants their data to remain private doesn't store it in the cloud. This is why the CIA got Amazon to build a custom airgapped on-prem AWS region at Langley, for instance.)
Do you think that “everyone that is serious and actually wants their data to remain private” is the default? In a random sampling of 100 people, how many do you think fall into this category?
> This is why the CIA got Amazon to build a custom airgapped on-prem AWS region at Langley, for instance.
Is your threat model the same as Langley’s? Or might there just be different levels of what people’s needs are?
There’s ideals and there’s practicality. It’s impractical in today’s world to completely avoid cloud services. If you can do it, congratulations, more power to you.
I had a heck of a time finding a proper POE recording DVR camera system for my mom's house without online or cloud bullshit, but still I isolated it on the network to not take any chances of UPnP port opening or dial-home crap.
The only system I would trust would be one that laid out their security model, source to their apps, and had a self-hosted server DVR option. The captological signals of 99.9% of security system websites do not instill confidence in my mind.
The cameras are a wide variety of IP POE cameras which is helpful as I am not stuck to a single brand.
First of all, google "Wyze RTSP firmware". It's the official firmware from the vendor that enables the RTSP protocol. Now you can enable RTSP via the app and give the camera a fixed IP address in your DHCP server.
RTSP is a pretty standard protocol, so you can now view the feed via VNC player, record it 24/7 via ffmpeg, use tools like motion, etc.
The camera will still try to connect to cloud, but you can move it to a local-only Wi-Fi network, or outright block it from reaching the outer world on the router side.
And if you want advanced stuff (multiple streams, organized recording, etc), there is a plethora of free/open-source security camera tools (iSpy for instance). It all takes time to learn and configure, but you can have your own fully closed-circuit surveillance network, while still using the Wyze's rather cheap hardware.
and don't overlook that user's other repos, as seems like there are quite a few fun things in there: https://github.com/koush?tab=repositories
As an software engineer who's dealt with caches for large high throughput services, this does not make sense to me why they are blaming a caching client. It's your own code that will decide what is the cache key, and what value to pass as the cache key. Did the caching library have a bug where when you ask for a given key, it returned results for a different key? Or more likely did your own code have a bug where you mixed up the keys? I think we need more details on what went wrong in here.
Yes, of course. Blame a third party library which was probably created by an open source maintainer instead of testing your own systems.
but even with this, it's still an security camera app that can't send push notification without cloud access.
Waze could come out with a line of dash cams, and then you'd have Waze cams and Wyze cams.
In the era of cloud and microservices why each user does not have their own dedicated resources?
Because race to the bottom.
People want incredibly cheap products that are internet connected. Your average home user should not have to worry (and won't) about cybersecurity concerns, so this will continue to happen. The only out I can foresee is government regulation stepping in to make these incidents actually hurt for the companies, but America has basically no appetite for that.