Apple gave Uber access to a secret feature that can record iPhone screens (2017) | Hacker News Reader

Apple gave Uber access to a secret feature that can record iPhone screens (2017)

wccftech.com

91 pointsminikites23 comments

Discussed at the time:

Apple iPhone Security Issues - https://news.ycombinator.com/item?id=15418558 - Oct 2017 (1 comment)

Apple gave Uber's app 'unprecedented' access to a secret backdoor - https://news.ycombinator.com/item?id=15417137 - Oct 2017 (3 comments)

Uber app can secretly record your iPhone screen thanks to special ‘entitlement’ - https://news.ycombinator.com/item?id=15414884 - Oct 2017 (39 comments)

Apple gave Uber app 'unprecedented' access to backdoor that can record screens - https://news.ycombinator.com/item?id=15413036 - Oct 2017 (4 comments)

Uber’s iOS App Had Secret Permissions That Allowed It to Copy Your Phone Screen - https://news.ycombinator.com/item?id=15411533 - Oct 2017 (21 comments)

Seems mostly reasonable to me.

“It was used for an old version of the Apple Watch app, specifically to run the heavy lifting of rendering maps on your phone & then send the rendering to the Watch app. This dependency was removed with previous improvements to Apple’s OS & our app. Therefore, we’re removing this API from our iOS codebase."

The explanation is good, but that's a heck of a permission to allow. I hope Apple wisely took special precautions to make sure Uber wasn't using other functions that permission allows.

I don't see how this is reasonable. I can grant needing to do the rendering on the phone, and pushing to the watch, due to resources. But why does it need access to the screen? Is the phone not capable of rendering to a buffer in memory? (Or, because that just doesn't sound plausible & given the name of the permission in the article, can iOS not separately permission access to the screen & rendering to a buffer?)

I believe the API gave you control over the framebuffer (r/w), so theoretically it would have been possible for Uber to read and store buffer data to record your screen.

I don't know how custom the GPU on the Axx series SoC is, but it might be hooked to the screen in a very non-standard way.

I don't know, but still sounds a bit fishy to me. Rendering to a buffer and then turning that buffer into bytes is a standard feature of any graphical OS - and iOS can't do that without hogging the entire framebuffer and granting security-critical permissions?

Also, how was this supposed to work if the user wanted to switch to another app while keeping the map open on their watch? Doesn't this mean the phone would have force-mirrored the waych the entire time? This feels like a pretty unpolished and impractical UX.

Finally, if the entire point of the permission was to draw on the screen, why would it require access to the buffer while in the background? Or was the permission simply not fine-grained enough?

The API sounds really bad. But it could be limitations of the GPU underneath.

I guess the argument then is "technical limitations excuse privacy lapses."

Further to this, Apple auditing that Uber is not abusing this permission is pretty easy: they could reasonably have Apple engineers embedded in the team at Uber – something that would make sense for the scale, Uber is a US company, based in CA, so dealing with it in court would be possible and a fairly well-defined process, Apple could even have engineers trace the application properly during App Store review to check what it was doing at runtime, or put very stringent requirements on _how_ the entitlement was used at the code level to make it easier for them to enforce usage.

None of that scales to the millions of developers they have on their platform, but for such a key application, for essentially marketing purposes for the Apple Watch product launch, sure.

> Further to this, Apple auditing that Uber is not abusing this permission is pretty easy

Are we talking about the same company using Greyball to fool local authorities not just abroad, but also in the USA? Taking into account total lack of moral integrity in their top management, I would not believe in anything they promise.

> I would not believe in anything they promise.

This is exactly my point – there's very little trust needed because Apple can verify/audit one company far more than they can the millions of registered developers.

I'm not saying I agree with it, I'm saying I can understand why they did it, why it works, and that it's likely possible to do safely.

There are many mobile analytics applications such as smartlook.com, hotjar.com that provide screen recording feature.

exactly what i was thinking. Why is everyone acting like its a priveleged access. Any app dev can integrate smartlook and watch how you use the app

although I get that they gave access to recording the screen, literally every app can integrate https://www.smartlook.com/ and record user sessions while using the app .

Are they saying that they needed the API to have the iPhone app drive the watch app? By rendering to the screen, screenshotting it, and sending that to the watch? Sounds like bullshit.

What does the watch have to do with the phone?

It says specifically for maps rendering. Keep in mind the original Apple Watch had a 520 mhz single core processor and 256 MB of RAM shared with the OS and anything else running in the background. And pre-SwiftUI, 3rd party developers were working with WatchKit which was a lot more limited than what Apple was using internally.

Could that render a single map frame? Probably. But if this was an animated map showing the car location with zoom/pan to follow along, I can definitely see it making sense to offload to the phone.

What surprises me is that they needed a private entitlement for this, since you should be able to render a view out to an image without any entitlements [1]. Maybe that solution doesn't work offscreen? Or if the map was animated, maybe this can't play smoothly enough?

[1] https://stackoverflow.com/questions/30696307/how-to-convert-...

More info on WatchKit, per [1]

> Can I use custom views in the Apple Watch? Can I customize the interface elements beyond their public API?

> No. You can’t use custom views. WatchKit only supports certain native interface elements. None of the interface elements can be subclassed or customized beyond their public API. The available interface elements are WKInterfaceLabel, WKInterfaceButton, WKInterfaceImage, WKInterfaceGroup, WKInterfaceSeparator, WKInterfaceTable, WKInterfaceSwitch, WKInterfaceMap, WKInterfaceSlider and WKInterfaceTimer.

In the original watchOS release, you can't even draw shapes, only display images. So to make a custom map you'd have to render the image yourself for display as a WKInterfaceImage.

With watchOS 2 you were able to draw to a WKInterfaceImage with Core Graphics [2], but Uber had written their app before this.

[1] https://www.raywenderlich.com/1950-watchkit-faq

[2] https://github.com/shu223/watchOS-2-Sampler#draw-paths-updat...

Yeah that's the part that seems off. It implies that the image for the watch was on the iphone screen first, which doesn't seem quite "professional quality". Did they render something high quality (to look good on three options) with the GPU and then subsample the image for the watch?

Not long ago the Apple Watch was very limited in what it could render on-device. So a bunch of apps had to do it this way.

But only Uber (allegedly) had access to the private entitlement. So how did other apps do it, and why couldn't Uber?

That said, this is certainly a "special relationship" with big partners like Uber to get their apps working at Apple Watch launch in 2015[1].

That the access "wasn't removed already if it was no longer required", well... I think we can all understand legacy code.

[1] "It is likely that the Cupertino tech giant had to give this access after it gave app developers a four-month window to develop apps for its Watch before the unveiling of the product."

I’m not sure why this access required a “special relationship”. I remember years ago reading public Apple dev docs that recommended rendering certain things on iPhone and sending them over to the watch as a rasterized image.

If I remember correctly, the "secret API call" was to the framebuffer. They were rendering the maps using the iPhones cpu/gpu, then pushing the rasters to the watch. The watch was not yet able to render the maps in realtime. Uber had to be able to create screens even in the background, so the API could have taken a snapshot of the active part of the screen buffer.

Reply on news.ycombinator.com