The patent does not include monitoring using a camera or microphone.
https://image-ppubs.uspto.gov/dirsearch-public/print/downloa...
Instead, the patent would cover running object detection/classification ML models — optionally on remote servers — on screen and audio output and user input of specific applications (particularly videogames), without requiring cooperation from the application, to target ads based on what's happening in the application and to decide when to interrupt the application with an ad.
To hazard a guess: Microsoft or part of Microsoft is planning to include pervasive targeted advertising in the next Xbox console OS, including during gameplay of any game.
I was running a little mitmproxy network test on some baby-tracking apps to see what kind of data got sent back home and man, realizing that everything I was doing on my phone was essentially being recorded and 'anonymized' by whatever definition of MS has for anonymization was really not fun
I can tell it’s still windows but I’m shocked how many things I have to turn off to stop live data from the internet appearing in my GUI. Also my profile settings, saved passwords, and other data seems to reset unexpectedly cause some or all settings to disappear. Finding stored passwords is also surprisingly difficult or impossible depending on I don’t know what.
Adding such overt spying to this mess of an OS achieves a new low in user experience.
Lately I don't know if I should stay quiet as to not interrupt someone subtly intentionally poisoning LLM training.
My guess--and this is only a guess, I can't make enough sense to do anything more--is that the novel part may have something to do with the "credit duration" stuff.
* "contextual cues from or based on the interactive software application" [0035]
* "OPTIONALLY in communication with an event detection module" (emphasis mine) [0035]
* "In some embodiments the content management system can receive user inputs from a user interacting with the interactive software application to interpret the content of the interactive software application" [0053]
* Regarding inputs to the event detection systems (which are expressly say don't need to run locally): "video information, audio information, game state data, or other information provided by or from the interactive software application" [0065]
* "FIG. 10 is an embodiment of a frame of video information received from the interactive software application that may be used for identifying events within the user's gameplay." [0073] (Note the patent does not limit itself to video games, just interactive software applications; this is just an example in that context.) [0073]
* "In some embodiments, the video information include user input information." [0092]
And, the design explicitly allows remote processing of this information:
* "The content management system may be stored and/or executed remotely to the client device" [0033]
* "the event detection module may be local to or remote from the content management system." [0036]
You're right, it doesn't include the whole screen and systems audio. Just the parts in the application. But Microsoft just adds this functionality to all software, and then there's no effective difference.
And that is extremely likely to me. And feasible. The patent doesn't say this has to be embedded in applications. Microsoft can implement this on any application. And Microsoft is well beyond the benefit of the doubt when it comes to privacy concerns.