Xbox co-pilot mode changed disabled sister’s life | Hacker News Reader

Xbox co-pilot mode changed disabled sister’s life

reddit.com

277 pointsmansarip56 comments

I love co-pilot. Throughout the pandemic my fiancé and I used the feature to play different sections of single player games as though they were multiplayer. I don’t think either one of us would consider ourselves to have a disability.

I used to work on games at Microsoft and they put their money where their mouth is when it comes to accessibility. I had spent multiple days in the accessibility lab interviewing and understanding how to make games easier for people with various disabilities. One of my managers encouraged me to test my features using the controller with one hand and farther away from the screen where bad font decisions were more obvious.

Two things I remember in particular from that time at Microsoft:

* The claim that making games better for people with disabilities makes them better for everyone

* The claim that most people will have a disability (not necessarily permanent) at some point in their life

I feel that those two lessons have permanently impacted my design lens.

Well then, will you not share what kind of game you've worked on? That's too good of a post to leave us hanging, hehe.

> The claim that most people will have a disability (not necessarily permanent) at some point in their life

I think it was Microsoft that came up with a great infographic illustrating that disabilities aren't a black/white thing. It broke things down by permanent vs temporary vs situational.

Something like:

- Touch / tactile - arm/hand amputation (permanent), arm in a sling (temporary), new parent cradling a baby (situational)

- Vision - blindness (permanent), cataracts (temporary), distracted driver (situational)

- Hearing - deafness (permanent), ear infection (temporary), bartender in a noisy bar (situational)

- Speaking - non-verbal (permanent), laryngitis (temporary), interacting with a heavy accent (situational)

When framed like this, it seems much easier to make the case for accessible products, versus as an afterthought.

I really liked that infographic when I first saw it on twitter a while ago. I found it again at https://accessibility.asu.edu/wa-helps-everyone-large-versio..., which specifies Microsoft's Inclusive Toolkit Manual as the source.

Also with touch: Arthritis, Carpal Tunnel and other RSI, tremors, using touch screen in bed or other awkward situations

Vision: Color blindness

And the list goes on.

> most people will have a disability (not necessarily permanent) at some point in their life

I've got an eye infection when I was a teenager. It has strongly tinted my vision white, so for a couple of days I could only see outlines of big objects highlighted by skylight. From this experience I found I could still walk my usual routes without wandering off the road, as I could recognize familiar telephone poles, trees, I even remembered all the ruts in the road.

Also, the Xbox Adaptive Controller deserves a lot of respect.

https://www.xbox.com/en-US/accessories/controllers/xbox-adap...

That’s not a token effort or a cash-in. The team behind it really gave a damn.

Yes, it's absolutely wonderful. I hope to never need it, but I'm so happy it exists for everyone that does.

It also shows why controller compatibility between consoles might be more important than one would think: the "native" controllers might just be inaccessible. If you want to play on a Switch, you can use an adapter [0] to use the Adaptive Controller. Similar products exist for the PS4, I believe. Hopefully someone reverses the protocol DualSense speaks with PS5 sooner rather than later and can make an adapter for that as well.

[0] https://www.mayflash.com/Products/NINTENDOWiiU/MAGIC-NS.html

IIRC Sony puts DRM in their controllers, there's a chip with a key on it that Sony only gives out to selected partners. I'm not sure how 3rd party companies did it without the chip but afaik the method is not open source.

More interesting is that whilst the original PS3 controllers had a full Bluetooth stack (meaning you can connect them to an Android device or Linux and they work perfectly fine), the bootleg clones only implemented the spec enough to get them working with the PS3. I didn't even know this until I tried to connect my old PS3 controller to my Pi running retro emulators, and all the controller would do is input to bottom right on the right analog stick. Same on Android.

PS4 Controllers can be connected via BT to phones and via cable to PC with almost all functionality. There's an extension port on the controller but I haven't seen anything that plugs into it. Also don't know how easy it is to connect your own controller to the PS4 but there is an "ext" port on the controller that is reportedly a USB2 port but a proprietary shape.

You absolutely can't just use a USB compatible controllers on a PS4.

This is true, unfortunately. A common way around that is that you must connect both the controller you want to use and an original PS4 controller to the adapter. The adapter will forward all DRM/handshake messages to the original controller, but forward the input from the desired one.

Unless Sony is now signing every message I see no reason why this approach couldn't be used on the PS5/DualSense. That said, my cursory search didn't reveal a working PoC.

You probably aren't remembering correctly: the protocol for DualShock 4 (PS4 controller) was reversed - there's a working userspace driver[1] that I have used for years to play games on Linux using a DS4 controller, connected via Bluetooth. More radically, Sony recently published an official Linux driver[2] for DualSense 5 - none of these require a "DRM chip" or additional hardware to use.

1. https://github.com/chrippa/ds4drv

2. https://www.phoronix.com/scan.php?page=news_item&px=Sony-HID...

There’s a V1.32 note in red on that page that says it’s now compatible with the PS5, but I’m not sure what that means exactly. It’s hard to say from the product page what this thing does.

I believe it means that you can now use the adapter to connect a PS5 controller to the Switch.

PSA: You can use the MAGIC-NS to connect multiple controllers at the same time while they appear to the Switch as one controller. This basically is co-pilot mode.

> Hopefully someone reverses the protocol DualSense speaks with PS5 sooner rather than later and can make an adapter for that as well.

Sony recently[1] mainlined its DualSense 5 driver to the Linux kernel, so reversing the protocol shouldn't be difficult. There previously were a couple of open-source 3rd party drivers for older DualShock controllers - I use a DS 4 controller for all Steam on Linux games and it works very well: most games recognize it as a PlayStation controller, display the correct button symbols on help screens and QTEs

1. Dec 2020 - https://www.phoronix.com/scan.php?page=news_item&px=Sony-HID...

Unfortunately this isn't the complete story. Being able to use a DS4/DualSense with a PC or other device is a different beast from being able to emulate a DS4/DualSense for a PS4/5.

The console actually authenticates the controller using some secret [0]. I find it very doubtful that Sony has implemented this authentication in their Linux driver and released the necessary secrets.

Unless this procedure is a part of a standard that I'm unfamiliar with that it may still need to be reversed for PS5/DualSense. We also need the secrets or use the workaround with a legitimate controller I referenced in a different comment [1, 2].

[0] https://fail0verflow.com/blog/2018/ps4-ds4/

[1] https://news.ycombinator.com/item?id=27378663

[2] https://www.cronusmax.com/manual/wired_controllers_connectin...

Wow thanks for sharing this, I had NO idea this existed!

It was a big source of internal pride when I was at Microsoft. As far as tech companies go, Satya’s version of Microsoft really does give a damn about people, including it’s employees. Great place to work.

Microsoft has done a lot of things that they didn't "need" to for the sake of accessibility and I think that's worth celebrating. Particularly when it comes to something as simple as this I think there's no reason why it can't exist on every other console. As mentioned in other comments, their adaptive controller is also great.

This co-pilot mode sort of reminds me of Starcraft: Brood War where there is a cooperative mode where two people can control the same set of buildings and units. The amount that each player contributes to the game can be as variable as you need it to be, as far as one player not doing anything at all. Or if you were both skilled, it could simply augment what you could do. One person could be making sure you're spending resources and building the base while the other focused on controlling the units fighting the enemy, for example.

Overall I think the video games industry is very very behind in accessibility, though. Even some relatively simple accessibility options are absent from some of the biggest games. I believe Ubisoft has made great strides to make their video games more accessible to everyone and there are some notable ones such as The Last of Us 2 [1] and Celeste [2] that go above and beyond to ensure that anyone that wants to play the game can.

Accessibility can be expensive to implement and gaming isn't really considered an essential, even in terms of entertainment, but I think it's worthwhile to try and encourage even smaller developers to try to make their games more accessible.

[1] https://www.playstation.com/en-us/games/the-last-of-us-part-... [2] https://celeste.ink/wiki/Assist_Mode

I am terrible at video games. It's not what my brain does well so I miss out on playing computer games in general, I'm more interested in board games or cards. Not long ago I was waiting for laundry to be done with a friend who had a Nintendo Switch on her, and we started a game of Mario Kart. The entire time we played, maybe 15 mins, I thought wow, I'm better than I think! Then after the game I told my friend and she says, the whole time she was racing behind me to kill the bad guys so I could enjoy playing. What a beautiful gesture!

I don't think another player can actually be that helpful in Mario Kart, but it does have extremely strong steering assistance, which can be enabled/disabled in the pause menu. Not trying to be rude by comparing you to a small child, but the main benefit is that little ones can join in and have fun while they're not quite co-ordinated enough to steer properly.

If you wanted to get more into gaming, a Switch would be a good choice. There are lots of games which are mechanically accessible (still not trying to be rude here) to six-year-olds, while still being a ton of fun.

No offence taken! It's not so much the coordination but that I just don't care about playing. My sense of being entertained is different than what computer game play provides, so I don't put any effort into learning the moves. So in relation to the topic here, it's nice if I can hang out with friends who love games while I don't need to invest effort to join in on occasion, as I could get quite a lot out of copilot mode.

Not quite true :) I’ve played a lot with my 2 youngest daughters and the AI adapts to the fastest player. So if you hang behind the “worst” player, it will become very easy.

Subsequently, if you race full-pull the other person will be in a permanent #12 spot. They will receive a lot of star and bullet power ups though ;-)

Mario Kart: Double Dash was actually a lot of fun with two players in a kart. A friend of mine who couldn't steer to save his life took over the weapons while I focused on driving. The best part is that the second player could make their character throw their weight left or right to shove the kart sideways and save you from an obstacle, or a tight turn taken a little too quickly.

Goose & Maverick. Very cool.

I probably would've played more video games if I had known about these co-op modes.

Most people when they say they are terrible at video games mean they are bad at reflex games. I recommend you don’t discount entire sub genres of non-real-time video games you might enjoy. Depending on your taste that can be anything from tactical/turn-based RPG’s to sports management games.

my youngest is 2.5, a survivor of bacterial meningitis at birth, the results are tragic. she has soft tone cerebral palsy and cortical visual impairment, imagine being stuck in a kaleidoscope for life.

as part of my journey, trying find answer and heal. i stumbled across the fact that Microsoft's current CEO Satya Nadella son Zain is a legally blind quadriplegic with cerebral palsy. his book refers to the first few years of his sons life as "a dark place". Satya is fairly older than me, I admire him very much.

I have no doubt that it bring great joy and comfort to Satya that he's been able to drive Microsoft to be a leader in accessibility. Most great efforts have great stories. This is one of them.

Co-pilot is a really great idea, and also so simple. Definitely one of those "why didn't we think of that before?" moments.

We did - I remember playing Doom with my friend as a kid - one of us would do the walking, another would do the shooting, on different peripherals. The idea is pretty fresh on the consoles, however.

one of us would do the walking, another would do the shooting, on different peripherals.

We used to do that sharing the same keyboard. My friend and I played though most of The Catacomb of Abyss like that. Good times.

That reminds me that when USB came out (or came to our, my parents' computer) I thought if only we got a new mouse, I'd be able to plug both in, getting two cursors, and be able to cheat at solitaire. Heh.

One game I particularly remember is Super Mario Galaxy 3D on the Wii. It had a two player mode where the second player didn't control Mario, but instead had the ability so interact with the environment and player in limited ways to help. I wonder why there aren't a lot of other games that do that.

In Sonic 2 and later for Sega Genesis/Megadrive player 2 can control a second character and also help in limited ways.

Game design is hard and asymmetrical game design is incredibly hard. Game developers can hardly make symmetrical cooperative systems that properly take advantage of the multiplayer aspect. Borderlands, for example, doesn't add unique systems that take advantage of the fact that there are multiple players. There's no unique mechanics to bosses or enemies. There's not really interesting synergies that you can take advantage of by the different classes.

When games do coop well, it's usually an inherent part of the game. If you can make the assumption that there is at least two players playing together, you can create unique and interesting ways for the two players to interact on screen. "It Takes Two" is a recent game that was designed explicitly for two players and you can immediately tell the difference from it to a game that cooperative modes were an afterthought to the solo mode.

Games requiring more than one player immediately alienates a lot of players and limits your audience a lot, so that's probably why pure coop titles are extremely rare.

As for asymmetric titles like Mario Galaxy where the second player is not essential, it's pretty hard to make the gameplay satisfying for the secondary player while not impeding the first player or making it feel essential to the gameplay. The "little brother" mode costs a lot with little perceived benefit for the developers.

> As for asymmetric titles like Mario Galaxy where the second player is not essential, it's pretty hard to make the gameplay satisfying for the secondary player while not impeding the first player or making it feel essential to the gameplay. The "little brother" mode costs a lot with little perceived benefit for the developers.

I'm not sure how frequent this is, but when I was little we (me and my brothers) had limited time to play, so we would play our time and watch each others, sometimes completing long missions in Ground Control by playing one after the other. When you look at it this way, having something to do instead of just looking at the screen (which was already plenty of fun) is huge.

That's a very touching story. Although most of us don't work in the consumer hardware business, many of us work on frontend web design. This post serves to underscore the importance of accessibility in our designs.

My 3 year old is very interested in playing video games, but her hands are too small to hold a standard controller and reach everything. We've been playing Carto a bit on gamepass (no enemies, moving around, no timing constraints).

Just out of curiosity I checked, and co-pilot seems to be available on Windows too (within the Xbox sphere at least). I will try that out, looks like it could have potential for us.

My neighbor has MS and can't physically move ( his head and he can still talk, but not the rest of his body)

Is there anything that i can buy to improve his quality of life with? Even something small could be useful.

If you haven't had this discussion with him already (and you think you can comfortably have this chat), he may already have some ideas that he hasn't thought to mention.

Otherwise, one suggestion would be controllers for quadriplegics. There are a few options out there, afaik most are controlled with your mouth but there are also gaze-tracking options. I believe the XBox accessibility controller is compatible with some of them.

There's one linked right on the XBox adaptive controller page https://www.xbox.com/en-US/accessories/controllers/xbox-adap..., though boy, that's an expensive piece of hardware.

I could have that talk with him. But i don't want to start it with no options or no suggestions.

Him and his wife don't have it easy ( they used to go walking in the mountains a lot, so the declining mobility in recent months are ... Depressing for them, although they don't really complain. I just notice it in the conversations when it comes up)

Can't really make any hardware recommendation, but as this post is about games, I would suggest trying out more puzzle like games like "The Witness" or "Outer Wilds". Despite being single player, I played both of them in tandem with my wife and I feel like it was a much better experience than playing alone as there was always something one of us would have missed or would get stuck on.

Well, it's game hardware for "disabled", so this is one of the best threads similar to my question. Where people could have the same interest or know about something already.

If he can comfortably move his head, might I suggest something VR related? Most of the first VR games that came out for mobile (i.e. using your phone as a VR screen) couldn't utilise a controller so they used head movement for controlling things. Usually you had e small crosshair in the middle with which you can select things by simply holding a position. I sadly can't remember the names of the games I tried but you might find them (or newer ones for newer headsets) through google. I know it's not a complete substitute but if they miss mountain walks, there are VR experiences (often videos) which simulate that. Might be interesting to try. I'm not aware of it but there might even be multiplayer ones, so that his wife can join him.

I wonder if they could make an AI co-pilot for those who don't have someone to company them as they play.

You're already overthinking it with AI and stuff; there's games (especially on mobile) with minimal input requirements, think flappy bird, runner games, some of the racing games, etc. Even Epic's first major mobile title, Infinity Blade, played more like a slow interactive story than something needing lots of control input.

I mean I won't claim they're fun, just that they're easy to play.

Your argument would also refute the whole co-pilot experience itself. It is not about "there are simple games, go play that". It is more to enable them to play anything.

I think anything that can be co-piloted so smooth that other player is not annoyed, can be also simulated by an AI.

It is not really overthinking it. It is taking the learning and bringing it to a wider audience.

Platinum Games and some other studios are now in the habit of putting "easy automatic" (not always using that term) modes in, where you can set the game to automate some or all of its inputs. If you turn all the assists on in Toylogic's "Nier Replicant" for example, the game basically plays itself during combat but you still have full control so you are effectively orchestrating the action on screen instead of mashing the buttons. It's nice for people revisiting the game (it's a remaster with updated content) who don't want to do all the combat again. For some titles, this is a great accessibility feature since you can turn it on only for particular mechanics that are hard for you to keep up with (dodging, aiming, etc) and still play "the rest" of the game.

I think their first game with this feature was Bayonetta (https://www.youtube.com/watch?v=_6nkc6GQ-B8) but at this point a bunch of other games have it. I think every action game should have this where possible, and you can find testimonials out there from people who love these features despite not having major accessibility concerns, like https://www.vice.com/en/article/xgx8nz/auto-battle-is-the-be...

Increasingly I think we're discovering that there's a class of temporary accessibility issues that can get in the way of using a website or video game - you have trouble focusing because your kids are distracting you, you can't remember what your current objective is because the last time you played was a week ago, etc. It's great for games to provide assistance in those cases too.

Won’t that be cheating ? (That’s how aimbot work on warzone and many fps on PC )

It is as much of a cheating as the co-pilot thing, yes. I guess it shouldn't be applied to multiplayer competitive games.

Only from the perspective of the 'inferior' player - 'co-pilot' seems fine for competitive multiplayer, since as far as other players are concerned you could have been performing better, if you were only the better of the two of you, 'unimpeded' by the one being assisted.

Many games have "assist modes" of some kind. Sometimes as part of a difficulty setting, sometimes you have fine-grained controls, especially in racing games.

Usually, when these modes are enabled, there is some indication that they are used, and you can't enter ladders with them, to prevent cheating.

We're not anywhere close to just having a general purpose game playing AI for the current gen (though there is that Atari playing one)

But there are a lot of affordances devs can make in individual games. Someone else mentioned Celeste and Last of Us, but Microsoft's Forza series has a bunch of options, like brake assist or turning assist, that can take a lot off your plate.

Reply on news.ycombinator.com