Hacker News Reader

Comment by andrewmcwatters | Hacker News Reader

They can already do this with the desktop composition software they use today. All the windows are virtualized onto backing layers that you can draw anywhere and add effects to. It’s how window shadows work, and how certain window effects are done.

They just haven’t done it.

View on HN

Yes?

The first iphone didn’t have copy/paste.

Apple will always prioritize critical scenarios over nice to have. None of these things are technically difficult, it’s just time. I’m willing to believe they released too early, but at some point you have to start learning from real users.

I'm aware. I've worked in the space. It isn't as simple as you're making it out.

Yes, it is.

In fact, there's already a project for it on GitHub. https://github.com/saagarjha/Ensemble

480 lines total, including comments, headers, whole shebang.

> Ensemble is currently "pre-alpha": it's really more of a demo at this point. There's a lot of things I need to work on and until then it is unlikely I will be taking any code contributions.... The code is definitely not designed for general-purpose use yet, so don't expect much of it :)

Tech demos are often easy to put together.

It's all of the edge cases and UX refinements that takes time.

Yeah, sure. I'm gonna go ahead and say Apple probably could have found a way to ship this over the visionOS dev cycle.

Some of the code is factored out into individual libraries. For example the networking and serialization code is separate and it is likely that screen recording will get pulled out too at some point.

Sorry but almost 10 years ago I could do this on Xorg where the worst problem is that compositors cannot redirect input (so you had to kludge new events from scratch). I cannot imagine it would take more than _half an hour_ for someone with macOS display compositor experience to implement it.

You nailed it, down to how you'd do it without any help from Cupertino: https://github.com/saagarjha/Ensemble

It's probably more down to the getting the UI right on apple's end.

Except that doesn’t actually do it. It is just a proof of concept.

Try it with a full suite of Mac Apps and you’ll find it falls apart because they aren’t all well behaved.

It falls apart how exactly? There's some app for which the popup menus are not shown at the right coordinates?

You are making this sound much more complicated than it really is.

Amongst many possible edge cases. Not to mention shearing with things that have rapid animations etc.

The trivial case is simple. You are imagining it's all the trivial case.

But they have already fixed shearing, otherwise there would be no MacBook display mirroring feature at all, multiple surfaces or not.

I’m talking about this open source project, not Apple’s implementation.

Unfortunately I happen to live in Cupertino

I doubt making the windows draw is what’s taking them time to get right.

I cannot imagine it would take more than _half an hour_ for someone with macOS display compositor experience to implement it.

What fools Apple engineering management must be, then!

It probably actually is a quick initial implementation, but like everything else, it needs to be planned, ticketed, developed, tested, and slated for release.

Compositor work isn't tremendously hard when the graphics primitives are already done.

UI state handling and input is a majority of the work from there. I've implemented this work before. A large portion of my background is in compositors and UI primitives.

What's really cool about having window backing layer handles, is you can do all sorts of crazy fun stuff in the office and show off to your coworkers or tech demo that basically will never make it to production because it's totally impractical.

The best example of that which actually did end up in a final product in my opinion was Windows Flip 3D. Totally fun implementation, I'm sure.

And the reason Xorg is now dying in favor of a system that doesn’t have this capability is because the architecture that enabled it, while cool at the time, severely limited the graphics performance and capability of the applications.

What? Doesn't Wayland work this way by design?

Indeed it does not. It’s local only.

I see. Do you mean like the Xorg fast path for local 3d rendering that became the basis for Wayland?

Exactly that. A local only path that provides high performance.

Remember the original comment was the claim that Xorg could easily do remote windows years ago. This is only true using the low performance path that modern operating systems have universally rejected.

No, the original comment is that by writing an Xorg compositor I could project separate windows into separate surfaces at arbitrary positions in the 3D world. There's no need for network transparency and you can very well do with an HDMI cable (actually, I was using an HDMI cable -- this was 10 years ago).

Network transparency is exactly what people want. If you're going to say just tether it to your computer with a wire then you just aren't understanding the product.

Network transparency is not required for wireless video, either.

Don't be letting actual experience get in the way of shitting on Xorg, now! Wayland will never win with that attitude!

> They just haven’t done it.

Literally nobody has done it. It's beyond ridiculous that you can't already show or duplicate an application window on any display you want and allow it to be controlled from anywhere it is visible.

Searching for ways to do this lead one into extremely niche software ecosystems. Please is there any collaboration app out there that makes it seamless to toss windows around like everyone actually wants?

I feel like multiple people did it back in the original X11 days, and almost certainly when compiz was the new hotness

Nearly 20 years ago there was an OS X screensaver that would capture all the window buffers and then float them around the screen as they rotated various ways. Another app would save a snapshot of all windows as Photoshop layers with appropriate transparency (before blur was added). X11 was always network transparent, though I don’t think windows could easily be moved from one client to another.

Even classic Mac OS, which wasn’t designed to be “rootless” could be run that way could have its windows mixed with OS X windows in the classic days.

As others have pointed out, there is already a PoC app that is doing it, so it seems if Apple wants to, it’s completely in their power. However, does this match their vision? (pun intended) Time will tell.

Isn't this how ordinary spanning monitors works? It might be slightly awkward with AR goggles since the relative orientation of the displays will be constantly changing as your head moves, and what happens to a window you have half on and half off of the Macbook's screen when you look away? Or do you want to have the application jump between devices, like appearing on your fridge when you go for a drink? With the old X11 protocol and a daemon in the middle this was possible but the use cases were extremely limited and the security issues made it a pain in the ass to actually use. With distros moving away from X11 this is only going to get harder, and you have to ask yourself how much you really want it.

This would mean the goggles would be basically just a dumb display for the Mac. It would also be weird to try to move an AR app onto your Mac.

So Apple could have done this, but did not. Why? (Speculation and leaks welcome)

- [Profit on basic innovation] Did they want to wait and see how their customers would adopt VisionOS's native free-floating windows, so as to avoid cognitive overload by commingling with MacOS windows?

- [Benevolence to fellow competitors] Did they not want to takeover the existing market of virtualized VR desktops?

Most likely because it wasn't polished, so get it out now in a polished and limited state and now you have a fancy update to tout when it is done.

I'm reasonably certain it's a combination bandwidth and tech issue.

The Vision Pro is effectively using AirPlay to mirror the whole screen. If you used AirPlay to mirror each window as a whole screen, you'd run out of bandwidth pretty quickly.

The windowing system in MacOS, Quartz Compositor, also isn't built to stream window information. Right now it has a big built in assumption that any windows its displaying are on a screen it also controls. It was probably too big a lift across teams to also re-write the graphics stack for MacOS for the launch of the Vision Pro. Hopefully they get it working in the future, but neither of these problems are easy to solve.

Makes a lot of sense, future will solve it someday.

Reply on news.ycombinator.com