Feasibility of low-level GPU access on the Web
kvark.github.io
kvark.github.io
As of GPU exposure to the Web ...
It makes sense only when we will have stable and unified GPU abstraction. As for now DirectX.12, Vulcan and Metal are close but different.
Like the WebGL that is (more or less unified) OpenGL. But even that looks too foreign to HTML/CSS/script - immediate mode rendering in architecturally retained display model of web documents.
And conceptually: HTML5 umbrella is large but not infinite. 3D rendering is too far from HTML "endless flat text tape" model.
I remember those <applet> days when browser was used as a component delivery platform for stuff that does not fit into DOM and structured yet styled text. That was conceptually the right way "to grasp the immensity".
These days, with WebAssembly, we have another incarnation of the <applet> idea and I think GPU functionality belongs to it rather than to HTML. GLSL to be expressed in WebAssembly bytecode terms but not in JS.
Web standards have to be backward compatible and I doubt that current still ugly GPU paradigms will survive on the long run. Like tomorrow someone will come with practical voxel based system instead of current abstract vector ones, what will we do?
Will it live on? Sure, we will still have games and Xboxes. But if you're going to pick a standard that can work on mobile and desktop, there is no contender other than Vulkan.
err, Intel iGPU, available in something as tiny/low-power as a LattePanda board… and all the Windows tablets…
2. Vulkan is not allowed on UWP apps
3. Even on Switch, Vulkan is not the primary 3D API.
2. Nobody (in Joel's on Software sense) is writing high performance 3D UWP apps anyway.
2. UWP requirements apply to any Windows store game and Microsoft already started to be more bully about it
3. There is hardly any Vulkan game worth playing that isn't DirectX 12 as well.
2. Both of them... If you are gamer, you are going to Steam (and Valve is not a friend of the Windows store idea). I'm not happy with this situation either, as I prefer GoG, and GoG is a distant second.
3. Doom, Wolfenstein II, F1 2017, The Talos Principle from the other side of genre spectrum or the upcoming Star Citizen. Vice versa is more true, there's no DX12 game worth playing, that's not also Vulkan game.
2. Lots of gamers just use XBox or PS4 (no Vulkan there). And on PC, not everyone uses Steam. Plus Microsoft already started to be more agressive regarding games on Windows 10, with Age of Empires remaster being the first example.
3. Well it is a matter of taste, not everyone craves for FPS, then there is also the small matter that Vulkan is not supported on XBox anyway, while DX 12 is. With much better developer tooling.
2. Age of Empires is Microsoft's game in the first place. Of course they will want their assets to use their technologies. 3rd party adoption is a rounding error. On the PC, except for 1) hardcore indie games players and 2) games locked to the publisher's platform, everyone uses Steam. In the second case, where the games are exclusive to a publishing platforms (Origin, Uplay), they have similar attitude to Windows store as Steam.
Windows store is an existential threat to them. They will ignore it as long as possible.
3. Sure, that's why I mentioned The Talos Principle (a puzzle game). Xbox API will be handled exactly as PS API is.
For one thing, any new approach will still have to solve the same problems around data transfer and formats that make up a lot of current APIs.
And for another, no matter how good this hypothetical new approach is, the old one will still be around to handle existing content and workflows forever anyway.
Also mentioned recently here https://news.ycombinator.com/item?id=16347216
A simple DOM representation could be used as a fallback, if possible.
The internal canvas compositing operations you mention are also GPU-accelerated in most cases.
In the quest for eye candy we're losing what made the web great to begin with.
A lot of HN comments dismissed my prediction[1] that WebAssembly will bring opaque "compiled applications" that treat the canvas as a "standard pixel buffer", allowing adblocking CSS and request filters to be bypassed.
A lot of people seem to be focusing on the benefits of new technological changes ("no longer a restriction"), when they should first be concerned with the potential risks that change will create.
They could already do that. It's called serving the whole page as an image
Can you elaborate on the arguments against a dialect of SQL being available for use with local storage? I can only think of arguments in favour of it. Would be good to understand the grounds on which the idea was dropped.
The only alternative is to make sure all new interfaces have multiple, popular implementations simultaneously, so that authors can't afford to take advantage of bugs in one implementation. This is the sort of activity you see in WHATWG these days.
The problem with "a SQL dialect" being available for use on the Web is that every browser intended to use SQLite as the backend. Nobody wanted to invest the time and effort to write a second, compatible, equally-reliable implementation; nobody wanted to exhaustively research and document bugs and flaws in that specific version of SQLite; nobody wanted to commit to back-porting security fixes from later versions, or forward-porting the required bugs to later versions.
And since nobody wanted to do the work to make it possible, using SQL for local storage remains impossible.
As much as I wanted it, and was bummed that Mozilla protested so much, they were/did/are doing the right thing by saying that SQLite cannot be used as a standard, it isn't a spec.
What one can do, is compile SQLite to emscripten or wasm, or write a SQL engine in JS and use that in the browser. That is totally fine.
https://www.html5rocks.com/en/tutorials/offline/quota-resear...
Wish we could've just had two SQL implementations. There are other ones out there - Microsoft ships JET and an embeddable version of MSSQL, someone could've embedded postgres or something. As-is people who care about performance are just going to compile SQLite down to wasm/asm.js and run it in a worker.
Everyone thinks that they can do better with an immediate mode API, then we start building some kind of data structure to track down what needs to be drawn and when, with the result being similar to the traditional joke of half-implemented Lisp or ORMs, but applied to retained mode rendering.
Sure, some experts manage to get it right, and those are the ones that get to write AAA game engines, but many don't.
I can't think of a retained mode API that works as well for RTS games as it does for FPSs, to say nothing of non-game uses like CAD.
... but they are not built on an abstract retained mode API. That would be the wrong level of abstraction for a Web API that people can build on properly. I think that's the point here.
Unfortunately, Web APIs tend to be far too high level while missing out on low-level hooks, like the disaster that is Web Audio (and media playback in general).
The browser is indeed quite complex (and not just because of the massive historical baggage), but its job is to give the user control while safely downloading untrusted code and running it locally.
So, if your goal is to just have a simple standalone app I think you could stay largely compatible with APIs available in the browser environment.
Also, I think users would have more control if documents didn't automatically gain the same privileges as applications.
Easier implementation of a browser? You might find it interesting to see what Servo has chosen to implement and what they have not. Some things you'd think would be easily removable (such as document.write) turn out to not be so simple to skip.
One of the most valuable things about the Web is the care taken around backwards compatibility.
I do think it'd be quite interesting if you had a user agent that did the DOM differently (not sure what you have in mind specifically re: "documents didn't automatically gain the same privileges as applications") and focused just on providing a GL canvas and audio APIs.
I think you might find that these APIs aren't quite as nice when it comes to re-implementing things that CSS and DOM make easy, and it'd be hard for such a browser to really compete with existing browsers given the backwards-compat situation on the web (mandating GL would leave some devices behind, and web authors as a whole don't really adapt all that quickly).
In any case I think it might still be useful as a reference implementation / proof-of-concept on how minimal a web user agent can be, if it was just focused on hosting applications.
There's interest in wasm-land about having "non-web embeddings", which wouldn't assume things like JS APIs exist at all.
I think in that sort of world, you could probably find nicer APIs to target than WebGL and WebAudio... however if you don't mind still having a JS interpreted available then it'd probably be easy to build this sort of thing today using Node.
This is just a subset. I'm mainly curious if this would be easier to implement or not. Using Node would be cheating.
Not released yet, but there's basic Windows, macOS, Linux, Android and iOS support. Contributions welcome!
But that's not what my idea is. My idea isn't really a browser. It's just a separate format/mimetype (.app or .game) for a app runtime.
Small projects could implement it. We can have it embeddable as a object in browsers. Clients other than major web browsers (Gopher, Dillo) could embed it. Embedded devices (Roku) could include support for it. It could even be used in a physical media like SDcards or DVDs.
Here's what I'm trying to reconcile: if the idea is to break off the awesome subset of multimedia web tech, because it would be easier to implement than a full browser -- I 100% agree! -- then why is there a need for a new mimetype/format?
Many things could be done better than the current HTML-as-laundry-list approach, but if we invent a new format, that adds the extremely difficult problem of getting everyone on board, rewriting things for it. The brilliance of asm.js (which gave birth to WASM) was that everyone had already implemented it before it existed.
Until browsers supported the mimetype they could just be served as .wasm files. Not that the mimetype is important at all I just think it more clearly states it's intended use.
Few sites even use WebGL for anything interesting.
Low level GPU access will just make it easier for them to run montero miners on my browser.
But in the end the problem with be battery life...
What's annoying is that you don't get double precision or any of the compute stuff on WebGL currently. The interesting stuff isn't just graphics, but also doing computations directly in the browser.
I understand that engine developers can get some (small?) percent improvement from an ultra-low-level API that exposes more platform specific details. But they almost always support multiple APIs natively, and already use the low level ones where available. There may be a small performance benefit (over a higher level API) in the browser, but I don't think the browser is a likely target for ultra heavyweight apps / games, that usually are multi-GB downloads anyway, in the near future. And keep in mind that if smaller software sticks with WebGL because of API complexity, that might be a big performance loss for the user.
But a higher level API would have great benefits everywhere! Right now the only real cross-platform graphics API is OpenGL ES 3.0, which is becoming more and more obsolete, with little sign of the situation changing. No compute shaders, no AZDO, etc. Any move beyond that feature set now requires multiple APIs, shaders etc. And the easiest way for a small developer to get those features is still to skip Vulkan and to stick with GL ES 3.1+ on Win/Linux/Android and choose Metal on Mac/iOS. Of course on the web for those features there are no options at all.
I think if a new API was available that was easy to use and truly cross platform (including web), it would be the obvious first API to implement for all new graphics software. And this would be a much larger benefit than an unknown performance improvement that is accessible mostly to engine developers.
At this point, so long as Apple is in your target market, there will never be an everywhere API because Apple does not want there to be one. The whole point of Metal is to make your life harder so developers currently writing for Apple first are less likely to port their software elsewhere.
I don't know how much bad faith I want to assume on Apple's part, since there are probably some legitimate technical reasons why they don't want to support a lower level API (and they came out with Metal first). Vulkan is such a pain to use anyway that it's probably mostly used by engine developers who generally don't have a problem supporting multiple APIs like Metal / DirectX etc.
But at the very least, it would be nice if they upgraded their OpenGL version, since they already support that and it's only them holding back some of the newer features.
I don't think anyone who actually worked on a serious engine thinks it's somehow harder to use. Sure, there's more boilerplate and it might get pretty difficult to port something to it, but that's something else entirely.
Which I guess, it is where the resistance is coming from, hence the need for such presentation.
Metal and Vulkan are not at the same level of abstraction. If you look up the code needed to draw a triangle on the screen (maybe the most basic graphics task), it is much, much more long and difficult to do it in Vulkan than Metal. Vulkan is a lower-level API. It gives engine developers more flexibility, at the cost of making basic things time-consuming and complicated.
OpenGL was the previous cross platform API. WebGL is almost identical to OpenGL. OpenGL still runs on Mac / iOS, but Apple has stopped supporting newer versions. The newer versions of OpenGL have closed the gap some with Vulkan and Metal (it won't catch up entirely, but it added some important features like compute shaders and it's easier to use it efficiently). OpenGL is still easier to use than Vulkan. The problem is the newer versions are not cross-platform, since Apple wants to focus on Metal.
Apple does not want to support Vulkan. Metal came out before Vulkan did, and it is a higher-level API. It's arguable if Apple should support it or not, but that's how it is. Microsoft also wants to focus on DirectX 12 (their API).
I was making the argument that a higher level API, Metal-style, would be a good base for the new web standard. Metal couldn't be used directly, at the very least it would have to be changed from Swift/Obj-C. But the idea of roughly basing it on Metal as mentioned in the article doesn't seem unreasonable, even Vulkan was based on a previous AMD technology called Mantle.
A low-level standard like Vulkan is hard to use directly, its adoption will depend mostly on people using frameworks / engines that use Vulkan. It's possible that due to its low-level nature there would be some performance advantage, although games using Metal also seem to get good performance on iOS. The disadvantage of Vulkan is that it is much harder to use than WebGL.
A fair amount of the WebGL content is not web specific. It is possible to write OpenGL content and compile it for desktop / mobile and the web. The most popular game engines, Unity and Unreal, both support compiling to the web and desktop from the same codebase.
In my view, there is no good replacement for OpenGL, now that new versions are not cross platform. Vulkan is much more work and does not work on Apple platforms, while Metal is easier but only works on Apple platforms. If they could provide a standard that is both easy and cross platform (by providing a C API library in addition to the web standard), it would provide the best of both worlds. The main downside is that it might leave some performance on the table compared to a low level API, and it would be yet another standard (which is why it would be important to provide a native library too, so developers have the choice of coding to only one API).
Before iOS, most mobile devices were having their own experiments with 3D APIs, Nokia N95 was the very first with OpenGL ES compatibile GPU, but it was thanks to iOS games that it ever took off.
Apple was also pursuing Quickdraw 3D before the NeXT acquisition, so not too keen on OpenGL anyway.
Their OpenGL's adoption was mostly survival related, now they are back on top, they can afford to dictate their 3D APIs just like all other console vendors.
We should also probably sort out the native low-level APIs before setting the standard for the web, because otherwise we're building on top of a big mess. Though, my impression is that the WebGPU initiative is basically just another battleground in that struggle. I don't have any faith that this is being done for the good of users. It's just strategic ground to capture.
On the flipside, the native low-level APIs have all diverged. Vulkan, D3D12 and Metal each made different decisions, so it's pretty much inevitable that a 4th standard will have to be created to unify them. It will probably be higher level than any of the 3, and it will still be subject to strong sandboxing limitations.
Personally I think the big issue is that people stare themselves blind on the traditional graphics pipeline. Modern renderers have evolved past this, with various compute-driven and/or tiled approaches common place now. They're a nightmare to implement against current APIs, because a small change in strategy requires a rewrite of much of the orchestration code. The job of figuring out how to map your desired pipeline onto the hardware's capabilities should be the job of a compiler, but instead people still do it by hand. Plus, for GPU compute-driven graphics to be useful for interactive purposes beyond just looking pretty (i.e. actual direct manipulation), you absolutely need to be able to read back structured data efficiently. It's not just about RGB pixels.
There's an immense amount of potential locked inside, but the programming model is a decade or two out of date. Only AAA game companies and the vendors themselves have the resources to do novel work under these constraints. Everyone else has to throw together the scraps. Even the various attempts at LISPy GPU composition fall short, because they don't attempt to transcend the existing pipeline.
Then the real problem is, since all shader invocations share those fixed-function units, if you need to reconfigure them to use a different set of textures, buffers, shaders, etc you have to bring the whole operation to a complete halt, reconfigure it and restart it. And, contrary to popular belief, GPUs are the exact opposite of fast - each shader invocation takes an enormous amount of time to run, which is traded for throughput. Stopping that thing means having to wait for the last pieces of work to trickle through (and then when starting back up, you have to wait for enough work to be pushed through that all the hardware can be used efficiently), which means a lot of time doing little work.
So if you're trying to deal with the above, any notion of keeping things separate and clean (in terms of what the hardware sees, anyways) immedietely goes out the window. That's why things like virtual texturing exist - to let you more or less pack every single texture you need into a single gargantuan texture and draw as much as possible using some God-shader (and also because heavy reliance on textures tends to work well on consoles). Then you also have to manage to make good use of those fixed-function units (which is where tiled rasterizers on mobile GPUs can become a problem), but that's a relatively separate thing.
Also: transfering data back and forth in itself isn't necessarily that bad in my experience (just finnicky), it's usually the delays and synchronization that gets you.
I would rather target WebGPU and write my program once rather than implement my logic three times in Metal, DirectX and Vulkan. But, Apple and Microsoft don't want me doing that, so why would they support WebGPU?
It makes me wonder if anyone has created some sort of port to a standalone app with no browser involved where you could use javascript/canvas-api/webgl to draw pixels on a canvas-like surface... without the fatness of the browser. Just spawning some window, that would be a lovely scripting/game-dev environment, maybe with some sdl-bindings or whatever. Anyway just rambling, does such a project exist? Anyone knows? :)
I've got sample using an OS X window but it should be easy enough to do the same for Windows
It's been sitting in a private repo since then but I can give you access if you'd like
That being said, I think a better alternative for a thin graphics scripting environment would be haxe with Lime or snowkit (which provides OpenGL, SDL and windowing). I've used these in the past and loved working then
A university or corp could push a tab open script to thousands of desk tops, while they wouldn't push Folding@Home client.
Also, a compute shader extension may be coming[1] for WebGL 2.0, it's mentioned in some meeting slides at least: https://www.khronos.org/assets/uploads/developers/library/20...
but it is needed for doing GPU compute efficiently for many workloads.
Compute shader extension for WebGL 2.0 would be cool, but it would require to port a large part of OpenGL ES 3.1: OpenGL ES 3.0 / WebGL 2.0 doesn't include even random access buffers (SSBOs)
For example, here's WebGL compatibility stats from a site that has counters on technical web sites and graphics programming web sites: http://webglstats.com/ - As you can see, WebGL 2 compatibility is only at 40%, despite having been enabled in stable Firefox/Chrome for over a year. WebGL 1, a 7 year old standard, is now at 97%. (And even for the nominally WebGL-enabled browsers, users often report browser or OS crashes, so the percentages are upper bounds).
From an armchair quarterback position, if I wanted to effect GPU compute uptake, I'd work on compiler tech and tools targeting WebGL GLSL.
[1] http://newport.eecs.uci.edu/~amowli/hpcfactory/publication/a...
The issue with this in the browser is that the API isn't part of the JavaScript/WebASM, and just exposing it in the same way will allow you to subvert the sandboxing of the VM>
I'd also wonder if you could share the power of any Xbox One Xs connected to a multiplayer session, given the gap between an original Xbox One and the X is rather large (seems like it'd be far more trouble than it's worth though).
Safari has "tab paused/reloaded due to high power consumption" (good), Chrome has "auto-mute tabs" extension (which I have turned on), I have "tab suspender" extension installed.
Basically, I'm trying to be a responsible consumer:
- this tab would like access to your hard drive files
- this tab would like access to your video camera
- this tab would like access to your microphone
- this tab would like to play sounds
- this tab would like to download more than 5mb of data
- this tab would like animation/movement
- this tab would like to use your CPU a lot
- this tab appears to be using a lot of your battery
- this tab would like to use your GPU (at all)
- this tab would like to maintain state > 24hr
the_internet.js is actually potentially really hostile (suck 9999mb at full speed, ddos@1.2.3.4, while(1){alert(1)}, mine_bitcoin( $hacker_wallet )), and I am much in favor of treating it as untrusted by default (low access, limited # of cpu cycles) until "trusted" (ie: android permissions swap: i give you the executable, you give me the permissions).
YouTube? Yes to whatever they ask.
ShadyWebsite.1234.some-random-domain.ru? You can d/l 300kb and can't do anything else (ie: web 1.0/no-script).
While each "tab" in a web-browser attempts to provide "safe" access to the computer resources, it is still not "permitted" access to computer resources. Whatever the browser defines as "safe" is 100% ok, which has to work equally well for WASM-unreal-tech-demo-castle as well as cnn.com.
I'd prefer cnn.com only had 300kb download, no external domains, no sound, no battery, no cpu, etc.
As I _trust_ cnn.com more (to the same level as youtube), I would then permit sound by default, permit large downloads, permit gpu, permit animation/video/etc.
It was no more than a year when a remotely exploitable WASM hole was exposed (derivatives of Spectre and co.) Knowledgeable people told that ISA level hole that can be exploited remotely over the web will be "a one minute global IT disaster" if somebody would resort to propagating it through a big adnet or paid traffic scheme.
As for WebGL as it is now, there were numerous sites on my memory that froze/crashed/rebooted both Linux and Windows systems, which means that the prime suspect there was a buggy shader as it is the only thing resembling raw instructions that can be passed to gpu through webgl.