Hacker News Reader

Comment by Athas | Hacker News Reader

I'm not sure why bytecode is supposed to be significantly better than storing the shaders as strings. Sure, you get rid of a complex parser and can more easily use it as a compiler target, but the core problem remains: the compiler can still not reason across the device boundary. The CPU compiler does not understand what happens on the GPU, and the GPU compiler has no idea what the CPU is going to do do it.

Although I'm not a big fan of CUDA, it does have an advantage in that it combines both worlds in a single language. This does permit optimisations that cross devices boundaries, but I have no idea whether the CUDA compiler does any of this.

Apologies for tooting my own horn, buI have been working on a language that can optimise on both sides of the CPU-GPU-boundary: http://futhark-lang.org/ (Although it is more high-level and less graphics-oriented than what I think the author is looking for.)

View on HN

The issue is that every driver implementation ever ships with a slightly different implementation of that complex parser. That means (every mobile device)X(every OS update) and (every PC ad-in card)X(every driver update) all have the potential to misinterpret/reject your shaders in slightly different ways.

Case in point: I recently learned that "#line 0" is a spec violation. A single OS revision of a single Android device refused to compile it. This is after shipping that line in every shader in multiple games for multiple years to millions of happy customers.

Apparently, Unreal Engine's boot process involves compiling a series of test shaders to determine which common compiler bugs the game needs to work around for this particular run.

This doesn't have anything to do with shaders as strings. It's been true since the beginning of shader programming, whether bytecode shaders or otherwise.

His point is that shaders-as-strings have a larger surface area for misinterpretation than shaders-as-bytecode.

Couldn't a bytecode format have the same problem? Probably not with respect to parsing, but the semantics could still differ.

In theory, yes. Neither system represents perfection. But, what's more important in practice is the likelihood of problems. Parsing bytecode is trivial. Thus problems with drivers inconsistently rejecting D3D shader bytecode has been relatively rare.

Parsing GLSL is quite hard. Thus, problems with drivers inconsistently rejecting GLSL text has been relatively common.

OpenGL's text requirement makes great sense in theory, but encouraged lots of problems in practice.

The software engineers who write drivers for graphics hardware have, unfortunately, an ill-earned reputation for being unable to code their way out of wet paper bags.

The fact that the GLSL standard forced every hardware vendor to vend a fully-functional compiler for a C-like language (even a simplified C-like language) is no small part of the source of that reputation.

You know what's great fun? Getting errors that only apply to desktop GLSL when trying to compile valid GLES GLSL. Can you guess how that particular vendor shortcutted implementing their ES compiler for that Android OS revision? Can you guess how likely it is that those Android customers will ever get a driver update? ;P

> The issue is that every driver implementation ever ships with a slightly different implementation of that complex parser. That means (every mobile device)X(every OS update) and (every PC ad-in card)X(every driver update) all have the potential to misinterpret/reject your shaders in slightly different ways.

I can confirm this happens a LOT. (At least with ESSL) In it's defense, it usually is a mistake I made, but some implementations are definitely more lenient than others.

Can empathize with that. The problem with OpenGL is not really the API but the impossibility to really have something correct. There is a sheer difficulty of finding combinations of GPU x drivers x OS to test OpenGL. It's quite common that the drivers lie or reject something valid.

I was poking around on your site, and I took a look at your publications list. I found this paper: http://futhark-lang.org/publications/icfp16.pdf

But it has no authors! I assume it was a double-blind submission to ICFP, which required removing the author names. When posting it publicly, though, I think you should put the author names back on. You don't want your work floating around the internet without your names attached.

I think the article's author does not necessarily bemoan the lack of a comprehensive integration, but the complete lack of any non-textual interface definition.

I agree, CUDA took a step in the right direction by allowing annotated C++ source to be compiled for the device. As an HPC developer, I'm using the Kokkos C++ library https://github.com/kokkos/kokkos to switch between CUDA, OpenMP, or Serial modes from a single source code.

Furthark looks really nice, but what does the future look like for it?

(I'm currently working on a Python hpc codebase with pyopencl and would be interested in reducing some of the manual labor required for cpu-gpu coordination)

Well, I still have a year left on my PhD, and I hope (with some reason) to get funding to continue after that. But I must emphasize that the language is still a little young and raw.

I just read the paper and it looks better than most of the alternatives for GPU programming.

Besides the good reasons brought up by other posters, game devs tend to be very secretive of their shader code, so they like the bytecode as an obfuscation layer.

Reply on news.ycombinator.com