WebAssembly, an executable format for the web
blog.octo.com
blog.octo.com
function readUtf8String(memory, pointer) {
let s = "";
for (i = pointer; memory[i]!==0; i++) {
s += String.fromCharCode(memory[i]);
}
return s;
}
Edit: Changed "LLVM" to "LLVM IR" to clarify as requested.
[1]: https://www.destroyallsoftware.com/talks/the-birth-and-death...
You don't have to use the Java standard library to target bytecode. Scala, Ceylon, Clojure provide their own stdlibs. Indeed not doing this is one of Kotlin's competitive advantages.
Although the JVM wants to think in terms of objects, this is only really mandated as a unit of linkage and garbage collection. You could equally criticise C runtimes for being too oriented around procedural programming semantics because they see the world in terms of top level functions and global variables. You can write non-OOP programs in Java bytecode and Eta is an example of using the Haskell Spineless Tagged G-Machine representation on top of the JVM. If your runtime doesn't have any concept of an object-like thing, then it gets rather hard to do garbage collection and certainly any kind of moving GC.
Finally, the JCK is encumbered to protect the Java trademark. We can argue about whether that's really been so useful given Android, but the principle makes sense - it's about ensuring that if someone claims they run Java bytecode, they actually can run it. If WebAssembly isn't trademarked and protected in the same way then it'll be a repeat of HTML5, where a claim of support is essentially meaningless and everyone has to constantly consult giant feature/bug tables to figure out what subset of the spec works on your particular implementation.
Now it may be that in the end everyone is OK with the word WebAssembly being more of a statement of intent than a precise claim, just like we muddle through with the word HTML5. And in reality we all know that only Mozilla, Microsoft, Google and maybe Apple will implement wasm or at least only these companies will have implementations anyone cares about. They will probably comply with whatever test suite is produced without needing legal/trademark incentives. Maybe.
There are already projects to implement WASM in other contexts beside the web--some even in production. The specification, which has been entirely mechanized with an implementation of a provably-correct interpreter in Isabelle, is designed to be implemented easily. We welcome other implementations of the core execution spec of WASM and other embeddings. I can only speak for myself as a co-founder, but I believe most people involved in the W3C groups hope the scenario laid out in this comment does not happen in reality.
The worst case scenario for wasm or any spec author is that someone comes along and either implements it very poorly, or adds vendor extensions that effectively become de-facto standards. E.g. Chrome might add new opcodes or APIs that rely on other Chrome-only features. And then people start claiming they've got a wasm module, but it doesn't work properly or well enough on other implementations.
That's what happened to the web, to Java and others, and is one reason why the TCK is "encumbered", as the OP put it. It's also why Android requires you to pass an encumbered test suite to be able to use the trademark/logo/get access to the Play Store etc. It's not like one path is inherently superior to another. Protecting a trademark by licensing the test suite and mandating it be passed is about risk management.
I'm not sure which proof you are referring to, but the Java and Scala languages both have unsafe type systems:
https://2016.splashcon.org/event/splash-2016-oopsla-java-and...
(Admittedly this is due to parametric types which are erased).
The JVM bytecode verification algorithm is over 100 pages in the spec.
> E.g. Chrome might add new opcodes or APIs that rely on other Chrome-only features.
We aren't doing that currently. We plan to continue participating in the W3C processes which have worked well over the past 2+ years.
We aren't doing that currently
All browser makers have long histories of forking the platform with proprietary extensions more or less whenever they feel like it. So what does this assurance mean? Indeed that's now how HTML5 is meant to evolve! Nobody even uses the W3C anymore do they? It's all just throwing whatever extensions get built by browser makers into HTML5.
I'm sure WebAssembly will have its share of compatibility matrices and vendor-prefix equivalents soon enough.
> Scala, Ceylon, Clojure provide their own stdlibs.
On top of the JVM ones when targeting the JVM. E.g. java.lang.String (Kotlin too)
> You can write non-OOP programs in Java bytecode
Everything is still an object and all code is in methods. There is no way around that. Also, there are many other things like forced GC and what not that marry the bytecode to certain paradigms.
> it's about ensuring that if someone claims they run Java bytecode, they actually can run it.
And about their impls of java.lang and other base packages. And a way to make sure that licenses that Oracle doesn't like, such as non-OpenJDK-derived open source stdlib impls, are not given the tests. Other languages with compat tests publish them for all and yet aren't worried about losing their name. This is dangerous reasoning to apply to keeping things opaque.
It isn't just those big companies implementing WASM. I implemented a WASM backend for the JVM [0]. The core test suite, which I use and comply with 100%, is available to all [1].
It's a head-scratcher as my code is nothing special and I don't see what I could be doing wrong. Then it occurred to me that these benchmarks look suspicious, with many scores below 100%. Are there fair benchmark results somewhere, measured by someone not trying to promote WebAssembly? What performance ratios are HN readers getting currently, and what performance ratios should we expect in the long term?
You can compile these samples natively and compare yourself.
We're all focused on the potential performance rather than what this will actually be used to deliver -- so maybe i'm just missing what prevents this technology from subverting the open web.
Back in the day we had Flash-only web sites. They died off because it turns out that it's easier to just throw HTML, CSS, and JS at people, and since these were open standards, there was a fair bit of competition between different vendors to make better tools for consumption, authoring, and delivery.
If you want to reimplement that experience in WebAssembly, the ROI is going to be horrible and if you render everything to a canvas your site won't be indexable by Google (goodbye traffic) or workable with screen readers (hello lawsuits).
Apple/Google/Microsoft would be stupid if they weren’t already working on a Swift/Kotlin/C# to WebAssembly compiler, as each of them would give them for their preferred language what JavaScript now has: the ability for developers to work in a single language, and move code between browser and backend at will.
Long-term, JavaScript is dead.
In practice, it does not.
It's too low level, and the APIs to access browser features are not available, so you have to go through seriously ugly APIs and just do it in JS anyhow.
So if you have to go to JS to do pretty much everything, why are we using WASM again?
Performance? Is performance the issue? Because if it is, isn't there maybe better ways of solving that problem?
It's certainly not for portability, we already have that.
To make 'web development available to C coders' ... Not sure that's it since they will invariably have to know web browsers anyhow ...
WASM is a great idea that won't gather steam until there are direct APIS. Which is on the roadmap I think.
I'd suggest just trying to build a simple WA, or even trying to make a simple app out of WA. Things look different then.
It's a nice dream to have, we're still a ways off.
However, I can think of two very compelling use cases for WASM.
As an addition to JS, it gives JS programmers access to a wealth of libraries and programs written in other languages. Codecs and compression algorithms, linear algebra libraries, image and audio processing, and a million other things have good libraries that just don't happen to be written in JavaScript. Adding a WASM blob as a library you can call from JS is a huge success. Suddenly you can do tons of client-side processing that was too prohibitive before (from an engineering cost standpoint).
As an alternative to Flash and Unity Player, you can write games for the web using WASM without having to deal with JavaScript.
Having used WASM personally, I can say with conviction that it is a compelling addition to the open web.
No, you have to go through a seriously crazy api wherein data is passed into a single managed array in wasm, and pulled off in JS and visa versa.
Ok - so accessing C libs is useful?
Are bindings from other languages to C considered to be really an important things for those languages?
Was there a specific set of libs that we were angling for?
Does it even make sense to do those things in the browser?
And what about the limitations - i.e. having to pass data into and out of the WASM container - does that play well with usage of 'codecs'?
Maybe not.
So why again does WASM exist?
What as the compelling use case?
Flash/UX is definitely not it.
I don't see any real uses cases for things like codes, nor was I seeing demand.
So where is the market/product need? And use case?
This is cool tech that's a long way off from being useful in the real world.
Having gone through the pain of writing my own little app, I realized that it's still fairly bleeding edge. I will go back to it when there are deeper APIs between JS and C, are the toolchain is more stable.
That's false. When writing a game, you interact with APIs in your implementation language (e.g. SDL).
> Are bindings from other languages to C considered to be really an important things for those languages?
Yes, it's a fairly critical feature, which is why successful languages almost universally have a C FFI. Some languages have several FFIs to choose from, and others can even integrate with the C preprocessor. So many libraries that we use on a daily basis are written in other languages than the languages we use, and that's just how modern programming works.
> So where is the market/product need? And use case?
> This is cool tech that's a long way off from being useful in the real world.
It will happen faster than you think. Some JS developer somewhere will want to use a native library. They will want to use it badly enough to port it to WASM, write a JS wrapper interface, and upload the whole thing to NPM. Suddenly, a C++ developer with no knowledge of JS will be delivering features to front-end developers with no knowledge of C++, all thanks to some third developer who put in the effort for the WASM port.
Which is how it already works everywhere else, so it shouldn't be a surprise. For example, you can use SDL from C#, even if you don't know C and the SDL developers don't know C#. All because some guy with a bit of knowledge about P/Invoke figured out how to copy data in and out of the C# runtime to make some SDL bindings, just the same way you'd have to do it with WASM.
I mean, you can complain about problems with copying data in and out of WASM, but it already works that way with other FFIs.
Ultimately - information must not only be passed through an ugly memory-reference API, but anything that happens on the screen, must happen in JS.
Ok, sure, you can abstract it all away, but that doesn't change the fact of the matter.
It would be more powerful and efficient to be able to do stuff natively inside the container.
" Some JS developer somewhere will want to use a native library. They will want to use it badly enough to port it to WASM, write a JS wrapper interface, and upload the whole thing to NPM. Suddenly, a C++ developer with no knowledge of JS will be delivering features to front-end developers with no knowledge of C++, all thanks to some third developer who put in the effort for the WASM port."
Agreed - but I still don't think we are quite there yet.
Yes.
> Are bindings from other languages to C considered to be really an important things for those languages?
Definitely!
> Was there a specific set of libs that we were angling for?
No, there are too many to count. For example, I am looking forward to using network visualization libraries that no one here has heard of. They would be very difficult to port from C++ to JS.
> Does it even make sense to do those things in the browser?
Yes.
Besides that, are modern sites actually accessible in that regard?
And yes, modern sites really are accessible. Screen readers traverse the DOM looking for the semantic information that's already there. Look at WCAG (https://www.w3.org/TR/WCAG/), you'll see how easy it is to follow the guidelines. Unless, of course, you stick a bunch of stuff in a <canvas>...
The degration in detach indexing and user experience, load time and battery usage would quickly make a first mover loose users.
Is the idea to write an entire application in WebAssembly, or is it to break out parts of the application that benefit from performance improvements and potentially cross-platform code bases?
Also, how do you do things like make network requests or decode JSON in webassembly? Do you need to implement these things yourself, or pass things back to the browser to handle them, or find C libraries that do that for you?
I've seen people do some very naive stuff, like:
var particle = {
color: randomColor(),
position: new Point(x, y),
// number of frames the particle lives
decayIn: 50 + Math.ceil(Math.random()*100),
};
This does not scale if you want to draw thousands of particles that each last a few seconds.
If they were smarter about it, they could create a pre-allocated "pool" using a maximum nr of particles and re-use them.
Likely to be even faster would be preallocating one Uint32Array (for color), two Float64Arrays (one for x and y), and one Uint8Array (for decayIn), and then use an index to look up an individual particle's color, x, y, and remaining frames.
Anyway, what I'm saying is: idiomatic JS definitely can result in garbage collection-related bottlenecks for game development, but a lot of this can be avoided by using non-idiomatic coding styles (like heavy dependence on TypedArrays).
Too late to edit, but this really should have finished with:
... at the cost of code that is more verbose yet the opposite of self-documenting, inherently frail even if you put wrapper-functions around those typed arrays, and likely to require massive changes all over the code-base for the smallest updates.
In my opinion, unless you have a really good reason to not use alternative options like WASM or Emscripten, trying to squeeze that kind of performance out of vanilla JavaScript it's unlikely to be worth the technical debt.
In most other JavaScript applications GC is hard to notice, because it typically runs once the application has finished executing.
Excessive allocation makes also other languages, such as C++, very slow.
The fundamental problem with a GC is that all threads must be stopped to perform the garbage collection.
GC pauses are a problem for realtime applications because an unpredictable GC pauses can blow your time budget of 16ms (or even less) per "tick". The only way to avoid this is by not allocating anything during the realtime segments. Since the GC "infects" your entire program you are also not allowed to allocate anything during the non realtime segments. So you might as well just write C++ code.
If the heap is large enough (several dozen gigabytes) the GC pauses may cause latency spikes that last more than a second.
I don't like to compile stuff. But improving performance by writing certain functions directly in assembler or in a subset of js would be awesome.
asm.js was a deliberately limited subset of JS syntax. By adding a "use asm;" statement, you signaled to the JS engine that everything after that was limited to the asm.js subset, allowing it to simplify the parsing logic and use a fast path for loading the code. You would rarely write asm.js syntax by hand, but rather use it as a compiler output format.
WebAssembly takes that further by defining a bytecode that the JS engine doesn't even have to parse, but can accept and work with more directly. This speeds up loading time. Also, loosely put, the way it's defined allows the JS engine to execute it faster.
WebAssembly is definitely not intended to be written by hand. Languages like C++ and Rust can now compile to WebAssembly, allowing non-JS source code to be run in a browser. (There _is_ a textual representation of the bytecode as well.)
I'd rather have the flexibility in those situations.
There are inevitably defects in the design and/or implementation which will open up browsers to gaping security vulnerabilities until they are found and fixed.
Adobe did an amazing job of making sure Flash was there, and seamless. To the point people barely had an awareness.
Java Applets failed because Sun was a 'debating society' (per James Gosling). Definitely not a consumer oriented company, thinking about end user behaviour etc..
Other reasons as well, but that's #1.
Source: me, deployed a bunch of Enterprise projects on Applets back in the day.
If you look at Java's security track record in recent years, there's been just a couple of zero days in the past five years. That's probably a mix of genuinely better security and less attention due to being kicked out of web browsers. But if you go look at the security histories of other sandboxes like browsers or kernels, it starts to look pretty good. A new paper that just came out introduced a new Linux kernel fuzzer and discovered, I think, over 30 zero day exploits across several different Android phones. So ordinary UNIX style process isolation is pretty useless if you can reach device drivers from inside the processes. Even the quite aggressive and resource intensive browser sandboxes that browsers use routinely have escapes, often because they're always adding large new attack surfaces like WebAssembly, WebGL etc (all written in C++).
IE, you'd have to convert the javascript to C, and then compile the C into web-assembly, right? Does V8 have a method to convert JS to C?
Otherwise, you will have a hard time to convert your Javascript code. You have to implement a JIT compiler and GC too. Webassembly was never designed to be a replacement for JavaScript. And especially, it was never designed to protect your source code. A minimizer/obsfucator does a great job of making your JavaScript code unreadable.
I hardly get that, all and every major backer of WASM did effectively admit at some point that "ip protection" was the one and only reason they back it.
The absence of a native dom interface or proper I/O implementation at this point means that WASM compiled code actually runs slower for everyday web apps than non-compiled code.
I'm sure you wouldn't mind providing sources for this bold claim?
This kind of obfuscation is harmful to the web, don’t do it. Also, any compsci student with a summer of free time can easily break it, so it’s useless, too. You’re just wasting your own time and the bandwidth and performance of your users.
In fact, if you tell me what site you want to obfuscate your code on with WASM, I (an average compsci student) will spend my next summer reversing it, deobfuscating it, and publishing the results publicly (just to demonstrate you how useless your obfuscation is).
Photoshop isn't on the web, but I oppose closed source in principle.
> Should they give away their source code?
Ideally, yes. If you do it is still your decision, but you still should be aware that if you introduce truly new concepts, someone will clean-room reverse them, and publish them.
> So, art programs kind of need source protection.
The Krita project begs to differ.
> I won't be making a web-version until this is the case. If you can crack NW.js source protection, let me know
Then you shouldn't make any build where the business code runs on the client. Even native binaries can be reversed, their functionality analyzed, and copied. Which is how LibreOffice/OpenOffice originally discovered how to parse the Microsoft Office formats.
When people paint, they want it responsive, no stream lag, no server hiccups. I must allow people to install the software on their computer. I would still like to protect the work I've done, because I'm just one guy making software, and I don't have marketing money. What would stop big companies from taking my source and spending more on advertising? I need source protection.
If your threat model is a company with billions in of dollars in revenue, then source protection is useless.
NW.js protection can be cracked, too, requiring maybe one or two man-months more. For a student reversing it, this is annoying, but not a problem.
For GoogleAppleAdobeZon? They dont even notice the cost difference. You can be Snapchat, have unique features, and have it all obfuscated — and Facebook can still copy it all in weeks.
The question isn't "will Adobe be able to copy my app" — the answer to that is always yes.
I think there are times when open source isn't the best. I never would have spent so much time making art software if I didn't think I could make it reasonably hard to crack. Eh, maybe I would have anyway, but I might have just made another mindless web app that wasn't difficult to code.
I think there are definitely cases where closed source works, especially when certain algorithms are easy to read but hard to contrive.
For native binaries, such as games, currently the strongest solution is Denuvo, from a team that formerly implemented copy protection for Sony. Decades of engineering went into it, at the risk of significantly slowing down the game, using several layers of obfuscation (including VMProtect), all wrapped around the existing, older obfuscation schemes, for a native binary.
The first version of it took a few hobbyists 6 months to break, by now games protected with it get usually broken within of one week or less by hobbyists.
That’s hobbyists.
If, as you truly say, your threat model is Adobe, or a similar corporation, your obfuscation is basically useless. If determined individuals can break some of the most advanced obfuscation schemes in days, how long do you think will it take Adobe to break yours, if they actually see you as competitor?
> I think there are definitely cases where closed source works, especially when certain algorithms are easy to read but hard to contrive.
They work short-term, but as soon as someone extracts the algorithm once, everyone has it. Especially considering that in most places algorithms are neither protected by copyright nor patents (including the EU).
Same story since the Amiga Days
As someone who worked for a while on painting tech at a BigCo, the truth is that the chance that a BigCo would steal an algorithm from your one-man-paint-app's minimally obfuscated JS is extremely close to zero (excluding if it was MIT/BSD OSS license, and even then I would be pretty surprised if it happened). Anything half as novel as you are describing would be better leveraged by talking about some of it as siggraph and then acqui-hire selling your company for millions to a Adobe/Microsoft/Apple type.
It makes perfect sense to not open source it, but if you are trying to build this as a business any amount of time you second you spend trying harden your source from some hypothetical reverse engineering boogeyman (that is likely to never even be trivially attempted) is time that would be much better spent on marketing or improving your product.
Almost every country on this planet has copyright laws.
If you're worried about them ignoring the law and taking your source code why are you not worried about them ignoring the law and physically stealing your computers?
[0] http://docs.nwjs.io/en/latest/For%20Users/Advanced/Protect%2...
Running JavaScript through an optimizing compiler and minifier will obscure pretty much the same information. But either way, you can step through the code in your browser's debugger to see what it does.
ActiveX didn't even have a sandbox, so not even related.