carlopi

```sql FROM 'https://raw.githubusercontent.com/hannes/fortytwodotparquet/...' LIMIT 10; ``` (or online: https://shell.duckdb.org/#queries=v0,FROM-'https%3A%2F%2Fraw...)

Here it is: https://carlopi.github.io/duckdb-wasm-unsigned////#queries=v...

That is, the duckdb-wasm web shell with loaded PSQL extension

DuckDB-Wasm, with extension enabled. Execute SQL queries within a browser tab, and share the scripts as a link.

Btw, it's already happening:

Go to https://shell.duckdb.org, and type

FROM 'https://hacker-news.firebaseio.com/v0/item/37663308.json';

Querying hacker news, from a browser tab (passing through a bunch of database and Web technology that make it possible for DuckDB to be executed within a browser tab)

Thanks, cool read, I took a couple of ideas, will see where we land.

I'd say find an angle that adds value for you (ideally both since you want the answer AND you can benefit from learning) random ideas can be: * C++: extension to interface with GitHub workflows results (say how many times job X passed) * JavaScript: browser-based library to visualize data * Python: crunch geospatial data (say your own GPS data) * ...

I just discovered Hacker News offers a JSON API, ducking awesome, if only there was a nice tool to crunch data as JSON files...

I think I have just found out yet another great DuckDB-weekend project!

Not sure whether they classify as obscure, but I haven't see cited already:

- Dominator tree (https://en.wikipedia.org/wiki/Dominator_(graph_theory))

- Single-Connected-Components-Graph

- Deterministic data structures (eg. a set that acts deterministic to the fact that addresses might be somehow randomly assigned, very useful for ensuring reproducibility)

Already cited, but it's clearly among the most elegant:

- union-find (!!!!)

and as a bonus one that is easily overlooked:

-std::deque, that when restricted to push_back() or push_front() guarantees not to ever move objects around.

I have to double check this, but the approach should be theoretically sound since we are quite strict with what functions are considered to have known call-sites: Function has to be internal (as in no visibility from outside) + no indirect uses (so no saving a pointer to the function somewhere). This should be enough to solve the problem you are thinking about.

I am not completely sure of what "classical" partial evaluation is, but probably yes, this is somehow a special case of it. I have now quite some material to read (see other links about partial evaluation or super-compilation).

Take this other example: https://gcc.godbolt.org/z/nebP68Tx8

Here by playing HumanCompiler it should be possible to prove that the if condition never evaluates to true, so removing the if is safe.

This is an example of optimization that PartialExecuter is able to do. Note that some combinations of other optimizations might also be potentially able to get to this result (say adding a tag "is always power of 2", but doing this in a general way it's what PartialExecuter does well).

Somehow similarly, clang might be instructed to enable a check to avoid including printf_float and somehow detect it and exclude it (this is what happens here), but this is hardly generalizable.

Devirtualization is always plenty of fun + has lots of potential. Added to the list of article to read.

Congrats, later will check & take inspiration!

> My respect to the entire team.

Same!

Yes, I wanted to be more nuanced but oversimplified. Thanks for mentioning this!

The step forward that I believe worked well here is that given some actual paths (and since they are simply a sequence of Instruction, they can be fed to an interpreter) they are merged or forked while entering or exiting SCCs, this allows to keep a low count of actual visit but exploring a big enough set of path to prove that it covers all possible executions.

On the second part, I have to do some thinking and coming back, thanks!

PartialExecuter happens at the LLVM's IR level, and in theory it's fully generic. Then has been only partially tested outside the Cheerp-pipeline, so I would expect it to rely on some implicit assumptions on the kind of legalizations or lowering that are done before-hand.

Target-wise, all information is kept encoded in the IR, so that part is easier (the only strange thing that might happens is bumping some alignment to 8, but I expect every target to be able to handle the prescribed IR alignemtn)

Thanks a lot!

I also believe it would be cool to upstream this, we will have to sit down and do some planning.

Compilation time could improve (I was actually working on this today), but it's already in line with other optimizations, taking < 10% of the time spent doing optimizations on a big codebase we use as benchmark.

Currently it's applied to all functions, since runtime it's anyhow somehow linear in the number of Instructions a Function has, but possibly in more costly versions of this (that we have on paper but yet to implement) some logic to filter functions in advance could be used.

WebAssembly implies static linking (malloc / printf & all are part of the shipped module) and code size matters since it directly influence users (since there might be delays in downloading big payloads). Both factors plays a role in deciding to plan putting work in optimizations like this.

There are some general gains to be had with this optimization, and probably the same ideas were already around for a while, but putting together in this way was helped by thinking about WebAssembly specific constraints.

A subset of this could possibly be applied at the wasm-opt level, but consider that at the LLVM's IR level there is more information to be leveraged (in particular PartialExecuter uses information on what memory ranges are read-only).

In general the whole concept behind Cheerp (C++ to WebAssembly + JavaScript compiler) is doing as much as possible at the LLVM IR level, JavaScript concept included, since it's easier and more powerful to do code transformation there.

Hi, author of the post here, I am not familiar with this concept, do you have any pointers?

Hi, I wrote the article, and I am somehow to blame for PartialExecuter, looking forward to any feedback and questions!

Pure JS unrolling + bit manipulations lead to roughly 40% speed up, using this function:

  function add() {
    var i = 0, d0 = 0.0, d1 = 0.0, d2=0.0, d3=0.0;
    for (i = 0|0; i < N; i = ((i|0)+(4|0))|0) {
      d0 = a[i^0] + b[i^0];
      d1 = a[i^1] + b[i^1];
      d2 = a[i^2] + b[i^2];
      d3 = a[i^3] + b[i^3];
      c[i^0] = d0;
      c[i^1] = d1;
      c[i^2] = d2;
      c[i^3] = d3;
    }
  }

Could you try it & possibly adding this (possibly instead of the slower unrolling options?)

Author here, questions are welcome!

Do you have any single example where Rust+WebAssembly are 100x (or even 10x) faster than JavaScript?

I did got to the page you linked, but also there is no actual "proof".

I am highly skeptical of any claim that anything can be 10x-100x faster that JavaScript. Do you have any concrete example where a similar program coded in (decently written) JS is that slower? I work on a compiler from C++ to JavaScript & WebAssembly (Cheerp), and I have basically never saw any such a difference.

Thanks, I wrote the article, ask away!