nigeltao

Thanks for the feedback. I'll add better build instructions.

If you just want the jsonptr program, instead of everything in the repo (the Wuffs compiler (written in Go), the Wuffs standard library (written in Wuffs), tests and benchmarks (written in C/C++), etc) then you can use "build-example.sh" instead of "build-all.sh".

  ./build-example.sh example/jsonptr

For example/jsonptr, that should work "out of the box", with no dependencies required (other than a C++ compiler). For e.g. example/sdl-imageviewer, you'll also need the SDL library.

Alternatively, you could just invoke g++ directly, as described at the very top of the "More details are at [link]" page in the grand-parent comment.

  $ git clone https://github.com/google/wuffs.git
  $ g++ -O3 -Wall wuffs/example/jsonptr/jsonptr.cc -o my-jsonptr

jj is faster than jq.

However, jsonptr is even faster and also runs in a self-imposed SECCOMP_MODE_STRICT sandbox (very secure; also implies no dynamically allocated memory).

  $ time cat citylots.json | jq -cM .features[10000].properties.LOT_NUM
  "091"
  real  0m4.844s
  
  $ time cat citylots.json | jj -r features.10000.properties.LOT_NUM
  "091"
  real  0m0.210s

  $ time cat citylots.json | jsonptr -q=/features/10000/properties/LOT_NUM
  "091"
  real  0m0.040s

jsonptr's query format is RFC 6901 (JSON Pointer). More details are at https://nigeltao.github.io/blog/2020/jsonptr.html

I'd be curious to know if SIMD cares about the big-endian design decision. You can always do SIMD shuffles near where you load and store the bytes, but it'd be simpler and probably faster if you don't.

Similarly, BSWAP / MOVBE might be cheap or free on x86_64 but IIUC RISC-V doesn't guarantee a dedicated instruction for that (and RISC-V is little-endian). "Does [endianness] really matter?" It might, for embedded devices a few years from now. I can't really say without real hardware to get real CPU profiles. But that question is entirely avoidable by just picking little-endian.

That's how WUFFS (Wrangling Untrusted File Formats Safely) works:

https://github.com/google/wuffs#what-does-compile-time-check...

> The point remains though that you need to somehow get both the encoder side and the decoder side upgraded to benefit from the advantage,

Well, for PNG, Apple have shipped exactly that. It was something they could unilaterally do without e.g. having to get all of the major browser makers on board.

Apple's PNG encoders (i.e iOS dev SDKs) produce multi-thread-friendly PNGs and their decoders (i.e. iOS devices) reap the benefits. And unlike trying to introduce a new but not-backwards-compatible-with-PNG format (JPEG-XL), other PNG decoders can still decode these images. They just don't get the speed benefit.

> Any system that has to take arbitrary png as input cannot rely on MT decode being an option, etc.

It's forwards compatible too. Apple's PNG decoder will use MT decode if the PNG image was encoded with MT metadata, but Apple's PNG decoder can still decode arbitrary PNG (using its pre-existing single-thread code path).

> I think you must be doing something very wrong if you measure less than 1 Mpx/s for fjxl. It should reach at least 50 Mpx/s or so,

In case it wasn't clear, the 0.630 fjxl encode speed number in the top level README.md in that commit is after normalization such that QOIR is 1.000. After all, absolute numbers are hardware dependent.

If you look at the doc/full_benchmarks.txt change in the same commit I linked to, fjxl encode speed clocks at 106.48 megapixels per second. It's just that QOIR encode speed clocks at 168.90 megapixels per second (and fpnge at 312.59).

> Did you compile fjxl with or without SIMD?

I had not. Thanks for the tip. Fjxl encoding speed is now about 20x faster than libpng, not 10x, but still mid-table. Decoding speed is unaffected.

https://github.com/nigeltao/qoir/commit/c2fec840

> If existing deployments need to be replaced anyway, you can just as well use a new format altogether.

There's still an operational difference between rolling out multi-threaded PNG versus multi-threaded $OTHER_FORMAT.

At the file format level, MT PNG can be designed to be backwards compatible. Older PNG decoders simply ignore the new non-critical chunk.

At the software level, you can roll out encoders that emit MT-capable PNGs without having to also roll out new decoders everywhere (or 99% everywhere, or 'on all major browsers'). You can roll out new decoders only partially, and those upgraded places enjoy the benefits, but you don't break the decoders you don't control.

Also, in terms of additional code size, upgrading your PNG library from version N to version N+1 is probably smaller than adding a whole new $OTHER_FORMAT library.

I agree that compression ratio is the primary metric (but not the only important metric). I just found "[better] in all ways", explicitly calling out encode speed, incongruous with simply ignoring decode speed.

As for which is more important out of encode and decode speed, I would personally rank decode speed higher. For example, app icons on your home screen were encoded once (by the graphic artist) but are decoded zillions of times (by every user every day).

fjxl is the "JXL_Lossless/f" lines in my full_benchmarks.txt link. It's about mid-table in terms of encode speed. I clocked it at about 10x faster than libpng, not 100x faster.

If you look solely at encode speed, fjxl loses to fpnge (also listed in full_benchmarks.txt) so I'm not convinced yet that JPEG XL beats PNG in all ways.

I believe that Apple have an unofficial PNG extension that facilitates multi-threaded decoding: https://github.com/w3c/PNG-spec/issues/45

IIUC, the quality knob (not the effort knob) isn't relevant for lossless encoding.

Knobs are also less relevant for lossless decoding, where there's only one correct output for any given input.

In any case, my code (https://github.com/nigeltao/qoir/blob/main/adapter/jxl_adapt...) copy/pastes examples/encode_oneshot.cc from the libjxl repository, plus an additional JxlEncoderSetFrameLossless call.

So I'm using whatever knob values that the libjxl example uses, i.e. the default knob values. If there are better knob values for lossless, let me know.

From the "Lossless Compression Performance" section of the article:

> JPEG XL can do lossless image compression in a way that beats existing formats (in particular PNG) in all ways: it can be faster to encode, produces smaller files, and more...

Does "in all ways" include any decode (not encode) speed numbers? Such numbers are dependent on the test suite, and I might be holding it wrong, but on my measurements, JPEG XL lossless encodes 10x slower and decodes 12x slower than PNG (the libpng implementation), and decodes 30x slower than PNG (the wuffs implementation). JPEG XL does admittedly win on compression ratio (smaller files), though.

https://github.com/nigeltao/qoir/blob/main/doc/full_benchmar...

Here's another SIMD implementation, with commentary: https://github.com/google/wuffs/blob/main/std/adler32/common...

Like the fpng implementation, it's SSE (128-bit registers), but the inner loop eats 32 bytes at a time, not 16.

"Wuffs’ Adler-32 implementation is around 6.4x faster (11.3GB/s vs 1.76GB/s) than the one from zlib-the-library", which IIUC is roughly comparable to the article's defer32. https://nigeltao.github.io/blog/2021/fastest-safest-png-deco...

https://github.com/facebook/zstd/tree/dev/doc/educational_de... is a self-contained zstd decoder. I get a 64 KB dynamically linked executable after running "make" in that directory.

  $ size harness
     text    data     bss     dec     hex filename
    24615     704      40   25359    630f harness

I agree that it's similar but ultimately different problems. Copy/pasting from https://github.com/google/wuffs/blob/main/doc/related-work.m... gives:

> Kaitai Struct is in a similar space, generating safe parsers for multiple target programming languages from one declarative specification. Again, Wuffs differs in that it is a complete (and performant) end to end implementation, not just for the structured parts of a file format. Repeating a point in the previous paragraph, the difficulty in decoding the GIF format isn't in the regularly-expressible part of the format, it's in the LZW compression. Kaitai's GIF parser returns the compressed LZW data as an opaque blob.

IconVG author here. Happy to discuss.

Animation is issue #2 on https://github.com/google/iconvg and I have some ideas but no code yet. I'm also midway through changing the current "version 0" format into a "version 1" format, dropping things like the ArcTo op (inspired by SVG) precisely with one eye on (future) animation support. The ArcTo large-arc-flag, like any boolean-typed value, is impossible to interpolate smoothly.

> Several times the compression/decompression speed of gzip

Just be careful that you're comparing against the best implementation of gzip. One recent re-implementation of zcat was 3.1x faster than /bin/zcat (and the CRC-32 implementation within was 7.3x faster than /bin/crc32). Both programs decode exactly the same file format. They're just different implementations. For details, see: https://nigeltao.github.io/blog/2021/fastest-safest-png-deco...

JWCC (JSON With Commas and Comments) gives you exactly that, without other things that are arguably a mistake.

https://nigeltao.github.io/blog/2021/json-with-commas-commen...

See JWCC (JSON With Commas and Comments) https://nigeltao.github.io/blog/2021/json-with-commas-commen...

> Only two levels of detail per file

There are multiple levels of detail per file. Each LoD is specified by two numbers: a lower and upper bound on the height-in-pixels that enables the subsequent drawings.

Hi. IconVG author here. Thanks for the attention.

It's still a work-in-progress and I'm actually considering some major file format changes in the coming weeks. As some of you have noticed, the docs aren't complete. I wasn't expecting to hit Hacker News today.

The primary goals are simplicity and security. The file size benefits are a bonus.

Why not .svg.gz? Well, if that (or another format) works for you, that's great. But implementing SVG needs a lot of code (the SVG spec is 400 pages long) plus dependencies like XML, JavaScript and Freetype/Harfbuzz (or equivalent) libraries. It's practically impossible to do a thorough security audit over a complete implementation.

For example, https://github.com/flutter/flutter/issues/1831 says "We don't want full SVG support. There is too much in the spec that is expensive, heavyweight, and/or duplicates what we already have in Flutter."

> Do you accept PRs for the Rust benchmarks?

Sure! I'll take a PR that adds script/bench-rust-png-decoder to the Wuffs repo.

I'll also take patches to script/bench-rust-png. I'm not as fluent in Rust as I am in C/C++ or Go, so I might easily be doing something silly there.

> I'm curious if it is at all competitive with the Wuffs decoder

You could try adapting https://github.com/google/wuffs/tree/main/script/bench-rust-...

> https://github.com/bschwind/png-decoder

Note that statements like:

let bytes_per_scanline = ((header.width * header.bit_depth as u32 * header.color_type.sample_multiplier()) + 7) / 8;

can silently overflow a u32 if width is 0x800_0000, bit_depth is 8 and sample_multiplier is 4. Part of the point of Wuffs is to catch things like that (at compile time).

Thanks for the kind words!

> smhasher implements both

Ah, I'm based on aappleby/smasher, which is what the original location (http://code.google.com/p/smhasher) redirects to. I didn't realize that you have a more active fork.

> not just 47x faster, but 5000x faster

It's 47x faster: https://github.com/rurban/smhasher/issues/189

Sorry, that part is relatively new and poorly documented.

Choosy is sort of vtable-ish. It selects between method implementations (which all have the same signature), typically based on cpuid (e.g. if the CPU is SSE4.2+PCLMUL capable, set foo = foo_x86_sse42; if not, use the default foo impl).

up (the private method) just implements update (the public method). Only private methods can be choosy.

That could get to parity with "Fastest", but it wouldn't match it with "Safest".

> Looks like there isn't an encoder for PNG

There isn't one yet, but I'd like to have one at some point.

https://github.com/google/wuffs/blob/main/doc/roadmap.md

> programmers must manage pointer offsets to each heap structure

I'm sorry but I don't understand this statement. Can you re-phrase it?

> I would be more interested how they plan to handle IO effects or if they want to omit that.

I'm not sure exactly what you're asking for, but see the Coroutines, Effects and I/O pages in https://github.com/google/wuffs/tree/main/doc/note

> Are there any compiletime benchmarks?

It used to take a couple of seconds to compile Wuffs' standard library. It's back down to a few hundred milliseconds.

https://nigeltao.github.io/blog/2020/dumbindent.html#compila...

Yeah, that's a renderer for the IconVG file format. As I've said previously:

IconVG is a compact, binary format for simple vector graphics: icons, logos, glyphs and emoji.

IconVG is similar in concept to SVG (Scalable Vector Graphics) but much simpler. Compared to "SVG Tiny", it does not have features for text, multimedia, interactivity, linking, scripting, animation, XSLT, DOM, combination with raster graphics such as JPEG formatted textures, etc.

The announcement is at https://groups.google.com/forum/#!topic/golang-nuts/LciLeI5p... and quoting from that:

"The SVG spec prints as 400 pages, not including the XML, CSS or XSLT specifications. [The IconVG godoc.org page, which includes the file format specification, prints as 26 pages]...

The Material Design icon set... consists of 961 vector icons. As SVG files, this totals 312,887 bytes. As 24 * 24 pixel PNGs, 190,840 bytes. As 48 * 48 pixel PNGs, 318,219 bytes. As IconVGs, 122,012 bytes."