Hacker News Reader

Comment by css | Hacker News Reader

Judging by the example string on the website, `LEHV6nWB2yk8pyoJadR*.7kCMdnj`[0], you cannot use these hash strings as filenames because they can have disallowed characters.

You could save a step by just sending which image to download and use the filename as the hash you use to render the result, but this algorithm requires you store the relationship between the hash and the image somewhere (unless I am missing something obvious).

[0] source code: https://github.com/woltapp/blurhash-python/blob/7469c813ea64...

View on HN

1. Using the hashes as principal indices seems unsafe in general (which includes image upload) anyway: it looks pretty simple to generate collisions. The suggested use case (save the hash in the database next to the original image or a reference to it) sounds fine.

2. Which disallowed characters? The dictionary appears to be "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz#$%*+,-.:;=?@[]^_{|}~". No slashes or null bytes, which are the common disallowed characters on server platforms. It does have characters your shell cares about, but that's only a problem if you're not quoting.

`:` is disallowed on MacOS and several of those characters are disallowed on Windows. Generally I view images on a frontend, not on my servers.

Fair! I would not expect the image be saved to a Windows path when displaying a blurred image in the example use cases (mobile apps and web pages). You're thinking Electron, perhaps?

Point of order though: I don't think that's right for MacOS, though you're certainly right for Windows. HFS+ lets you use any Unicode symbol including NUL (because the filenames are encoded Pascal-style). I don't know the details how how APFS does it, but it also appears to support any Unicode code point, though it additionally mandates UTF-8. [0]

[0] https://developer.apple.com/library/archive/documentation/Fi... grep "filenames"

Interesting, I was not aware about MacOS. If I run `touch :.:` it will run fine and create the file, but Finder disallows this for whatever reason:

    The name “:.:” can’t be used.
    Try using a name with fewer characters, or with no punctuation marks.

Finder exchanges : and / in filenames, for compatibility with classic MacOS, which used : as a directory separator and allowed / in filenames. You can create a file named / in Finder, and it will show up as : on the file system.

A file name with an embedded NUL character couldn't be passed to fopen(), or to any other function that takes a pointer to a C-style string. (Not directly relevant here.)

':' works fine in MacOS file names for me.

: was disallowed on classic MacOS. On modern macOS, it is allowed in the POSIX layer just fine. However, Finder and other parts of the UI will translate a : in the file name to a / on display, and vice versa.

This is because classic MacOS used : as a directory separator instead of /, and this behaviour is preserved in the UI, but obviously not in the underlying layers.

Is it not present in the underlying layer anymore? Applescript displays paths with : separators unless you specifically tell it you want a “POSIX path”. (This is actually really annoying.)

No, the underlying layer is POSIXy, and uses /. The highest UI layers do the mapping between : and /, and apparently also AppleScript.

Simple solution: run a POSIX-compliant OS on the frontend. All Linux distributions, all BSD distributions, modern macOS, and Windows Subsystem for Linux all support ':' in filenames.

They mention it's a custom version of base83: https://github.com/woltapp/blurhash/blob/master/Algorithm.md...

With this charset: 0123456789ABCDEFGHIJKLMNOPQ RSTUVWXYZabcdefghijklmnopqr stuvwxyz#$%*+,-.:;=@[]^_{|}~.

I imagine you could tweak that to replace problematic chars with safe substitutes.

These are the safest I could find. They are perfectly fine for using in filenames.

That depends very much on the platform ...

Well, Windows is very picky, yes.

Windows "A file name can't contain any of the following characters":

    \ / : * ? " < > |

which means "*:|" would need to be removed if you want to support that use case

> you cannot use these hash strings as filenames because they can have disallowed characters.

You can't really use them as filenames because they won't be collision free, too. They're not meant to be an identifier, but instead a compact representation of the image that can be stored in a database.

This is a tempting idea, but as was already pointed out, the problem is not the character set, but that similar images will encode to the same string.

Anyway, chances are you are already storing a filename in your database, so you would need a second field for the blurhash string.

  assets/example.png/LEHV6nWB2yk8pyoJadR*.7kCMdnj
  assets/example.jpg/LEHV6nWB2yk8pyoJadR*.7kCMdnj

s/g\//g#/

    assets/example.png#LEHV6nWB2yk8pyoJadR*.7kCMdnj
    assets/example.jpg#LEHV6nWB2yk8pyoJadR*.7kCMdnj

No need to send the extra data to the server for each request.

Hm, that's an interesting choice. I also wonder why they didn't use an encoding that has an alphabet with 2^n characters (that way you can directly map 1 charater to n bits when decoding).

Coincidentally, I've just finished some work on a project[1] that is in the same space (identifiers for images).

For the reasons you pointed out, I found Douglas Crockford's base32[2] encoding to be a good fit.

[1] https://github.com/pablosichert/ciid

[2] https://www.crockford.com/base32.html

That choice is explained in the README: https://github.com/woltapp/blurhash

(83 is about as many safe characters as you can reasonably find, and it allows some nice ways of packing values together.)

Could you explain what "AC components" refers to? I couldn't figure that out just by reading your README.

I wonder how the efficiency compares to just encoding on the bit level.

It is a term often used for DCT-transformed data. DCT, in this case, breaks the image down into basically an average colour of the whole image, referred to as the DC component, and a bunch of waves that make up the detail of the image, referred to as AC components.

https://github.com/woltapp/blurhash/blob/master/Algorithm.md

> Could you explain what "AC components" refers to?

All but the first component of the Fourier transform. (The first component is the average of the data.) The term comes from electrical engineering, but Fourier transform has lots of applications also outside of electrical engineering.

I quite like that idea. Maybe you could run base64/base32 on the blurhash. That brings two potential issues I could think of. Is the size of the result less than filesystem max file name length? And what about collisions where two slightly different images have the same blurhash?

As I mentioned upthread: none of those are disallowed on common server platforms, but ironically canonical Base64 _will actually_ break, and it'll break rarely enough that you're likely to miss it in testing! (The last character in canonical Base64, for symbol 63b10, is /, which is disallowed on (*nix) filesystems and does not occur in the native BlurHash dictionary. Unless you're suggesting encoding the encoded format, I guess.)

I accidentally made a folder (in my pinephone's / folder) the other day called '', and I was about to delete it with rm -rf .

Maybe that shouldn't be allowed, but it is :P

If you want to do that just use a different encoding which outputs a string safe to be used for filenames.

This one is picked very carefully to be as safe as reasonably possible with this number of characters. As was pointed out elsewhere, it is actually safe for filenames.

Reply on news.ycombinator.com