Hacker News Reader

Comment by tyho | Hacker News Reader

Be warned that this reads everything until EOF into memory, doing something like:

    hashpipe QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 < large_file

will produce unexpected behaviour.

View on HN

So too much in the pipe and things get wonky? Still a perfect name!

It doesn't have to read into memory; if that becomes a problem they can fix it. The hashing algorithm just needs a small, constant amount of state. While it is hashing, it can divert the data to a temporary file. Then it can just pass the file to its standard output.

A "hashexec" command could also be produced which passes the name of this temporary file to a subprogram:

   whatever | hashexec <hash> arbitrary --command with args

This "arbitrary" command's standard input is redirected from a temporary file created by hashexec (so no cat-like loop has to execute to feed the data).

Also, since this is for scripts, there could be an argument which limits the size. This could have default value, say one megabyte. Anyone pulling down scripts which are anywhere near one megabyte has to add explicit overrides for the size:

   whatever | hashpipe --max=4M <hash> | ...

The real problem isn't that hashpipe buffers everything, because the scripting language running the script will also do that; rather that because it buffers everything, it can be DDoS'ed with an infinite stream or whatever.

By incorporating a size limit, hashpipe could provide an additional protection measure to the next pipeline element: it protects against content which doesn't match the hash, and against content which is too large.

Idea: the size of the input could be encoded as a few digits of the hash. Then the argument is unnecessary. The hash itself tells you that the script is exactly 6059 bytes long; if you read 6060 bytes, the input is not the right one. Or the program could just stop reading at 6059 and check the hash at that point, and either pass on the 6059 bytes or error out.

All great ideas! mind adding some to https://github.com/jbenet/hashpipe/issues ? over time we can implement some

I don't use github; I have my own server with git and cgit.

yeah, hashes are decided by every single bit including the last one, and not a single bit should be output until the hash matches. It currently buffers everything in memory, but might do this: https://github.com/jbenet/hashpipe/issues/1

(some settings dont have disk though).

hashpipe is intended for most executable use cases (usually under <50MB)

Loading everything into memory at once shouldn't be necessary to produce a hash of the entire input. All of the hash functions currently supported allow for incremental hashing. That means you can hash in blocks instead of all at once.

The input still needs to be cached for eventual output in case the hash matches I assume.

In that case, you could still cache if you want but at least you now have the option to manage your memory instead of always crashing with large input

More importantly, so nothing is output if the hash doesn't match.

You should be able to detect if the stream is seekable (by checking the result of 'lseek(fd,0,SEEK_CUR)') and only buffer if it's not.

Of course, if you're really paranoid, the file could get changed out from under you. But honestly you're probably screwed either way with an attacker who can do that.

Another fun option would be to use a tree hash. The distributor of the content-to-be-verified would then envelope it in a format (to-be-defined) that includes proofs-up-to-root every N bytes. Then the verifier could stream, and know that everything it emits fits into the target hash, needing only N bytes of working space.

Caveats:

The source doing-the-enveloping will need two passes (and enough working space for the remainder-tree).

An attacker could still choose the moment-when-content-goes-bad; in the envisioned use of immediately-executing the verifier output, this might leave things in a problematic/resource-consumptive state. (Scripts could be hardened against such partial-execution failures.)

Reply on news.ycombinator.com