Hacker News Reader

Comment by joosters | Hacker News Reader

Without the virtual file system, does this work by hashing the files? That would make it very slow to use on larger files. If I was to tag my movie collection, would TMSU have to load every byte of every movie in order to show the file tags?

View on HN

It seems that this also hashes the files to produce a fingerprint. Check out the fingerprinter [1] for the raw details on how it's done, but basically, if the file is below the "sparseFingerprintThreshold" of 5 MB, the whole file is hashed. Any file above is "sparsely hashed" where the first 512 KB, middle 512 KB, and end 512 KB are hashed and combined to produce the fingerprint for that file. It shouldn't produce any performance concerns on large files, so it should be safe to use for your movie collection.

[1] https://github.com/oniony/TMSU/blob/master/src/tmsu/common/f...

You may want to check out the repair command. It seems to show that TMSU keeps track of file locations on disk. If you move a file, it can find it if it hasn't been modified. If you move and modify a file, it can't find it and will mark it missing.

https://github.com/oniony/TMSU/blob/master/src/tmsu/cli/repa...

Ah, I see, thanks. The FAQ also explains some of the strengths and weaknesses of this approach:

https://github.com/oniony/TMSU/wiki/FAQ

I wonder why he doesn't use EA's where available (or does he?)

What is EA?

Probably Extended Attributes (http://en.m.wikipedia.org/wiki/Extended_file_attributes)

They stick better with files, as long as your tools know about them and handle them fine. Weakness is that they can (will) get lost in tools that do not. There are many of them: zip-unzip, tar-untar, http requests, ftp, etc.

> Weakness is that they can (will) get lost in tools that do not.

Including, frustratingly, Linux's NFS implementation. A WONTFIX apparently.

Ah. Also, depending on the FS, they are probably too limited, e.g. in size (only 4kb for ext2/3/4).

Reply on news.ycombinator.com