Hacker News Reader

Comment by xkcd-sucks | Hacker News Reader

Somewhat related, the question of "what IS a file's 'type' really" is pretty messy and AFAIK doesn't have any satisfying solution or any satisfying single answer [0]. A satisfying solution might look like a (long) feature vector. Libmagic goes in the right direction at least although the binary-fingerprint-at-the-beginning approach doesn't work well for everything / past some level of detail

[0] where different use cases might be satisfied by e.g.

    "a text file" 
    "a file exclusively comprised of alphanumeric UTF16 characters" 
    "a CSV file"
    "a file with tabular data and key-value metadata that calls itself CSV but is not spec-compliant so internally we'll call it CSVprime"
    "A file with tabular data written as ASCII text that represents $this_kind_of_measurement"
    ... etc

If you have any resources on this topic off the top of your head I'd appreciate it if you shared them

View on HN

GNU file doesn't just check magic numbers against libmagic, it also defines a scripting language and a series of tests/printers written in it.

That's what allows it to do complex things, eg identify all the flavors of ELF objects even though they share a magic number or determine if something is JSON or CSV without one.

The point is that, fundamentally, the concept of a file type is undecidable or not well defined.

Think about it: A JSON file can also be considered a text file. It could also be some higher level type of file, depending on whether it conforms to some application-specific JSON schema. Thus the kind of file it is has more to do with what you want to do with it; it's not some intrinsic property of the file.

Ok? I was replying to a comment asking about how such systems can work. GNU file is example of a program that makes a best effort to classify file type in a useful way.

A JSON file is also valid YAML.

> A JSON file can also be considered

A Curly braces Separated file filled with Values

so much of our mathematical theories are based on the idea that we have objects which are certainly of one type or another.

what would happen to type theory if, say, the type of an object has a probability attached to it?

I'm pretty sure the file command is not GNU btw, and I can't find anything about a GNU version.. do you have a reference?

https://en.wikipedia.org/wiki/File_(command)

My bad, I thought it was a part of coreutils and didn't check. I've only ever dug into the Linux utility and assumed.

Too late to edit!

https://www.darwinsys.com/file/

This is the authors website. Apparently yeah its not part of GNU utils, I had no idea, I knew it came with most Linux systems so I looked for the Debian package and found the site linked above.

https://packages.debian.org/bookworm/file

Not sure why it would think a module javascript file is java. Maybe just not updated very often.

    % file crawl.mjs 
    crawl.mjs: Java source, ASCII text

probably looks for "import" at the start

Edit: yep, the test for Java source is just /^import.*;$/

I just think of every novice who uses Java when referring to JavaScript, even software does it! Heh

> If you have any resources on this topic off the top of your head I'd appreciate it if you shared them

I wrote a Ruby library that attempts to be good at this https://github.com/okeeblow/DistorteD/tree/NEW%E2%80%85SENSA...

A file is "of a type" if there exists some agent (program X or human H) that creates/writes such files, and whereby there exists some agent (program Y or could be Y==X or human I or could be I==H) that reads/modifies such files, and whereby the entire contents of such files can be made use of or understood in a useful/meaningful way; and other files not of the type's contents either can't be made use of meaningfully nor updated. Some file types may be sub- or super-sets of other file types (ex: .C source files are .TXT files, but .TXT source files are not necessarily .C files, they could be .CSV files) And files could be streams.

might wish to discuss files containing type information, and data/value information both, thus numerous but different valued files can all be of the same type.

you can get fancier if you want and talk about valid C source files or C source files which contain bugs, etc.

you could consider a file "weakly or adhoc typed" if there is only one or a pair of agents that use that type, and "strongly or standard typed" if there are numerous "different/distinguished" agents that "rely" on that file type.

this was off the top of my head

Well, you can define a file's type in the same way as you define in-memory data structures, with a type system.

See my comment: https://news.ycombinator.com/item?id=38117891

A C struct can also be addressed as bytes if one really wants to. The structured way of looking at the data is still more valuable, most of the time.

Another example: you can easily grep through a JSON file sometimes. But what if that JSON doesn't contain newlines? In that case it would be great if the system knew how to convert the data to a multiline string that makes sense for grep. This cannot be done if the data is "just bytes".

Further, JSON data can be represented with less bytes if you know the structure.

Are you suggesting putting the (I assumed versioned) type system into the file's metadata, so that others can make sense of it? Or, some type system UUID, in the metadata, that can be used? I think anything like this would require many new concepts, like the move away from files being a collection of arbitrary stored bytes.

Well in theory if we are only doing an inheritance, we can just combine file extensions hierarchically in the name.

For example, for CSV files we can use .csv.txt extension, so if shell has no software for CSV files, it will treat them as a plain text, and if it has a software for viewing CSV files, but not for editing or printing them, it will add these verbs from records for txt files.

This is backwards compatible and would require minimal changes to file managers and graphic shells.

Reply on news.ycombinator.com