Hacker News Reader

Comment by unilynx | Hacker News Reader

More importantly, we need to get rid of the ability to put line feeds, tabs in file names and also disallow odd starting characters such as tab, dash and $

I wish someone would add a mount option for that and have eg fedora be a trailblazer to fix the few apps that break

View on HN

Nah.. we need to use object graphs as streams instead of whitespace "(un)parsable" text. The output to the console (ui) or gui (ui) can be different, but the data should be structured

Yes, exactly. A number of newer shells take this approach. The one I wrote pipes File objects out of its ls command: https://marceltheshell.org

looks great

Sounds like Powershell to me. I'm down, as long as the syntax is as simple and terse as on UNIX-based systems and not what Microsoft did (were they paid by the character for flag names?)

Absolutely. For example: why can't "Get-TrustAuthorityKeyProviderClientCertificateCSR" simply be "takpccc" as $DEITY intended?

If your keyboard's tab key is still legibly labelled then you aren't trying hard enough or have an eidetic memory and fast typing skills!

Amusingly HN cuts off the end of the command you typed, I assume using css overflow attributes (don't have an easy way to tell on my UA). I assume it stops at "cate"[0]. I see this sort of chopping a lot, which naturally makes sharing PS commands frustrating -- although there may be workarounds like using `backticks`.

0: Nope, had to paste it to see it ends with "cateCSR".

That is basically integrations, there is never going to be nice integrations to my Cobol mainframe linked to a Springboot fuzzbuzz. As is stated in other comments the big issue is usually about being cross platform, and that is a subset of the ls problem: Most of the time you have control over your inputs, until you haven't. This is true for every language even Python which is obnoxious about that. What I mean is that you will always hit edgecases in integrations and you never have time to write new ones.

I always felt that powershell was tab unfriendly, the Get- prefix is hard to get used to. I may be wrong that they have a good way to deal with one-off integrations in a sane manner.

They could at least change the names order and start with the specific part (TrustAuthorityKeyProviderClientCertificateCSR-Get), so the (braindead) MS version of tab completion would be useful.

Powershell is usually terse enough as one uses aliases for interactive? (Not to mention tab completion)

E.g list files

Shell: “ls” Powershell: “ls”

Show sizes of files in size order Poweshell:

ls | sort length | select length

in Unix:

find -maxdepth 1 -type f -printf '%s\n' | sort - n

Lovely.

I use the long form stuff for scripting in Powershell (tab completed in the editor) but it’s not like anyone writes “Get-ChildItem” instead of dir/ls/gci.

If you do this, this will break every program which takes text based filenames on command line.. which is most of ghem. It is an interesting idea, but I don't think it would be Unix anymore.

'Unix' is too low level anyway.. Unix is about reading/writing byte streams..

I don't think the "interactive shell" was meant for scripting anyway. It's like writing your scripts in selenium similar tools. Someone only needs to change the structure or order of the webpage, and you have a problem, depending on how you do your scraping / interacting with the output.

No experience with powershell, but sounds great.

I dont think unix was every truly about byte streams.

  Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".

Is the core point. Later editions went on to specify text as the preferred language for these programs to communicate in but I don't think that's key to upholding the unix philosophy. It was just the easiest to work with at the time

We just need to agree upon a common framework for these programs to communicate with. There will definitely be a lot of churn though

> Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".

That's a quote from '78, by the Doug McIlroy, the inventor of Unix pipes. Pipes are exactly that... reading and writing bytestreams.

Also him:

- (ii) Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.

Later:

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

(https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s...)

I mean come one. "ls" grew from a simple tool with 11 parameters to 58 parameters. source: https://news.ycombinator.com/item?id=29568042 / https://danluu.com/cli-complexity/

This only happens because people are scripting in their ui. They shouldn't. "Unix admins" laugh at people who do the exact same things within office or other GUI solutions.

Using "text streams".. yes, for performance, a stream is better. Same with SAX vs DOM. nobody likes SAX

This can work with something like nushell, but obviously breaks the entire current universe of coreutils.

In the normal world we can solve this problem without breaking everything by adding --jsonout or similar to all the coreutils and then we can have sanity by piping to jq.

Not necessarily though, as filenames aren't required to be valid strings, so that would break json syntax. And json doesn't have a syntax for "just a blob of bytes", besides the fact that wrapping bytes in text just to be decoded back to bytes seems silly to me, but that's an opinion

> This can work with something like nushell, but obviously breaks the entire current universe of coreutils

Good, because these utilities suck. Half of them only exists because the data is unstructured in the first place, the other half are mostly made of parameters that only exist for the same reason, and most of their names have no apparent relation to what they do. It is time to move out of the 1970s.

Hi. Author of Next Generation Shell here. Totally agree. Also UI of the shell is stuck and ignores pretty much everything that happened in last decades.

Here is my plan for the UI: https://github.com/ngs-lang/ngs/wiki/UI-Design

Edit: but I do try to keep interoperability with existing bullshit.

> use object graphs as streams

graphs aren't streams. You could, of course, serialize an object graph to a stream (such as by reducing it to linear text representation of the objects with IDs and links.)

Why? The name of a file is none of the filesystem’s business. If users choose names that make using software difficult it’s on them. It’s not like there aren’t ways to handle any kind of “weird” character in a file name, as the linked article states.

Furthermore, if the kernel/filesystem starts prohibiting certain characters this is more code to maintain and test. User space programs that previously worked fine will stop working. All of this just to prevent someone shootings themselves in the foot by misunderstanding how filenames should be manipulated.

If you’re looking at file system in the purist possible sense, you’d be right. But equally a lot of other meta data is stored beyond the inode that file systems “understand”. And there is already precedence of file systems having code to intelligently handle file names (eg options for case sensitivity). So pragmatically it’s not an unreasonable suggestion to house any code defining legal file names in the file system driver too.

The biggest argument against that in my view isn’t down to testing but DRY methodologies: if the code sits in the kernel then it should work against all file systems and not just supported ones.

In practical reality, the name of the file is the filesystem's business. It would be nice if it operated similar to cloud filesystems, where you could version based on GUID that is disconnected from the filename, but the practical reality is that users and developers have long accepted the local operational mode.

Try and name a file "(//^-^//)"

You can't because certain characters are prohibited.

I already can’t use NUL and slashes in a file name. And win32 limits me even more. It’s always been a compromise

And the amount of feet that have been shot by weird file names is staggering.

Programs will stop working, but that’s why we need a bleeding edge distribution to find them. In the short term things will break, in the long term quality will increase. Just like memory protection broke some DOS apps in the short term

I hope you won't be in the position of handling the non-ascii file names. Whitespaces, symbols and other complicated glyphs are widely used in file name since Windows 95.

Or... ls could just escape filenames.

I'm not sure why it seemingly doesn't in the year 2021.

ls -Q

--quoting-style=xxx

From “man ls”:

    -b, --escape
        print C-style escapes for nongraphic characters

I'd also add:

    -1  list one file per line. Avoid '\n' with -q or -b

to make sure you can easily split the list by just the EOL, in case for some reason it thinks it is talking to a terminal and tries for format things for a human.

Thats not a bad suggest per se but it would only work for Linux and thus you still have other POSIX systems that wouldn’t follow suit.

So the advice here of not parsing ls is still prudent.

It would be almost trivial to create a fuse filesystem which completely hides these files if they exist (and doesn't allow the creation of new ones).

And how would that be even remotely useful? Unless something changed recently, FUSE has so much overhead it's only useful for niche applications and prototyping.

It would be trivial to code but any such wrapper would add overhead to file system operations. FUSE is a fantastic set of APIs (I’ve used it personally) and performs remarkably well considering it is constantly swapping memory between kernel and user space but for wide spread adoption any feature like this would need to be part of the native file system options.

A thousand times this. There is absolutely no reason to allow newlines in filenames, and it is pathetic that there isn't even yet a mount option to disallow totally idiotic filenames (at the minimum I don't want programs to create filenames with newlines or invalid utf-8).

It's great that file names in the user/kernel ABI are treated as unstructured NUL terminated byte streams so I can do what I want with my file names even if you don't like it. And you can do what you want with yours, including not creating ones you think are idiotic, or using filesystems with code or options that restrict what names can be used.

But is there any real use case for that? For me I've only encountered this when something else went wrong, I'd rather have an error at that time than later trying to find out what this garbage is and how to remove it.

So why not give a mount option for this behavior?

> But is there any real use case for that?

Compatibility, at least. Which is actually a big one and is basically never broken in Linux.

> So why not give a mount option for this behavior?

Not sure, maybe just nobody yet cared enough to code it up and submit it for inclusion. It's never caused me problems.

Compatibility with what? Is there any software that relies on this behavior?

> Compatibility with what?

With existing applications and filesystem images.

> Is there any software that relies on this behavior?

Possibly.

Possibly, so possibly even not. The kernel actually does "break compatibility" from time to time if there's no software relying on the behavior.

> Possibly, so possibly even not.

Possibly so.

> The kernel actually does "break compatibility" from time to time if there's no software relying on the behavior.

Certainly not something like this by default though.

To be clear, this wouldn't somehow solve shell / scripting / terminal issues with file names. There are many other special characters and escape sequences and other whitespace like spaces that can trip up incorrectly written programs. These can certainly not all be removed by the kernel so the incremental advantage of just filtering out a couple of such cases doesn't seem like it would be very big.

Can you give a plausible use case? Filenames can't be arbitrary bytes anyway, since they cannot contain '\0' and '/'. What's a realistic example where it's really useful to be able to stuff arbitrary bytes into a filename, just not '/' or '\0' and where C or URL escaping would somehow be onerous enough to justify all the other problems these pathological filenames create?

Do you really think disallowing pathological filenames (at least as a mount option) would be more expensive than the countless security exploits allowing them has already caused or massive tax nearly all software that tries to deal with filenames robustly needs to pay for it?

Forget shell scripting, almost no software can afford to just pretend filenames are arbitrary bytes.

They typically still somehow need to be displayed to and be editable by end users somewhere along the line, and this means (in unix-based systems) some conversion to-and-from utf-8. Which is going to cause problems[1].

And even if you don't directly need to handle this yourself (but you do, even for a simple shell script or command-line utility or a library that wants to provide an error message with a filename), there is now a whole lot of extra bloat and complexity and edge cases no one handles in practice. With weirdo types like special filename strings, which are neither bytes or proper unicode like python's unicode surrogate encoding (which effectively leaks into all text handling). And of course different languages and eco-systems solve it differently (e.g. whereas python bends its general unicode string for this, Rust has a OsString).

[1] Even the utf-8 compatible subset causes problems of course. E.g. if you have a terminal program that needs to display untrusted filenames to an end user, you now have to deal with problems like terminal escape injection via filenames.

Compatibility. And a mount option seems fine if you don't need compatibility.

That does not relieve applications of the requirement to robustly handle paths and file names though.

> Forget shell scripting, almost no software can afford to just pretend filenames are arbitrary bytes.

Much non-script software can actually treat file names as arbitrary bytes and just pass them through its typical input and output mechanisms. Shells and terminals are very special classes of application, and they need a lot of I/O sanitization whether or not the filesystem restricts file names.

Reply on news.ycombinator.com