Hacker News Reader

Comment by artifaxx | Hacker News Reader

That is quite the obscure and interesting issue to run into! Who puts colons in their filenames though? I haven't ever seen that used...

View on HN

I do: "find ~ -iname ":" | wc -l" yields 49 entries. Some of those are Xorg-related (which identifies displays with a syntax like ":0.0", which ends up in some log file names) or gvfs stuff. But most are PDFs, usually research papers which I tend to save as "title: subtitle - authors.pdf".

It is certainly interesting to see the different assumptions people make. Thanks for the details.

High score! In all my filesystem I get about 150,000 files with a colon in their name. Many of them are part of the filesystems of containers which seem to use colons in the names of their libraries and dpkg packages. Many archlinux packages have colons in their names too. I also have some media files and documents with colons in names too.

It's actually really common, for someone who's used *nix for ages and expects every character to be a valid filename.

I use colons when indicating the ISO 8601[1] timestamp for stuff; it's much more readable with ':' than not, e.g. 2016-07-20T17:04:30Z vice 20160720T170430Z.

I also use them when naming stuff with hashes or UUIDs. Not having colons in filenames seems just weird to me.

Heck, even on Unix I'm annoyed that I can't simply escape slashes! It'd be nice to name files with the URLs they are taken from.

[1] https://en.wikipedia.org/wiki/ISO_8601

Lots of people, that's who.

Besides the other responses, colon is a standard path separator on URIs. If you need more than one kind of them (the obvious one being a slash), the colon is often the most reasonable option. And if you decide to save data on disk, with parts of the URI as file-name (what is also very reasonable)...

Probably, the main reason this problem does not pop everywhere is that people hacking completely new tools rarely do that on Windows. And when they port, it gets hidden together with another hundred other little incompatibilities.

The Maildir format uses colons in filenames, which has created problems with running certain email software on Windows.

Why wouldn't you put colons in filenames? Unless of course you use Windows. Colons, spaces, backslashes, whatever.

The colon should be reserved in Unix to separate path names a la PATH.

It isn't, though.

The PATH should be a list of strings not a string where : means something special.

And then how do you escape it?

In a sane system, PATH=/home/me/bin:/usr/local/bin:/usr/bin would be (setf bin-dirs '(#P"/home/me/bin/" #P"/usr/local/bin/" #P"/usr/bin/")) and if I wanted to have a quote in a file I could just write #P"/really-weird-name\"-isn't-it台北/bin/".

Transmit it as a JSON list or whatever. We're already assuming we're breaking UNIX userspace compatibility, so any option for reliably transmitting lists of strings is fine, and we have lots of those.

By list I mean an abstract data structure, you don't "escape" it. You could encode it in a specific format like JSON, if you want.

NUL

Useful if only for timestamps. 2016-07-20T16:27:00Z!

Artificially restricting filenames is an antipattern that makes things harder to read.

I use software that sometimes (but mostly not) needs files in dos 8.3. Because of this people seem to think it a good idea to use really short acronymed file names as a matter of course. If it makes sense to use a special character then people should be able to.

I agree if you are designing for a single operating system. The restriction isn't artificial if the software under design is multi-platform, since Windows for example doesn't work with colons in file names.

This would be the way to go if all program invocations consistently used some kind of common, higher-level data structure a la powershell. As long as programs rely on parsing their command line according to some syntax the developer just made up, I'm very glad there are commonly agreed sets of "safe" and "unsafe" characters. Dealing with shell escapes is horrible enough today (e.g. quoting rules under windows, filenames that start with a dash under linux...) and this would make matters even worse.

Quoting is a much bigger problem than differentiating flags from paths, and on Unix that's a solved problem: the shell always handles quoting, and unix programs only expect a list of words which can contain arbitrary characters (except NUL, of course). If you invoke a program directly using the exec-family of syscalls, you don't need to quote anything.

Whereas AFAIU Windows programs expect quoted words to be passed via main(), and must parse them. The only benefit is that you can disambiguate a filename with a dash (or slash) based on whether it was escaped, but that's a quite rare necessity, and of course still relies on the caller quoting them. (Does cmd.exe quote pathname expansions?)

The dash problem is also solved as long as programs use getopt() or getopt_long(). First, getopt() knows which flags take arguments and which don't. Knowing this, if a flag takes an argument it doesn't matter whether the argument begins with a dash or not. One consequence is that there's no such thing as an "optional" argument to a flag when using getopt and friends, as that ambiguity cannot be handled cleanly. People who roll their own argument processing code just so they can get "optional" arguments to flags invariably don't appreciate the security problem.

Second, a double-dash (--) terminates the argument list. getopt stops consuming command-line arguments at that point, and optind will index the first non-flag argument. So if passing a list of filenames to a command, the correct idiom in Unix is something like, `foo -- /path/to/*`. Of course, that presumes that the foo is using getopt or getopt_long, or a compatible argument processing implementation. Fortunately the vast majority do.

Smart programmers should rarely if ever roll their own argument processing code. Any headaches (real or imagined) related to a mismatch between the semantics offered by getopt and what the application might want is usually dwarfed by the usability and security benefits of adhering to the system facilities.

On a related note, I've always disliked the way GNU's getopt and getopt_long permuted (reordered) argument lists. I have an inkling it could introduce needless security issues, though I haven't thought it through carefully.

I assume any files I make should be usable by any major operating system, so to be sure I avoid any special characters in file names.

Mac OS 9 and earlier used colons as path separators and some support for that made it into OS X, although it might be gone by now. Apple's guidelines https://support.apple.com/en-us/HT202808 recommend not using colons.

It's not really gone, and is the path separator in the "UI layer" (Cocoa&al.) but the POSIX layer uses `/`. This gets automatically mapped back and forth.

Try to create a file or folder containing `/` in the UI (Textedit, Finder) and look at its name via the terminal. Now, `touch foo:bar` and look at it in the Finder.

Also from the Apple lineage, GS/OS allowed either slashes or colons as valid path separators.

See the last paragraph on page 109 of the GS/OS Internals manual. http://www.brutaldeluxe.fr/documentation/gsos/Apple_IIgs_GSO...

Could say the same about spaces and a plethora of systems fail catastrophically on those.

Not any well-written ones. Software should be able to handle all valid names on the platform. For Unix, any name that excludes forward slashes and null bytes.

And what about file bundles that are deployed cross-platform?

They can only target the common denominator of file names. It would be safe to avoid all special characters.

Do you also use slashes? what about backticks and quotes?

As a lazy user, I often copy paste the document title as the filename. Windows generously trims any disallowed character, but if it didn't, lots of my files would contain colons.

I actually wish colons were supported, since it's so prevalent in document titles. Question marks, too, while at it.

I think I read somewhere that Microsoft were investigating ways in which to get rid of the FAT32 backward compatibility issues with NTFS like the 255 character path element limit and the character limits in filenames. You can manage such files using the NTFS way of addressing them, "\\?\C:\Example\file:with?illegalstuff"

Is there an official name for those kinds of long paths in a single namespace? I found UNC, but it seems to apply only to the network variety.

https://en.wikipedia.org/wiki/Path_(computing)#Uniform_Namin...

I'm not sure Microsoft gives an official name for "\\?\".

The closest I can find is "To specify an extended-length path, use the "\\?\" prefix." (from https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...).

So maybe it is the "extended-length path prefix".

You can also use that prefix to access volumes that have no mount point, as in \\?\Volume{<UUID>}, among other things like UNC paths (\\?\UNC\...). So that's not a good name.

Who puts colons into their filenames? Oh, short-sighted people like the designers of OBS (OpenSUSE Build System) who introduced a colon convention into project names.

http://lists.opensuse.org/opensuse-buildservice/2008-12/msg0...

This causes a problem even in the POSIX environment, because the colon is used in PATH and PATH-like environment variables. Usually there is no escape mechanism.

Well, it isn't uncommon to create a log file with a timestamp and a name like foobar.$datetime.log with $datetime expanding to something like 2016-07-20T22:08:38

> Who puts colons in their filenames though?

Twitter does

https://pbs.twimg.com/media/Cn1tGFoXgAApMt3.jpg:large

I try to save papers sometimes and give the filename the same name as the paper, which often has a colon in it.

Reply on news.ycombinator.com