Of course there's always a trade-off.
I can think of dozens of edge cases in all kinds of security controls that remain security weaknesses despite a large chunk of problems being addressed. The point is to address the bulk of the cases and not allow perfect to become the enemy of good.
> a) someone uses this library for validating whether files contained are conforming a given layout, then passes the tarball to tar - differences in behaviour (tar strips leading .., while eg. the Rust module ignores all files with '..', no matter the position in the path) might cause a security issue.
I would argue that any library that does not already fully resolve relative paths to do this is already faulty and this makes it no less safe. To draw an example, if at the time I pass in a relative path to say, "all files should exist under this directory" and provide "/tmp/somedir/../../usr/lib/" as that parameter, I would expect that this already resolves to "/usr/lib/" before doing any processing. This would eliminate the entire problem presented here.
> b) silent removal of 'invalid' paths might cause unexpected behavior for when these paths actually are valid and expected (you might not have used ../ in tar, but I have); can imagine one hell of a debugging session coming from that one. I can also imagine codebases that filter tars for malware, and if they happen to skip ../-paths (because of the tar parser silently skipping them), they could let through the tarball unaffected, with some files unscanned, but extractable in GNU tar.
I agree that silent removal is a problem. There should be errors/warnings attached to indicate that this has been done.
This isn't a good counterpoint to actually having a sane default though. The very issue here is we have a clear security bug introduced in part due to the fact that the library does something seemingly unexpected precisely because it does not line up with existing tooling.
> c) silent rewriting of 'invalid' paths might cause inconsistency between the behavior of different tar libraries, or might cause internal inconsistency due to two 'invalid' paths sanitizing to a single valid path; imagine someone relying on ../-stripping behaviour at some point, then that behavior changing due to switching libraries or a rearrangement of the files in the source tarball. Sanitizing is a complex issue, and any kind of universal logic might actually introduce more bugs than it solves.
Again, I'm not advocating it be an invisible process, just that the default should be the safer of the two options for the bulk of cases.
> d) false sense of security for more complex classes of bugs, like extracting a tarball with a symlink pointing outside of the root of the archive. What should the default behaviour be then, and why? Sometimes symlinking to `/etc` is exactly what the user needs, sometimes it's going to cause a bug further down the line. What about relative symlinks? What about cross-filesystem symlinks? What about the setuid bit? If you make users expect the library to do the right thing, it might cause them to think even less about possible security implications of more tricky situations. Writing user-controlled data to the filesystem is _always_ dangerous and there is no right way to do it, it all depends on the context.
What you describe is not a code problem but a programmer / documentation problem, and is really about the idea that fixing one problem does not inherently fix another problem that it's not necessarily aimed at fixing.
Clear documentation that outlines the steps that are taken, as well as the option to disable the feature if truly needed (as I suggested) would solve this entirely.
Also, this is a weird argument to make given this already exists with existing tooling. I expect if I name a file "../../../../etc/cron.hourly/evil.sh" and stick it into a tarball, then unpack that tarball to /tmp/ I will get a file named "../../../../etc/cron.hourly/evil.sh" in /tmp/, not a file named "evil.sh" in /etc/cron.hourly/. At the very least I expect to get a heapload of warnings/errors telling me about this and that I should unpack things with "--unsafe-please-never-do-this" or similar flags.