Tar and Curl Come to Windows
blogs.technet.microsoft.com
blogs.technet.microsoft.com
Specifically, I need this functionality to make my backup program (Snebu) more useful to Windows users, as it relies on an installed tar implementation on a client to gather files.
If anyone here is skilled in the Windows API and is interested in helping with this, let me know.
The main problem with backup/restore these things is SIDs. Windows ACLs are much more sophisticated then these 6 bits in Linux.
It will work in some cases, e.g. when the owners are, and all the permissions are granted/denied to, a well known (1) group such as "Everyone". Or when all these SIDs are domain users SIDs, and you’re restoring to a PC that’s on the same domain.
It won’t work in many other cases, e.g. when these permissions were granted to some local user and you’ve reinstalled Windows between backup & restore.
That’s why in Windows it is typically considered OK to not bother backing up/restoring these ACLs.
(1) https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
As for "permissions were granted to some local user", in the tar file format it stores a UID/GID number, as well as a username/groupname, so when restoring to a different system the username takes precedence over the UID number. Could the same idea work on Windows? (Granted, I'd probably have to store this in the PAX header's name/value pairs, due to potentially longer length of the UID numbers).
The other item I've read is that Windows file permissions can be inherited from parent objects, so it may be useful to store those permissions, but not restore them (or give a warning if the effective permissions end up being different due to restoring to a different location).
To backup the complete C:\Users directory to be restored later on the same PC, I’d use specially-designed APIs for that, BackupRead/BackupWrite.
See this article for an overview: https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
That article was written in the assumption that you’ll be using a real hardware tape for backups, but you can backup to any other place.
This will handle file data (including alternative NTFS streams) and file security, but you need to handle file names, file attributes, and timestamps somewhere outside of that. And TAR format can’t quite fit all that info. File names in TAR are limited to 100 characters, there’s only 1 timestamp in TAR but NTFS has 3, resolution of TAR timestamp is 1 second but NTFS keeps 100 nanoseconds, finally there’s no place to keep file attributes. I wouldn’t pick TAR format for that kind of backups.
Also note that NTFS hard links and EFS encrypted files both need special care to backup.
You’ll surely be able to create a PAX-based format that your own tools will be able to backup and restore correctly. But it won’t be compatible, i.e. no other tool will be able to restore or read the data.
BackupRead API returns zero or more streams, each prefixed with a variable-length WIN32_STREAM_ID structure. For normal files without security descriptors, you could just unwrap the content of the BACKUP_DATA stream to the file content inside the TAR and you’re good.
But even if you’ll find a way to store BACKUP_SECURITY_DATA stream in these PAX extended names-values, there’s other stuff to keep somewhere. Alternative data streams, usually (when written by windows explorer) they’re quite small, dozens of bytes, but nothing prevents them from being gigabytes, I’m not sure PAX will be happy about values that large. And also there’re sparse files, BackupRead won’t return you these zeroes like ReadFile, it’ll return you a BACKUP_SPARSE_BLOCK stream instead, with just the non-zero portions of the file.
If you won’t bother parsing the backup data and unwrapping the streams, it’ll work for your tool, but your TARs will be incompatible with standard TAR tools which won’t see the content of your backup.
Also, unrecognized pax data would be ignored by a regular tar.
And in my use case, when data is submitted by the client to the Snebu backend, it has its own tar implementation built in for separating file data from metadata, and re-synthesizes a tar file upon restore. (The reason that Snebu uses tar format for transfering, is it was a good way to make it agentless for normal Unix/Linux servers, yet extensible enough to support other use cases. Also I figured that most tar implementations were fairly well optimized, at least more so then I could do in a short amount of time).
Thanks for your input, really appreciate it. Been a little bit tough learning the WIN APIs, after a couple decades lost in Unix land.
I think that’ll work. Just be sure to name these separate streams in a way so they never conflict with other files that might be in the same directory or with other streams that may be in the same file. E.g. in Windows, there’re characters forbidden in file names which may work fine in Linux: https://stackoverflow.com/a/31976060/126995
> with a property of "nt_stream_count=2"
Read this: https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
I don’t think just count is enough. Alternate data streams have names. Other data streams, extended attribute, security descriptor, etc., are essentially unnamed.
You can implement some naming schemes for them, e.g. “file” for the main data, “file:xxx” for “xxx” alternative stream of the file, “file?sd” for security descriptor of the file, “file?sparse” for sparse file data. This can result in TAR backups that are more or less readable by the standard tools, but still allow your own tool to restore the complete thing.
You have two problems to solve.
1. Directories. Quite often, they contain at least security descriptors. But AFAIK directories just aren’t stored in these TARs.
2. File names and their encoding. AFAIK modern Linux is UTF8 all the way down. Probably TAR is OK with that. But if UTF8 representation of the UCS2 name (WinNT isn’t quite UTF16, UCS2 is a subset of that) is longer than 100 bytes, you’ll have multiple files in your TAR with the same name, they’ll only differ by your extended PAX attributes that contain the complete name. If you combine that with the above fake names with security info & alternate streams, it becomes even more complex.
P.S. I’ve been programming for windows for decades, only occasionally for linux or other platforms.
These days you can use clang so a nice C compiler has returned.
A laughing stock turned into a respectable trend. Well done.
I think with this, PowerShell needs to stop aliasing curl to Invoke-WebRequest by default (or at least try to deprecate that alias).
A deprecation warning would be a good idea, though, to suggest to developers that are using the curl alias to move to iwr.
(Probably a while: command line familiarity is far from a given in Windows)
"GNU Windows" ;-)
(Not all GNU software is under GPL.)
Porting simple UNIX tools from the 70-80s to Windows and making a big fuzz about it? Tar? unzip.exe or whatever equivalent with a gui has been around for ages.
I just can't see how any self-respecting developer would be lured to Windows by this...
I'd still rather run GNU/Linux from the metal up, but my work depends on Windows for legacy data-acquisition systems. Right now, I have an experiment that's running a linux box and windows box side by side in order to get the best of both. Working out hardware access in a VM sounds unpleasant, so I'd love to see the native-windows-GNU implementation extended to X.
TL;DR: I want to 'apt-get install lyx octave gnuplot chromium darktable' with full functionality.
* MobaXTerm was usable (on Win7-8, a couple years ago), but IIRC it's surrounded by a faint stench of upsell and I recall being annoyed by its attempt to be a full desktop environment (tabs, file manager, integrated editor, etc). It feels very... windows-y, the X server equivalent of PuTTY.
* VcXsrv had (on Win10, about two weeks ago) a show-stopper bug - windows would fail to redraw outside their original bounds when resized. It also had trouble using modern UI toolkits or themes.
* Cygwin/X is the best, no issues at all if you're in cygwin, but I was unable to get it to cooperate smoothly with bash on windows and ended up deciding that the environment was worth more than the tiny missing features. If you don't need WSL I'd recommend this.
* Xming is what I've settled on. I use the free version, which is a major version behind (6.9 instead of 7.7), but I haven't noticed it lacking anything compared to Cygwin/X. Seems to be just as reliable as Cygwin/X too. Getting it configured was somewhat annoying (had to create xauthority manually, etc), but now that it's set up it's working smoothly.
I am mostly annoyed by two things, both of which are common to the setup and not the chosen tools:
* I have to put up with Windows for window management. I would much prefer something like i3, but I haven't gotten the fullscreen mode to work to my satisfaction.
* Network issues. You must be wired and you must have spare bandwidth. If you're running gigabit ethernet this isn't really an issue, but if you try to do it over wifi you're going to have a bad time.
Compared to running a virtual machine... it really comes down to picking a set of annoyances. VM you have to deal with configurating another machine, it devours resources, integration with the host isn't as good, and it can't tolerate monitor switching at all. Native X you don't get a usable WM. shrug
The Verb-Noun verbosity of PowerShell is great for A) discovering new verb and noun combinations that you hadn't considered before (and Get-Verb gives you a list of common verbs), and B) for keeping scripts readable in the long term. Most every Verb-Noun has at least one shortcut alias, most based on common cmd/bash-isms (Get-ChildItem has gci, ls, and dir), and Get-Help on any Verb-Noun will list the aliases for you.
They tend to work without any issues.
I'm not sure if Windows still does this. Probably does.