DTrace on Windows
techcommunity.microsoft.com
techcommunity.microsoft.com
There is some sort of a bitterness in the Linux community when people talk about ZFS or DTrace. But it's the consumer who is affected by it. For their sake, I hope they don't take the 'We have BtrFS/eBPF/X' path.
Merging DTrace implicitly means they are fine with both the CDDL and Oracle's patents, and thus that ZFS wouldn't be an issue for them.
It's available with new kernels (including 5.0) on github as well.
https://github.com/oracle/dtrace-utils
https://github.com/oracle/libdtrace-ctf
https://github.com/ezannoni/dtrace-linux-kernel
We are working on a Fedora version of the kernel patches.
We are also discussing with the BPF team (on netdev) how to provide DTrace functionality integrated with the Linux Kernel.
Also, NTFS already has snapshots via Volume Shadow Copy.
ZFS is great but it’s not going to be a viable replacement as it exists today.
Microsoft strongly recommends developers utilize alternative means to achieve your application s needs. Many scenarios that TxF was developed for can be achieved through simpler and more readily available techniques. Furthermore, TxF may not be available in future versions of Microsoft Windows. For more information, and alternatives to TxF, please see Alternatives to using Transactional NTFS.
https://docs.microsoft.com/en-us/windows/desktop/fileio/abou...
The Sun Storage Appliance used ZFS snapshots for updates, and it worked quite well.
Obviously.
The super annoying thing is when the OS doesn't try to tell you which process(es) keeps it open and doesn't even ship with built in tooling to let you find out on your own.
I think that is the point GP tries to make.
It lets you see which process locked a file, and optionally remove the lock.
PS> $lockedFile="C:\Windows\System32\wshtcpip.dll"
PS> Get-Process | foreach{$processVar = $_;$_.Modules | foreach{if($_.FileName -eq $lockedFile){$processVar.Name + " PID:" + $processVar.id}}} PS C:\> gps |? {$_.Modules.FileName -match "wshtcpip\.dll"} | select name, idOr, imagine every copy of Word you have running is using a different set of binaries.
No, thank you.
I use it in Win 7; I think it works most of the time (not always).
Yeah, it should come with the OS, but that's Windows for you.
NTFS's USN change journal (which is separate from the transaction journal) is also useful as a robust offline alternative to ReadDirectoryChangesW file monitoring. Toy example: https://gist.github.com/pervognsen/bcc610d6a5ae6cbc3b2b4f7aa....
Anyway, the problem I've always had when using non-standard file system metadata like alternate streams or sparse files is that they interoperate poorly in practice. There's lots of code and programs out there that wants to treat everything like a plain old file and hence won't preserve alternate streams or sparseness. From my recollections this is true even for several of Windows's built-in utilities. Same issue if you want to transfer files over a socket or pipe or a non-NTFS storage medium like a FAT-formatted USB drive or a NetApp filer or a source control system. As a result this kind of thing is only really useful when deployed in a bubble; I believe WSL uses alternate streams for Unix permission bits and other Unix-specific file properties. Even though those WSL guest files live in a directory tree on a normal NTFS volume, they strongly caution you against touching them directly outside of WSL, presumably for the aforementioned reasons.
Aside from features, NTFS has some performance issues compared to other modern file systems, notably with lots of smaller files.
- For permissions WSL uses EAs; for capabilities it uses ADS
- I'm not sure if it's really NTFS that is slow with lots of small files or the I/O subsystem in general... I got the impression it's the latter but not sure.
All of these except shrinkable volumes (somewhat) are already part of ZFS, also assuming that you replace "BitLocker integration" with more general "encryption integration"
ZFS and NFSv4 were both designed to be able to serve the entire subset of NTFS features over the network, to interop with Windows computers.
As for ADS: basically xattrs. They can be arbitrarily named and each xattr on ZFS can be up to 16EiB large (same as the primary bin for file content). (As an aside, they aren't fully usable on Linux since Linux itself imposes a 64KiB limit on an xattr's content -- but that's not a ZFS limitation)
Absolutely not! Reparse points are directories that are associated with drivers that extend the capability of the filesystem itself. They can be used to implement symlink-like behavior, but that's only a single use case that scratches the surface of their potential power.
See https://docs.microsoft.com/en-us/windows/desktop/fileio/repa... for more information.
It doesn't sound like it is really an inherent property of the file system, but just controlled by a driver. I don't see why they couldn't be made to work on any other file system, such as FAT or ZFS.
When you feel slow, it is the abstraction layers and plugable interfaces above it. - and they are API.
Replacing the core won't help the performance. They need to remove some abstraction and break some application.
I think that filesystems like ZFS and Btrfs are simply one step ahead than anything else, not only in features but also in tooling and UX. Send/receive, CoW snapshots, checksumming are all tremendous features that I think would fit fantastically into the Windows workflow, if Microsoft really managed to integrate them well. For instance, I think a ZFS-based Windows update would be able to do away with transactions and rely instead on datasets to snapshot the system, apply an update and rollback everything if something broke, exposing to power users the tools to clearly understand and rollback their systems themselves by hand.
Obviously all of this can be achieved by rewriting everything from scratch as in house, specific platforms, but I just feel that continuing to replicate in NTFS/ReFS features other open and widely supported filesystems have had for almost a decade is just not the best way forward, and it's not really better for anyone.
Does anyone actually use this on Windows these days? I'm guessing the reason so few filesystems support it now is because the gains aren't worth it for the types of files that take up most of the space on filesystems these days, such as video files. You can't compress x264 video with a general-purpose compression algorithm. Storage space is plentiful and cheap, and (general-purpose) compression yields little gains, and the stuff we're storing now is already compressed (images, video, audio, etc.).
Mind that DTrace was ported by Oracle (at least partially) to Oracle Linux first. They haven't done that for ZFS (instead they created bttrfs, before owning Sun)
To my eyes at least they are pretty much helpless to do anything about it because of the licences. Do you have any ideas as to how they could work around the licencing issues ?
Edit: I am talking about Linux not Windows.
I'm not a lawyer or anything close, but who would possibly have standing and motive to sue? There's, what, a dozen people at most who have copyright on those struct definitions? And API copyrightability is still unsettled?
Maybe the lawyers were too conservative, and the engineers who listened to them were applying the wrong model?
Someone tell me why I'm wrong.
The source of incompatibility with ZFS and DTrace on Linux (until recently) was from the GPL, not from the CDDL. Windows (and, indeed MacOS - which has shipped DTrace for years), have no such restriction - the CDDL allows inclusion in larger proprietary works.
DTrace has a linux port which is GPL licenced since last year. So they are not helpless and there is nothing to work around.
Personally, I'd rather see more resources poured into bcachefs[1].
[citation needed]
Simon Phipps, Sun's Chief Open Source Officer at the time, has stated this is not the case:
* https://en.wikipedia.org/wiki/Common_Development_and_Distrib...
The only person who has made this claim is Danese Cooper, and she has been directly contradicted by her boss at the time (Phipps).
(Improving bcachefs, or whatever, is independent of the CDDL question.)
Of course he did. There's nothing to gain from admitting it. It's the sort of statement akin to 'Facebook deeply values your privacy', as in of course they're going to say that.
On the other hand, submitting the license to the OSI, but not the FSF to be consulted on compatibility before using it suggests otherwise.
""" > > It would be a critical mistake to underestimate her knowledge on > the CDDL > issues.
Nonetheless she is wrong to characterise the opinion of the Solaris engineering team in the way she does. She is speaking this way because she lost an argument inside Sun, not because her view is representative of the views of Sun or its staff in the way she claims. She, along with many actual engineers, was an advocate of using GPL for OpenSolaris but the need to release rather than wait for one of {GPL v3, Mozilla license revision, encumbrance removal} meant that this was not possible. I am still furious with her for the statement she made at DebConf, which was spiteful and an obstacle to a united FOSS movement. """
* https://marc.info/?l=opensolaris-discuss&m=115740406507420
Cooper is, AFAICT, the only one that made the claim: was it ever corroborated by anyone else? Phipps and Cantrill dispute it: are/were there others?
Edit: it seems that not even GPLv3 (which they considered) is not compatible with GPLv2 (which Linux uses), so even if they did that it would be a problem. One main thing (IIRC) that was of concern was patent license grants.
From your own quote:
> She, along with many actual engineers
The CDDL-GPL drama seems to be one of an attitude of "fuck-ZFS because fuck-Sun/Oracle and their fucking CDDL license".
The validity of the claim that the design of CDDL was 'malicious' is the thing I am skeptical about.
It may have ended up incompatible because of technical-legal reasons, but was that the intent? Cooper says yes, Phipps/Cantrill say no.
Same as with kqueue vs epoll and so on.
There have been better, cleaner, leaner, FOSS operating systems out there for decades, why support, or even give mindshare to something like Windows?
Also: https://itvision.altervista.org/why.linux.is.not.ready.for.t...
Notice that a lot of this one is about NVIDIA. The solution is simple: don't use it. Desktop Linux works great on Intel and AMD, where the driver support is excellent. NVIDIA isn't the only GPU vendor out there.
Printers work much better in Linux than in Windows, and it's been this way for a very long time. The exception is cheap, crappy inkjets. Solution: don't use cheap, crappy inkjets. Inkjets are a scam.
Games are always on this list. I don't care about games.
Many of the other points in your list are completely obsolete.
For business use, most of these factors are a non-issue. Being able to play AAA games is not important for a business computer, nor for many (most?) home computers. I know gamers can't conceive of this, but not everyone plays recent AAA video games.
- spend more money
- I don't care about it
- something undefined is obsolete
- ignoring that most business runs on MS products and compatibility issues are a thing
This attitude is one reason that was not on the list. It's one that one should not forget though as it hits hard for those who are determined to make it work. They found themselves confronted by a self centered community of semi-gods who think the world should arrange around what they assume it is.
Nvidia isn't the cheapest option by far, so no, you should change this to "spend LESS money". If you're talking about printers (your post was poorly written and didn't refer to anything point-by-point), then yes, and also "spend more money to save money". Inkjet printers are a scam, full stop. They cost a LOT of money in consumables. You will save a lot of money by buying a laser printer instead. It's really that simple. And every laser printer works just fine in Linux. There is no good reason to buy a <$100 inkjet printer unless you really want to be ripped off.
>- I don't care about it
Yes, and lots of other people don't as well. If games are the only thing keeping you on Windows (and I've talked to a bunch of people for whom this is the case), then stop complaining about all the problems and spying in Windows: you're making a choice to enjoy a luxury, and put up with things you find abhorrent, just because of your addiction to games. I know this is news to a lot of younger techies, but not everyone cares about modern AAA games. I guess I shouldn't even bother writing this because apparently this concept is so utterly foreign to them, but it's true. There's large swathes of the computer-using population (which these days is most of the population) that don't buy or play video games, and haven't since they were kids. Amazing huh? So no, the fact that Linux can't play every single AAA game just isn't an issue to them.
>- something undefined is obsolete
I have no idea what you're referring to here, perhaps you could try being more descriptive.
>- ignoring that most business runs on MS products and compatibility issues are a thing
In my experience, all my work is in Linux, except for MS Office and Outlook, and companies are wasting a lot of money buying me a separate computer (or VMware licenses etc.) just so I can read email on Outlook.
That's not the point here. Usually people have problems with Linux and Nvidia because they already have Nvidia in their computer. So changing hardware for Linux to make it work properly involves spending more money. Same goes for peoples printers. And no it's not just about inkjet. This is about drivers and unforeseeable issues you face if you switch and I know what I'm talking about since I made my parents laptop switch over to Mint and been confronted with issues on their laser printer...
> Yes, and lots of other people don't as well. If games are the only thing keeping you on Windows then stop complaining about all the problems and spying in Windows
I'm not sure if you misread the thread here. The original argument was about someone complaining that people have "love for Microsoft". Nobody was complaining about anything you bring up there.
> you're making a choice to enjoy a luxury, and put up with things you find abhorrent, just because of your addiction to games.
So what you did here was to blow out a straw man to infinity. Who's talking about addictions? Not everybody who wants to play a game after work or from time to time is addicted. What is this generalizing overdramatization?
> I have no idea what you're referring to here, perhaps you could try being more descriptive.
I didn't know what you meant by obsolete. That's why I wrote that. If you don't know what's obsolete, it wouldn't be much of a surprise to me considering the way you crippled the other points above but it's still kinda weird.
> In my experience, all my work is in Linux, except for MS Office and Outlook
So I guess you just don't have much experience then. It's not just office and outlook. It's a endless amount of small programs in different regions of different businesses and hardware. There is a huge world out there beyond your narrow world view of addicted kids, inkjet printers and outlook. The fact that you still think you can impose this narrow worldview on strangers enforces the point I've made at the end of my previous comment. You know, the one you ignored. You might want to sit down and think about this point alone. It may give you a hint why people like you are not helpful to the whole migration away from Microsoft. You are rather another reason why it's happening slower for some or not at all for others.
Bye.
Ok, then suppose you decide you want to run MacOS. Do you think it's unreasonable to have to change your hardware for that? Then why do you think Linux should be able to magically run perfectly on all PC hardware out there?
>Nobody was complaining about anything you bring up there.
The Win10 spying is the #1 complaint I hear from people these days about why they're "really" going to switch to Linux this time.
>Not everybody who wants to play a game after work or from time to time is addicted.
Yet the anti-Linux people constantly bring up games as some huge reason that desktop Linux "just can't work" and is "completely impossible". No, not everyone plays games, but people like you just can't conceive of this, can you?
>So I guess you just don't have much experience then. It's not just office and outlook
I have decades of experience, unlike you apparently. In all my engineering jobs, the only thing Windows is used for is Office and Outlook, and maybe running a hypervisor.
I think it's unreasonable to change to Mac at all but that is a completely different topic regarding a walled garden. I don't think Linux has to run on all kinds of hardware. Actually my argument was that Windows does that and that's one reason why it's everywhere and it will stay so for quite some time. You should read the comment you've originally answered to again.
> The Win10 spying is the #1 complaint I hear from people these days about why they're "really" going to switch to Linux this time.
The narrow group you are talking about here are also not the topic here. As I said before: Windows is everywhere. In regards to your argument here: most people don't care or don't even know about this.
> Yet the anti-Linux people constantly bring up games as some huge reason that desktop Linux "just can't work" and is "completely impossible". No, not everyone plays games, but people like you just can't conceive of this, can you?
Look at you. Just one comment before they were all addicts now everybody questioning the actual usability, compatibility and all those other issues form my first comments link are "anti-Linux people" and of course everything is my fault now...which brings us back to the topic of attitude. You know, that issue that you keep ignoring.
> I have decades of experience, unlike you apparently. In all my engineering jobs, the only thing Windows is used for is Office and Outlook, and maybe running a hypervisor.
Just like above you've displayed you extremely narrow experience and or world view and you keep doing it. I work for a huge worldwide engineering and architecture company and we have all kinds of programs that have been written for and run on Windows. For example software that collects and manages logger data, ever heard of AutoCAD? Revit? I mean, seriously how could you in your DECADE OF EXPERIENCE never heard of that? Everything around Sharepoint. Everything the architects design for VR. Also client ports and client software is exclusively Windows in this field. Ever worked for any branch of the military or state entities? All this potentiality increases if you look at our subs and their subs.
So I guess your decades of experience are either in a very narrow field or you are ignorant or you just lied to me and hoped to get through with that. None of those options shad a good light on you and the Linux migration movement. As I have mentioned above: you are hurting it.
This also means they have done a lot of the platform work needed to support eBPF one day, to really catch up to Linux.
It seems to be on a project which has virtually no contributions since last august.
In addition, I'm not sure where Microsoft is going with this: your DTrace scripts probably won't be portable anyway. The only common point is the language. Is that sufficient ?
That said, there are _many_ points of commonality which will apply to Windows too - USDT probes, function boundary tracing etc.
On the other hand, using a foreign syntax can hurt too, had they used something that looked more like powershell for example, it might have been easier for Windows admins to get accustomed to.
[0] https://wiki.netbsd.org/tutorials/how_to_enable_and_run_dtra...
Is the performance on par with running dtrace on solaris?
whereas DTrace could tell you "these are the file accesses that took longer than 5ms" or "here's a histogram/heatmap of file access durations". DTrace can correlate — file accesses by a given process in-between a given function's entry and exit — telling you _why_ a file was accessed.
well, maybe you're just asking how performant DTrace would be at achieving the same output. sadly I can't answer that.
Makes me more enticed to switch everyday.
I'm hoping a lot of it gets more solid in the next couple years. As it stands, I'm happier to see .Net Core running outside windows more.
What's new is that they now acknowledge the existence of the UNIX world and open source.
Microsoft surely also needed it for other reasons. Developers, sysadmins, and power users -- all need DTrace or DTrace-like tools.
We're finally leaving the dark ages of unobservable operating systems! Hooray!
We should all give huge thanks to the creators of DTrace, Mike Shapiro, Bryan Cantrill, and Adam Leventhal, as well as Brendan Gregg and all of the crew at http://dtrace.org.
Sadly due to licensing, they don’t.
Edit: I stand corrected. GPL should be fine.
I know ZFS works fine on Linux, because I use it everywhere, but there’s no way around the fact that the incompatible licenses are causing issues.
> system tap, ktap, lttng, ftrace, etc
That's why I'm talking about NIH.
Even with ebfp, they intentionally choose different tracing syntax (just look at it, it's comical). Didn't use ready-to-use CTF. Etc etc etc.
ZFS is still problematic, specifically because of the GPL-incompatible license. Various distros have found ways around that (or are deliberately ignoring those issues at their own peril), but it's still very much "a real problem", and it took years (if not decades) to get to the point where ZFS on Linux is actually as viable an option as it is today.
If the CDDL simply didn't matter, after all, they could just add it to the source tree and be done with it. I wonder why they haven't done that...
ZFS is not in the Linux kernel, and Linus has stated that he will never merge CDDL licensed code due to license incompability.
Those Linux distros that distribute ZFS do so as a separate module, not part of the Linux kernel.
DTrace was GPL licensed too late (2018), something even Brendan Gregg stated as he finds eBPF more capable (and he is a leading expert on DTrace).
The BPF people cannot be blamed for not using CTF. Even if it had been relicensed to the GPLv2 way back in 2005, until recently (https://github.com/oracle/libdtrace-ctf release 1.0 or 1.1) it was impractical to use CTF on larger projects because the file format could only encode a strictly limited number of types (2^15 in each of a parent and child container): it had a lot of related limitations as well, but that was the big one. This is not enough for a largeish enterprise kernel, even assuming that you share types used by multiple modules to reduce the overall type count. Also, BTF and CTF serve rather different purposes: CTF specifically encodes knowledge about C types, while BTF is specialized for encoding information about BPF maps. You can't use CTF for that: C doesn't even have a map type, nor anything like one, and the bits CTF spends on things like the details of floating-point formats are wasted on BPF.
As for the other part... obviously, as someone working on DTrace and using it ever more, I think it does hold value in its own right. It operates at a different level of the stack from BPF, in any case: it's a user-facing tool like a kernel-level awk, which is nothing like BPF.
In my opinion, saying that DTrace doesn't hold value because of BPF is like saying that C doesn't hold value because ARM assembly language exists as well as x86 (in this metaphor, C == DTrace, ARM assembly language == BPF, x86 assembly language == DIF, the DTrace intermediate format). It certainly seems possible to replace the DIF portion of DTrace with BPF, but this will not obsolete BPF nor DTrace: instead, DTrace will drop DIF and the DIF interpreter and build on BPF, generating BPF instruction streams the way it now generates DIF instruction streams. We get to improve BPF if needed and drop a redundant interpreter and BPF tracing gets a hopefully-nicer user interface in the shape of DTrace, and wider usage. Win-win!
(I'm not doing most of the work on this, so my opinion is far from authoritative, but I did do a preliminary experimental conversion of the hand-rolled DTrace code generator to emit BPF instructions instead, and it seemed perfectly practical: the two encodings are remarkably similar, and BPF is pleasant to generate, as such things go. That's only a small part of what needs doing, of course...)
Just think about it for a second. It’s ridiculous. Oracle has plenty of open source code. Merely suggesting that something be open source could not possibly lead to termination. There is obviously way more to that.
Whatever reason he was fired for might be unjustified, I have no idea. But it was absolutely not due to suggesting that they open source something.
If you’re basing that on this tweet, not only does it come well short of the claim you’re making, but it clearly indicates that Cantrill is going on third party hearsay.
https://mobile.twitter.com/bcantrill/status/9313425004463513...
i guess eBPF verifier is a bit scary because a small bug anywhere in the verifier creates a big hole. i think project zero found a lot of bugs in the verifier.
its pretty scary putting an interpreter in the kernel. i'm not surprised linux didn't want to have two completely different interpreters in there.
Yes, that would be the D programming language :)
(1) Those who consider Microsoft to be reasonably trustworthy based on recent behavior.
(2) Those who will need Microsoft to continue its current behavior for a longer period before they're ready to trust.
(3) Those who consider Microsoft's current behavior to be at best a partial improvement over the past, or at worst a charm-offensive meant to distract from their ongoing bad behavior. E.g., Windows 10 with unavoidable snooping, and ongoing stealth patent attacks on Linux.
Let's hope the good guys win this one, but I personally don't see it happening unless they get hit with governemental regulation on a regular base.
Seriously. Everyone else is doing it, why shouldn't they?
If it wasn't for our iOS projects, they wouldn't be even around the office.
Disclaimer: I work at Microsoft on Windows.
Also the availability of kernel headers, config tools cross compilers and the scripts compatibility.
When I first set up my Mac, I thought case-sensitivity would be a good idea just in case I ever had to copy files from another case-sensitive file system. I never did need the case-sensitivity, but I did have to repartition to get Steam working. It thought me a good lesson about switching away from defaults "just in case."
That doesn’t make Windows suddenly any less of a mess.
I agree that there are all kinds of issues with Apple’s Mac hardware pipeline, but this is just hyperbole.
There's something about computing that makes us want to cheer for these big companies as if they are sports teams, no matter how many times they screw us.
But for the uninitiated the biggest problem for some time has been that Microsoft was and still is a big patents troll. They also continued fighting open standards such as the OpenDocument format, even under Satya Nadella. To their credit they did join the Open Invention Network, but it's still unclear what parts of a Linux OS are covered and what aren't. The FSF rightly asked for clarifications on whether the multimedia components are covered and to my knowledge there have been no clarifications thus far.
But forget the years of trolling Android phone vendors with patents such as for the FAT filesystem, here's a story from 2017: http://techrights.org/2017/02/27/microsoft-novell-v2-via-azu...
Also Windows 10 is a piece of spyware shit.
Windows 10 has been intercepting Chrome and Firefox installations and harassing people into using Edge (and this has been my own experience): https://www.ghacks.net/2018/09/12/microsoft-intercepting-fir...
Microsoft is reportedly planning to make Edge indispensable: https://www.theguardian.com/technology/2018/mar/19/windows-1...
Windows 10 has been displaying intrusive ads: https://www.theverge.com/2017/3/17/14956540/microsoft-window...
Windows 10 has been blatantly disregarding user choice and privacy: https://www.eff.org/deeplinks/2016/08/windows-10-microsoft-b...
Windows 10 collects everything you do: https://www.techworm.net/2014/10/microsofts-windows-10-permi...
---
> "Investing in making Edge work like Chrome (more important to developers and consumers than your ideological quandries with Google owning the web will ever be)"
That's incredibly naive. First of all they aren't "making Edge work like Chrome", that's an overstatement, the new Edge is going to be only a shell on top of Chromium and nothing more.
Microsoft adopting Edge is either them throwing the glove and admitting that they can't develop a browser, or them pulling another embrace, extend and extinguish. After all, as we've established already, they don't really like that users have a choice when it comes to the browsers they use. This is important to mention, because even for the Chrome fans out there, Edge switching to Chromium provides the perfect opportunity to disallow alternatives on top of Windows. This is just a possibility mind you, when it is far more likely that Microsoft will simply place Edge on life support.
In either case, whether they end up contributing to Chrome's ecosystem, or not, consumers lose for a myriad reasons. And heralding this move as something good is ignorance for the whole history of computing.
I use a Windows 10 desktop for 1/2 my work. The other half I use macOS on a MacBook. They're both great (and the latter is fantastic for virtually any sort of work, whereas with the former I am constantly encountering Windows-specific limitations that demand that I fire up a Hyper-V instance --- WSL is very limited in a variety of deadly ways).
I'm curious to see how they will act once they regain sufficient market share. Will the old MS appear and will they implement the extinguish phase?
1. https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
Does that include Wow64[0]?
[0] - https://docs.microsoft.com/en-us/windows/desktop/WinProg64/w...
[0] - https://docs.microsoft.com/en-us/windows/desktop/WinProg64/d...
Anyone have an insider build to test?