Thanks. The implementation is very stable because WinFsp is tested regularly and rigorously.
Please read about the WinFsp Testing Strategy here:
574 karma · joined April 27, 2016
Please read about the WinFsp Testing Strategy here:
Alternatively you can show me that there is a lock already taken within Dokany for this purpose (protection of the data structures of the user mode file system) and I will be happy to remove the lock and rerun the tests.
https://github.com/billziss-gh/secfs.test/blob/60f9f44b87731...
I do not see a big difference to the fsbench call. It may be that fsbench attempts to make a mapping that is 4MB long. Now you can fix that problem in Dokany ;)
I do not understand this.
Are you familiar with how memory mapping works on Windows? CreateFileMappingW is it. If that API fails you do not have memory mapping.
Furthermore I challenge you to show me how Dokany/MEMFS makes this API fail. If you find such an issue I promise to fix it and re-run the tests.
>The lock exists on WinFsp/MEMFS. It is just taken in a different place: It changes nothing on what I said. You added it on dokan side without reason and not on winfsp memfs.
I added it on Dokan side, because I needed to protect the Dokany/MEMFS data structures against corruption. On the WinFsp side this lock is taken automatically as a convenience (although one can remove it if the file system can cope by using a custom "operation guard" (FspFileSystemSetOperationGuard) -- it is not needed for WinFsp operation).
The exact same lock is taken on both sides. There is no unfair advantage to WinFsp. If you design a file system that takes no locks and works on both WinFsp and Dokany, I will be happy to run the tests against that file system.
> A pull request will not change that a real application on a non-virtual machine will not see the difference.
All the "real application" has to do is to open a file and issues some I/O. Just because Dokany does not cache I/O it will be 9x-10x slower than WinFsp.
Examples command lines:
WinFsp: memfs -t 0 -n 100000 -m Z: # no caching memfs -t 1 -n 100000 -m Z: # enables metadata caching memfs -t -1 -n 100000 -m Z: # enables data and metadata caching
Dokany: memfs-dokany -n 100000 -m Z:
It was me that ported FSX to windows so I am quite familiar with this test.
Here is the code that fails in fsbench:
Mapping = CreateFileMappingW(Handle, 0, PAGE_READWRITE, 0, FileSize, 0);
> You add lock (that will slow down the result) on Dokan implementation only https://github.com/billziss-gh/memfs-dokany/commit/03ebaebb1....
The lock exists on WinFsp/MEMFS. It is just taken in a different place:
https://github.com/billziss-gh/winfsp/blob/master/src/dll/fsop.c#L60
The WinFsp/MEMFS had indeed a problem which I fixed, but it was not WinFsp related (it was MEMFS related - aka my test user mode file system). I attempted to apply the same fix to Dokany/MEMFS, but I could not because Dokany does not pass an Offset parameter to its FindFiles operation.
> You also improve memory allocation on winfsp memfs implementation and again not on dokan https://github.com/billziss-gh/winfsp/commit/d12234bb01b3620....
Actually that is incorrect.
https://github.com/billziss-gh/memfs-dokany/blob/master/memfs-dokany.cpp#L88
No. WinFsp is faster, because it is faster. Feel free to send me any pull requests that you want for Dokany/MEMFS and I will incorporate them into the project and re-run the tests.
It is interesting to see that NTFS is doing extremely well in a few different tests that are entirely satisfied from the cache: the list directory operation and the cached read/write operations. OTOH it does not do as well in tests that could potentially be satisfied from cache (like file open tests).
In any case I intend to retry the tests using NTFS on a RAM disk when I get some time.
The deception is not intentional. For me it was important to verify that my project is not slower than NTFS when using my test file system, which is an in-memory file system. You may notice that Dokany is actually slower or on par with NTFS in many scenarios even with an in memory file system.
Trying NTFS on a RAM disk is a good idea. I simply did not think of it.
There is no on disk implementation of WinFsp that I can share at this moment.
Please let me know if you have other questions.
Please also note that I want to convey that my solution is faster than Dokany, which is a competitor solution.
<< This performance testing shows that WinFsp has excellent performance in all tested scenarios. It outperforms NTFS in most scenarios (an unfair comparison as NTFS is a disk file system and WinFsp is tested with an in-memory file system). It also outperforms Dokany in all scenarios, often by an order of magnitude. >>
My intent was to measure performance of WinFsp against NTFS in order to establish a baseline and to show that building file systems in user mode is not as inefficient as people think.
You should also note that a hacker news submission is only 80 characters and it is hard to clearly communicate one's intentions fully.
1. The only Dokany thread that I have explicitly mentioned WinFsp is this one (and only after someone else mentioned it). I avoided doing so on your Reddit thread, for example. You may consider that my original message to the Dokany mailing list was promotion, but from my perspective at least it was simply a message to like minded people.
2. That until Lyrina's message(s) in cygwin-apps, where she effectively sabotaged my ITP submission, I was willing to work with Dokany and I said so in a message to the Dokany list.
The truth is that until that ITP submission point I was not bothering anyone and was just working on my project. The only two times that I made comments on Dokany vs WinFsp was when I was explicitly asked: one time by you on the Dokany list and one time on Reddit in WinFsp's own promotion thread.
So I agree with you: let's stop the arguments here. I am willing to stop hijacking threads or trying to sabotage your project's progress. However I also want Dokany to stop insinuating that WinFsp is not being tested (where my argument about the test suites stems from) or sabotage WinFsp's progress.
Bill
OTOH, here is how I got involved with this. I was developing a file system on OSX that I then ported to Linux. When I looked to port it to Windows I did not find any satisfying solutions and I ended up writing WinFsp.
Why port to Windows? Because it still has the biggest market share and because it was an interesting technical problem.
> 1. the license
I am pretty convinced that this is a major issue. I am looking for a solution that would allow unrestricted open source use while restricting closed source use. For example, freely combine with a BSD licensed open source project without imposing any restrictions on it (GPL), but disallow combining with a closed source project (unless a commercial license is used).
I am wondering if such a license (or license strategy) already exists.
> 2. windows
Ok, I hear you there :) This will sound weird, but I do not even consider myself a Windows developer anymore (despite being an ex-MSFT and writing Windows kernel mode drivers). I am writing to you from a Macbook Pro and I make a living these days writing Mac and iOS apps!
That's actually a cool idea.
The API's are not dissimilar as both API's were designed for Windows. If you decide to look into this you can count on my help and support.
BTW, please consider contributing to WinFsp :) You do not have to contribute to the core project if you find the license unpalatable or any other reason. But there are other ways that one could help. Creating a C# or Java wrapper would be very useful, for example.
The cygfuse project also needs help and this one intends to work with both WinFsp and Dokany:
https://github.com/mgeisert/cygfuse
> Many of your potential contributors may also lack the required operating system to do the work...
That is an interesting thought. I wonder if I could spend some time to try making this compile on Win7.
That is what I do for development, between releases.
BTW, I am not really trying to make WinFsp into a product. I simply created it because I am a file system writer myself and wanted a stable FSD for Windows.
I was originally planning to put it under the BSD, but I spent too much time and effort to simply give it away.
I suspect that there is more than that. I suspect that people no longer like the GPL. I am actually one of those people.
My intent with WinFsp is to allow any open source project to use it without restrictions (commercial or otherwise). The only thing I want to restrict is closed source commercial uses (in which case I offer a commercial license).
Unfortunately I am not aware of an open source license that is compatible with other open source licenses and does not impose any restrictions on them. The only open source licenses that I know that allow unlimited open source use, also allow unlimited closed source use (which I would like to restrict).
If anyone points me to such a license I will be happy to consider it for WinFsp.
Yes, I did have savings. Yes, I was single at the time. I recognize that it would not have been possible otherwise.
Years ago I (semi-)retired from the corporate world for similar reasons and because I thought I would have more fun working on projects related to my hobbies. Turns out that even that got old eventually.
Then a couple of years ago I started working on something else and got renewed passion which continues until today.
Can you take a long break (say 6 months) and go do something completely different (like spend time in the Sahara or something)? When you come back you will have a new passion for your work. Or perhaps you will have discovered a new passion :)
Good luck!
[Nothing against C# BTW. I used it for years while employed at MSFT.]
Dokan calls IoRegisterFileSystem on disk file systems, but does not appear to do so on network file systems. FsRtlRegisterUncProvider is the pre-Vista MUP and does not do it for you.
WinFsp has multiple integrated test suites, some of which Dokany uses (with tests missing/disabled). It also has hundreds of users at this point.
https://github.com/billziss-gh/sshfs-win
Download binaries from here: