Show HN: WinSpd – create “SCSI disks” as user mode processes on Windows | Hacker News Reader

Show HN: WinSpd – create “SCSI disks” as user mode processes on Windows

github.com

46 pointsbillziss33 comments

Note that creating virtual CD-ROM drives attached by SCSI and then loading them with virtual game CDs so that said game will play, was the entire basis of the drive emulator market for a while. :-)

I have also known people who use this technique to create an encrypted disk image which is distributed with their program which provides the 'secret bits'. Sort of like mount the image, decrypt it by giving it the key, and then launch the app which loads its .dll files from the newly decrypted attached disk. A bit of obfuscation which minimizes the impact on the application that you're trying to keep out of the hands of people without a license to it.

I am the WinSpd author. Feel free to ask me any questions you may have.

What is the advantage of your framework compared to implementing the iSCSI protocol (SCSI over TCP) and using standard tools like Microsoft iSCSI Initiator?

Good question. Unfortunately I do not know enough about iSCSI to answer this question fully.

In my case I wanted a simple way to create "SCSI disks" using a user mode process. Since I had already created WinFsp, which has a solid IPC (Inter-Process Communication) protocol, this felt like the best way forward.

This might sound like a dismissive question, so I should say upfront that I don't mean it to be. I'm truly curious and not overly familiar with this technology. With that said...

What is this for? What itch were you attempting to scratch which lead you to create this? How could it be useful in a project? / What sorts of projects would it it lend itself to?

I think it is a perfectly legitimate question. I had two reasons for creating this: a practical one and a theoretical one.

The practical one is that I wanted to be able to store disk data in a cloud storage bucket (i.e. "format" a bucket with NTFS and use it as a disk). For various reasons I did not want to build this as a WinFsp or FUSE file system.

The theoretical one is that I am exploring the possibility of writing a new OS where all/most IO/services/etc. could be implemented either in kernel mode or user mode. Not bolt it onto an existing OS like I have done with WinFsp and WinSpd, but build the OS like this from the ground up.

To finish here are some use cases for WinSpd as food for thought:

- Mounting virtual disks in a format other than VHD.

- Mounting a cloud storage bucket as a disk.

- Serving a read-only disk from an internet location for use by multiple users.

- Tracking disk I/O for backup, snapshotting or other purposes.

Great reply, thanks! I did wonder if this would be useful for integration with things like cloud storage, so good to hear that it was one of your primary driving factors.

How bug-free is it?

I tried Dokan and CBFS (Call Back File System) a long time ago. They were both great, but I ran into bugs fairly quickly with each.

MS is actually working on their own replacement for Dokan that is called Projected File System [0]. If I understand correctly, they need it internally for handling large git repositories, so it's probably there to stay.

[0] https://docs.microsoft.com/en-us/windows/desktop/projfs/proj...

As far as I can tell it is not the same thing. Projected File System overlays a part of a real file system, and files are cached to the local file system.

Agree. The Projected File System is very limited compared to WinFsp, Dokany or CBFS. I have looked at it quite extensively as my own WinFsp open source software is in the same space.

Dokany, a fork of Dokan is a very active project these days https://github.com/dokan-dev/dokany

Dokan and CBFS are more like WinFsp (another project of mine) in that they allow you to create file systems rather than "disks".

I like to think that WinFsp is very stable software. I am working to bring WinSpd to the same levels of stability.

Dokan and CBFS are at the filesystem level. WinSpd is at the block device level.

Yep, but all involve Kernel drivers. Is a block-level driver less error prone by its nature?

It has a smaller surface area (4 fundamental operations).

For WinFSP (not the software in question, but the same author, and maybe closer to what you want anyway), there's a testing strategy here[1].

[1] http://www.secfs.net/winfsp/develop/tests/

Thanks.

Here is the most updated version of those documents:

https://github.com/billziss-gh/winfsp/wiki/WinFsp-Testing

What advice would you give to a programmer wanting to get into this kind of low level Windows programming?

- Be enthusiastic about learning! - Focus on creating something that scratches an itch of yours.

Does it work over network shares (SMB or NFS) and how would the performance look like in that case?

WinSpd is not a standalone application, but rather a set of libraries and components that allows others to create applications that can do what you want (serve a disk over a network).

The included sample, called rawdisk, memory maps a file and uses it as a disk image file. This could presumably be done over SMB or NFS, although I have not tested the performance.

Yeah. I thought of putting my Steam game library on my NAS so I can use the same library across multiple computers, but Steam doesn't like having its game library set on a network share.

Does it support async/overlapped I/O? Command queues?

Yes. Async I/O in Windows happens at the file system level, so the answer is yes, by default.

Command queues are supported by default as well, as this is a Storport virtual miniport driver (which supports command queues).

Why this over your own WinFsp? How does it compare to ImDisk?

WinFsp allows you to create file systems. This project allows you to create disks that can be formatted with existing disk file systems (e.g. NTFS).

They both have their uses but in general I think WinFsp is more useful for most.

I am not familiar with ImDisk.

Sorry for the second reply, but I read a bit about ImDisk.

I think the main difference is that WinSpd is a general tool for creating disks as user mode processes, whereas ImDisk is a tool for mounting image files only.

It looks like WinSpd is a more generic tool. But I know very little about ImDisk so I may be mis-representing it.

Just want to note, that ImDisk, mentioned here in the thread, while technically GPL too, can be stripped of non-GPL code easily, according to its author.

Interesting. I was not aware of ImDisk. Reading about it, it appears to be a RAMDisk only tool. Did I misunderstand?

"ImDisk is a virtual disk driver for Windows NT/2000/XP/Vista/7/8/8.1/10 and Windows Server 2003/2003 R2/2008/2008 R2/2012/2012 R2, 32 and 64 bit editions. It can create virtual hard disk, floppy or CD/DVD drives using image files or system memory."

http://www.ltr-data.se/opencode.html/#ImDisk

Thanks for the additional information :)

That still misses the point, that it provides a protocol dubbed "devio", that enables userspace programs to create these disks virtually, and handle access via own implementations of write/read primitives.

They too have a .NET wrapper BTW.

Reply on news.ycombinator.com