Hacker News Reader

Hi Lucas, I'm a bit sceptical, as most people will probably be. As mentioned by other here - more concrete benchmarking data will greatly help. Preferably include a comparison with other technology bench results using the same data.

On the topic of data - I wonder what was actually on those compressed disks you've shown in the videos, because as you probably know compression algorithms performance depends on the complexity and entropy of the data.

Also, NTFS and FAT filesystems are mostly zeroes, so they do shrink down a lot when compressed, and virtual disks can even be "sparse" to begin with..

In short - more information is required.

As for reaching out to research groups / investors - It can be difficult, and I know this from experience running my own startup.

Between you and me - most investors won't even understand what you've created. Assume they are non technical people. If it were the 90's again - most of them would be working in the stock market, not technology.

So dumb it down, publish more papers/data, and get people excited - In this business it's more about appearances than substance I'm afraid.. If there's hype - they will be interested.

Hope this helps

Hi Nihil,

This definitely helps, in fact this was the nicest post I've seen and I will try to reply equal care.

Sceptical - Yes I expected this, I didn't expect as much skepticism though.

benchmark - It is difficult to compare to something else as there's no one doing something similar (afaik) and would be comparing apples and oranges, nonetheless I will provide a comparison details at the bottom which is easy to understand to general public.

DATA - The disks have a clean installation of Windows XP, chosen due to relatively small size of 1.5GB compared to Windows 10 with 10GB and makes processing of the model much easier. If you compress the Windows XP image with Winrar it shrinks to ~500MB so around 30%, my method requires only 200MB physical space so down to 13%.

It also differs from traditional compression in that with more powerful hardware I can shrink the storage further preserving fast I/O, but it gets exponentially more expensive for each %. Still large companies can afford the benefit which is multiplied for thousands VMs.

Sparse file system - no, it's much better :-)

Additionally I'm having trouble to name my technology, it has much similarity with compression but it is not traditional compression, it uses some math functions such as number pairing function and others (I can't disclose the sweet secret) and achieves an I/O much better than a traditionally compressed file system.

Performance is in fact better than plain NTFS as can be seen in the third video, a Windows 10 boot in 5-8 seconds is better than an SSD can do (around 25 seconds). Ultimately apps and systems can be written to take most advantages of the technology characteristics.

Your last paragraphs were the most helpful I agree I need to get people excited, I was hoping I could do this mostly anonymously but perhaps not (don't need people to know my name). I will need to think what is best strategy.

Some benchmark. HOST ----------------------------------- OS: Windows 10 Enterprise Processor: Intel Core i5 8 Gen Memory: 16GB DDR Base Storage: SSD ----------------------------------- GUEST 1 (Normal) ----------------------------------- OS: Windows 10 Education Processor: 8 Virtual Processors Memory: 2GB dedicated VM Storage: SSD Boot time: 18 seconds ----------------------------------- GUEST 2 (New technology) ----------------------------------- OS: Windows 10 Education Processor: 8 Virtual Processors Memory: 2GB dedicated VM Storage: Enhanced File System Boot time: 8 seconds -----------------------------------

Lastly, this is not only application for VMs, same technology can be used for e-mail servers, databases, etc. Any more comments are greatly appreciated.

Thanks

sorry need to learn how to format text here.

Thanks for the detailed reply. Was a very interesting read, and definitely helped soothe my scepticism.

Regarding benchmarking - I think boot-time is not a very good test, as it varies greatly depending on the guest VM. A fresh install will be pretty consistent, but it's not a very reliable and repeatable test.

You need some hard data - graphs, charts and counts. Maybe check out DiskMark or some other tool like it, Then run test inside normal VM and your encrypted-disk VM.

Second - I wonder if there's a way you could let people experiment with your solution without "giving it away". for example-

1. Compression-As-A-Service - a simple website, people can upload a file, get the compressed file back. This way they can compare the size to Zip/7Zip/whatever, And you don't actually give them the algo (You'll probably say - "without my algo they can't open the file or use it, so they won't see the crazy fast IOPS." , But it might be enough, at least as a first step).

2. Compiled algo, limited distribution - I have a feeling you wrote this in some low-level programming language, not Python and such. Could it be compiled to a binary? If so it might be safe to distribute, as it will require serious reverse-engineering to decipher. You could also "watermark" each executable you send someone, so it has to "dial-home" via internet before it works, and will help you identify who released it to the public in case someone does.

Hope this helps, and please feel free to connect with me on other channels as I'd love to help some more :) my mail is nihil75 at gmail.com And I'm "nihil" on mastodon.sdf.org

Can you win the Hutter Prize? https://en.wikipedia.org/wiki/Hutter_Prize

It's a well known challenge and it's a simple way to have an independent validation of your method. The problem is that most people think it's too good to be true, so they'd like to see an independent validation.

Actually no, there's no one size fits all with compression, in order to win that prize you need an algorithm specialized in compressing natural language text. Mine is specialized in binary data and fast I/O so it's a different nature.

Maybe I will try to email the organizer though to see if there's interest in what I'm doing. Thanks!

Any reason the technology isn't explained in text? A video how fast a program starts under Windows isn't exactly a scientific measurement.

thanks for your comment, added second and third paragraph to the question. I choose video to demonstrate materially that this is real without disclosing too much implementation details too soon.

Again, trying to figure out the best approach.

The paragraphs help a lot, thanks for adding.

What is the goal? I mean you don't want to disclose the implementation so it's a teaser of sorts. Of a coming product? Patent? Selling the idea to companies? Getting a job or funding at a research firm? Does it have a name?

Yes correct. I can understand when you hear "technology breakthrough" the first thought that comes to mind is BS, but it is what it is.

I intend to sell it as a trade secret to an investor or company and they do the patent. Why don't I patent myself? Because I don't have a board of lawyers to do the fighting When it is copied.

As for name, I don't have a final name yet but NextGen Hard Drive sounds fit.

Hi Lucas, please keep us updated about progress in the future. I, and others, will be very happy to hear your story, as well as this breath taking invention.

I will, I got bogged down a little trying to win the Hutter Prize but it might take a while, at the minute my adapted model can only match winzip for that task.

Is this not DriveSpace all over again? https://en.wikipedia.org/wiki/DriveSpace

How interesting the new stuff I can learn in an open forum like this!

Well no it's not the same it has several differences but the two major are:

One, it can shrink storage up to 100 times, unlike traditional compression which saves 50-70% with prohibiting performance loss. Two, data access can actually be up to 100 times faster, while in traditional compression data access for many small files can be too slow.

It would be helpful if you could provide some benchmarks.

Here can I find more info about this project?

PS: Windows XP?!

Third video is windows 10 :) People claim on a fast SSD Windows 10 boot in 25 seconds. That is 5 seconds on the video.

I'm looking for guidance to take this to right attention and get a deal.

Show HN: New storage technology