Show HN: Minimalist Object Storage Project | Hacker News Reader

Show HN: Minimalist Object Storage Project

github.com

33 pointsy4m4b429 comments

The first questions which I look for on objects stores are how it approaches data integrity – things I'd like to see on the homepage:

1. Does it have the S3/Swift/etc. semantics that you will always get the bits first stored or an error but never a corrupt or truncated file?

2. Does it have the S3 and (IIRC) Swift feature that allows you to provide an expected hash so data corrupted in transit will be detected and an error will be returned instead of storing the object?

3. How does it verify data integrity? Only on access, background scrubs, via API request, etc.

4. Which strong hashes are used for all of the above? Are they configurable / selectable so e.g. if I wanted something strong than S3's Content-MD5 header for uploads can I use SHA-256 instead?

5. The classic easy-to-ask, hard-to-answer one: what level of testing backs up all of the previous answers?

> 1. Does it have the S3/Swift/etc. semantics that you will always get the bits first stored or an error but never a corrupt or truncated file?

Yes you are right the behavior is exactly similar to how an S3 API would behave for Get/Put().

> Does it have the S3 and (IIRC) Swift feature that allows you to provide an expected hash so data corrupted in transit will be detected and an error will be returned instead of storing the object?

Yes, just like how Content-MD5 is set through a client request for per uploads in S3, Minio honors that as well.

Minio provides three modes of operation

-- memory (in memory object storage)

-- filesystem (point to any directory or disk, all files are contigous and not erasure coded)

-- donut (experimental, distributed erasure coded on disk format)

In Donut individual parts or chunks are separately sha256 summed along with erasure coding on top of these individual parts.

https://github.com/minio/minio/tree/master/pkg/storage/donut

Donut also implements rubberband erasure which is a dynamic erasure coding style where in cumulative parities are created for all objects depending on the available disks, total usage etc.

> How does it verify data integrity? Only on access, background scrubs, via API request, etc.

Donut also does data integrity checks on a regular basis as an operational process, where even silent data corruption is fixed automatically without the need for any user intervention.

We are in early stage of our project, as of today Donut is experimental - currently runs on a single node, does sha256 sums for erasured chunks and erasure codes your data on disk.

Much of the operational process and distributed donut is a work in progress, we are planning to get them ready by end of this year.

> Which strong hashes are used for all of the above? Are they configurable / selectable so e.g. if I wanted something strong than S3's Content-MD5 header for uploads can I use SHA-256 instead?

Yes while Content-MD5 is what we do now, along with erasure coding and sha256 summing individual chunks. We can provide the similar semantics of ``x-amz-content-sha256`` from client side. We would be happy to implement it feel free to open a bug - https://github.com/minio/minio/issues

> The classic easy-to-ask, hard-to-answer one: what level of testing backs up all of the previous answers?

I agree with hard-to-answer :-). Minio is written in Golang, which largely drives us to "write tests first" style and we have rightfully followed it on a day to day basis. Not being limited to that we are also working towards building a continous integration testing framework which would be baked into the product, which would help any user perform self diagnostics of his/her setup. In-turn generating reports for quality of their setup, any unknown parameters which might show up etc.

Also testing is an ongoing process, so we are striving to be better everyday :-)

Thanks for the questions, feel free to hop on to - gitter.im/minio/minio and find us there.

Thanks for the reply – it sounds like you're definitely hitting the areas I care about.

One quick thought: I was at USENIX the year that ZFS was announced and one thing which was very popular was when the Sun team talked about how much random testing they'd done with billions of runs exercising the major features with simulated failures, memory corruption, etc. so they could stress-test the detection and recovery mechanisms:

https://blogs.oracle.com/bill/entry/zfs_and_the_all_singing

You could see a lot of sysadmins nodding their heads as they described each class of failure mode and how they were trying to simulate it.

These days I'd probably toss in something like running afl-fuzz against your storage code, too.

> Thanks for the reply – it sounds like you're definitely hitting the areas I care about.

Thank you, appreciate the inputs :-)

I'm new to the concept of object storage. Is the thing which distinguishes object storage from key-value storage the ability to also store metadata?

I.e., Instead of providing (key: value), it provides (key: (metadata, value))?

You can think of object storage as a type of key value storage optimized for storing unstructured data at scale. These include but are not limited to images, video, backups, and binary blobs.

You can contrast this to other key value stores such as memcache which is designed to cache data in memory or Cassandra which is designed for storing large quantities of small, structured data.

I've seen this an an outgrowth of the file storage world:

1. Flat data model – i.e. no directories, just a single globally-unique identifier 2. Much larger sizes than many key:value stores are designed to handle, typically bounded only by physical storage 3. Updates are complete and atomic: you don't have to support all of the crazy permutations like streaming writes from a file handle which could be kept open for years, dealing with simultaneous writes, etc. which make life interesting for filesystem implementers 4. None of the more complicated filesystem behaviour like locking, alternate data streams, etc. which add so many edge cases

I don't understand what is this about.

Why the downvotes? I am genuinely curious also, but there is next to nothing in terms of details (uses cases and such) on their site!

We don't have any yet, we are just few months old project. We will will have a distributed server by end of this year.

We have stand alone server now which includes currently three modes of operation

-- memory

-- filesystem

-- donut(experimental)

A small video to help you download and start a memory mode - https://asciinema.org/a/21575

Client tools and libraries are ready to be used by community to give us feedback on.

- https://github.com/minio/mc - Minio client collection of tools like `ls`, `cp`, `sync`, `cat`

- https://github.com/minio/minio-go - Golang

- https://github.com/minio/minio-js - Nodejs

- https://github.com/minio/minio-java - Java

What's "donut" storage?

Donut - is a distributed erasure coded backend format, designed specifically for immutable data.

We call it "do not delete" in short donut.

https://github.com/minio/minio/tree/master/pkg/storage/donut

It is currently experimental, but a distributed version is a work in progress.

It is an open source alternative to Amazon S3. Just few months old project. Client utility 'mc' and language libraries are already usable.

While the concept is interesting, it hasn't built the only real detail that matters from an Object Store perspective:

Cheap, Multi-Datacenter, Performant, Highly Available storage.

"YAS3I" / Yet-Another-S3-Implementation isn't really compelling even if it is open source.

I can pull that out of https://wiki.openstack.org/wiki/Swift/APIFeatureComparison#A...

if I really need that.

We are in an early stage, we will have a distributed version by end of this year.

It isn't about "YAS3I". It isn't compatible just for the compatibility sake.

Minio implementation is opinionated: "There is only one way to do it". Given a choice between Amazon S3 and OpenStack SwiftAPI compatibility, we picked Amazon S3. As SwiftAPI gains adoption, we may switch over to SwiftAPI, but will have to drop S3 then. Let the community decide.

Tiny piece of advice, declaring a possible total revamp of the API as soon as you launch might not be the best way to attract early users.

Advice accepted. The choice is for the community to make, we will not make that choice unless community strongly wants us to do so.

From working at an Openstack startup for a while, I can say that customers definitely prefer the S3 API since it works with their existing tools. That being said, they are largely compatible until you get into the authorization models, and the fact that S3 is globally namespaced, whereas Swift scopes names to tenants.

What specifically separates this from Swift or Ceph, both of which have S3 compatible APIs and are considerably better documented/hardened/maintained + distributed?

Minio is about minimalist design, take a look at the blog post to get an idea https://medium.com/@minio/minio-no-knobs-to-turn-no-button-t...

At this early stage of the project it is hard to explain why another approach is necessary.

Why not just use Swift?

http://docs.openstack.org/developer/swift/

It can be run as a stand alone app so it's generally pretty simple and can include support from https://swiftstack.com/.

How do you run it?

https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...

First time I'm seeing Swift.

Are there use cases where Swift is a better fit than Cassandra?

Swift is part of the larger family of OpenStack. Swift should provide (haven't verified this myself) better integration with all of the other OpenStack components (Keystone, Cinder, Neutron, etc.).

Swift can integrate with OpenStack, but you can very easily run it stand alone, and many do. It has erasure coding, geo replication, and its a fairly well baked product.

Here are good overviews:

* http://docs.openstack.org/developer/swift/

* https://swiftstack.com/blog/2012/09/16/globally-distributed-...

And how Rackspace runs it at scale;

* https://www.openstack.org/summit/openstack-summit-hong-kong-...

As mentioned elsewhere on this thread, you'd use Swift (and other object storage systems) to store larger pieces of unstructured binary data (eg. images/videos/backups). The easiest way to do that in Cassandra without rolling a whole bunch of new code for yourself would be to store these things as gigantic rows in Cassandra, which isn't where it performs well. Typically object storage systems also come with a whole lot of features such as being able to control who views your files, the ability to upload files in piecemeal fashion, etc.

Free software and open source is all about choice. Gnome vs Ratpoison. GNU Emacs vs Vim .. :)

Personally I'm enjoying playing with Skylable SX server (and its libres3 s3 interface) - I'll be honest, I never really gave Swift enough of a chance. I've yet to get my head round it quickly enough so far. Found this one while researching other things

"Skylable was founded in 2012 by Tomasz Kojm, Luca Gibelli, Edwin Török and Alberto Wu, best known for their success with ClamAV, the leading GPL anti-virus for UNIX running on millions of servers and protecting billions of users."

http://www.skylable.com/products/sx/

"With SX Cluster you aggregate the disk space available on multiple servers and merge it into a single storage system. The cluster makes sure that your data is always replicated over multiple nodes (the exact number of copies is defined by the sysadmin) and synchronized. Additionally Sx has built-in support for deduplication, client-side encryption, on-the-fly compression and much more"

Very responsive in their IRC channel on Freenode too. I know I sound like a walking advert reading this comment back but I have no affiliation other than end user

Reply on news.ycombinator.com