BeeGFS, the Parallel Cluster File System | Hacker News Reader

BeeGFS, the Parallel Cluster File System

beegfs.com

56 pointsrandomname244 comments

Can someone explain how this differs from something like HDFS? I'm kind of confused as to why there's no comparisons or mentions of HDFS here which makes me think I'm missing something important about what this provides that's special. Is it the easy to install part?

HDFS is designed mostly for immutable files, stores three copies of everything in the default setup and has the whole hadoop calculation stuff.

BeeGFS is closer to a normal NFS share. In the default setup it just stores the data once, striped over multiple machines and it supports efficient updates in-place etc.

http://www.beegfs.com/docs/BeeGFS_EULA.txt

Umm, how is this compatible with GPL v2? It seems more restrictive (enterprise features, parts that would be better enforced by a trademark policy).

Speaking of trademarks, there is a band who would want to talk to these guys.

How is there any potential for commercial confusion?

You could store music in the filesystem ;-)

It appears that only the "client module" is GPLv2.

found this interesting:

https://indico.cern.ch/event/346931/session/4/contribution/6...

> Plain BeeGFS is as reliable as your hardware

currently i would give MooseFS, orangefs or lizardfs a try.

Had experience with ceph and glusterfs and both I can not recommend.

Could you elaborate on this please?

I'd be also interested in a bit of info why Ceph and GlusterFS disappointed you. I tried GlusterFS a few years ago and it took a good while to figure out the right setup but in the end had disappointing small file performance.

Ceph was very interesting but wasn't product ready when I looked at it and since then it somehow never got to a stage where people would rave about it so that alone makes me think it's maybe too complicated. A bit like BTRFS which also for years and years is being developed and sounds really promising but never seems to be finally ready without gotchas.

I also tried ceph and gluster before settling on moosefs a couple years ago -- gluster was slow for filesystem operations on a lot of files and it would get into a state where some files weren't replicated properly with seemingly no problems with the network for physical servers.

Ceph at that time was unstable -- once I had a couple boxes kernel panic in testing, I jumped off that boat immediately.

Moosefs is pragmatic -- written in pretty tight C, performant, and the web UI for seeing the state of the cluster and how files are replicating is very nice. We have a small cluster with maybe a dozen nodes running for almost 2 years now with no hiccups...

Kernel panic? Were you using ceph or cephfs? Ceph block storage shouldn't have any loaded kernel modules.

We were trying out the filesystem -- it was clearly alpha/beta quality at that time, so it was no surprise it didn't work, and perhaps it's improved in the last two years.

cephfs is clearly marked as not for production. It still has more than its fair share of bugs and has single points of failure. From the very top of the docs page on ceph FS:

"Important CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters"

RedHat is focusing on making the block and object storage bulletproof. They're content to let the opensource side twiddle away on ceph FS.

Ceph block storage or CephFS?

MooseFS and BeeGFS don't really seem to be in the same category of filesystem though.

BeeGFS comes from the HPC world where it is all about performance, while MooseFS seems more focused on high reliability even in the face of entire machines coming and going.

BeeGFS stores files striped over multiple machines and can use infiniband natively which gives us a system where bandwidth to individual files can reach a bit over 1GB/s (best case) and aggregated bandwidth can reach 30GB/s from a very mixed and unoptimized bioinformatics workload, a lot more when just testing the raw bandwidth.

Since BeeGFS uses an underlying filesystem on each storage target you can of course run raid, zfs or whatever it takes to make you comfortable that the individual storage targets aren't going to be lost - which is what it takes for data to be unavailable.

If you want some extra reliability in BeeGFS it also supports mirroring so you only lose data if you fully lose two storage targets. We can't really afford to run with full mirroring for our 3PB though.

We are very happy with it for our HPC environment but I'm not sure how well it works in an AWS setup.

The company I work for has been running BeeGFS for 6 years with no issues. Write and read speeds on Dell storage hardware match or exceed the NetApp appliances we use.

We do occasionally have issues where ALL storage nodes must be rebooted but other than it works great.

The only thing I wonder is if the name is a pun on https://en.wikipedia.org/wiki/Bee_Gees, the band from the 1950s.

or if you closely look at the logo..

Allegedly it's good at Staying Alive...

Actually there isn't a easy file system which spans 3 computers and scales to thousand and is eay to install. There are only one's which are hard to install, or needing more than 3 computers, etc..

Honest question: how does GlusterFS not meet these goals? I know that it's easy to install, and can easily run on three computers. And I believe it can scale to thousands.

Gluster provides files storage but not block storage.

True enough, thanks!

Actually GlusterFS is close to my needs, however the "easy to install" and actually bad performance on many small files is a road blocker for us. However it runs well, even on just 3 servers.

http://www.ceph.com/

pretty hard to install (ceph-deploy makes it easier) but I've seen some pretty spiffy claims like really easy scaling and provisioning.

I've run a ceph deployment for about 18 months now, it's good. ops-wise it's like anything, a few quirks to learn - but it's very solid. I played around with a couple of puppet modules and ended up on ceph-deploy as well, makes life quite easy.

Out of curiosity, how big is your cluster in terms of nodes, capacity, throughput, etc?

I will say that I know two people that have independently tried fairly large trials (400TB+) of Lustre, GlusterFS, and BeeGFS, and BeeGFS was their eventual favorite.

Nice anecdote, but can you provide at least ANY detail on why they picked BeeGFS?

This doesn't fill the huge gap for Cloud NAS. GlusterFS is hugely complicated and a fragile install, SoftNAS has (anecdotally) poor performance on high-io... but it's closer. Amazon hasn't launched their EFS product to General Availability Yet.

So wtf do we do when we need a 5TB shared storage platform across some EC2 nodes?

At this time you will need to use an EC2 instance serving NFS over a big EBS volume.

Check out Avere systems. They have a clustered NFS appliance that you can spin up in EC2, which translates back-end S3 object storage into NFS for your clients. It supports between 3 and 50 nodes, and uses local RAM and SSD cache to accelerate performance. Avere has also been in the NAS acceleration/caching business since 2008, so their product is very mature.

Thanks for the heads up on Avere. Use case for us was for file parsing as part of a batch data pipeline. The latency we saw on alternatives was killer compared to directly mounted EBS. That would be my fear with an S3 option as well, especially when it presents like NFS.

Weird, the URL to the GPL'd source code behind a registration wall.

For what I read, this is to know a bit who is interested in getting the source code. I give you the code but you tell me who you are. This is not prohibited by the GPL.

Still seems a bit weird though? Probably enough to put a lot of people off checking it out.

Yeah, sounds like they would be better off putting up the sources on a public git repo and then develop it in the open. Closed FS development for Linux at least does not seem to work well based on history. They can always have paid support but the closed development part doesn't bode well.

Given that they would like to keep copyrights on the contributions of third-parties, I doubt many outsiders will contribute.

Shame though...

The obvious response to this silliness is for one person to download it and then make it available by torrent.

Or mirror it on GitHub

The source is at: http://www.beegfs.com/source/

The source is released under the BeeGFS license... which is highly not GPL compatible (or copyleft) ... Only the clients are GPL'd

It's a tragedy to think of how deep is your love for the project. Wonder how long it will be stayin' alive? Gives me a night fever just thinking about it.

Reply on news.ycombinator.com