Hacker News Reader

Comment by cies | Hacker News Reader

Cannot find much on mounting storage to the container. It exactly that --storage-- that has been recognized as a difficult problem when moving an infrastructure to containers.

Given the title "The Hard Way", I kind of expected it would dive deeper into this topic.

View on HN

Of course - following the first law of technical documentation, the amount written about it is inversely proportional to the complexity of the problem.

Seriously though, it is getting a bit silly how almost every single kubernetes demo involving persistent data just uses emptyDir or hostPath and we are all supposed to pretend that those are a production-ready solution that won't lose data. AFAIK the PersistentVolumeClaim and StorageClass stuff will be the solution for this?

To me it seems that there are many approaches. A storage cluster seems to be a necessity for anything serious. But then the questions is: which one? Open source solutions are not too many (Gluster, Ceph, FreeNAS come to mind), and from the closed source camp ScaleIO looks really promising.

But non of these are part of your K8s tutorial :)

These storage systems are meant to be SAN devices. Meaning either Amazon EBS volume or Google volumes, on their respective clouds.

I recall reading that EBS is being [partially] implemented and tested. (Read: The documentation is full of "known bug: ..." and "please don't try that in production").

Probably depends on the scale/scope of your deployment. I know NetApp has a docker plugin for their storage platforms, I would imagine other vendors do as well:

https://github.com/NetApp/netappdvp

And gluster at least has horrible performance and stability issues. And with containers, kernel panics kill an entire host…

Do you have more details on performance and stability issues? The recent releases of gluster have been working very well.

Gluster is completely in userspace. How would stability issues, if any, in userspace cause kernel panics? Can you comment more about that?

> Do you have more details on performance and stability issues?

Tried using it for a while (last November until ~April) on Debian Jessie, with latest versions. Lots of bugs with SSL mode, lots of desynchronization issues, crippling performance issues, …

And no matter what I ran into, the bugs were already known and filed for months at that point, with zero developer reaction.

> Gluster is completely in userspace.

Unless you try using its NFS server to get not entirely disastrous performance.

What workloads were you using Gluster for? How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed.

> Unless you try using its NFS server to get not entirely disastrous performance.

Wrong again, Gluster's NFS server is in userspace too. Not sure how that can cause a kernel panic.

> What workloads were you using Gluster for?

Sharing config files. Wordpress sites. Django sites. The performance was too shitty for everything.

> How did you reach out to the developers? Usually the mailing lists are very responsive and most bugs brought up on the lists are addressed.

IRC. I ended up finding all my errors unresolved by it in the mailing list archives with zero developer attention, and since I didn't have the time to help RedHat make their product actually usable, I dropped Gluster.

> Not sure how that can cause a kernel panic.

I guess it's technically not a kernel panic if I/O just stalls so hard that you can neither mount nor unmount nor otherwise access any of your NFS targets… but you have to hard reset the whole machine anyway.

we used to run 3 gluster nodes, with a few hundred gb of shared storage..mostly used to keep logs and as war file directory for jboss/tomcat. from what i can tell gluster itself seems ok..but someone also enabled ctdb..which seems to hang every half a year or so..

Lol, this is how I've been feeling about every virutalization and container/jail solution since the dawn of this century. The early jail-type solutions were somewhat excused, because the answer was either "local filesytem", "nfs" (later nfs and/or iSCSI) - or for the adventurous some kind of cluster filesytem.

On that note, are anyone using encrypted NFSv4 in production? On paper it ticks most of the right boxes, but I'm not convinced it actually works...

It's mandated by our security team, but no-one has ever got it working...

I can understand the frustration. Been watching this space to see a plausible story on having storage volumes available to containers from any host.

Histpath and emptydir aren't really acceptable as solutions since they add complexity to your cluster set up.

There are some cluster storage technoloies out there but there are no tutorials, or overviews detailing, the pros and cons, and their performance limitations in a k8s environment.

Networking was the hot topic this time last year but storage is the topic no-one is willing to talk about. Tectonic is attempting to build a solution designed for the container use-case[1]. But that's a long way off.

If you are building cloud-native applications on-prem and you need state you're on your own for now.

[1] https://coreos.com/blog/torus-distributed-storage-by-coreos....

> If you are building cloud-native applications on-prem and you need state you're on your own for now.

Same conclusion I've come to.

I guess K8s is just one part of the solution, it needs to be paired with a storage solution to be really able to replace existing infra. When looking at open source solutions the landscape is pretty empty with Gluster, Ceph and FreeNAS. Where only Gluster and Ceph provide some level of HA.

Check out the "persistent volumes" section of the manual:

http://kubernetes.io/docs/user-guide/persistent-volumes/#par...

as well as the "volume management" section:

http://kubernetes.io/docs/user-guide/volumes/#types-of-volum...

Those two should give a good overview of the available options. Basically, you can use local paths, NFS shares, vendor-specific storage (e.g. AWS EBS) and various other things.

> Those two should give a good overview of the available options. Basically, you can use local paths, NFS shares, vendor-specific storage (e.g. AWS EBS) and various other things.

Yeah, but which of them actually work?

> Yeah, but which of them actually work?

And on top of that, which of them work when scaling the app out. In other words: how does each option deal with having multiple containers mount it.

edit:

From this link -- http://kubernetes.io/docs/user-guide/volumes/#types-of-volum... -- I found that the following volumes allow mounting by multiple writers: nfs, glusterfs and cephfs.

But which one of these works well under (production) load?

There is no single "production load"; everyone's load is different, so you'll have to try it and measure. This is always the case with any storage system.

It depends on the environment: If you're on AWS, you can use EBS, if you have an NFS share you can use that. In the simplest case, you can just a `hostPath` volume to mount a local directory from the pod's host.

If you're on GCE or AWS, GCE PDs or AWS EBS works out-of-the-box (i.e. using kube-up, kops or kube-aws on AWS, kube-up or GKE on GCE).

Everything should work, but it's harder to validate / work around every NFS server's possible set of configuration options (for example).

Petsets (http://kubernetes.io/docs/user-guide/petset/#what-is-a-pet-s...) are adressing the problem of storage persistently linked to the container.

One way to handle persistent container storage is to use a shared filesystem such as glusterfs or objectivefs. It's easy to set up and can be very helpful when moving your infrastructure to containers. Once created you can mount your shared filesystem either on the host and share it with the container or directly in the container, see e.g. https://objectivefs.com/howto/how-to-use-objectivefs-with-do...

If I understand correctly you are suggesting a block storage solution. But that would then prohibit a scale out, right? (I'm reading on the ObjectiveFS site that it does allow the concurrent access of shares... Hmmm.)

The other solution is object storage, which is the real way forward (IMHO). This route make a app more adherent to the 12-factor principles.

Both GlusterFS and ObjectiveFS are shared filesystems instead of block storage, so they work with POSIX files/directories (like NFS) making them easier to scale out. If your application can work directly with an object store that is of course also a great way to go.

Check out Quobyte: https://www.quobyte.com/containers

Reply on news.ycombinator.com