Hacker News Reader

Comment by yobert | Hacker News Reader

I've been trying to use podman in production and it is not working very well. I'm excited for this technology but it is not ready.

- podman layer caching during builds do not work. When we switched back to docker-ce, builds went from 45 minutes to 3 minutes with no changes to our Dockerfile

- fuse-overlayfs sits at 100% CPU all day, every day on our production servers

- podman loses track of containers. Sometimes they are running but don't show in podman ps

- sometimes podman just gets stuck and can't prune or stop a container

(edit: formatting)

View on HN

I could very well be wrong but Podman seems to have missed the time-frame of opportunity. It was always a knife fight between Red Hat and Docker with regard to tooling. Red Hat wanting to own the toolchain for containers so they didn't have to deal with, and so they could box out, competitors like (now basically defunct) Docker Enterprise.

I've taken a look at podman from time to time over the years but it seems like it's just never formalized, never been polished and almost always has been sub-par in execution. On this list the builds and container control are things that I've run across. I guess - what's the point? The rootless argument leaned on so heavily is pretty much gone, the quality of Podman hasn't (seemingly) improved and now IBM owns Red Hat (subjective, but a viable concern/consideration given what's recently happened with CentOS).

You're more than safe leveraging Docker and buildkit (when and where needed). Quite honestly, given the relatively poor execution of Red Hat with these tools over the years, I don't see the point. I'm sure there are some niche use cases for Podman/Buildah, but overall it just seems to come up as an agenda more than an exponentially better product at this point. Red Hat could have made things better, instead they just created a distraction and worked against the broader effort in the container ecosystem.

> Red Hat wanting to own the toolchain for containers so they didn't have to deal with

This is a misrepresentation. The situation was that Docker didn't take patches, as some were very specific changes for systemd and lack of unionfs, etc but over time it applied to most patches from RH associated people.

I'm not sure that's entirely true. If you look at the history of container tooling, all of the FUD Dan Walsh used to spread and Red Hat pulling support for Docker packages on RHEL I believe what was stated to be fairly accurate.

Edit: You also may have different perspective given you work for Red Hat / IBM.

Not to mention podman's historic[1] lack of compose support (or similar), which is frequently used in CI and local dev environments. I think has really held it back for a lot of folks.

[1]: Apparently initial support planned in version 3.0?

I don't know if I'm misunderstanding, but if by 'compose support' you mean docker-compose, then podman-compose has existed for a lot longer than v3.0 (which got released yesterday). First commit was nearly two years ago.

Sadly it doesn't feel as polished as docker-compose, I can only assume it's due to the lack of API. Specifically, podman-compose just translates all the docker-compose.yml file directives into podman cli commands, which doesn't seem to be handled gracefully.

I've been running multiple services through podman on a pi without any issues. Certainly not the same as production though.

Layer caching seems to work for me. Note that rootless stores images (and everything else), in different place from rootfull. It may be that you're caching the wrong directory.

I'm surprised it's using fuse-overlayfs instead of the overlay2 driver. Is it that you're using rootless containers in production? For rootful containers I think you'd have the same experience as docker here (or at least much closer).

Reading through the comments though it looks like I should try making a BTRFS mount for the container storage and that might help the fuse-overlayfs problems though.

I know it tends to be a controversial technology around here, but I've had great luck with OpenZFS for container storage.

It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho.

Now I understand that btrfs is not really a requirement.

But I've been thinking about using podman to replace docker-in-docker on our fleet of gitlab runners where we build our images, and running btrfs is a deal breaker. We really don't want to add complexity to the mix. Fuse-overlay burning through the whole CPU is not really good looking either. Mh.

I still have to properly dig deep into this, but I'm keeping my hopes high.

>It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho.

Why? Storage is literally the backbone of technology. From databases to webservers there are unique and real requirements. There's a reason why the enterprise storage market is still worth several billion dollars, and it's not because storage is easy.

In 2021, with billions of dollars of R&D at their disposal - neither Amazon, Google, or Microsoft have even a mediocre NAS stack in comparison to the likes of Dell/EMC (Isilon) or NetApp (ONTAP). Heck they can't even compete with startups like Qumulo or Vast.

Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves? Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. They make far more revenue leasing storage to the enterprise. Cloud storage is also recurring revenue where enterprise storage is considered perpetual. The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise).

>Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves

They have "competition" in the market, it's just horrible. Because storage is hard.

>Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend.

I'm going to let you go back and review the earnings reports from the aforementioned companies. Razor thin margins? You wouldn't survive in the market with razor thin margins.

> I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes.

I'd argue Azure Stack, AWS Outpost, and Google Anthos readily prove you wrong.

> Cloud storage is also recurring revenue where enterprise storage is considered perpetual.

Just... no. Enterprise storage has a shelf life of 3-5 years before maintenance or technology make it obsolete.

>The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise).

https://www.google.com/finance/quote/NTAP:NASDAQ

5 years ago they were at $20/share, today they're at $69/share. Reality doesn't appear to match your claim as to the street's opinion of enterprise storage.

First, if you review the financials of the storage companies compared to earnings of all cloud companies they're abysmal.

Yes, razor thin margins - comparatively. The cost of buying physical storage is commoditized. I've worked in technology from the pre-sale engineering side for a number of years. Margins on hardware are fractional compared to software and subscription offerings.

Stack, Outpost and Anthos are not revenue drivers today. They're vendor lock in tools.

3-5 years is a horrible argument in the position you're attempting to make considering it's an eternity in technology. Consider that you sell the storage once in that 3-5 years. Your cloud provider bills you monthly and you don't own it. It's recurring revenue vs a singular sales event.

NTAP over 5 years is a decent return. 215% over that timeframe. Amazon returns 543% in the same timeframe. And let's use a hot space that's 100% subscription based, like Crowdstrike: 265% in less than a year. Software and subscription margins crush the legacy perpetual hardware model.

Storage is an old, stable, and commoditized market. There's no huge growth (any storage earnings report show that very clearly). And while the demand for storage continues to increase, the NetApps of the world aren't capturing where that growth is.

>First, if you review the financials of the storage companies compared to earnings of all cloud companies they're abysmal.

I'm sorry but you're just flat out wrong and didn't bother to look up margins like I suggested you do.

NTAP 2020 margins: 66% https://www.macrotrends.net/stocks/charts/NTAP/netapp/gross-...

AMZ 2020 margins: ~41% https://www.macrotrends.net/stocks/charts/AMZN/amazon/profit...

An all time high... and not even close.

>Storage is an old, stable, and commoditized market.

Which is why... per my original post... the cloud providers struggle to provide basic features in 2021 that have been available to enterprise storage customers for 20+ years.

Your chart takes all of Amazon into account, not just AWS. You'll have to actually dig into earnings to understand that.

Second thing is gross margin is not the same thing as net margin. Net margin is a far better indicator in the case of pure profitability / revenue. Just because a company shares $1.42B in revenue doesn't mean they have $1.42B in profits. If you look at NetApp net profit margin it's off 45% year over year. Latest net profit margin is 9.68%. So on that 1.42B we have $137M in profit. Not too hot.

The reality is hardware sales have thin margin (comparatively) and NetApp's software model is much weaker than comparative storage offerings in the cloud. It's very expensive to design, build, ship and house inventory of hardware for purchase. There's no way around this, which chews through that profit margin.

>Your chart takes all of Amazon into account, not just AWS. You'll have to actually dig into earnings to understand that.

Yes, I'm aware Amazon continually buries their AWS profitability. It's 57% of the overall margin, it's not significantly better than NetApp.

>Second thing is gross margin is not the same thing as net margin. Net margin is a far better indicator in the case of pure profitability / revenue.

Net margin is a dial they can and do turn by shifting money in and out of R&D and sales staff. Furthermore you can't simultaneously say that for Amazon we only take into account AWS and not the rest of the business... which is a part of the engine that drives the AWS business. That's no different than saying we shouldn't account for hardware in NetApp's number because they also sell software.

>It's very expensive to design, build, ship and house inventory of hardware for purchase. There's no way around this, which chews through that profit margin.

What exactly do you think Amazon's datacenters are full of? Hardware they design, build, and ship inventory around the country for. They are working with the exact same ODM's that NetApp or any other storage vendor works with to design their custom hardware variations.

I'd wager heavily on AWS (i.e. Amazon) outperforming NetApp over the next decade. Why? Their operation model is far better positioned for profits. Honestly, if you consider storage vendors a significant growth market you and I have a very different opinion of what's driving profits in technology today.

All balance sheets can and are manipulated to differing extent. The reality though is net margin takes better into account than what you were proposing, which is a cherry picked angle to make a point. Growth companies move profits to R&D. NetApp isn't highlighting R&D on earnings calls. Amazon highlights R&D very publicly multiple times per year. Low R&D costs and investment are indicators of a stagnant market.

Amazon's hardware model is not the same as NetApp's. You realize Amazon consumes almost 100% of their own hardware offerings for AWS, right? They have a much more palatable JIT model. They can build as they need. NetApp has to house dead weight inventory for potential sales so they can recognize revenue, especially for end of quarter / end of year sales pushes. Amazon doesn't have this problem.

Also NetApp has to package their product for customer distribution. You may ignorantly shrug this off but it's an additional cost that adds up quickly. NetApp also has to maintain documentation and support staff for customer facing hardware related issues. Amazon has a much lighter requirement because they have specialists within their DCs.

It's not even remotely comparable. Amazon isn't working with "the exact same ODMs". Many ODMs NetApp is forced to work with Amazon doesn't need. Take a look at what Amazon is doing internally and it's clear their stack continues to evolve more towards in house designs and build. NetApp is far more reliant on external support than Amazon is given their positions in the market and have far greater control over the hardware stack from top to bottom. Maybe peruse the career openings at AWS vs NetApp for some insights.

> Which is why... per my original post... the cloud providers struggle to provide basic features in 2021 that have been available to enterprise storage customers for 20+ years.

Netapp is love, Netapp is life.

I have fond memories of cloning a volume, from a snapshot, in seconds. On EBS it takes minutes.

Have you tried Kaniko? That's how I got hid of DinD in my Gitlab Runners.

Could you maybe provide a small example of the relevant part of the gitlab CI definition? I assume you run kaniko inside a Docker container on the runner?

What are you exactly excited about?

It's just minor difference to me except it doesn't work well.

Not sure what the laziness was about from RedHat on podman development.

What are the Github issue IDs for these 4 problems?

Where are the four unit tests in podman's codebase that should be catching these issues?

I find it very strange that your reaction is, "the software is infallible, it is _YOU_ who have failed the software by not logging bugs!" It is perfectly reasonable to have a conversation about the quality or issues with a piece of software SEPARATE TO the logging or finding of bugs in said software.

What are you trying to say here? Your comment feels a bit passive-aggressive.

> What are you trying to say here?

I want to look at the problems. But I don't want to do a guessing game with the 159 open issues, which one may relate to one of these points or not.

I agree with the sentiment that I should go find them. But I also agree with the sentiment that if RHEL 8 is going to make it hard to install docker and promote podman instead as a drop-in replacement, it's a bit frustrating.

> I agree with the sentiment that I should go find them.

I can sympathize with feeling overwhelmed when searching for similar Issues in a bug database, as I was in a similar situation in the past.

What happend was this: I went to the central starting point of the repo's database (e.g., https://github.com/containers/podman/issues), removed the filters like `is:open` from the search bar, and entered one or two keywords that for me sounded similar to the problem I faced.

As I didn't find any Issues that seemed similar, I just created a new one. Of course, I was worried about creating a possible duplicate, but I figured that it would still be a valuable contribution by adding more keywords (with the maintainers linking my duplicate to the ticket where the problem gets resolved) for others to search for. And last but not least, I thought this would also add additional debugging information for the developers, as well as an indication of the community-wide impact of the problem for the release management team.

In the end, I was quite happy I went this way, because I didn't just make a small impact on a project that would be useful for me, but it also enabled me to connect to the amazing people that build the tools I use on a daily basis.

I hope that you can also find a way that gives you some happiness in dealing with your situation, yobert.

Reply on news.ycombinator.com