Hacker News Reader

The thing that makes distributions like Red Hat and SUSE "boring" is not that we only do maintenance work, it's that we've effectively solved the problem of maintenance and release engineering -- though of course there's still improvements to be made (and on the SUSE side, openSUSE Leap and the plans for SLE15 are quite interesting developments).

We do a lot of work on exciting and interesting projects, some of which crash and burn as well. But part of the benefit of working for an enterprise distribution is that generally if some project is not ready to ship yet, we don't ship it. Having an out-dated version of some package is fairly common, and as long as we can do the maintenance work on it, it doesn't really matter. So we have a lot of time to improve upstreams to be stable enough to ship.

[I work for SUSE.]

I am super interested to see how the SUSE stemcell and rootfs work turns out, now that SUSE is contributing to Cloud Foundry.

> The thing that makes distributions like Red Hat and SUSE "boring" is ...that we've effectively solved the problem of maintenance and release engineering

Hmm.

There are a lot of developers that could probably really do with some exposure to this kind of environment.

I could definitely see myself hugely appreciating, say, a 6 month stint doing... something relating to the nitty-gritty of keeping a distribution going, learning what the pitfalls are, what the unintuitive stuff is, etc. Basically being some key point-person's assistant (the one who deals with major disasters), or something.

It would be even better if some of this work could involve direct enterprise support: learning what customer requirements are, getting a sense of scale, etc. I can see this kind of information (enterprise support) being the most valuable - actual bespoke software would likely be a major focus of such training, and this would tech would-be developers about the disasters that can happen 5 years or 10 years on from "I have an awesome idea" "wow it's so elegant" etc etc. (Continuing from the previous point, perhaps there could also be an opportunity to be the assistant to the on-call person who sometimes wakes up at 3AM.)

You could position this as something for people looking for DevOps jobs, and maybe make some kind of cert for it as well (this would likely help with/create traction for/be needed for enterprise-level interaction).

I personally would have zero issues with signing a small book of NDAs to get this type of training myself. The NDAs would just cover how I learned, not what I learned, and in this context the how would be largely irrelevant (but at the same time incredibly invaluable). FWIW, I'd personally greatly benefit from a good debrief at the end of the course to help me find all the little ways I might accidentally leak something (eg a few too many words fall out before I stop myself etc) and learn to catch myself quickly.

Agreed. I would add a second question to your first: What are the risks of implementing this into production? Get them thinking as a matter of habit about both of those in tandem - what's the customer benefit, and what are the risks of doing it, and does the former justify the latter?

Also, all programmers have a burning desire to build stuff from scratch, which is part of what's driving yours to incorporate cool new stuff like this at a risk to your revenue. No can do. But it may be worth exploring side or 20% projects specifically dedicated to building stuff from scratch. Get their itch scratched on non-revenue projects that serve as potential useful on-the-job training and tech evaluation, while possibly getting something of value out of it.

The burning desire to build from scratch is simply because it is easier and because it tend to be more verbally valued by management then working on older product which tend to be snarked at as merely maintennance.

The moment management praise also people working on less new and gives them appropriate salaries and autonomy, burning desire changes - except in few people who are truly naturally like that. It is win win, because the remaining people tend to be trully better in cutting edge tech problems.

I'm starting to have doubts about this line of reasoning. It just doesn't explain enough of the risky behavior I see.

I'm starting to think it might be the same thing that gets young developers to put up with 10+ hour days. That feeling you get when you lose yourself in a problem for hours and hours.

Being in a Flow state is seductive. Nobody gets that feeling when looking through the bug database or knowledge base, or hunting through the source of an OSS module trying to figure out how to ask it the right questions. But they do get that feeling when trying to rewrite it themselves. All that open space to run as fast as you can... in the wrong direction.

It's thrill seeking. BASE jumping for programmers.

Programming is based on hero worship. Anyone who makes even half an effort to better themselves as a software developer will encounter famous names from the past - those that "made a difference" by creating languages, operating systems and algorithms. Keyword here is "create". This is not an industry that celebrates maintence programmers. There's no glory for the guy that spent 20 years keeping the system that your banking transactions depend upon.

So naturally young programmers will see to be the next Richie, Torvalds or Knuth and to do that you gotta create something new. Gotta change the world

There are some wannabe-heroes, but they aren't so widespread.

The vast majority is just being pragmatic. Trendy toys and 'designed and implemented new X' in resume looks so much better for career than 'maintained something old so customers were happy'. Of course, there are some technologies that helps to make pretty good money even being ancient, like COBOL, APL, AS/400 and IMS. But for how long? So it seems rational to move forward instead.

That said, if you've got COBOL, Fortran, etc on your resume you're pretty much guaranteed to get asked about it in an interview.

Maybe discussion on mainframes transaction processing and COBOL features helping to implement it the way we know it thesedays as 'Cloud Functions' or 'AWS Lambda' might be a little bit more fruitful than reversing linked list pointers on a blackboard.

As a fairly young programmer, I think it's not necessarily hero worship (though that does play a part). I think it's because a lot of young programmers define their identities in terms of what they do. This is especially prevalent in the free software community. If you're going to define yourself by what you work on, it doesn't seem very fulfilling to just keep an existing project running without having the freedom to make drastic changes that would improve the project.

It should noted though that the above points about Red Hat and "boring" (as much I hate calling people's work boring) don't sit right with me. I know quite a few people who work at Red Hat on very interesting technologies. At SUSE, I work on fairly interesting technologies as well. The key aspect that makes SUSE (or Red Hat) "boring" is that we have effectively solved the problem of release engineering and maintenance (though in recent years SUSE has been improving on it further through openSUSE Leap and the recent plans for SLE15). The funny thing is that if upstreams could do their own release engineering correctly, SUSE and Red Hat would have a weaker value proposition.

The "boring" comment isn't meant to denigrate people's work. Rather, it should be taken as a complement for a job being done well. In fact I've heard the "boring" comment from a couple RH employees myself. So, I think they understand what is being said, rather than taking it in a derogatory manner.

You think that Torvalds or K&R are famous because of their creation and not for the long-term maintenance of their projects? That's pretty shallow thinking in my opinion. Yes they created things that we use and love, but we use and love them because those creators took years of their lives and dedicated them to the maintenance (both technical and political) of their projects.

I for one admire them for their dedication and vision far more than for their creations and wish more programmers would try to be like them.

I love the feeling of being lost I solving and how it can extend for hours in any particular day. It can happen with both old tech and new tech, old project or new project - the only requirement is large task. (Impossible in agile max 4 hours long task environment).

I havent seen people in that state for weeks on end. It explains peeks of productivity followed by drops of productivity. Does not explain long term crunch.

And I mean, programming is not nearly as risky as Base jumping. You don't die coding in wrong direction and you still get paid for it. It is annoying to be wrong, but imo, we are overusing word risk.

Also, all programmers have a burning desire to build stuff from scratch,

I don't think this is true at all. 80%+ of software work is maintenance related. People who have a burning desire for green field development won't last long so they are few and far between. In fact, this is one of the biggest differences between real projects in industry and fake projects in CS undergrad education.

Amen.

Sadly it seems in a sense RH is funding both sides at the same time, as many of their employees are the very source of the "exciting" developments happening in the Linux ecosystem these days.

Thing is though that in a sense the Linux community has painted itself into a corner. Thanks to the likes of RHEL, Ubuntu LTS and Debian Stable, there is little reason for the upstream to be anything but "exciting".

Out of curiosity, what are the exciting developments happening in the Linux ecosystem these days?

Checkpoint-restore of processes (CRIU) is a fairly cool technology that is quite unique to Linux, and has gotten a lot of work in recent years. eBPF (both for seccomp and tracing) is also quite exciting in terms of its applications. User namespaces are also allowing for developments such as rootless containers. There's also some interesting stuff for trusted computing that are happening with IMA and TPM developments in general.

In user-space there's even more interesting developments, you just have to look a little closer. ;)

CRIU looks amazing, live-migration of LXC containers :)

There are rough edges to be worked out before we arrive at solidly productionised systems, but I believe Red Hat and Docker are both heading in that direction.

Mind you, ask yourself: why do you need it?

If it's for apps, isn't the point of these platforms to not need to care about individual instances?

If it's for data, don't you have an existing way to manage HA or parallelism?

Yep, every time I hear people get excited about live migration of containers I have to give them a sideways look. If you are running only one instance, than availability must not have mattered that much to you.

You’re making assumptions that I’m thinking of Docker, which couldn’t be further from the case. I’m looking at using LXC/LXD to replace full-fat VM’s for highly stateful, single-instance services - game servers. Being able to live-migrate a LXC container to drain a host for maintenance is indeed a useful feature to have.

I usually assume all sorts of things[0]; I've become accustomed to living in a world where strict division between state and logic is assumed, even for relatively intensive stuff like trading platforms and very large ticketing systems.

Kubernetes has been more open to absorbing sticky, less-factored workloads than more opinionated platforms are. Red Hat have definitely made that point to folk whenever we and they are competing for business.

I had occasion to look at CRIU relatively closely in recent times. It is, as I said, early days, with annoying corner cases that need someone to patiently fix and polish (we ran into problems with networking, environment variables and process IDs).

In your case it will probably become attractive in the next year or two, because both Red Hat and Docker Inc (which is why I mentioned Docker) are touting checkpoint migration as a feature and I expect both will devote engineering effort to productionise it.

As an aside, if you're using the JVM, there was a really clever paper on coordinating the JVM G1GC and CRIU to greatly improve migration time: http://www.gsd.inesc-id.pt/~rbruno/publications/rbruno-middl...

edit: [0] Which is why as usual I am a dimwit for starting with answers instead of questions.

> Checkpoint-restore of processes (CRIU) is a fairly cool technology that is quite unique to Linux

DragonFlyBSD has had process checkpointing for more than 10 years.

https://www.dragonflybsd.org/cgi/web-man?command=checkpoint&...

While the functionality is admittedly & unfortunately quite limited, there are no technological reasons this couldn't have been furthered in the meantime given sufficient funding/dev time, etc.

CRIU has so many more features and is so much more powerful that I don't think it's a fair comparison at all to make. You might argue that you could have implemented all of the features on another system like DragonFlyBSD (and in fact I would argue it would've been easier than doing it on Linux) but I don't see why that statement matters. What makes CRIU "unique" is that it does have those features and it is used on production systems for live migrations. I remember there used to be very old Linux tools that did the same thing in the same timeframe as checkpt, but they weren't widely used either because they didn't support any of the things CRIU does.

For example, not being able to restore sockets of any form (such as TCP sockets) is a massive limitation that CRIU doesn't have. userfaultfd allows for CRIU to have a slave process that is used as a source of lazy page loading from another machine (allowing for viable cross-host checkpoint/restore with on paper no downtime). And you can use CRIU to checkpoint/restore entire containers (DragonFlyBSD doesn't have Jails, but it looks like you can't even checkpoint process trees).

Flatpak is pretty damn awesome - a layered container-like attempt at tackling application distribution and Dependency Hell.

I've said it before but basically every other post on HN makes me want to learn Ada out of spite.

Also people don't remember how the old proprietary Unixes were very boring

Few things changed between versions and they would tell you exactly what would change between versions so you could be prepared.

(however I have much less trust in a distro running SystemD, it is not stable or reliable)

I thought the same thing. RedHat has become the open source Solaris. I guess that's not such a bad thing. Solaris had some truly astounding engineering -- dtrace and ZFS felt like using software products that had been shipped to me from the future.

Preach it!

Current job shows all the scars of this, every 6 months people hate the current system and want to rewrite something using some framework/lang/library that will really make everything better. The one positive aspect is that my resume now has every single buzzword. The downside is that the current team inherited 5+ years of failure to focus and address the problem at hand and instead try to run away from their problems.

> A few years ago, I started challenging people at work when they were trying to pull in the latest language/library/toolkit, with a simple question. "How does that help our customers".

I take a similar approach. Anyone trying to win my vote has to prove what is the business value of adopting technology X.