Hacker News Reader

Comment by badgersandjam | Hacker News Reader

No it's not just one data point. This is the final straw to use the old phrase. After a few years of serious problems, our most recent being CIFS VFS problems causing panics and mounts locking up on CentOS 6.5, hard locks on RH certified hardware, power management hell and so much incredible churn with no progress and the sudden "fuck POSIX" approach, it paints a really bad picture of the current state of things.

There is a distinct lack of engineering prowess and quality control. It originates at the core GNU + kernel + freedesktop teams and waterfalls down through the distribution houses.

That's the problem and it's endemic within Linux.

View on HN

I caution you to not apply the word "engineering" to software, especially in the apparent manner you've been applying it to this thread. You seem to have an idea that there are universal software development practices that are so well understood we can make regulation around them. But that isn't how software works. There are some practices that are known and some that are unknown. For working software "engineers", we are all, daily, put into a position for having to figure out for ourselves even what tools and materials to use to make anything.

Imagine you were an architect of buildings. Your day to day job is to design mundane strip malls and gas stations. You have building code on your side for much of the process. As long as you don't violate the regulation, you at worst can only make an inconvenient building, but not a dangerous one.

But imagine instead that you're building a large office building every six months, your clients demand you don't reuse any design principles on your future clients, and you not only lacked the building code, but also 1/4th of the heavy machine equipment, 1/2ths of the tools, and 3/4ths of the raw materials. I don't just mean you don't have them in stock, I mean nobody has invented them yet. And of the ones that have been invented, we don't even know all of their material properties, say nothing about what material properties we should be looking for. Will this particular bolt we are using with these particular cross beams hold up to the stresses placed on them? The answer is unknown, and I'm some cases unknowable.

It's really easy as a user to say, "they should have tested this more". While strictly true that more testing may have found your issues ahead of time (presuming the right tests were done), it is inefficient engineering to exhaustively test things. Even mechanical engineering bases a lot on statistical modeling, which will always, always have corner cases that don't match reality.

In the real world, people had to learn the hard way about things like lightning rods and sacrificial electrodes. They didn't come about from "testing during development". They came about from testing live, and seeing which buildings and boats did or did not burn down or sink. That's not bad engineering. That's just the nature of unknown problems.

That sounds like an excuse which I don't accept. I have a formal engineering background and whilst you're fundamentally right, engineering is based upon cumulative experience gained. We have a hell of a lot of experience as a society of writing software that works and is of merchantable quality.

What the general state of affairs shows is the following traits:

1) There is no thought and research going into the design of a piece of software. Ergo, we do not learn from past mistakes.

2) There are isolated individuals writing vast swathes of software which are trusted unconditionally. Ergo, we do not learn the benefit of multiple eyes on a problem, review and discussion.

3) We assume that software is correct from one person's viewpoint and opinion. Ergo, we do not test software properly nor cover those tests with objectives.

4) We work to deadlines, not quality objectives. Ergo, we trade quality for tolerance from others.

In this case someone came along and didn't think about the problem, didn't work with others, assumed they were unconditionally correct and chose tolerance over quality.

To use your analogy, they're now selling stainless steel lightning rods (poor conductivity), are the only vendor of them, are a vocal marketing front and houses are catching fire everywhere.

Or more specifically, in one example, the entire process was above and from the author to the distributor, no one even noticed that loginctl doesn't work properly.

Wait, so let me get this straight. You're an engineer-engineer, not a system administrator or a software developer. And you think there is a fundamental problem of qualification in software development, yet you lack qualification in software development.

On your points:

#1 is patently false, to the point of being extremely insulting. You've lost all sympathy from me at this point. Go peddle your baseless opinions somewhere the audience doesn't know better.

#2 is also laughably false, as that is the entire freaking point of open source and often considered the greatest strength of Linux. You think because your highly qualified opinion wasn't consulted before you had to spend a whole four hours, OMG on a problem that means that no review or discussion was done?

#3 is false again, because software is tested. You use the word "properly", so I will sit here and wait for you to bestow upon us your great wisdom on what we could be doing better.

#4 is false on both presumptions that software is not built to quality standards instead of deadlines, or that other fields are not dictated by deadlines.

10 years in EE (embedded systems, defense industry), then 15 in what we now call devops/architecture. Experience is fine. I have no problems being arrogant about that. I've fallen down a lot more holes than a lot of people and know what I'm talking about.

When I say tested properly, I mean tested completely. If you miss an entire functional unit of the software and a client reports it as broken, its pretty obvious what the problem is.

Our senior software guys sat down for the other four hours and presented all our findings together and cumulatively said "we're not supporting that shit; we can't trust it".

Regarding #4, it's plain to see that RHEl was released with a broken systemd implementation due to deadline...

> When I say tested properly, I mean tested completely.

I am still in the learning phase, but even I know that complete testing of any complex software is practically impossible.

So, how do you guarantee completedness in "proper" testing? I know you can't without redefining the word "completely". What's your definition?

Also see Impossibility of Complete Testing[0] by Cem Kanen, co-founder of http://www.associationforsoftwaretesting.org/

[0] http://kaner.com/pdfs/impossible.pdf

Yes you're right.

When your system consists of functions "A, B, C, D", I'd expect to see test suites for "A, B, C, D". In this case there were test suites for "A, B, C". The client found D therefore the test suite was incomplete.

Now if a bit of A, B, C or D suites were missing that would be different and entirely expected.

You should at least be testing the happy path and most common failure mode of every component of a software system, whether manually or automatically. The most visible components to the user should be tested most. Imagine an ecommerce site where nobody ever tested checkout, or an OS where nobody tested logging in.

If there is one thing that HN has taught me it's not to make broad, hyperbolic statements just because I don't like something; especially if it is a divisive issue. There is always a "moron4hire" who will rip you a new one (and rightly so).

Nah, they're pretty much right.

As for testing--notice that when a lot of people here are reporting issues with systemd/pulseaudio, their reports are pretty much dismissed out of hand, or they're told "no, you've done something wrong".

For #2, a lot of times somebody with the right political position (say, Lennart at Redhat) or just the ability to shout louder and longer than anyone else will get something put in, regardless of technical advantage. Don't even try to claim otherwise.

I read him as leveling his complaints against the entire field of software development, and if he was being more specific than that, it was at least as big as the subset who develop Linux.

It seems to me that OpenBSD stands for everything you want in OS dev, except maybe point 3, Theo seems really at the center of everything.

You're 100% correct. In fact my own mail server/web machine uses it. Theo is not right at the centre; there's a large group of people who work together as equals from what I can see. I can't recommend it for our "enterprise use" though because we need some of the more friendly features that FreeBSD offers such as ZFS.

I run a mixed environment right now with OpenBSD, FreeBSD, Red Hat, and Windows. I use Red Hat and Windows because certain software requires it (basically they are single app servers[1]). The OpenBSD servers are all doing basic utilities and the FreeBSD servers are doing stuff the requires a big file system (ZFS). There are a lot of enterprise tasks that OpenBSD is fine with doing, and I just use FreeBSD for tasks that require the file system or really heavy load.

1) government contractors are so fun when they get their software required to deal with certain parts of government

Hmm, I don't see any of that, odd isn't it?

Not really odd. You just haven't poked the bits I have.

Reply on news.ycombinator.com