Microsoft has been making quality assurance mistakes lately | Hacker News Reader

Microsoft has been making quality assurance mistakes lately

computerworld.com

48 pointstanglesome58 comments

One of the main points:

> how in the world can a patch to the Web browser lock up the operating system when you have an office suite installed?

In a perverse way, I enjoy stuff like this. Why? Because, someday, Raymond Chen will write about this and I will get to read about it on The Old New Thing, and while Chen will do a wonderful job of explaining the immediate causes, they will point to other technical oddities so bizarre that I will be reduced to a kind of mute incomprehension usually reserved for people who deal with the really interesting anal insertion cases in our fine nation's ERs.

I enjoy that feeling.

http://blogs.msdn.com/b/oldnewthing/

Too bad there's only one Raymond Chen. I'm sure there is enough material within Microsoft to keep 10 of him busy.

This is exactly what I thought would happen when they gutted their QA staff and reassigned the surviving testers as developers. Your employees act in accordance with their incentives, and the incentives aren't there for QA. Microsoft made it clear that they only value feature development and shipping, and their employees now associate "doing QA" with "being fired en masse".

To corrupt a phrase, they've crapped their bed and now they have to lie in it.

Edited to add: I'm not some hardcore "M$FT SUXXXX" guy. I like and use their products, but I'm deeply concerned about the direction the company seems to be heading.

I think QA is still rewarded, but it probably will still not be done as well as it was done before. Eg. I wonder if the problems with KB3004394 for Win7 showed up on the dev machines with the test root installed.

I disagree. Back before the reorg and layoffs, there was a clear prestige difference between Test and Dev, where most SDETs were levels 1 or 2 with one Senior per team, while the SDEs had 2-3 times as many Senios, and one Principal per team. It was clear that the ceiling for promotion was much lower in Test.

I left before they began doing away with the SDET discipline, but I'm sure that QA work is still not very prestigious.

I also get the impression that they have been handing out promotions like candy recently. I know that the system was broken before and people were withe leaving or threatening to leave to get promoted, but the flipside to fixing the review system too late is that you may be promoting the wrong people.

Put another way, many of the level 64 SDEs I knew are now Principle (65+) for some this is long overdue, but for others it is unwarranted.

Over-promotion can do horrible damage to a team, particularly when it moves someone into a broader decision making position than they ate capable of handling. Some of the QA issues may well stem from this

This is something that deeply bothers me in the software industry, the segregation between Dev and QA, and the low value placed in QA. "Oh whatever, QA are low-paid techs that just run scripts!" No. QA is vital to the functionality of the product, and a proper QA person should be just as capable a programmer as a developer in order to 1) write and maintain a test suite and 2) understand the tested program and how best to test it.

My previous company in Europe understood this, and that seemed the norm in the surrounding industry. My current in America doesn't, and that seems the norm in the Silicon Valley.

I'm a Quality Engineer, and have worked for a number of large and small companies in the Valley after moving into quality from software engineering 13 years ago. I've seen a lot of teams.

Problem is it's a feedback loop. A lot of companies maintain (or did maintain) armies of unskilled people who basically were script executors, with maybe a bit of barely-trained skill in finding equivalence classes and boundary testing so that basic test plans could be written.

I wouldn't even call this QA, even if it was misidentified as such--it's Quality Control to catch problems after the fact, with no front end assurance whatsoever. It's honestly really only good for pissing off people when you stop the shipment at the last minute. And that lack of value has been broadly recognized. When you combine the high cost associated with the brute-force documentation associated with verbose test scripts, result templates, and other prehistoric process artifacts aimed at this level of tester (and which are usually simply unnecessary and actually detrimental to agility and pace) it's no wonder they're being cut.

Problem is as classic brute force QA teams get gutted, the market has been flooded with these testers, and most of them kind of suck. Hiring a decent quality professional has gotten damned difficult. And when everyone you meet with "QA" in their resume is a non-technical, uninfluential yo-yo, you get a pretty bad impression of the discipline.

In the meantime, everyone as skilled, interested, or immersed as you describe has been moved into software engineering, because QA was treated as an "entry-level" position and nobody wants to be a non-technical, uninfluential yo-yo. The rise of the Software Engineer in Test is as much a collective gambit to keep skilled people in quality as it is identifying a new skill-set. And for people in my line of work, identifying yourself as a SET has about a 30% pay differential over identifying yourself as a QA Engineer, so there's plenty of incentive to move in that direction.

But even then, the rise of the SET reflects an over-reliance on automation, and an under-reliance on process-based quality improvement and the type of fuzzy testing and intelligent guessing only a human is going to reasonably provide. The net result is that edge and corner cases escape, because automation is pretty shitty at catching those unless a fuzzer or an automated monkey (which is essentially a UI fuzzer) can be used. Generally, you only find problems where you've programmed it to look, which will be necessarily be pretty narrow, and the automation cannot use discretion to vary off the script to where new problems actually might lie. People have conflated the value of QA with the value of bad QA, and have chucked it out the window without understanding the cost. I'd guess that many have never even seen good QA to understand what it can do.

Nevertheless, this particular story is egregious. However much you cut back on quality control or assurance, the ONE FUCKING THING you always guarantee is that the user can get out of any problems that arise with minimum damage to money, data, time, and customer goodwill, roughly in that order of priority. Bugs in updates that kill your code signing stack and block further updates are just plain unforgivable.

Unless I'm missing something, this article really seems unreasonably harsh. He cites two bad updates within the past months (one for Windows 7, one for Exchange) that have had to be pulled due to issues, but it seems like well over half the article is devoted to complaints about patches for Windows 10 - an (officially) unreleased product.

You can look through the author's blog history. One in three is an anti-Microsoft diatribe.

As soon as I saw the author's name, I closed the browser. Author is rabidly anti-MS

Yeah, this article is FUD.

The Windows 10 piece is stupid - it's a pre-release OS, these things happen. It's why they haven't released the OS yet, so I'd say their QA process worked. Pre-release OSes don't go through the same level of QA - if you don't like it, you should use a supported OS.

Second, with Exchange updates, there's kind of an unspoken understanding that there is no way MS can test every scenario their users will throw at Exchange. So updates are often released that can cause edge case issues, with the understanding that their large customers will test the updates in a test environment before deploying them. If issues come up, Microsoft can pull the update. These kinds of retractions are not uncommon with server software.

How long can you responsibly run Exchange on the internet after a security patch was released? The recent Drupal SQL injection [1] showed that attackers have databases of fingerprinted services, just waiting for an update that shows them how to exploit these systems. Reverse engineering the binary patches for Exchange and building a reliable exploit out of them probably takes longer than it did in the case of that SQL injection in Drupal. But how long a testing period is too long?

1: https://www.drupal.org/PSA-2014-003

In the case of something like a corporate Exchange server, you can run an IDS/IPS system in front of them to block any specific exploits. Even then, enterprise configurations of Exchange are usually quite unique to the company, so there's a low likelihood of a security problem being exposed to the outside.

This. It's a bit harsh, sometimes arrogant, filled with 'I don't like MS' kind of speak instead of focussing on facts. The article does does state some real problems though, that cannot be denied. However, extrapolating that to a general trend like "there is no QA anymore" (which is at least how I interpreted the article) seems just wrong.

He makes a valid point about the seemingly unnecessary ties between components during an upgrade.

The same pattern can be seen when updating Office for Mac. Every single time the Office updater insists I close all browsers, including Chrome and Firefox, to apply a Word or Excel update. Nothing from any other vendor exhibits this pattern on a Mac (a Unix system BTW).

The author may have an anti-MS history, but he is spot on in questioning these unnecessary tie-ins. I use Mac, Linux desktops, and Windows regularly, and these simple user interaction failures make the latter a pain to work with.

Ordinarily I would brush off complaints about prerelease alpha software quality. But the same problems are affecting mature flagship products.

Last month a Microsoft Visual Studio update intercoursed all our Azure development machines. We never got any reply from Microsoft, and had to reimage all the machines. I see from https://connect.microsoft.com/VisualStudio/feedback/details/... this has nearly a dozen repos, and still no official reply or workaround.

Forget Connect. It's a barren dustbowl full of tumbleweed. If it's urgent shell out for a PSS support issue. Can't remember what we pay per "urgent/high priority" incident but it's small potatoes compared to waiting around for a response on Connect.

I've emailed Scott Guthrie a couple of times in the past about the sorry state of Connect. Maybe they'll just shoot it in the head and stop pretending it's useful to anyone.

I stopped reading when he suggested MS buy Stardock and "call it a day". This is what passes as news/journalistic content at Computer World? And how in the world did it bubble up onto HN?

I hold out hope that Microsoft has bigger plans for it's operating system than simply pandering to the tech-media's demands for, of all things... the start menu to return.

The Start8 Windows 8 dock (in windows 8 mode) is more in place with current UI of Windows than Microsoft's Windows 10 one.

The title seems to apply to Apple, too (see iOS and Yosemite WiFi bugs), and could probably be applied to Google, too (Hangouts bugs, Lollipop problems, etc). But mistakes happen and many bugs are less obvious to reproduce than others, so I wouldn't point fingers too hard, personally.

If a car blew up and left you stranded whenever you filled it with gas, or unlocked its doors whenever you left it out in the rain, you'd demand a refund and perhaps compensation.

It's a testament to the usefulness of lawyers that the big players in the software industry don't even pretend they can work at the level of professional and managerial competence offered by almost all other consumer and B2B industries.

Seriously! You never hear about a car would accidentally accelerate when you try to brake, or that the tires would spontaneously blow out, or that they would just plain roll over. I mean, the phrase "automobile recall" is practically an oxymoron!

[1] - https://en.wikipedia.org/wiki/Toyota_Prius_(XW30)#Recall [2] - http://www.forbes.com/2000/08/18/mu3.html [3] - http://www.cbsnews.com/news/toyota-will-recall-lexus-suv-for...

The fact that "automobile recall" is something that exists and is expected when defects occur, but "software recall" largely does not exist, was his point.

It's almost as if software and automobiles were completely different things! These comparisons are pointless.

Reliability of hardware or software is a trade off. If your software or hardware is responsible for keeping people alive then more time and money is spent on ensuring its stability and that is then factored into the cost.

Automobile recalls are necessary because it's impractical for automobile manufacturers to remotely deliver and install a fix to your car. Software can and does work that way, which is much more convenient for everyone.

His point, as I read it, was that the software industry does not operate to the same quality standards as the auto industry. Which may have some validity, but is not as obvious or clear cut as he was implying.

"Software recall" does exist, they just call it "applying updates" instead.

I worked for a company that did a voluntary software recall once. That was back in the days when it meant shipping out CDs.

Well, if my car did that, I would just back up, reformat, and reinstall. So I don't see why the auto industry puts that much effort into preventing problems that are so easy to fix. It's not like you're in danger or anything -- just use some cheap redundancy.

This was my reaction as well.

I've had a lot more serious problems with my Lollipop-updated Nexus 5 than I have with my Windows 8.1 desktop PC over the past month.

Software quality in general is IME pretty bad these days and the problems go way beyond just Microsoft.

Having worked on and off in software testing for the last fifteen years or so, I'm becoming increasingly convinced that there's a correlation between the recent trend of calling software test teams "Quality Assurance" and general purpose software that seems to get less stable as time goes on. Bear with me on my admittedly whacky theory that I haven't completely fleshed out yet.

What is called "quality assurance" is really "quality control" in the places I've observed. The difference is subtle. For example, at my last position I carried the title "Director of QA". That was a made-up title that didn't mean anything. It didn't mean anything because if I were truly a manager of QA, then when dev throws something over the wall that has no code reviews and no unit tests, I'm empowered to say, "no, we're not shipping this". But as we all know, that's not how it turns out. Instead my team is "quality control" because we're just testing the output and have no empowerment over the creation process. In other words, we're a test team, so quit with the "QA" crap just because it has the word "quality" in it. As the saying goes, you can't test in quality.

With that, I see two trends. One, which has been going on for a while what with TDD and the like, is an increase of the testing burden on dev. This is, generally speaking, a good thing IMO, especially if dev can be backed by a dedicated test team. Where it goes off the rails is what I hear from my Microsoft friends: there is no more test, "devs" (many of whom might be converted testers that may or may not be qualified "devs") are responsible for testing, too. Don't ever rely on your own testing if you're a dev. You'll test it the way it was written, and you'll not only miss edge cases but likely scenarios as well.

Could this explain the steaming pile that was the Xbox One (the only MSFT product I have in the house, or use, these days) in its early days? Could it explain the unbelievably buggy, nearly unusable, _Halo: The Master Chief Collection_ we got from a Microsoft studio? Dunno, could be coincidence, but it conveniently fits my theory; might want to grab some tin foil.

Trend #2 is a regression bug: throw shit over the wall and hope test finds the bugs. Call 'em "QA" so that when the quality isn't there we have someone to blame. That's what we used to do 20 years ago, then we as an industry got a clue and put some quality onus on dev while test got brought into the process earlier. Maybe it's just my corner of the world, but many seem to be going back to just that (I left my last position mainly because of that reason, after they bait-and-switched me between the interview and the actual work.)

Could be that tools that help with quality haven't kept pace with the increasing complexity of software systems. Could be, but I'm not sure I buy that. We've got tools I could only dream of 20 years ago (Xcode's point-and-click profiling tools, code coverage tools, static analysis). Test modeling tools need some work, but at least they exist.

I think an overly simplistic answer is that the industry doesn't know what to do with software test. We know we need some, but damn does it cost money. So how can we minimize that cost without appearing to just throw up our hands? After all, most of us are not building the guidance control for space probes, so who cares if Angry Confectionary Squashing Avians crashes once in a while? Instead we throw an understaffed and unempowered team of victims at the problem, then scream "why didn't test (pardon me: "QA") find this?!"

That's my take on it, anyway. I'm astounded at the poor quality of most software I encounter, but I think the general user population is used to thinking that it's either something they're doing wrong, or that's just the way software is. There are no late night pitchfork parties along Microsoft Way or One Infinite Loop, and people keeping paying, so there's not always a huge incentive to do otherwise. Many of us have said we'd pay for an upgrade that included nothing but bug fixes. Companies think otherwise (and they're probably right). Apple had their Snow Leopard, which they had to give away, but I can't think of any other examples.

Solutions? Oh, I've got a few but this is getting long-winded and rambling, so I'll spare those of you that stuck with me this long.

I totally agree. What I think Test should mean includes testing the spec and acting as a user advocate; testing the design spec to ensure the dev is considering all relevant scenarios and cases; manual UI testing (and scenario-based API testing) to find the corner cases; writing test automation and scaffolding to catch regressions; and writing test automation using combinatorial techniques to cover as much of the input space with a minimal number of test cases.

Now that I'm a dev I really wish I had some competent testers to work with. It's really impossible to test your own work, because you're inherently optimistic about your code. Testers need to have the mindset of "there's a bug in here somewhere, and I'm going to find it."

The Windows 10 Technical Preview is available to author as part of Microsoft's quality assurance program. Technical Previews are alpha software [1] by another name and Release Candidates are beta software in Microsoft speak.

[1] see: http://www.hanselman.com/blog/IntroducingASPNETVNext.aspx and http://blogs.msdn.com/b/cesardelatorre/archive/2014/05/12/th... for examples

Are you making the point that "it's beta/alpha, deal with it"? Because the author adresses exactly that, and the headline incident is only a one (egregious) example of a series of recent problems out of Microsoft.

My point is that the software is equivalent to an alpha release.

People can build there own points based on the author's calling it beta software.

My own WTF moment came recently when we dropped XP support for our product. Suddenly the window frames started painting improperly. One of the system calls we use in NCPaint returns different values depending on whether the executable is marked as XP compatible or not, even with no other change to the program.

How are you supposed to create reliable software when you can't even count on the raw API that's been unchanged since forever?

They go through a lot of effort to make sure their software is backward compatible. (You're not supposed to make reliable software, you are supposed to make broken software and convince your customers to pay Microsoft to make it work on their systems.)

I've been using Visual Studio since it was called Visual C++. VS 2013 is one of, if not the buggiest versions I have used. The only responses I seem to see are "It'll be fixed with Roslyn, just wait!"

I'm curious as to what has shifted inside Microsoft. Part of me believes I'm seeing the work of new devs as the old devs have left. Another part believes this is the result of agile taking over. I liked my visual studio with waterfall releases. At least it worked without constant hangs, locks, and crashes.

Interesting, I use VS2013 with C++ and C# all days and rarely have any issues. In my team with 10 devs the only guy who run into problems often is the one who installs all extensions he can find (his VS looks like my mothers IE with all the toolbars). Compared to how often VS6 crashed for me in the late 90s and how buggy VS2003 was, VS2013 feels pretty solid to me. Guess we are not using it in the same way or for the same things.

I agree, 2013 is pretty solid and feels a lot snappier compared to 2012 and 2010 (I have all three installed).

I found it a bit irritating that there were no news about the various Microsoft update issues on HN last week. The article at least mentioned the various problems, that's good.

Some articles that capture the whole timeframe:

> I found it a bit irritating that there were no news about the various Microsoft update issues on HN last week.

Why didn't you post one?

(Edit: I hope that didn't sound accusatory; I mean it as a cordial invitation to post good stories that you think belong on HN.)

I read about it only in german IT news portals like http://www.heise.de .

I really like HN, it replaces /. for me.

Therefor I try to not submit controversial posts (is it in US?). I assume it's not optimal to submit Google Translate links of german articles (like my links above).

Yes, that's not optimal; the only exception is when there isn't yet a substantive English version of a major story, or something particularly interesting.

I don't think this particular story would have been unduly controversial, but more importantly, thanks for your concern for the quality of stories on HN.

My favorite change lately is the one where they increased the version number on System.Web.Mvc in the GAC, thereby breaking almost all old ASP.NET MVC builds.

http://blogs.msdn.com/b/webdev/archive/2014/10/16/microsoft-...

The comments on this article have some great vitriol.

I think this about sums up the quality and obvious bent of this article: http://cdn.meme.am/instances/500x/57216937.jpg

Come on, the Preview is not even close to beta! That's why it's not pushed to everybody, but offered to enthusiasts who are fine with things being broken for a while. I mean, you can be upset all you want, but blaming an early preview for being unpolished is reaching.

As an aside but related - I bought a Surface Pro 3 and was shocked to discover major issues with its WiFi when you have Hyper-V enabled. Basically WiFi "breaks" whenever you wake the device, and can only be re-enabled through a reboot. (http://winsupersite.com/surface/will-microsoft-ever-fix-surf...)

That's the new Microsoft - less thinking about backwards compatibility and not breaking things, but faster development, reduced management overhead. Faster time to market HAS to come from somewhere.

Since the HN guidelines ask for linkbait titles to be changed, we changed the somewhat baity title to a representative sentence from the article.

dang, have you considered aggregating these kinds of notifications onto a separate page, or maybe using a specific account for administrator action? It would be helpful for those of us who value transparency (and thank you for your continued transparency - it has been great).

Yes, we consider that and similar ideas from time to time, but I worry about unintended consequences.

Just a plain log would do.

It's the unintended consequences of a plain log that I worry about.

Thanks for letting me now. I appreciate it.

> Microsoft has been making quality assurance mistakes lately

Well, that's news to me!

I've always thought that everything that came out of MS is a quality assurance mistake. Heck, even MS itself is a quality assurance mistake.

lately? since 1980s

Reply on news.ycombinator.com