Comment by wtallis | Hacker News Reader

Comment by wtallis | Hacker News Reader

The biggest lesson to take away from this is probably that they thought they knew how to test a SSD, but were quite obviously clueless:

> we run a fairly comprehensive set of block-level tests using fio, consisting of both sequential and random asynchronous reads and writes straight to the disk. Then we throw a few timed runs of the venerable dd program at it.

Running dd as a benchmark is a major red flag. It show that they didn't know what they were doing with fio, and didn't trust its results. They later started using IOzone and a custom-written tool to accomplish stuff they should have done with fio in their initial testing.

They also did not mention pre-conditioning the drives or ensuring that their tests run long enough to reach a steady state. This is one of the most important aspects of enterprise SSD testing and they would have known that if they'd consulted any outside resources on the subject instead of making up their own testing guidelines from a position of extreme ignorance about the fundamentals of the hardware they were using and the details of their own workload.

They really should stop calling any of their tests "comprehensive".

I think you just expected to much from this article.

This is not a comprehensive guide to testing SSDs, it’s the story of what the author went through when trying to test SSDs. It’s well written and the author seemed to really engage with the topic and describe all the setbacks he had and research they did. I did not think he presented himself as an expert, just a software engineer tasked with upgrading their SSDs. And who knows, maybe this was only a 20% time project.

There are a lot of blog posts on HN that have much less actual content and where the authors have much less of a clue, yet often the response is overwhelmingly positive because someone took the time to write it up. You should really be more charitable here.

And to adress the calls for “outside experts”: If everyone called in outside experts for everything hardware (or software) related, we software engineers would never get to do anything cool or learn some new framework. We’d just be watching an outside expert do their thing. And outside “experts” are not necessarily better, often they might just sell themselves better. And who is going to check their work if the knowledge is all outsourced?

I think it’s great that the BBC lets their engineers do this and learn along the way, and a place where that is possible sounds like a nice place to work. It’s not like they had any downtime or anything because of this.

> And to adress the calls for “outside experts”:

You completely misinterpreted (and misquoted) me on that one. I wasn't implying that they should hire a consultant for this kind of thing, but they should at least have bothered to read anything about the methodology used by SSD reviewers or the industry standard storage testing methodology freely published by organizations like SNIA. It's clear the BBC guys didn't even spend an afternoon trying to read up on how to evaluate SSD performance; they just jumped in and started re-inventing the wheel, hitting all the foreseeable problems along the way. It looks like they now have a clue and have learned a lot from the process, but this is not how you should handle this kind of upgrade.

EDIT: no need to downvote wtallis, he has a valid point!

Sorry I misread and misunderstood that, I did not mean to misquote.

What you’re saying is of course absolutely true, they should have done much more research at the beginning.

However, at the start of a seemingly easy task, research may not be first thing to spring to mind (although it should be).

BBC published a technical (but really PR) article written by amateurs posing as pros, instead of consulting reputable experts?

The problem isn't that the amateurs wrote the article, it's that the amateurs made the purchasing decisions that created the story in the first place.

No. They were duped by their own IT department. The reporters thought they were talking to experts on all things IT-related. The reality was that their team, while no doubt experts on many things, knew little about testing SSDs. This wasn't PR but a simple mistake in journalism. They should have talked to outside experts before publishing.

To be clear, this isn't a news article written by our journalists - it is a piece written by the team themselves that we felt may be of interest to others, and that might help us do things better in the future. While I enjoyed reading it, I can assure you that SSD performance testing doesn't move the BBC PR needle compared to the identity of the new Doctor Who.

We're acutely aware that we've still got much to learn in this space, so if there are thoughts you have on how we could do better we're all ears.

Finally, while I assured you it wasn't a PR piece we're always looking for engineers in this area (and across the whole BBC) so if you'd be interested in helping us improve, get in touch.

Do you accept American Engineers? /s

I found the piece to be wonderful. I don't do large scale storage work, so I'm very un-knowledgeable in the area, but it's great to see someone else's struggles other than Amazon or a backup service. And it is yet another indicator that the BBC cares about quality content instead of just pushing up some stock price.

Thanks for you write up

Don't let the gratuitous negativity get to you! I may only see a very small slice of what the BBC does, what with being one of those continental imperialist and all. But what does find its way to me has always seemed to be excellent.

(Not a job application. Unless you know something about the EU/British future that I don't.)

Reply on news.ycombinator.com