The BBC's hi-tech failure: Don't Mention It | Hacker News Reader

The BBC's hi-tech failure: Don't Mention It

economist.com

99 pointschrbutler90 comments

Was wondering if anybody here would notice that the "CMS" being described here was a project to handle every single second of radio and TV footage from ingestion through recoding through editing to broadcast, and then beyond that to transcoding everything for whatever devices want to watch or hear it once it's finished.

People here hear "CMS" and think we're talking about text.

Incidentally the parts of this boondoggle which actually exist are being used as components in the BBC. The headline last week was about the decision to terminate that particular project because it had got far, far out of its depth. Also to put the highly-charming person responsible for it on gardening leave indefinitely pending the investigation.

It's a massive cockup, but it has had some useable outcomes, and it is a product class that doesn't exist outside the world of the Beeb, or other national/international broadcasters. There's nothing off the shelf to buy here. In the 90s this sort of thing wouldn't have got cancelled, so I'm seeing this as a sort of progress.

Real headline: Publicly funded broadcaster makes same mistake most large private enterprises make in more or less the same way.

It seems to me that months or years of auditing and hearings and cries for heads to roll are probably wasted. Just like a private company a few mangers should probably get tossed or reassigned and the second version of the same project should get started to address the businesses needs the first one was supposed to.

>Publicly funded broadcaster makes same mistake most large private enterprises make in more or less the same way.

Actually large private enterprises don't usually make this kind of mistake. This is a typical big government project. Software development is hard. Building large, complex, poorly defined software systems is hard.

//EDIT

I should clarify that. I don't think a private enterprise would engage in a project that would have it build an internal system, at that scale ($100 million), having so little impact on it's primary revenue driver, over so many years!

Big enterprises fail with big software, that's a fact. But they wouldn't fail in this specific way. This is a quintessential big government software project failure.

There are a lot of numbers tossed around without a lot of hard data backing them up, but it's generally accepted that over half of all large software projects 'fail' at both private companies and government institutions.

"Actually large private enterprises don't usually make this kind of mistake."

Really? You sure about that? Because hard numbers are hard to come by, but I'm pretty sure they do with some frequency. It's notoriously common for large consultancies not working for the government to have the same outcome too.

It's possible private enterprise makes this mistake at a different rate than the government, but I'd despair of trying to prove that claim, or even give it a solid definition. In general large software initiatives seem to fail, regardless of whose bright idea they are.

> Actually large private enterprises don't usually make this kind of mistake. This is a typical big government project.

Your last two points contradict your first two. There's nothing unique about a government's ability to make terrible decisions. I'd say one difference is that private enterprises typically go under if the mistake is big enough, and they have no great interest in telling the public about their mistakes.

I just don't think a large enterprise would engage in an internal $100 million project to build a CMS.

I'm not saying private enterprises can't fail or don't fail. They just wouldn't fail in this specific way at that magnitude.

Massive write-offs for failed projects and acquisitions are pretty common in large enterprises.

Failed acquisitions are pretty common, failed projects too. I'm not disputing that, and that wasn't my argument. I added a clarification to my post.

"I just don't think a large enterprise would engage in an internal $100 million project to build a CMS."

Isn't this a circular argument? If a company doesn't want to build a system in-house then its only other option is to contract another company to build it. At some point, someone has to build the system.

There's nothing special about private enterprises that makes them immune to risk. As others have said, many IT projects fail [1]. In the private sector this is just seen as a business risk, but when a public sector project fails then its "wasting taxpayers money".

For some reason the Economist article doesn't mention that the project was originally awarded to Siemens and Deloitte [2], but technical failures and cost overruns by these private companies caused the BBC to bring the project in-house. It could be argued that the BBC lacked the capability to deliver the project on its own, but thats also a mistake found in the private sector.

[1] http://www.computerworld.com/s/article/9116470/IT_s_biggest_...

[2] http://en.wikipedia.org/wiki/Digital_Media_Initiative#Initia...

There's something special about private enterprises making them reconsider their approaches, though - the fact that they lose their own money (which is often combined with having a smaller pool of money to begin with). Few enterprises can spend $150M and fewer will keep spending upon signs of things not working out.

As to private contractors doing a poor job when paid with taxpayer money - makes some sense, as long as said money keeps flowing. In a different setting the flow would stop more quickly.

Can you justify this? More software projects fail than succeed, private or not, VC's base their decisions on it, everyone knows it.

This is untrue of large private enterprises, which are quite capable of throwing good money after bad, engaging in wishful thinking, and all the rest of it. As another commenter pointed out, the difference with the public sector may rather be that private failures can be more easily hidden.

I did one stint at a BigCo, during which they blew over $100M on a "platform of the future" project on which all their disparate software products were going to be developed going forward. It was obviously never going to work, yet they kept funding it and kept pushing it on all their software teams. Even though this was a massive failure, it was never announced as such or even explicitly canceled. It just crawled off to die alone somewhere. Official rhetoric was that the project had succeeded, and the executive who had championed it was promoted to run the entire software division of the company.

Edit: so I think the salient difference here is not private vs. public, but large vs. small.

In what sense is it "their own money"? The programmers at a private enterprise are not doing anything with "their own money" besides collecting a salary. The same detachment from the outcome that happens in government work can happen in businesses, and it does all the time.

The people at the top of a business software product might be stockholders, so they could have some risk. The people at the top of a government project also understand that it will be their photo in the media along with the public outrage. And both understand that the consequences - whether public anger or a dip in the stock price - are temporary and will go away in a few weeks.

I know of a private newspaper with a circulation of around 30,000 that spent $3M on a failed Print software system.

How does that compare to the BBC's viewer & listenership numbers for $150M for a Audio/Video production system?

According to barb.co.uk, BBC 1 (the flagship TV channel) had some 45M viewers over a recent week. We could consider other channels, radio stations and the website, but that's already well over half the UK population. By cost per customer, that's more than an order of magnitude better than your newspaper example.

> Actually large private enterprises don't usually make this kind of mistake.

I would say "Actually large private enterprises don't usually [admit to making] this kind of mistake." I swear the ERP and document management are the two areas where every enterprise tries to "standardize across organizational units" and fails badly[1].

I have come to believe that this type of software is hard because it imposes rules foreign to the entity on it and doesn't value a companies existing workflow[2], or it requires thousands of developer hours to adapt.

1) oh, they keep using the new system, but their customers feel it

2) often a very successful workflow for the entity and its customers since the entity can now afford a multi-million dollar software package

ERPs are a great example. but I don't necessarily agree with your view that a large enterprise's existing workflow is superior. My take:

Most large enterprises have workflows which could be vastly improved by automation of ANY kind, even one that is imposed by a minimally customized ERP. ERP implementations are most successful, on both an ROI front as well as success in implementation, when they are brought in and the existing workflow, at a high level, is redesigned to fit the tool. Why?

The other path is chaos, because you are essentially asking people whose jobs revolve around doing things manually to "build their own replacement." A simple way of putting it is getting the assembly-line workers to have input on a new, highly automated assemblyline. They don't want the robots to do their jobs for them. Instead they will all insist on the value of their interaction in the process, and the robots will simply be tasked with helping them do their job instead of simply replacing them.

When ERPs succeed, they lead to lots of admin people getting laid off or reassigned. These are the same people that have to be consulted if you want to customize them for a company's existing workflow. They fight tooth and nail to keep everything the same as it was.

Good points, and my thoughts exactly when I left university with a business degree.

My take now:

The workflows of large enterprises are not really well understood at all. With lean management and how management careers work, even the lowest management levels has only a vague idea what is actually happening in their departments.

Designing even a tiny system for a small number of users requires a lot of time and a lot of love. People do many more things that they could ever put into words. The new system breaks many things in ways that are difficult to describe. The users are defenseless because they are up against huge projects with a lot of momentum and are not prepared to communicate those kinds of problems.

Uhm, so let's hope that your ERPs are actually succeeding.

I'm not sure if you are assuming that I'm a naive guy who is fresh out of university or not. In case you are, I am not. I worked for a small company (I was the 6th employee, and it grew to 150 people) whose largest project was an ERP implementation for an extremely large enterprise which I shall not name, but you would immediately know them. I was not involved on the project directly, but was frequently pulled in to advise those who were. Said organization went down a rabbit-hole trying to map out all of their workflows so that we could customize the ERP for their needs. What a nightmare. I am now witnessing a successful ERP implementation in another large organization. The first priority is to minimize customization.

General Electric is a model of how to implement an ERP. They followed the minimal customization model, and forced their workers to change their workflows to take advantage of the tool. The key is to identify all of the data an org needs, and make it accessible. If your developers don't have an excellent metadata analysis/storage tool (homemade or bought) they are screwed.

> General Electric is a model of how to implement an ERP. They followed the minimal customization model, and forced their workers to change their workflows to take advantage of the tool.

Yep, that's the model, but I contend it is a pretty poor model for everyone but the ERP vendor. It is basically saying the ERP or CRM vendor has a better grasp of a companies business process than the company. I have seen companies adopt a ERP and use its process, then anger customers because of the change. Since they follow a new path, many times it is IT telling the rest of the business how to run things[1] and much of the institutional knowledge is thrown out.

1) something as simple as what's on the invoice or payment voucher can have huge affects on customer experience in some industries.

I was not assuming you were naive or fresh out of university. I was assuming however you came from the consultant's perspective.

I am sure that you must recognize, that when people complain about ERP implementations not fitting, they refer to implementations that you describe as "model"?

I come from the other end of the spectrum, making my first steps building business-process relevant tools while employed in various functions. What really shaped me was having carefully discussed the requirements, building it as specified, then being called because "it is not working", and realizing then and there what the actual business logic was, and how deep it was buried and how dysfunctional it all would have been had I insisted that I had implemented as specified.

I'd say (to the consultant profession in general), grab the proverbial pistol and flashlight (http://en.wikipedia.org/wiki/Tunnel_rat), and go down the rabbit hole, because that's the only way the job can be done right.

    Actually large private enterprises don't usually make this kind of mistake.

counter-anecdotes, 24.4k hits

https://www.google.com/search?q=lawsuit+failed+erp+implement...

4M hits, but lots of dups:

https://www.google.com/search?q=lawsuit+failed+crm+implement...

Your edit: ..having so little impact on it's primary revenue driver..

Sorry, you are still wrong. The BBC is specifically in the business of video content and their failed system was a video content management system. You can not get any more core to a business than that! This is a quintessential big software project failure I do not see how it being in government has anything to do with it or how you have made any kind of case for that belief.

This is not actually true, I am not aware of much info on CMS roll outs, but large CRM and ERP software projects at private enterprises fail all the time.

This is very common, I think perhaps the hacker news crowd probably have more experience with real software behemoths like google, facebook, microsoft and/or scrappy smaller start ups. Perhaps in those realms this doesnt occur because software delivery is their bottom line. In run of the mill fortune 500 companies major software projects fail commonly, and CRM and ERP projects fail even more frequently and have price tags in the 10s of millions, and when they fail the company has no choice but to roll out a new one

Actually, there are lots of stories of millions and billions being blown!

http://www.insearchofstupidity.com/

I have this book, and I'm sure there are plenty more like it.

>Actually large private enterprises don't usually make this kind of mistake.

Are you new or something? Do you know how many projects I've been brought in to rescue? Do you know how many times my advice to "Stop throwing good money after bad" has been ignored for one reason or another?

Big enterprises fail in exactly this kind of way. All. The. Time.

Like many of the comments below I don't agree with this statement.

I have been at several mid to large sized private companies and have watched at least 4 $30mm+ projects go up in smoke over the years. Fortunately I have not been involved in any of them directly but I got a front row seat. Surprisingly more then half didn't result in extensive turnover of the key parties involved.

A productive use of this trollpost is to list failures of giant software initiatives by private companies. You could fill up a thread w/just SAP.

Levi-Strauss, SAP, and $100M: http://www.zdnet.com/blog/projectfailures/levi-strauss-sap-r...

"Levi’s is bucking the industry-wide trend by going ahead with ERP now. 'It’s expensive, high risk and requires a lot of cultural change, and unless you have a really compelling reason to do them you don’t [during a downturn],' says Paula Rosenblum, an AMR Research analyst."

Waste Management, SAP, and $100M: http://www.zdnet.com/blog/btl/promises-promises-a-look-at-wa...

I've personally worked at 3 large organizations that have tried to formulate and develop an all-encompassing content management system. One of those organizations is a broadcaster that was also trying to build a digital content management system.

Every single one of those projects has been a total boondoggle. Always over budget. Always changing scope. Always missing every single milestone and deadline.

Now, I know that this is not a statistically significant sample size, but this sort of thing seems to be more common that I initially thought. It's like the worst-case scenario of design-by-committee.

The same organization that couldn't be arsed to host some of their historical web properties due to "high costs": http://853blog.com/2011/01/25/pulling-the-plug-on-the-bbcs-i...

It strikes me that the BBC is hilariously bad at archival. They scrapped dozens of Doctor Who episodes in the name of saving space (http://en.wikipedia.org/wiki/Doctor_Who_missing_episodes).

The BBC is, today, very good at archiving programming material. All output is kept for at least five years, and the overwhelming majority is kept permanently (exceptions are things like long-running quiz shows, where samples of material are kept post five years, although any disposed material is offered to the BFI and other third parties before being junked).

Doctor Who wasn't scrapped in the name of saving space, it was scrapped because nobody thought it was worth keeping. The shows couldn't be repeated (because at the time, Equity had negotiated limits on repeat rights), and quad tape was very expensive and easily re-used.

More importantly, nobody really thought that Quad was an appropriate archiving format. Programs would be tele-recorded onto 16mm for overseas sale and archive. This is not, in of itself, a crazy idea - try finding a quad tape player today vs a 16mm projector. The problem at the time was there was no mandate for archival, so the engineering department were junking the tapes, and the commercial department (who were selling Doctor Who on to other broadcasters) weren't retaining the 16mm reels.

Even in the 1970s, many people were still not convinced of the value of archiving TV output. It's not totally dissimilar to the early days of the web - it was seen as ephemeral. This was happening at pretty much every broadcaster around the world - Wikipedia has a pretty big list of lost TV programs, most of which were wiped for space.

In the name of saving cost.

Tapes were (and still are) really expensive. at the time the BBC were really strapped for cash (as ITV were not long after.)

Not just the BBC. Everyone did that.

Read about how Groucho Marx's grandson saved "You Bet Your Life" from NBC's trashing.

Even into the 1980's Ted Turner made a large part of his fortune by buying the back catalog of movie studios who saw almost no value in their archives. Turner saw the value of playing these old films on cable and video tape sales.

EDIT: forgot this link

http://boingboing.net/2013/03/12/the-day-my-grandfather-grou...

That was common.

https://en.wikipedia.org/wiki/DuMont_Television_Network#DuMo...

"By the early 1970s, their vast library of 35mm and 16mm kinescopes eventually wound up in the hands of ABC, who reportedly disposed of all of them in New York's East River to make room for more recent-vintage videotapes in a warehouse."

Nowhere near as much as the £12Bn the NHS spent on an impossible moving-spec IT system that they also scrapped(1). Oh wait, it's a tenth of the amount.

(1) http://en.wikipedia.org/wiki/NHS_Connecting_for_Health

A tenth of 12B would be 1.2B. This is a hundredth.

Apologies, physicists are often off by an order of magnitude ;)

This is widely reported size of the waste, but as I understand it, because the program was cancelled 'only' £2bn was wasted.

The US has contributed some nice failures as well. For example, caBIG ($300M+):

https://www.google.com/search?q=cabig+failure

Saying the program was scrapped is a bit off. It makes it sound like nothing has come out of the project at all.

Choose and Book is widely in use, the Spine exists, as well as the N3 network.

Data format and transfer standards are coming into play now. So instead of one system to rule them all, it's working towards interop. Which probably should have been the overall goal to begin with.

People like to think of the NHS as a monolith, but it's really hundreds of tiny fiefdoms.

Tiny fiefdoms all on 172.16

They could easily have picked ipv6 and done it the smart way. Oh how things could have been.

Relevant recent news:

http://www.nao.org.uk/report/review-of-the-final-benefits-st...

I vaguely remember reading uplifting, grandiose articles and presentations about domain driven design at the BBC; search for "BBC domain driven design". From this article, I cannot deduce whether this failed CMS is the same system that has been glorified in these presentations or something else; maybe someone can shed some light on this.

First thought when I read this: Is this the system das was the base for all their DDD talks?

It probably is. I've experienced the fact that DDD scales up only until the consultants have gone :)

Same as NserviceBus only works until you've paid for the training.

Previous discussion, including info from someone close to the project, here: https://news.ycombinator.com/item?id=5762116

Related further reading for trainwreck geeks:

EDS (now HP) infamously botched a 1-year ~50million-turned-4 year-multi-hundred million dollar CRM project for BskyB a decade ago, and eventually lost a years-long lawsuit.

http://www.linklaters.com/Publications/Publication1403Newsle...

I had the chance to see some of the work Adobe is planning to do with their CMS (some actual demos from the "Adobe Labs"), and i cant deny the resemblance to BBCs case. I am wondering whether Adobe was involved...

A side matter, but handing the BBC's governance over to Ofcom? They haven't held the Independent Television programme contractors to their remits since 2001 under the threat of any sanction - or, for that matter, put their contracts up to tender (not that it worked or ever will work if we want a One Nation Tory-style public broadcasting system again - and not that I remain suspicious of their inconsistent singling out of Thames Television) since 1993.

The BBC Trust, at the very least, has an image to mantain.

I did a quick check to see how Wikipedia defined a "One Nation Tory" - after all they have been pretty thin on the ground this last 30 years (especially here in Scotland!).

I'm somewhat amused to see that "One nation conservatism" appears to now be the policy of the Labour Party:

http://en.wikipedia.org/wiki/One-nation_conservatism

Note that the content in this CMS is hours and hours of raw video footage rather than the usual snippets of text.

Duely noted, but I don't think it ought to make a difference in the price tag unless the CMS was doing conversion and distribution.

...which it was. One of the big features was the ability to edit video in a lower bitrate (i.e. faster to work with) then send it off to a server to process on the full HD video. Great idea, but apparently it was a total mess to work with.

IIRC, their CMS was supposed to be the Holy Grail of video management. Centralized and decentralized storage, processing farms, search-by-image, advanced tagging, etc. Sounded quite impressive but from what I understand, there are now commercial solutions that have some of those feature sets. The way they explain it, when they started no one had these features.

hours? This is the BBC. It is probably months, possibly years worth of material.

Very definitely years. It wasn't just for finished programmes (i.e. ~ number-of-channels*24 hours of footage/day): it was for all rushes. All the footage that was filmed in the course of making the final programme.

We're talking stupid quantities of data here and I've never seen a detailed answer to how it was all supposed to be stored.

I'm not going to pretend I know what I'm talking about because this project was VAST - but would't a mkv type solution over P2P (BitTorrent more specifically) been more practical? I can't help but think they tried to make it too complicated and it just collapsed, not much different than your typical startup.

Hours and hours and then more hours until you get to years.

I'd say "well this is what happens when government runs something technical" but I also watched similar stuff happen at MSN when I was there. Not to say that the Beeb is like MSN, but I do know that there were a lot of dim bulbs to go around...

As a big fan of the Economist, I'm surprised they didn't catch the typo in the name of the project they're reporting on. "Digitial Media Initiative (DMI)" Especially since a rudimentary spell-checker would have caught it.

I guess Economist's CMS is not as sophisticated, as one at BBC

From what I understand (and I work in the public media sphere), this was a monster CMS project. A way to integrate audio, video, web content, reporters notes, etc. Is it any wonder a project like that failed?

The only reason failed content management system isn't an oxymoron is that content management system is three words.

In the US I think noone even blinks an eye at $150 million anymore

It's funny you mention that. My first thought was, "wow, that's a lot of money." But then I thought, "but how much is it really?" I actually have no idea how much it should cost for them to do what they were trying to do though I suspect it is probably a fraction of that $150MM.

For a more concrete perspective, if you assume that a good developer costs about $150,000/year, then this is one hundred years of development time for a team of 10 good developers, which is quite a lot.

I don't know just what this thing was supposed to do, so I don't know what the scope is.

no one working for the BBC in the UK would make that sort of money as a developer

I don't know about that. I've seen a few tech jobs around the £70-80k mark at the BBC, which is about $120k. Throw in the additional benefits and it's probably not far off.

Hah, I wish. I was a senior engineer there and earned less than half that.

You don't join the BBC for the money in IT... for the lax culture perhaps.

a few, and they'll be in London so your living costs can be truly massive.

Sometimes the rates you see advertised are also 3 month contract jobs where someone is looking for attention by stating the contract amount as if the job was 12 months salaried.

To 10 developers add a few dozens of Business Analysts and Managers. Outsource some work to India. Oracle licenses and IBM fees. That will easily get you over 100M.

I expect that the lack of a clearly defined "what they were trying to do" was a large part of the problem.

$150 million roughly equals to 1000 developer years.

When I was at BBC Online ~10 years back they wasted a couple of million on a content management system through a "leading" consultancy that never made it to production.

I'm surprised more public projects don't run as bounty-style "build x and get $1 million" type systems. Rather than picking who's going to do what (or worse, building all in-house), having many different small times competing for one pot seems like it might be a better way to get a good starting foundation. Or perhaps that's a crazy idea for something as big as this.

Do you think any decent consultancy is going to be willing to invest millions of £ into building a system where they will most likely be able to recover none of it?

OTOH , more uk government code is being released as open source. So in theory any developer could study the source and offer to fix bugs/add features and give a more accurate estimate. This might increase competition.

It might not lead to savings though. Suppose that the normal contract price is $1,000,000. If you have a bunch of small firms try to build the system, each on only gets the reward with probability p = Prob(they succeed and they get picked out of all the successful firms). A firm would only participate if the reward were greater than $1,000,000/p.

Because the best aren't going to put good work into a project that will probably just get thrown away.

If you entered a bounty project with two hundred other entries (all of which have systems similar to yours), would you put hundreds of hours and all of your effort into the project in spite of the fact that there's only a 0.5% chance you'll get paid? Not if you want to stay in business.

The size of the project is barely even relevant, take a look at bounty programs that are up now for areas like design (99designs, etc) and how they are chastised for pumping out bad products from designers who are only putting a fraction of effort into the projects.

This type of system sounds great for those looking for cheap labor, but in practice it falls apart.

That's spec work.

Yes, that seems quite obvious, which is why it puzzles me that you took the time to post it.

Who has time to build a $1M system and only potentially get paid for it? I know the basement Lisp hacker[1] probably thinks he could cobble it together in a weekend, but the reality is that these are huge systems. How could you get a team of people to build a system with a $1M budget without knowing if they'd ever get paid? How would the bounty be awarded? How would they know the winner produced software that is up to snuff? Would there be multiple parallel betas? Would this system encourage people to cut corners? Would the person choosing the winner be able to just give their buddy the prize and turn down everybody else?

[1] at this very moment I'm in my basement, hacking on some Lisp

Article doesn't actually tell us what it is, beyond a traditional CMS.

More important, what is in it. A large difficulty with content management systems are the data types; pairing down or normalizing metadata while trying to map old to new.

They were trying to build netflix?

Sounds like a typical hospital EMR/EPIC/Cerner/Meditech deployment!

Now they didn't. They "transfered" $152 million of public money into private hands.

It's a success. Plus, they get to do it again.

Reply on news.ycombinator.com