Europe’s biggest city council faces £100M bill in Oracle ERP project disaster
theregister.com
theregister.com
Councils are I think too small to access GDS? If they could I'm sure the superior GDS project management would mitigate this kind of disaster.
I remember when the University of Cambridge had a similar disaster and ultimately published an inquiry: https://www.admin.cam.ac.uk/reporter/2001-02/weekly/5861/1.h...
Councils probably don't want to access GDS. Why lose all those sweet vendor kickbacks? Free ticket to this, free travel to that...
There is a lot of corruption in Britain at the lower level of government. It's all in plain sight, it's all legal, and because it benefits the same networks (someone would say "classes") that are structurally in power in Westminster, nothing gets done about it.
Besides, corruption isn't needed to explain this. ERP projects fail all the time, with SAP as well.
Maybe you don't even need to do that, maybe you already knew them from school, so it was a given that they'd call you as soon as they got in a position to bring you in - and you'll be so grateful, you'll probably let you stay at your cottage in France for free, any time they want.
Etc etc etc.
This is stuff that in other countries would be called organised crime, mafia, defrauding the state, and so on. In Britain it's just how the ruling classes go about business.
Honestly, it's not that surprising.
Government salaries aren't competitive with the private sector, and especially not with FAANG. Same government workers are now tasked with identifying requirements and managing consultants. But are they even qualified to do that?
I've been in the private sector for the last 10 years and most of the folks I worked with sure aren't.
Birmingham cannot cease to exist, however, so failures need to be recovered from.
In my experience, Oracle only ever built one product that was good, and you can replace it with PostgreSQL for significantly less money and gain greater flexibility to boot.
I think the source of Oracle's execution woes lies in part with their Fusion Middleware. It's garbage. Even though JEE has become somewhat outdated, it's still viable. That isn't the issue. The problem is the horrible memory management, continually stuck threads, and their poor messaging implementation. It makes solutions built on the platform prone to constant failure and requiring lots and lots of hardware - which of course increases licensing costs back to Oracle.
I've given up trying to figure out why people keep buying their stuff.
Enterprise software is always *enter-pricey*.
Lots of $$$ poured into consultants and very little to the folks who actually do the work and implementation.
And of course, a project (or program in corporate speak) of this magnitude will have a lot of meetings involving a lot of people that usually ends up in very meaningless decisions.
If Apple had been in the B2B enterprise space, the iphone would be an exact clone of the blackberry
I tend to think that’s true. I have been involved in some process improvements at my company and it was almost always that:
* Nobody really knows how things are working now
* A lot of people don’t want to adapt to new processes
* Execs give you a lot of verbal commitment but don’t do anything to really help
It’s really hard to implement a good software system within these constraints.
People are incredibly siloed and very incentivized to make the issue someone else’s problem - if you have a vendor you can shift the problem onto, even better.
Big cities in Turkey have a single metropolitan municipality each. Below the metropolitan municipality each district also has its own local mayor and local parlament. They have a duty sharing system.
For example water and public transportation is responsibilty of the metropolitan municipality. Collecting property taxes is responsibility of the smaller local municipalities etc.
1) this is a huge bill, but it is relevant that it is a bill for a large city council (as opposed to a village facing the same bill)
2) they are big enough they should not really be considered as too small to afford good counsel on how to do this sort of project properly
Some people inside these customer organizations are fanatically committed to it, since what choice do they have? A disaster like this?
The better idea is to take a more modular ERP imho, and slowly transition.
...or just imagine what £100M in bounties to open source developers could accomplish. It would probably get their requirements met and, as a side-effect, contribute billions to the world economy by liberating many others from lock-in, besides just Birmingham City Council. As it stands, all they seem to have accomplished with this ludicrous amount of money is getting rid of one evil by inviting another.
With that amount you could've reasonably employed a 50 person department with roughly 100k cost of employment for 20 years.
Contracting disasters are always because the customer is trying to outsource thinking. Usually it ends poorly. Either way, you don't solve the problem with a configurable OSS tool.
From the article: "Standard SAP is the same, but BCC customized SAP to get it working really well and apart from some minor annoyances, SAP was a good product that should never have been ditched," the insider said.
I've used two open source ERP, both as end users and as maintainer (granted, in smaller companies). Both were objectively better. Props to ERP5 and the composition it allows by the way, especially for the web pages that are easy to modify without prior knowledge (controllers and the database especially are a pain to work with however, but unless you want to create a new module yourself, you won't need it).
Open source ERP can neither cover the functional range of main ERP systems like Oracle's or SAP's, nor can you expect the same level of domain knowledge being applied to it in development (with a small fraction of the developers the large ones have, lacking the specialization at the large vendors), nor do at least I know of success stories in that regard with Fortune 500 companies, as their main, driving ERP system.
Finding consultants who could help was like pulling teeth, though. The offshored Indian ones were terrible, and the European ones often didn't speak English well enough. Mexico City was had good offshoring options but couldn't supply enough bodies.
Never had a strictly Oracle project go well though. Exadata is a dirty word in these parts. Only thing that was more of a PITA was Azure...
In theory, you should be able to have your software support the workflow that works for your specific organization, business, and context. Rather than having to change your workflow to what the machine demands as universal. The latter seems really inhumane.
In practice, it never seems to work out very well. Does _anyone_ like their ERP and it's implementation?
These products are designed around being customizable -- if they gave that up, they'd be much simpler products, which means they'd probably work a lot better (for what they are designed for, which may not be what you need, which is the problem), and be a lot cheaper (which is not necessarily in the interests of the companies that sell them -- the complexity is fine with them, as long as they can keep charging for it, including with their consultants).
It almost makes me wish yo ucould do things without software at all -- but that's a disaster too, this isn't 1970 and pretending it is doesn't work.
There's really no way out. We've built a society where you can't do this without software but the software that would be _good_ takes more skill and resources than society can possibly afford.
Software around workflow (an ERP) is one of the most challenging areas. Because it is hard to shoe-horn it into a universal pattern, or figure out what a given organization actually needs (not always what they say they need).
To do it in-house succesfully requires a level of healthy functioning in the organization as a whole (to be able to figure out and decide functional requirements, to prioritize, to give engineering the environment where they can be succesful, etc) that few organizations have.
And with current software engineer salaries (and UX designer, product manager, etc), even if all your cards are in order for success (very rare), few could afford it. It am confident it would not be cheaper than the big ERPs; you'd hope it would at least reliably end up better/less bad than the big ERPs for the cost, but I'm not confident of that either.
Big enterprises pay a lot more and will have to maintain all sorts of operations folks anyway, so I'm pretty sure you could have a team of, say, 5 SWEs for less than and ERP system. Especially for less than Oracle.
Also there are at this point plenty of open source ERPs you could start from.
The supposed extra features are usually 20-year-old disasters. For example, one thing they all have and are very proud of is a standard UI system. That means the "same form" works on Windows, on the web, on phones (these days), with a centralized auth provider, ...
Except to say that ABAP GUI sucks is like saying first hammering a spike into your own skull and then having it pulled back out by a bull is a suboptimal experience. The question that joke screams "why would you do it like that" is one you'll be asking a lot when using these ERP systems. In case there's any doubt: their UI most definitely does not really work on mobile without extensive changes ... also known as "development". And yes it's been improved and is better. Which is to say at one point I believe Oracle had a "mobile UI" which was an android "RDP client" (sending screens pixel by pixel) connecting to a windows server which ran the windows binary of their UI. Even required per-user windows licences. I don't know how they did it, but it actually had windows 3.1 style scrollbars on the phone display. These are improvements in the sense that HIV is a big experience improvement over gonnorhea.
And you might think "surely that means the form is going to be very responsive !".
Heh.
These installations require DB admins, require SWEs, require managers, oncall, ..., require colocation/dedicated/cloud costs (may God help you if you're paying cloud prices for your DB, and whilst I agree admin is easier it's not so much easier that you don't need a DB team). Oh and I've never seen a SAP ABAP installation go 6 months without a consultant getting hired to solve some issue the company couldn't solve.
Just about the only really unique thing about ERP software is that the code for the "applications" lives in the database itself (MS access, or Dbase used to do that too). This is also just about the worst thing about them, as it means you don't get integration testing, or standard version control, or files, or ...
Yes, I know, somehow Oracle has convinced every MBA to do this rather than pay 5 SWEs to just make them PHP apps. And no, I don't mean those PHP apps will be worse, they'll be way better. It's standard so you can get consultants to attempt to fix the worst disasters.
I think it actually requires the organization to morph into being a tech company first. In order to have good software to run a City Council, it effectively requires the Council to become "a software-making organization that delivers council services". In order to have good software to run a bank, it requires the bank to become "a software-making organization that delivers banking services".
Unfortunately, it's both politically difficult (nobody goes into any type of non-IT activity because they want to work with software) and really challenging: not even IT organizations have figured out the best way to address problems like technical debt, language churn, documentation etc, and non-IT organizations face them just as much.
I don't actually want my city government to be a "software-making organization that delivers government services".
Every organization needs an ERP (maybe?), but not every organization that needs an ERP is or should be a software organization. Saying that in order to get a good ERP every organization has to transform itself into a software-centered organization... seems a bit much.
Its mission is to deliver council services, but if the best way to do that is through efficient IT (and in this day and age that's pretty undoubtedly the case, since most communications are pushed via websites, payments are made via automated systems, etc etc), then the organization should definitely pack the necessary skills and organizational tools to achieve efficient IT first and foremost.
> I don't actually want my city government to be a "software-making organization that delivers government services".
Let me understand this.
You want your city government to overcharge you on Council Tax because they didn't see the form you filed two months ago about being a single occupant.
You want your city government to miss your bin because they actually performed the collection one day earlier than usual and didn't alert you in some way.
You want your city government to be at the mercy of unfaithful civil servants who abscond with money by applying dodgy accounting.
You want your city government to dig themselves into massive financial holes because their planning and forecasting software is subpar.
You want your city government to be unable to analize expenses and contracts to identify where they're getting gouged.
You want your city government to ignore complaints about this or that item, since they can conveniently forget to file them when people call.
These (and more) are not hypotheticals - this is what goes on in many (most?) local authorities, today like yesterday, and it's all stuff that can be fixed with software by organizations with the right skills and attitudes; but they require putting software (and people who know how to make software) at the heart of "how we go about doing things".
But services provided by IT are only one part of what I expect my local municipal government to be able to do competently, and not necessarily the great part. While I would like the services provided by IT to be done competently (ideally beyond competently), it does not necessarily follow that they should be focused on to the exclusion or de-prioritization of things that are not IT, which it seems to me would be the consequence of the local municipal government reorganizing itself and considering itself as fundamentally and mainly a software engineering organization (or an IT organization in general, which is not necessarily the same thing either).
Convincing your auditors that those systems were representing your financial position correctly would be hell on earth. Like them or not, auditors know SAP, Oracle and the like - and have well-defined processes & experts to certify them.
There's large amounts of risk and cost on both sides of the equation.
At that point you’d might as well compete with the likes of Oracle/SAP yourself, or the employees themselves will probably start their own startup to do so.
Yes, for a number of reasons:
1. An ERP system consists of thousands of modules, out of which a smaller subset is needed by the customer. One typically pays by module so you want your number of modules to be as small as possible.
2. A customer this size has already implemented a number of business processes that they are not willing to sacrifice, even though an ERP provider will do their best to have them import "their" way of doing things.
3. Customers want to acquire an edge, not be on the same level playing field as the competition. "Kill you darlings" - eh, no.
4. If a municipality buys an ERP system tailored to manufacturing/trading, well then you have substantial customization ahead.
Birmingham City Council doesn't have competition.
But they may have specific municipal / local / state / national rules to comply with in ways that commercial entities do not.
These differences imply customizations.
You can run your business / org exactly as the ERP designers envisioned, but that means you'll have to adapt to your ERP, which is wrong-headed and certainly much more expensive in a large org.
Even the most vanilla implementation for a small business will have a degree of customization.
Bingo. I'm not going to tell my CEO to change how they run the business just to conform to how the lowest-bidder ERP decides I should do things.
Unless it's ServiceNow, for some reason.
Redundant headline. Oracle & ERP almost always implies disaster.
Seriously, never again. Ditto for IBM, and a lesser extent, SAP. I wish Red Hat had a real competitor, cuz Canonical ain't there yet.
The technical problem is wanting one end-all goto system instead of like 20 different utilities I guess.
In my experience it only takes one or two levels to get to that point.
Regarding the insane requirements, it's also my experience that a lot of companies think they're snowflakes, thinking they do something really special, when they're really just doing what everyone else is. For historical reasons they ended up doing it this way instead of that way and now they'd rather have the world change around them than change their ways.
At least with anything government, there is another problem: changing processes often requires a lot of work, which means explaining to (completely clueless) politicians why something is necessary, get cost approval for each change, convince majorities... it's easier to have an omnibus "migrate to newest version of XYZ" package and to spend the money on making the system operate according to your whims than to fix the underlying bullshit.
This is my experience working in government IT. The people whose job it is to go to meetings and talk end up being the ones representing the businesses interest and the vendor ends up running circles around them because they aren't in any way qualified to talk about something technical.
At some point these systems have to share info, dump info somewhere (data lake, backups, whatever), and output info.
Inevitably there will be one or two systems that connect them.
Plus the overhead of dealing with 20+ different vendors, systems, updates, vulnerabilities, interconnections, etc. is a nightmare. There is a point where a "single source of truth" is too much effort and too expensive, but a clusterufck of different systems is not a great solution.
ERP vendors generally have a whole bunch of standard applications ready. Things like inventory, sending out stuff, payroll (for X number of countries), ...
Most of these are built on a gimmick, the same as in the Open Source world. Every database has "their thing". Mysql "is fast and simple". PostgreSQL supports everything you could possibly want.
Oracle has their "fantastic" SQL compatibility and their fast clustering solutions. SAP has their "live indexes". Of course, both have implemented each other's features, and so has PostgreSQL at this point. So that apparently makes them great developers of timesheet software, right? Apparently to MBAs this makes sense.
These companies are essentially consultancy software development companies that have an in-house "core" software (Oracle, SAP, ...) that they can work with really well (in theory), and have standard solutions for many problems on that core. But 3 ABAP developers aren't that different from 3 PHP developers: they all do things slightly differently. And, yes, I do think SAP ABAP is "kind of" PHP. You can even make it look pretty similar, though it uses their internal stuff, not HTML. These companies are selling "quick" and "standard" business software development.
There are major differences too, though. SAP stores all it's code in the database itself, which prevents all development tools, from vim over git to CI/CD from working.
Of course this would mean Wordpress is ERP software ... and it kind of is, with enough plugins.
Long before I ever heard of ERP, I worked on the Burroughs accounting systems. They had inventory management, accounts payable, accounts receivable, payroll, subcontractor accounting etc., all feeding into the general ledger.
So is that really what ERP is? Is it what used to be called an accounting package?
Whether it makes sense nowadays to continue stuffing these all into one system centred around the general ledger is left as an exercise to the reader.
Basically, the totality of "back-office" operations software. At minimum, it's usually accounting+finance, often including HR, sometimes also CRM. And in niche spaces, other stuff too (an ERP for higher education/colleges would also include student enrollment management, etc).