CDC website built by Deloitte at a cost of $44M is abandoned due to bugs
technologyreview.com
technologyreview.com
Can’t make this stuff up.
https://beta.sam.gov/opp/2f5fc512fdde4e22860832337aac420f/vi...
edit: Speaking from direct American government experience.
However, I'm familiar with the term ‘kickback’...
Well, it turned out it was, because the person posting the add forgot to take out the phrase “Make sure Björn meets these requirements” in the middle of the ad.
Sounds like they could save a lot of money getting rid of the training requirement.
https://www.gp.se/nyheter/västsverige/militär-blunder-när-ny...
Under the first bullet in the screendump in the article it says:
* Experience from personnel management positions and documented good leadership qualities (Here Henric lacks the clear experience why I think it should be moved to meritorious)
In French it’s called being "fléché", as in attributed for someone. It’s of course totally contrary to the rules. (It’s most of the time somehow well intentioned, from the point of view of labs doing it, like there’s this great person doing a lot of work and they deserve this new opening the lab fought to get budgeted.)
The sad part is that some poor naive young researchers will prepare and cross the country to interview (hopefully now there will be more video calls) not knowing that the call is already custom fitted to someone in the lab.
The fun part is that, as there has to be a majority of external jury in the commission appraising the candidates, the external jury sometimes don’t even know that the call is rigged and pick another, clearly better candidate Then, a few months later the candidate starts their new job and is really puzzled at the attitude of this one person… the one that was supposed to win the call
When the French government puts out an EU wide bid for something (As is the rule in the EU), the bid will be won by a French company. Every. Single. Time. And it's not by accident.
Finland, on the other hand, is playing things stupid.
We've had cases where a city's laundry services was won by an Estonian company(!) because they were the cheapest. They actually shipped all laundry 300km+ south, over the bay and to Estonia. Wash it and send it back. For some reason the people handling the bid were all suprised_pikachu.png when it turned out the arrangement didn't work AT ALL.
That's all down to knowing how to write the bid so that the winner will also be the best.
Joke aside, it seems one of the winner of the Finland->Estonia washing contract is actually a Finnish national that incorporated a company in Estonia: https://www.aripaev.ee/article/2011/2/22/soome-pesupesijad-u...
We are talking about permanent positions guaranteed for life with fantastic perks. They're extremely competitive to get.
When a permanent position is opened, it's often for a position that's been contracted out for a while.
Want to hire the contractor who's been doing the job perfectly for 2 years? It's simply not possible because of the law above. Gotta advertise the job nationwide and run formal interviews. An interview panel will run interviews and pick whoever they like (it takes months).
It's not as rigged as the poster makes it out to be. The panel is 5+ random people selected among a very large company. They don't know that the position is tailored to someone and they couldn't care less either way, they're all people with a cushy position guaranteed for life, no favor to give. (It's really the opposite of Russia/China for those doing the comparison).
It's rigged in the sense that if there was somebody doing the job for year, they're probably more qualified than you are.
This rule exists to prevent middle managers from doing empire building and growing their headcount indefinitely. It works as you can see above where a better person was chosen.
It is called L-1 Intra-Company Transfer Visa: L-1A and L-1B for transfer of managers and specialists
https://workpermit.com/immigration/usa/employees-guide-us-im...
(I should clarify that I’m fully in favor of immigration programs that serve to increase the density of highly qualified individuals in the US and to fill positions which would otherwise go vacant. If as a small side-effect, those programs depress wages via increased supply, that’s still an excellent trade for the population. I am not in favor of visa programs substantially/primarily aimed at decreasing wages.)
That is why George W Bush and older generation Republicans always supported legal AND illegal immigration. Their constituents were the businesss that thrived and made more profits from a cheaper laborforce.
The Democrats were historically hostile to immigration because their base was the working class. Now that Wall Street and Big Business has gone Democratic since Bill Clinton and the New Democrats there will be more supply of labor and thus lower wages ceteris paribus.
Today I learned another useful latin expression.
The job postings are useless, if you want to work there what you need is an introduction (someone to give your CV to a lead that needs hands on deck). Then 2 to 3 months later a job posting with an almost complete copy of your CV will appear and they’ll hire you.
I used to work in a role where I had to respond to tenders for the software I led, and it was obvious that if the requirements list wasn’t written by our competitor, the institution had decided they wanted their software and listed every feature they knew only that company offered.
I guess this could happen with hardware too. But adding more features is much easier in software. Marketing can then argue that it makes a certain product “better”.
For a while my entire job ended up writing proposals and all the related nonsense for federal an state government contracts. I did not like that one bit.
However spending week after week trying to divine the meaning behind phrases and terms.
It shortly became apparent that some parts of the government would consistently have requirements only applicable to one vendor.
That saved a lot of time since I could just skip them whenever they came out.
These types of «arrangements” often go all the way to congress promised pork for donors and secure job for voters.
If I have given the impression I was doing all the work myself that is not intended. We had a big team of all sorts of disciplines
Meanwhile auditors (unless burned out) would love to get the contractors and buyer staked in court, but technically all checkboxes on the checklist are checked (pun intended) and there's no obvious better candidate to use as counter example.
No corruption to see here folks, move along.
The problem here isn't even necessarily with the government: Had they gone with anyone else at a sane price tag, Deloitte would have sued that company to oblivion and the Government as well, probably for all sorts of IP issues.
I can assure you that the GSA does not treat its construction contractors so preferentially. Lowest public bid wins, with preference to companies owned by service-disabled veterans, women, and minorities. I’ve never been directly contracted by the GSA, but I’ve been a sub for GSA projects several times. The closest you get to “one party fits the contract” are systems like fire alarm, where if a Siemens system is in the building, you must sub out the work to Siemens, or if there’s a Johnson Controls BAS system, you’ll need to sub out that work to JCI.
I know there is a decent amount of corruption in local/state construction contracting so I cannot claim there is 0 fraud for federal construction contracts either.
Construction projects are more understood than software projects, as you have a tangible result that either meets the specs and passes inspection, or doesn’t. It’s a bit different than contracts software or a new plane where the specs can be interpreted in looser ways.
I've seen this with SunGard/Ellucian, and also Oracle. They would not make their documentation & technical manuals publicly available. A third party arose offering general maintenance & other services, and used customer's documentation copies to do so. SunGard sued them, and the customer [0]. Oracle has done something similar [1].
Unfortunately, this is a common practice by providers of large proprietary systems, and whether or not the vendor prevails is almost irrelevant: customers now know that competitive bidding brings a risk of costly litigation.
Personally I don't think suing your customers is a very good business practice for long term customer loyalty. Then again some industries only have the choice of a few vendors, roughly equally litigious, to choose from. The alternative is to create your own custom system, but that's an option only available to very large customers.
[0] https://web.archive.org/web/20150319233444/http://blog.thehi...
[1] https://www.chronicle.com/article/a-small-company-promising-...
If you can’t get change orders to cover your budget overages, your company loses money.
Making the CDC look bad appears to have been part of that program last year. Shame the price has been so high.
When people say privatizing things could be a solution, they don't usually mean "take tax money and award it through a no-bid contract to a private company for laughable reasons which could be either stupid or corrupt, your choice". At least if those aren't the people who are getting no-bid contracts, those people mean exactly that.
> Making the CDC look bad appears to have been part of that program last year
You sound like somebody outside CDC on purpose went and made CDC look bad. As far as I can see here, CDC did it to itself - and it's way not the only messup they had recently, they also totally bungled covid tests rollout. There seems to be serious trouble there, and if they think they will be excused because of the pandemic - they unfortunately may be right, but this only means the rot will continue.
This is what is being referred to when a politician used the words "public/private partnership.'
By paying lump sums you are basically encouraging your contractor to screw you over. You should compensate them based on metrics that align with your goals. For example, pay $1 (random number) for each vaccinated citizen.
I think government is just sitting on the extreme end of bloated bureaucracy, with everthing defense related taking the cake.
This is combined with how various voter-friendly "cheap government" initiatives go on for all kinds of internal purchases, not just big dollar projects like this.
Printers that needs servicing all the time because "cheapest bid" means you have low-quality shady consumables used.
Cars that end up constantly in service because the provider is cheap and thus couldn't do the right job servicing a Land Rover Discovery 2, taking away critical machine for reaching accident sites for aviation accident investigation.
Purchase of new cars for the same org and purpose - the freshly elected government leaked "appropriate" story to one of the newer tabloids to make it look like excess that they are curbing.
I still tell the story of SCAAI investigator buying a plane ticket for international flight because office responsible for finances in appropriate ministry refused to arrange diplomatic courier. So an investigator booked an international flight, left the plane, gave the CD full of confidential data to his local counterpart, turned 180 and went back to the same plane back.
However, if a large outsourcer like Deloitte gets the gig, they should have the right experience and knowledge to work with the customer to determine what they really need/want, and to steer them through the whole process. They should also ensure they use best practices, such as tests. If a company fails here.
I think there is blame to go round on both sides.
Well how the hell do we have a £40M contract for supply of laptops to schoolkids filled by a noname laptop supplier using the jankiest (really low quality, really poor spec) laptops worth maybe £100 but being charged to the taxpayer at >£300 each.
School IT people are getting them delivered, a lot are broken, and the cost of repair is given as more than the cost of buying _retail_ a better replacement.
I don't know how these people live with themselves, ripping off the taxpayer and lumping in-need schoolkids with poor equipment.
Anyone in that decision chain could spend an hour reading "budget laptop" articles on trusted websites, then look on the website of a high-street supplier for a top price. Boom, saved the taxpayer £20M.
Mind you at least something was delivered, current UK government tenders appear to be typified by non-delivery by completely inexperienced companies for costs well above market.
They already exist. Mysterious why CDC didn't use them. Maybe they are hollowed out after the Trump years. I know I wouldn't have stuck around.
So enevitably requirements are tailored to what I want. It seems someone wanted Deloitte, why (and how expensive was the dinner) is another question.
This is how hospital software is handled (at least in Finland). The one making the decisions is the head doctor in the hospital, because they have the most power.
They know fuck-all about software and fuck-all about the day to day work of the nurses and doctors who need to use the product every day.
And this is pretty much how we ended up giving Epic Systems 774 million for their MUMPS-flavoured piece of shit rebranded as "Apotti". And the project - started in 2015 - isn't even done, nor bug-free yet. The American model of everything being cost first didn't fit the Finnish model of medicine - to the surprise of no-one outside of the organisation.
It was so bad that other cities refused to use it and went with their own solutions instead of using Apotti.
https://a16z.com/2019/10/03/anduril/
Basically, build first at the VC’s risk and then deliver a finished product to Gov.
That was our nickname for our worst customer. Some wannabe nerd promoted to CIO, probably to keep him away from patients and the other staff.
His most substantial contribution was divining the precise shade of the highlight color used by our theme.
Our "portal" had labs, scripts, meds, notes, referrals, and on and on. Stuff like lab reports were industry standard (we had domain experts on our team). Dr Corn Flower had opinions on all of it. Not detail too small to argue about. Every pixel mattered.
We eventually got to the point where two team member's full time job was babysitting Dr Corn Flower, while others were permitted to do real work (skunkworks style).
And that was just one hospital. We had dozens of other customers, and 100s of partners.
Imagine hiring web developers with only HR on the interview loop, no manager or team rounds.
Every single piece of electronics I own was bought that way.
How else would you buy stuff without being influenced by some subconscious bias caused by some stupid ad?
He just explained he picks the cheapest item in some database based on some specs.
Imagine if you had to describe in exact detail every possible aspect of a mission to Mars in 1940. You are guaranteed to wind up causing the death of everybody involved, because there is an enormous amount of research standing between where you are now, and having enough information to create a workable design.
The real problem is that the nature of the contracting system doesn't allow for open-ended / minimum-viable projects with iterative development, you have to do your best to write the requirements then throw it over the wall and hope for the best. Of course you're going to wind up with a mess.
The problem comes in when things like this happen - that a company forces itself to be the only option through unnecessary infrastructure and proprietary software in the public space.
The biased/corrupt person that ultimately is forcing the decision gets their pick, but doesn't face the full consequences of their decision because "hey it was a bidding, everyone got to offer". People making multi-million dollar decisions should be exposed with plenty of skin in the game for things to ever not be as rigged as they currently are.
Government, public offers, invitation to tender must not only play by the cheapest but also by the long term impact that changing contractor every 4, 5 years has on their own management and system. This is getting as absurd and stupid. Force contract to use standard/boring technology and leverage a large market of companies for support. Being the cheapest never was a fair decision maker.
Clearly not. Unless... the requirements were garbage to begin with?
At some point it will become obvious that software complexity is a national security issue. I feel that point still hasn't arrived, for whatever reason. Perhaps the realization has arrived, but those offering the realization have biased the reports to favor themselves (i.e. Deloitte, et. al.)
"U.S. national security increasingly relies on software to execute missions, integrate and collaborate with allies, and manage the defense enterprise. The ability to develop, procure, assure, deploy, and continuously improve software is thus central to national defense."
We already know it's a national security issue. Nothing is done about it.
1. competent technologists
2. competent technology managers who can communicate to the suits who are in charge of the budget that these things are needed for a functional quality tech environment
#2 is a much rarer skill than #1, and those people will get paid way more and will have to deal with way less BS working in tech companies than government positions.
Within a year or two the good staff end up working at the consulting shops and influence their former employers
In software world things change so fast that someone with 6 years of experience developing software will not have enough experience with newest tools, languages, methods, etc. Newer graduates with experience is newest applications in their resume get hired and they sometimes lack real development experience.
Building and delivering software is not like building a road or sidewalk where technology does not change by much every year or decade. They still construct sidewalks like they did say 100 years ago.
So major software contractor's success is pretty much random depending on who gets hired and how good the chosen packages are for the given application and how experienced are the staff in putting it together and having good test specs.
Another problem in software world is that software developers move to new IPOs and so the staffs are ever more transient. Government jobs with complex hiring practices are actually becoming unattractive to developers.
They don't even seem to be saying "because Deloitte was successful in the past". It's literally saying "because they're the only ones who have access to their proprietary system that we already locked ourselves in too.... they get the business".
"...so the staffs are ever more transient". Yeah, there's some idea that "company X" does ABC. The people at company X do ABC, but you are at the whim of institutional knowledge and tribal info systems within that company. The superstars who made ABC work 4 years ago may all be gone, and the current people are trying to make a name for themselves by making 'The New Process' so any success from ABC isn't at all guaranteed to translate to your new project.
Nothing happens for no reason. Few cases of success in big projects in the past must have got them in. History has impact on future.
Because it is so difficult to launch a project and provision the requisite infrastructure in the federal government due to various IT regs and laws and subsequently obtain an Authorization To Operate (ATO) in a timely fashion an existing platform is routinely sole sourced especially when speed is necessary.
Why? This is both legit and utter bs. Legit: the fed gov actually attempts to protect a persons’ private data. BS: after the drain of the acquisition process (led by non-technical folks), ATO, slide decks to more non-technical management (Deloitte and their peers crush this), the competent folks find other jobs either in or outside gov that free their creativity.
Positive: groups like USDS are reducing the drain, but once an exec gets burned they go back to checkbox management and shun the learning organization culture that has made tech the anti-thesis of this type of acquisition in other sectors.
*edited misspelling to show gov tries to protect not product privacy data
The British vaccine is being rolled out by public bodies and is working pretty good indeed (half a million doses a day).
Meanwhile our track and trace system, misnamed "NHS Track and Trace", is, despite the name, almost completely outsourced to private industry (and barely even speaks to the NHS). It's universally acknowleged to be an absolute mess.
The deeper lesson is to look at incentives: where things go well, the structure encourages the outcome society wants. When they’re misaligned, you get global climate change or the American healthcare system underperforming our peers at whole number cost and stress multiples.
I'm also curious, anyone know where the development actually happened? Did Deloitte further subcontracted out? Was it outsourced?
That's also why you see some government sites operate on a 9-5 basis
I’ve always wondered why this was the case. But surely inconveniencing the majority instead of making it more convenient for the minority is a step in the wrong direction. Whatever. Off topic...
Don't contractors do everything they can to milk government contracts as much as possible? I'm pretty sure I've even seen people provide advice on how to do this effectively right here on HN. So I can definitely understand how it happens...
I realized this once when I started attending (Canadian) gov ‘innovation’ events and ‘startup’ funding initiatives.
There was hardly anyone I’d consider an engineer that is benefiting from these systems. It’s a giant well organized grift for washed up “industry” execs and others hangers on with friends in the right places.
And I don’t mean to be ageist saying that. That would be over simplifying, but there is a inherent preference to those with safe names in their resume, regardless of relevancy. But there’s plenty of spaces in between for youthful people who understand how these meat factories work. Especially ones who understand their value, usually the unnamed ones working on the edges.
But by the time it comes to actually making the meat most of the money has already been allocated to the aforementioned people and they rush to find outsource the “real” work to some overpaid consulting company who are also experts at fleecing these gov contracts with endless meetings and perfecting the right checkmarks get checked.
I remember going to a Zuckerberg talk around 2010 at YC school at Stanford and he mentioned how he was hiring engineers even for the typical ‘executive’ jobs and I remember feeling like that’s the difference between startup tech companies and these failed hundred million dollar imitations.
It all comes down to the people and which ones are getting the power. Basically which side of the “iron rule” they come from https://www.jerrypournelle.com/reports/jerryp/iron.html
Like is there bloat on these projects? 100%, and a lot of it, but saying "just get 73 solid engineers in a room" doesn't really get you very far when the requirements are really really unknown.
Get them to sit next to the users (especially UI/UX) and see how the process happen. Ask questions. See how data entry people work.
(Ideally. In the real world it will be a very thorny and boring process)
"This is asinine!" you say. "Just make sure you dont break our existing workflow" they reply.
Most of the effort is actually managing the customer. That is, their contracting requirements, the whole requirements process, procurement hoops, change process, optics management, cost of "sales", reporting requirements, various additional compliance requirements...
I would blame government, but I'm simultaneously experiencing all these issues with a Fortune 100 company. They're almost indistinguishable in the degree to which their cumulative organizational headwind is a hurricane I'm supposed to fly into. It's simply hard past a certain scale for purely people and procedure reasons.
[1] https://www.koin.com/news/oregon-oracle-settle-cover-oregon-...
I’m just curious to know can you get government contracts let’s say if you are running a software agency
This is the overview, there are other websites that go more into the details if you google it
But basically it is survival of the most boring and the most detailistic. Then of course you lose the tender because you weren't "in the know", or you missed the requirements of the Toilet Procurement acts of 1974 which specified the thickness and traction resistance of toilet paper used in government purchases.
Oh and if you're "lucky" your Health contract management software will suddenly have to be able to order pizza as well, you know, it's "just an extra thing". But you won't get paid while that doesn't happen (ok this might not apply to the USA specifically, but you get the idea)
(As a new contractor you might be able to get a small-money contract from e.g. the SBIR but those max out at roughly $100K.)
I think it might also have had to interface to state systems in the states that were not running their own ACA exchanges, and to the systems of numerous insurance companies.
$44 million is an egregious number and I don't know how it's that high, but I'm betting at least part of it has to do with the fact that this was a website with an accelerated timeline, which probably necessitated a premium.
(I'm not taking the dismissive 'this is easy' side of the argument here, but the project failed, the lessons learned need to start there, not with the terms of the contract)
I was hired for a particular engagement with Microsoft years ago by a similar firm. Microsoft needed x, the firm had nobody so they put out an urgent call and passed me off like I was one of their long-time employees.
I think it’s safe to say nobody has a bench of 146 people ready to go at a moment’s notice.
And with all that said, companies like Deloitte do have hundreds of people ready to go. They may not have a deep bench for certain niche projects, but I used to work at a Big4 and for something generic like "build us a website for people to sign up", we had entire offices of people around the globe in our offshore delivery centers just waiting around for work like that. Oftentimes we would still subcontract because it works out to be cheaper if 10-20% of your resources on any given project are subcontractors, but they weren't always necessary.
With that said, the start of the pandemic was ~1 year ago. Even if it started right then, 12 months for a country-wide, government-run IT project is a rush job.
A lot of the debate about pandemic management boils down to criticisms of, "did it wrong," without saying what would tip the threshold in the other direction.
This is problematic because it inspires a lot of talking past one another, resentment, and miscommunication. Ultimately it turns into a churn of frustration, defensiveness, and offense.
But even if you manage to get a response to agree on what, "right," means, the depth of their example is likely shallow. (e.g. "Such and such had only 12 infections because they such and such.")
This, too, is problematic, because it presupposes a ton of equivalent variables between the two countries. Population demographics, geographic structure, legal structure, culture, economic stratification, etc.
Conflation between a, "did it right," and a, "did it wrong," country is practically impossible, at least in the context of pandemic response. And anyone who tries to suggest otherwise is either ignorant, lazy, or obtuse.
If you get a satisfactory answer to your query, I'll move to the moon.
Isn't the whole article about how they failed to mobilize quickly?
There are lots of great people who would rather be doing something meaningful than whatever they’re currently doing and governments need to do better at tapping into this.
It’s a different problem, but for a more recent example, look at how fast vaccinateca.com got going.
I wouldn't be surprised if these proposals take into account headcount as a positive for both quality and job creation purposes.
I think you will find that if only there was someone in charge to put their foot down, you can do without a whole lot of these things. But that's part of the problem of course: the government contracting out is just a bunch of clueless people making up a wish list with a partner on the other side whose only inclination is to say yes to everything.
So now what, you are reporting this adverse event to VAMS who reports it to VAERS? There is no flow, it's all just duplicate effort from architecture astronauts.
There is no excuse for this kind of cost for things as simple as tracking g appointments and completions regardless of data transformations and storage. There are no magical new protocols or data formats it’s using, it’s just a lot of people charging too much for them to have week long meetings to talk about their area of expertise. It’s an embarrassment.
The 22 year olds might not be able to do it (I beg to differ), but can management consultants/accountants who barely took typing in school who have Excel as the height of their digital literacy be expected to have all the answers?
By the time this gets to technical folks, 70-80% of the budget has been burned.
Edit: Beware Management Consultants and Accountants encroaching into "IT". Their worlds are drying up because tech folks can learn business easier than business folks can learn tech.
Edit 2: removed age reference.
Can’t agree at all with the ageism though, which is wholly uncalled for.
What that may leave is decision makers who don't always have a high enough level of digital or technical literacy.
Folks who are not competent in technology decisions and implementations being in positions of decision making and spending is the point I was trying to make.
There is a difference between "ageism", and "being honest about how hard some problems are to solve".
There are a lot of unbelievably brilliant 22-year-olds.
None of them have been building enterprise-scale architectures that turn messy, vague, and incomplete requirements that model even more messy, vague, and incomplete business processes for 20 years.
Exactly! If you’ve been doing that for 20 years I do not want to work with you, since you are apparently perfectly content with that status quo.
I’ve been doing it for two years and it’s already too much. My highest priority is fixing their chronically useless requirements.
But it’s clearly not that after all, which is a relief - I’m glad it’s just a case of everyone else that’s come before you being useless.
Boomers, am I right?
The resistance is ridiculous, but it’s still the single most valuable thing we can be doing to improve our project delivery.
Given the current division of people that care/not care (or are disinclined to act on it?). It is entirely likely that none of my predecessors did.
That said, if they had been doing it (and made any progress), I really wonder what the previous situation was. Maybe there just weren’t any requirements at all?
It's still an absurd situation, but I doubt very much it's the result of apathy - or at the very least the result of apathy from your direct predecessors.
I’m sure some of them are. I’m sure a lot of 22 year olds were brilliant in the 80’s and 90’s too.
Unless your thesis is that there’s been a sudden upsurge in the proportion of superintelligent earthlings in the last 25 years I’m unsure what your point is.
Sounds plausible but I sadly only observe the opposite of management consulting “drying up”
My hope is leaders with tech backgrounds can grow into that space. Its far more lucrative than most software only endeavours.
Apple is about creating usable technology for the masses, not the few.
Apple also simplified access to legacy systems, like buying music and making it digital. Or maybe using your watch or phone to pay.
IPhone is geared for ease of use for anyone to adopt, including folks who arent super tech saavy..
Apple's innovation is charging a premium to create beginners in tech adoption who might not be experts at tech, who want yo feel empowered by everything just working.
Their market share figures would show that if that’s their aim then they are failing. IMO Apple makes premium products for a specific kind of customer and have little regard for issues like backwards compatibility. Which is absolutely fine. But probably not a model you’d want the government to follow.
https://www.statista.com/statistics/236550/percentage-of-us-...
iPhone only bring affordable to richer people who mostly want the latest means lots of second hand items. I'll warrant Android users at the lower end keep their phones longer.
Was the Apple II for beginners? School children?
Was the original Macintosh easier for beginners to use than other computers?
Was the Apple Quicktime Pro one of the first digital cameras that anyone could relatively pick up and start using?
Was the iPod the first digital music player that just worked to buy and download music from end to end?
Was iPhone.. despite arriving as a smartphone a decade after Palm, etc, was the first usable smartphone for the masses?
Was iPad the first tablet that also just worked? Seems like anyone who can't use a laptop can use their finger on an iPad.
How about that AppleTV? Can anyone use it? It's easier than my AndroidTV.
What about that Apple watch, how many people who were new to wearables had it as their first? Seems popular with the masses.
Experts are usually the few, not the masses.
There's no doubt apple charges a premium. Their devices on average last 4-6 years, longer than the Android I'm typing this on.
Rather than being first, they do try to get it right for the masses. I'm not a happy Apple user, or fan with the keyboard fiasco. Apple's history though remains what it is.. easier to adopt by beginners.
Could say the exact same thing about tech recruiters
In Germany we literally use fax and enter data by hand. A project that successfully moved things to an emailing Excel sheets based workflow would be absolutely revolutionary.
“The next 10 years of software will be very boring. The entire field of CS will be employed turning paper process into digital process, at great cost. Almost all of these projects will fail because Americans are admirably earnest but tragically over-confident. You underestimate paper.”
“Good money for us though. But have a hobby to occupy your mind or you will be miserable. Zum Wohl.”
And that’s how I ended up order an ill-considered 4th martini. He was correct, although the interregnum was perhaps 2-3 years shorter than he predicted. Or 15 years longer depending on how one judges these things.
https://www.zdf.de/nachrichten/panorama/corona-test-digital-...
And it wasn't really true before either, most of the Ämter used the Demis system.
The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience.
Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before anyone sits down to code.
If money is left over, apply to the next available item via (logic).
Unless money is sent twice, for two separate sub-items on the same item, in which case if the first amount is insufficient to cover the entirety of the first sub-item, the excess (if any) from the second amount, after being applied to the second sub-item, must be applied to the remainder of the first sub-item.
There are legal ramifications of doing the first way, without the second. And only the actual in-the-trenches people were able to tell us about the second.
Using outsourced process documentation is @&$&ing useless.
In reality, there's one edge case where things needs to be applied in a different, distinct, and very specific order, otherwise (legal consequences).
That said, if I get requirements like that I always have a strong feeling the case can be generalized. Laws generally aren’t as specific as that.
The high bill rate is for your pain and suffering from having to deal with them.
Granted, that was a pathologically bad customer, but it happens.
The ironic thing is the question you usually get from them is "Do you have access yet?"
It's your system and your company! How do you not know the answer to that question?!
This has not been my experience, which is really to say that the world is big enough that you can find any combination of blame imaginable somewhere. Once you have misaligned incentives no part of the dynamic can be relied on not to cause problems.
This is true, too often, in poorly run and ineffective consulting firms who optimize best practices to mean what is best for them.
I have worked in tiny very early stage startups and old dinosaur companies. What comes across loud and clear is that early stage companies could use a bit more organization and dinosaurs could cut huge portions of what they “do” and then both would be in a similar situation to get things done reasonably well.
The bulk of the waste is an attempt to get people who don’t know what they’re doing into a position of getting things done. Sometimes it works.
On one hand, you limit (beneficial) mutation.
On the other hand, you limit (harmful) mutation.
An organization usually doesn't scale past a certain size without stabilizing its ability to copy itself. For better and worse.
also it is good that corps and governments try to innovate and spend money on it.
Can it be more efficient?
Always(!)
Can it be more focused?
Sure.
But a big innovation takes a lot of factors to be well balanced and that alone is often not guaranteed nor possible given the circumstances. You just might not find the kind of devs it really needs (and not even knowing about it). So you hire the best thats available but it won't work...
Bad luck. At least you tried.
I see it that way. Whether vcs pump millions into 100 startups so that one succeeds or corps pump millions into 100 projects... Who cares? The money is not burned. It hopefully ends in my pocket
People spending their lives pushing a boulder up a hill.
People paying taxes to pay for people pushing boulders up hills.
The problem is wasting human life on wildly inefficient endeavors.
because that's what you will get!
This is the single most important and insightful sentence that a lot of HN-ers are going to read today, and they don't even know it.
The contract was given to a team used to do intranet websites in classical JEE format and the letting them lose in such a project, designing as they knew.
No, of course not. But it's undeniable that these threads on HN (and other engineering-dominated forums) always include a healthy dollop of "I'd have done that better, because I'm a good software engineer" disdain. It's often paired with ignorance of how large-scale enterprise and government contracts and procurement work, what roles are necessary for large-scale projects with a significant discovery component, and what the planning process looks like when a team can't afford to launch-and-get-feedback or fail-fast-and-learn their way to a full specification.
I mean, this Deloitte project and the rollout are clearly a shitshow on a million levels. They got a no-bid contract for an eye-popping sum, then more than doubled it before the building even started! But the approaches boldly prescribed in most of the comments here wouldn't have produced a working system at the scale needed, either — only a cheaper broken one. A cheaper failure is still a failure.
This one is more insightful (in the big non-tech enterprise / government spaces). Because the takeaway for most HN developers who flip between projects should be: (1) learn how to recognize one of these projects & (2) stay away / find a new job when you do.
You'll save years of your work.
which works very well if the developers and the customer really know what they want when the project starts.
- is the scope well defined?
- do I understand the value add and agree with it?
- do I feel as if all stakeholders are aligned and adequately involved?
- do I think the team can pull it off ?
- do I think the mood and style fits myself?
- do I think the timelines and budgets are realisitc?
- do I agree with the level of transparency from management?
Etc etc
Ask yourself these questions - and more - and you will discover the answer for yourself.
In the end listen to your guts.
One important nuance: A project can be a major fail overal but a big success for you. Big check, new knowledge, rich connections...
- Does the customer have an IT org capable of fulfilling basic tasks you won't be authorized to do (app / db access, AD / rights modification, firewall adjustment, AV adjustment) in a timely manner?
- Can the customer provision necessary machines in a timely manner? If using VMs, is the virtualization team competent with their chosen platform?
- Does this project directly threaten the job of anyone whose blessing / help is required to successfully deliver the project?
- Is your leadership / your customer champion intelligent and willing enough to say "No" to scope creep when the customer gets excited?
- Do the PMs attached to the project have a basic understanding of how software is built and the systems involved?
- Does their management understand the actual processes being interacted with? (Major red flag! If their management / leaders say things in meetings that demonstrate a basic lack of understanding in what their own people do, run)
"Because we don't have the expertise to do software development."
The comments in this thread, which I fully agree with by the way, just cement my belief that if government wants to deliver high-quality IT projects they need to start developing the expertise to do it in-house.
I've also been a consultant cog in a mega-corp and run my own small consulting business, so I've seen it from both sides.
Tech folks underestimate non tech challenges (basically anything outside of their domain), and clients don't actually know what they want. It's an age old problem in lots of industries.
“We need a new system, but it has to function exactly like the old one.”
Including any bugs and craziness.
Perhaps worst of all, when failure has no true cost, avoiding failure matters far less. The CDC won't go broke because they blew through $44 Mil. Most every other organisation would. That's one of the challenges of government, the lack of creative destruction.
They should be diligent in what capabilities they choose to support. And ruthless in continually developing and pivoting these capabilities.
This is true of any large organization or enterprise that operates in a software enabled domain, so yes, most industries. The idea that software development is too difficult and should only be contracted out to fleety consultants is one that needs to be properly put to rest.
And to be clear, I'm not saying that consultants have no place. But for the type of technology and delivery as required by the project in the OP - there's no excuse.
Camunda is not as well known in North America, it will be interesting to see what emerges in the digital transformation/rpa space.
If the hardest / longest part of any project requires domain knowledge and personal relationships, then why are you outsourcing that part?
I don't care if it's not a core competency / something you can do. Make it one.
A high-quality consulting partner works to build the relationships and internal lines of communication that are necessary for the client organization to do more of the work — maybe all of it! — in the future. A cash-grab consultancy will just make themselves the organization's "glue" and bill until the money runs dry.
I think that's the reason you tend to see these unhealthy dynamics in extremely conservative, quasi-monopoly industries (government, financial, healthcare), where there's no real question of if they'll be a going concern. And no real rewards / attempts to outperform at the cost of increased risk.
Only an outsider can safely talk to everyone. They aren't a threat because they aren't part of the company and therefore can't take over.
Also, the end result and the path to get there isn't clear for FTEs – let's say they did the functional domain requirement gathering work – then who will implement it – they will have to contract out the development work – they may not be empowered to think and see through to the end goal fully.
On the other hand, for the top management folks – it is easier to temporarily empower the external consultants and terminate the engagement with them at any time – but they cannot do the same with FTEs due to employment laws etc.
Except I usually just stared at bad movies in hotel rooms to decompress after dealing with clients, rather than blow and strippers. Sue me, I'm boring.
After 8 hours in a windowless conference room with a parade of stakeholders, making RACI matrices on a stained whiteboard and scarfing down cafeteria sandwiches during "bio-break," I want to _talk to my wife on the phone then stare blankly at the ceiling_, not go out for a night on the town.
... did get really good at reading between the lines of Yelp / maps reviews. (Objectively, friends have since commented on it)
Upvote just for this line of poetry. Mondo 2000, haven’t heard that name in decades.
I am definitely biased by my past experiences with tech consultants. I’m trying to sort out if it’s a service provider issue or if it’s a customer side issue.
From my limited perspective, it seems management consultants are focused on 80/20 solution approaches. While it seems technical consultants are focused on delivering MVPs that don’t seem to map well to the business space.
I have no idea if that's the way larger firms do it (would expect not).
Consequently, profit = contract price - consultant hours. Every bad happened from minimizing the latter.
Even if you do straight up billable hours with the client, your time was typically scheduled 1-2 projects in advance. This lead to being prematurely (in terms of things being "done") rolled off a project, onto a new one, and finishing / supporting the first from your hotel after-hours.
This was in a pretty cowboy area. I imagine different parts of technical consulting behave differently.
But yes, MVP quality code seems to be a constant. I saw that delivered from a top-tier technical consulting firm (ML solution) to a T100 customer.
Nearly every time all the knowledge and skills necessary are hidden and fragmented inside our client's org. With our access through all the management layers and silos we could work to fullfil the contract and also enable change within the client organisation and culture.
But why kill the golden goose? Our higher people' bonuses depend on us being the glue as someone else wrote in another comment. And a lot of my colleagues actually on the client's really do a great job in opening lines of communication. They work their a*es of to help the client.
Sadly helping isn't a monetary long term strategy for upper levels.
In the end
And once a year as the government decides its budget you get to chew your nails as you wait to see if the money is in the next budget.
To what extent is this due to neo-liberal "small government" policies, where from the highest political levels - through lobbying - it is decided that outsourcing is a win-win that leads to financial savings and budgets that are cut, and later on cut some more, etcetera?
I've witnessed the crazy dance of enterprise consultants in large corporations (not government) and it is astounding how much money is wasted in inefficiency.
I was in a small tech shop and we had just won an RFP at a big corporation, with a very practical solution and for a limited domain. Our down-to-earth approach led to so much enthusiasm with corporate employees, that they sent all their consultants to us to investigate interfaces to ever more back-end systems. Scope ballooned, and we - being a small party - were dictated to support all this stuff. It was utter madness to such extend that we decided to more-or-less graciously wringle out of the contract. Just-in-time before we were committed to a monstrous project that was 100% certain to fail. And that was not the only time we experienced this phenomenon in big corp.
It's better to have an in-house dev department in these cases even in the complexity side, if you realize that complexity in the contractor side is just as bad as complexity in the basic org.
I notice that they spend oddly large amounts of money on anything for which we know what the cost roughly should be -- e.g. CDC web sites -- and don't seem to do so very quickly or very well. Should we assume that they're a lot more competent at things we know less about?
Unless an organization recognizes the core importance of software and carves out a specialisation stream and C level ownership of it, they rarely achieve more than brief or isolated competence.
This is the reason why SV is so appealing -- nowhere else do SWEs hold sway over entire companies and industries.
Govt can do excellent software if it becomes a priority, they could pay competitive wages (there are special pay scales for doctors and lawyers), but they choose not to because of risk aversion from politicians, who mistakenly believe they can outsource the risk.
For something like this, paying Eventbrite to do it wouldn't have been a bad place to start. But the fat cat consulting firms never have domain expertise beyond getting Govt contracts.
In my experience, a lot of the time it's not so much that the contractor is trying to make money by prolonging the problem. It's that the customer doesn't know how to be a competent product owner that prioritizes and scopes things properly.
Then, frequently the software being designed ends up replicating broken/inefficient business processes, so it ends up being difficult to make cleans and efficient software. This becomes worse when there's legacy software in the mix where the customer wants the contractor to port over all of the clunky functionality of the old app. Which of course isn't documented in the first place, so you've got to perform software archaeology to figure out what it did.
Some other show stoppers I've seen are when key stakeholders act as gatekeepers and prevent developers from talking to SMEs directly, so you have to build your requirements based on what the person with the 10,000 foot view knows. So of course you get incomplete and sometimes completely wrong requirements. And you might not find out how wrong they are until after you've spent a lot of time building things
Just those dynamics alone can add an enormous amount of cost or outright doom a project. I could probably fill a blog with all of the dysfunctional stuff I've seen in federal IT, but suffice it to say that these factors can easily balloon costs and timelines to a comical degree. This $44 million failure is actually small compared to some other government IT failures like the billion dollar Air Force accounting system.
This always comes down to piss-poor and craven senior management decisions.
This is basically true for any organisation.
There are a few issues with authentication, but there is support. Everything that's related to dealing with the UK government btw is all in ONE website, where everything is explained to you in Plain English - gov.uk
Coming from a country where this doesn't exist, I was blown away by every interaction I've had with the system. It's simply amazing.
Their UX documents and implementation are for my money some of the finest in the industry.
I've actually looked into getting a job there even though I'm from a start-up background
One of them is high-tech DYI solution, the other one is super simple yet not used at all by any other govt service I've interacted with.
It's because they have to shoehorn their solution into a morass of garbage software, that this turned into such a complex task.
The real answer is to dump ALL these huge contractors, and their 'solutions', and make something new and better, with proper engineers directing the process.
Imagine if Deloitte had been contracted to 'invent' the Internet...
This is usually not the solution. You have to understand the requirements/know your actual users. Engineers don’t necessarily do this well. They build with the latest tech but as you see in this example, the primary users are older people who can’t use latest tech
As far as I’ve seen there’s people that build first, and ask questions later, and the reverse. Only the latter are engineers.
The business team schedules an hour meeting with the project manager once a week for two years to figure out the requirements.
I spend a sprint implementing the backend in parallel with a frontend person. We spend another sprint testing and fixing bugs.
Then the project sits in UAT for 6 months while the business suggests contradictory things.
Finally the project is shelved indefinitely due to budget issues.
Edit: fixed typo
Sure all the requirement gathering and other costlier project parts can doom the success of the project because you built the wrong thing.
Here however the tech debt and bugs is why the project is abandoned
This is a booking management system for appointments, the initial problem statement is trivial, the difficulties lie in interfacing with other systems, simplifying UI, getting data in and out, providing some customisation but not too much, etc. These are all things you don't know how to do till you start using it.
The process you describe of getting 'domain experts' in a room to decide on a spec without contact with the real users then throwing it over the wall to build is fundamentally broken. You won't get to even 50% done without real users - that's the problem, and it sounds like that's what happened here.
You simply can't build a system like this without frequent contact with real users (old people booking appointments, volunteers entering data in a hurry, pharmacies who need to add extra info for patients), experts on the other hand will often add negative value unless they are trying to use it.
If there was a will within the org commissioning the work to appoint one project lead and then get out of the way apart from demanding weekly builds from week 1 onward this would be tractable. Collect feedback from real users and deliver every week. This is not rocket science and a small team could and should do it if allowed the space to do so.
Wash, rinse repeat and it won't take long for things to run very nicely.
Bonus, code can be published, standards made, all that and bot rented by the year.
At the city of antwerp we flipped over in 2015 from a typical assortment of poorly integrated consultant-built monoliths to a microservices api-first approach. It became a hard requirement that all functionality provided by new systems had to be offered by a documented api, and most of the newly built systems dogfood their own api’s from the ui. In 5 years this caused a fundamental shift, where it gave us the ability to set up a covid contact tracing app for schools in weeks, because it was mostly the piecing together of well documented and battle tested api’s.
One of my previous employers had a very good software and it sold such software to various other companies around the world. Such software was highly complex and really the top money could buy in the field. A yearly seat for a single user could cost up to around 5-7 k$.
Yet for years the company made most of its money from "installation projects" and "professional services", that is working with the client to understand its needs and implement customisation to the software to make it work well in the customer environment.
Then there are challenges around integrating with existing APIs, workflows etc.
Let's not forget feature creep. Sometimes the client might not be able to visualize the final UX in its entirety (often simply because multiple people are involved),and therefore, they might only begin realizing its impact as and when parts of the new system becomes available.
Essentially, you are not paid to build an isolated system, but to understand well and play nicely with the larger context - human and technological - within which your system must live.
The real problem is that requirements are written in such a way that very few (in this particular case just ONE) companies can satisfy, so essentially there's no competition - exactly a kind of situation which lets to pump up prices.
And if you asking yourself are those requirements really benefit users, or specifically created for money to go to, you know, "right hands" - you're probably asking the right question.
Even the part you mentioned: 'wrangling data for legacy systems' does not cost that much, it's still a small piece of the pie.
The 'bulk of it' goes to sales, BD, marketing, profits, layers of excess management, specialist consultants, accountants and lawyers.
This contract is an example of 'modern world graft'.
Healtchare.gov cost $1-2 Billion and it was also a mess until Google Engineers came in to save it.
Software is not like Construction where the costs can at least be well estimated and you don't need geniuses.
It's frankly very depressing and this kind of graft and stupidity is all over the system.
There are actually not that many people who are both: 1) technically/project competent enough to figure it out and 2) have enough gravitas on the business side to make proper bids. Partly that's the fault of government.
To be fair, it's also bad buyers who change requirements all over the place and throw projects in dispensary often for what are ultimately political reasons.
It's a real failure of leadership.
Are you sure? Because I think it might possibly involve:
- interfacing with an old COBOL system
- interfacing with an old DB2/ORACLE/(legacy database) with a "very creative" Schema and even more creative indexes (or maybe none - who knows). More bonus points if it's not even ASCII encoded
- Production system runs on a VM serving an old classic ASP or PHP app
- Data can be exported through an XML from Hell but that takes 2h to export.
Are there governance requirements around the data in them? I'm not familiar with US law, but we're talking about PII in this example so what legal and privacy hoops need jumping to copy the data?
Thats before we get to "Can an person with this specific learning disability, or a person who doesn't have a fixed address navigate through this form?"
Construction is notorious for cost overruns?
And when it goes way over, it's usually easy to identify the problems, it's just often politically infeasible to address them.
With construction - the numbers are on the table.
And FYI - aside from massive project, municipalities and construction is happening all the time. There usually are not massive overruns for such projects.
Usually, though not always, this helps to give a road map for the organization to follow.
Bugs will arise of course, but usually, it becomes localized, and can be addressed individually.
But for large systems, with lots of moving parts and data integrations with other systems, then you normally need a good, near-air-tight specifications document.
A recent project was valued at around $10M USD, but for that the customer got a fully-featured mobile app that their entire blue-collar workforce will use every single day, replacing gigantic paper work packs.
The backend was hybrid-cloud, interfacing with multiple on-prem systems, including a legacy ERP system which had started life in the early 80's.
So, it was a big and complex project - but it was also a successful one, because we know how to run projects like this. There were various factors that led to success. The BAs were probably most integral. We also held regular week-long workshops with the customer to work on the backlog, with the architect/tech lead (me) and dev team leads all present too. There was a group of pilot users, that always had access to the latest features. And of course, we used automated tests.
Any big corp, like Deloitte, should be able to deliver a relatively simple website, even if it has multiple integrations with legacy systems. And they should be able to advise and steer the customer if they are woolly on what they actually need - there is only so much blame you can put at the customer's door when systems like this are a disaster.
I get your point, but also yes it is. The issue is superficial, it's like when I try to Zoom with my mom, it'll take 10 times longer as I need to explain and bring her along every single step of the way. This is the same for these type of contracts. A lot of the slowdowns come from people just not letting development take its course as it should.
Just look at the article, the states ended up choosing to use a bunch of SaaS startup not even designed for this use case in its place, like eventbrite.
And look at the issues:
> “It won’t work on Internet Explorer; it only works in Chrome. The ‘Next’ button is all the way down and to the right, so if you’re on a cell phone, you literally can’t see it,” says Rowe.
That's not a project management issue, that's a full on development issue, it doesn't even work on mobile in 2021 !!
The user that needs help finding a button isn't worth the development cost. Gov services cannot operate that way.
I don’t get why people don’t just use Whereby, no login no plugin. It makes conferencing so easier.
For example, for each day/week/month you deliver past the estimated delivery date they discount 5% from your invoice. Sound like great motivation to deliver on time, right? Well, no, it's a guaranteed 100% (or whatever you can get away with) markup beyond what you would have charged because if anything goes wrong you'll go broke.
That said, this issue of super-expensive government websites done wrong feels all too common. You have to wonder how much cash we burn repeating the same mistakes.
The only answer I can come up with, even after trying to be generous, is corruption. The fact that there was no bidding should be enough for that contract to be highly suspicious. People in the top are pocketing most of the 40M and everyone involved knows it.
A commercial service is optimised for profit, not broadest accessibility. For example Eventbrite won't spend money developing special features for very specific niche user cohorts because of diminishing returns.
Government websites can't work that way, they have to be accessible and usable by all citizens. Think of people with different learning abilities or language proficiency, the likely environments the users might be in, and their general technical abilities.
Here in the UK there are strict rules about use of clear language, well thought out user flows, how to ask for certain types of information etc that has all been backed by years of research. We have a whole government department that covers this and will audit your new service to make sure it is good enough.
Having done this as a web developer for our Government, it was a massive eye opener to have a copywriter pick through the language I used in a form flow, and point out how confusing it could be to some groups of people. Or indicate that the method by which I was asking for an address made no sense for certain remote locations, and it needed to.
A few specific things to point out on Eventbrite in particular. It seems to me that you are likely have a whole bunch of other requirements too. Off the top of my head you'd need more powerful management tools to control: simply setting up so many locations and dates, who can book when (patient cohorts, priority groups), the logistics of vaccines and vaccinators, tracking vaccine batches used for each person.
You need to be big enough to apply (requires investment), apply to many government contracts, you need to understand the legal terms, price them, understand the regulations, price them in, then price the software development itself and price any other provider (hardware happens very quickly), and if you guess wrong, like on 1/3rd of the projects, you’re in debt... and you have 100% chance of getting the contract if you price it wrong.
Getting the contract done is another 1/3 of the contract, and the lawyers to deal with disputes and prove that you’ve done the contract, is another 1/3.
It’s probably exactly that. I always strongly wonder if you wouldn’t get a better result if you gave a pair of 22 year olds 1M and a year to develop and figure out all the same things.
You just have to find the correct pair.
Doesn't actually matter how old or young they are, just that they don't actually need more experience than you could get from two top-performing 22-year-olds that could easily make more than a million in progress over a year.
And it would have even been worth it to spend the first year doing nothing but intensively searching for these two so you could have operators so far above average and so well suited to the task, and so well agreed upon over a period of a whole year, that nothing else could even come close. It would probably be best with a pair who are equally motivated whether they would be getting the half million apiece or not, because a lot of that could be spent by now.
When everyone creating the software has the rare extreme performance ability that very few companies ever get to have even one employee that good, the more suitable the performance the fewer operators it should be expected to require.
This just has to be put into perspective compared to what you can get from a typical bureaucratic team for $44 million.
Average bureaucrats have never been good enough for average things.
a) The article talked about the software not working on Internet Explorer (IE). Older people still use IE. Software Engs with fewer years of experience tend to optimize for FF, Chrome and latest version of IE.
b) The article talked about 'wrong' placement of buttons (probably the software is not responsive or not optimized for phones or older phones)
c) Input fields seem to be rigid (older people will make mistakes)
d) Once you're registered, you can't switch from patient to staff. This should have been designed like a 2-sided marketplace with the ability of one person to sometimes play the 2 roles
If you don't have the right spec/requirements, the output of the engineers will still be bad
That said, building something that works well on mobile (re your point b) but also works on internet explorer is a pretty tough ask. No reason you couldn’t build a basic bitches version of the site that works on old devices, then route traffic these based on user agent.
Idk, points a through c are basically already solved problems. I’d be disappointed to see problems like that if I was spending $20,000 let alone millions. Issue d though does feel like the sort of thing that you didn’t know was on the horizon up front, would be a pretty major rebuild to add in later on.
That thinking is exactly the reason why this system turned out to be problematic. Even if it wasn't for healthcare purposes, bottom line is - you have to know who your 'actual users' are. Elderly people (60s and above) are still heavy IE users and so are quite a few government offices. If these 2% of overall browser users are the majority of your users or your early users, then not supporting them becomes a problem.
Probably cheaper overall, and certainly a better user experience for the tech-challenged seniors.
In short, anything below IE 11 is de-facto out of the picture.
Why isn’t this just the site?
I love Spartan, ugly, functional sites. That’s what the government should build, always. They should have style guidelines that are basically “no style”.
You’re not competing with anyone, there’s no need to look good. Function, function, function.
The article talks about the fact that a lot of the seniors get help with their online stuff from the libraries (which are currently closed). What is the probability that folks like these have multiple devices at home and/or they can use them?
Years ago, a lot of my bosses at work (in Tech) still used IE. Lots of our clients used government issued computers which were locked down to IE, sometimes several versions behind.
At the end of the day, it boils down to - know who the actual users are.
This article got posted here recently: https://shkspr.mobi/blog/2021/01/the-unreasonable-effectiven... even outside of a pandemic people need access to gov sites from whatever device they can get their hands on
> But the GOV.UK pages are written in simple HTML. They are designed to be lightweight and will work even on rubbish browsers. They have to. This is for everyone.
This is true but if you are a Government agency in a Country of 300M+ people, that 2% is still several hundred thousands users, who are calling your tech support etc. etc.
If this was a private app then it would be different, IMHO Government websites have accessibility concerns that the rest of us can ignore.
If you let qualified engineers drive those instead, in my experience, the end result is always better.
It's possible there's context we don't have that exponentially increases the complexity of their tech requirements and justifies the price here but I'm skeptical. Looks more like misaligned economic incentives commonly at play in engagements between consulting firms and the gov.
https://www.google.com/search?client=firefox-b-d&q=canada+fe...
example news URL: https://www.itworldcanada.com/article/phoenix-payroll-system...
Trust me, this money did not go to waste. Someone got paid, someone has a new yacht.
https://vams.cdc.gov/vaccineportal/s/login/
It didn't load earlier (infinite redirect), right now its a Salesforce error page, of course. Salesforce and Deloitte, a match made in heaven.
Whilst I do realise some of these quoted costs involve more than just the development of the app itself, some of the numbers are utterly ludicrous IMO as a software developer. e.g. Germany's app apparently cost approx €20 million.
https://www.dw.com/en/germany-launches-best-coronavirus-trac...
Despite the fact that I was working on critical code implementing core requirements of the system, I was only paid $20/hr. I believe the firm was charged $40-60.
Long before I started working in the product and before any working software was deployed, the majority of the budget was spent. I believe they had at least three civil engineers, and a hands-off software architect who I only saw once, developing a document with hundreds of pages of studies and requirements.
I remember one severe issue we had was that I was not allowed to test on the real database or one of a similar size. So a main query was tested the day it was deployed (on some extremely expensive Oracle server which was very outdated). And unfortunately my query did not perform adequately on the larger dataset. But the senior programmer decided not to tell me or to try to fix it. He just truncated my query with LIMIT and said nothing. This resulted in the query breaking and showing an error, and totally incomplete and invalid data. It is amazing it actually continued at all from that point but unfortunately it did. I only found out about it several weeks later when the project manager decided to take me on a visit. Strangely, no one ever mentioned the issue, nor addressed my immediate suggestion that the software could not operate without the problem being fixed. I assume the other programmer just said "well that guy is inexperienced" and they had other projects to work on and no budget left so they just left it for quite awhile. Anyway I could not believe that so I left shortly after.
It’s Deloitte, they probably hired 100 developers for 80k and 50 project managers for 300k
In short, the above comment is spot on. Companies like Deloitte, Cap Gemini, PwC, etc. all make their margins purely on their reputation. Just like back in the 80s when the phrase "nobody ever got fired for buying IBM" became popular, I think from the mid-90s all the way through to even 2010s there has been a similar sentiment with large, multinational consulting companies.
These consulting companies know they have a strong reputation for large scale, enterprise software projects, and so they will milk margins by charging high rates, upselling all sorts of additional features, and fulfilling projects with the lowest cost resources, whether it'd be outsourcing overseas, utilizing H1-B workers, or bringing in recent college grads. (Accenture is probably one of the most notorious for this -- not sure if they still do this now, but back in the early 2000s, they were known for bringing in college grads from all sorts of usually non-STEM majors, and they would put them through a 2-week bootcamp (er "Core Analyst Training") out in St. Charles, where more focus was on the partying and hooking up than anything else, but then shortly afterword, they'd be put on projects where they'd be expected to do technical design and even coding, while being billed out at $180+/hour)
Now, in all fairness, for every highly publicized, disastrous rollout that one of these companies gets notoriety for, there are actually still dozens of much less public, relatively successful implementations that gets deployed without any issue or fanfare.
But despite this, the costs for even the successful projects are still ridiculously high. As the main reason for their huge margins are primarily in their reputations, then as their reputations continue to erode from things like this news piece, clients will start to wonder if the large costs can continue to be justified.
And I'm beginning to see this sentiment directly -- my small dozen-person consulting company has not only been able to compete directly against these larger companies, but we have started winning contracts against them as well. For the latest contract we won, we bid at about 20% of what one of the larger companies ended up coming in at -- they actually thought they were a shoe-in because they were the incumbent, having built out several other projects already for the client. But we ended up being selected not just because of our pricing, but because of our overall proposal, favorable terms and expertise on the domain.
What's even more surprising is that even though we offered "bargain basement" pricing relative to theirs, it will still end up being the most profitable project we have had to-date -- it just makes it crazy to think what kind of profits/margins that they can make.
I've seen projects where bad requirements led to cost overruns and/or delays due to required rework.
I've seen projects where due to bad/strict requirements the customer's product basically became the 'learning platform' for a team to use 'the hot new web ui framework toolchain' of the year. I made sure to completely forget what I 'learned' about react/redux after leaving that gig.
I've also seen projects where the contract house threw bodies on that didn't even know the language used. I've seen horror story houses that are code-genies that overcharge to give you exactly what you didn't want.
But in this case My guess it would be a combination of requirements fustercluckery alongside what sounds like an integration nightmare; from my time working in an environment where we had to do a lot of 'etl loads' from different partners and/or acquisitions, you see a lot of snowflake cases.
Maybe they did and the too many chefs to cooks ratio played out, seen and experienced that.
The bigger issue is that it’s a no bid contract. However there’s no real process for verifying individuals who bid in a timely way.
The worst part is that we know government software contracts are done completely wrong but the government doesn't want to change. Too much money to be made.
You can, but the government has to jump through a lot of hoops. Even government contractors have to jump through a lot of hoops. Sometimes there are regulations affecting the hiring process even for government contractors, e.g. drug tests, citizenship requirements, background checks, location requirements, etc. Every relevant law named after a sympathetic child adds $5B to development costs. And so forth.
This is projected to cost 95 million dollars over 12 years.
Yep.
Organisations are like people. Their culture is like someone's personality. Dealing with easygoing people is much smoother than paranoid delusional people with OCD. In the same way, dealing with government agencies is difficult because their corporate culture -- their personality -- is not what most people would categorise as normal.
These deals aren't normal deals. They're the equivalent of specialist treatment for someone with mental problems.
Take for example an organisation like SpaceX: They simply do things. If some manager goes to Elon with a rational but inherently risky proposal, he'll weigh the risk and then more than likely give the goahead. Other companies have a lower risk appetite, but they will take occasional risks. Sometimes huge ones.
Large federal government organisations are the diametric opposite of this kind of agile, low-overhead, no-bullshit private company. Their appetite for risk is precisely zero. That's difficult for people to wrap their head around. It's like a superconductor. It's not just a "really good conductor". It's not a "little better than copper". No, it's something fundamentally different. This kind of zero tolerance squeezes out entire lines of thought, just like a superconductor expels a magnetic field.
Notice how in this article the blame is on Deloitte and the "problem" is that there was a failure. This isn't a fault, this is the system working precisely as designed. Nobody will get fired for picking Deloitte, their purpose all along was to take the blame for the low-low price of a mere $44M so that a government employee making $100K won't have to have a couple of uncomfortable meetings.
The problem isn't that there was a failure, that happens all the time with software projects. The problem is that risk in government agencies is managed by people with essentially infinite source of free money that they can draw down on in order to protect themselves.
If a magic genie gave you a million dollars every week to spend on your own safety with no strings attached, wouldn't you? Would you say "no"? Really?
Seeing orgs as people with characters, oh my, my eyes are opened now. Sounds weird but it makes sense the few times that I have worked on contracted projects.
For me, an added complexity in gauging the other side of the contract is that their personality changes depending on the actual people that are involved. When some higher-ups join the project it in later stages, the personality may turns even more mental than it already is.
"The vaccine management solution is a mobile-ready application that is built on our existing GovConnect Salesforce platform."
https://www2.deloitte.com/us/en/pages/public-sector/solution...
Either way, the deal should be possible.
A decade of that, and a whole lot of government would be awesome.
Of course, these arguments have been made many times. Money in politics prevents it.
I have presented them. I have watched large donations and a couple nice dinners make the whole idea go away.
:(
I fired them and refused to pay their bill (they hadn't even produced a plan for how they were going to proceed). Hired another outfit, who did the entire job front to back for $5000, and got it done in a couple weeks.
It reminded me of a university style group project, constantly astonished that they’d turn over a few 10's of thousands of lines of python code that (usually) some government department was paying millions for. Not to smear my friends who worked there who must have to work incredibly hard given the conditions and the stress they were under, I always suspected their code must have been absolutely riddled with bugs, and I shuddered to think what it must have been like for the next poor grads who had to work on it after them.
And it was always some project where I was amazed there wasn't already some mature software out there that accomplished what they were doing anyway.
But anyway they seemed to have fun so whatever
I understand the Deloitte prefers heavily documented and waterfall development with overseas developers doing the work. Basically top heavy and expensive management and cheap dev resources.
In order to qualify to even bid on a contract there is a pretty high entry barrier. The amount of time it will take you to cross that barrier would be more than amount of time it will take you to start a real business. Bidding is another expensive process which costs money. If you bid on 10 contracts in a year and win one the cost of bidding on rest 9 contracts must be covered to the won contract.
Then comes the compliance costs of various government regulations. In a startup world you can take up accessibility as a P2 item but in government contracts it is a P0. So there are all these complexities involved which bloat the contract. Government has passed probably thousand regulations on whom you can hire and how you work, where the data is stored and what kind of background verification you have done on your employees.
Of course if you and I were to develop a similar website we could do it for $4M and spare some cash to buy couple of Ferraris.
PS. There is also bribes. Of course these are never called so but you have to spend good amount of time golfing with right people, throwing parties, offering front row seats to NBA finals and those sort of things to get your files moving.
We also ended up being in contact with Serco about a tender they had to redo one of the goverment websites. We ended up not getting any work from that because Serco was under pressure to stop outsourcing all their contracts.
Goverment contracts just send lots of money to companies who already have lots of money.
How was the code obfuscated?
That brings back memories of php obfuscator. What a strange world that was. I think there was even a deobfuscator that tried to rename the variables sensibly, or something.
At Scottrade, I was horrified to find a 5k or 10k line Date C++ class. They'd copy paste functions to handle each leap year individually. The function names themselves contained the specific year.
They also yelled at me for downloading the source code to `touch`.
It was nice to get to see the crazy side of software. Everyone was deadpan serious about it too.
That would result in a bug report. The bug report was that Joe Blow saw that outcome. In other words, the bug report didn't contain anything about the code itself.
They had an entire division of support people whose job was to track these bugs, and to fix them.
In order to fix the bug, they would manually edit the database, or do whatever was necessary to make sure Joe Blow saw the right value again. But they never changed the code; they weren't programmers.
Once Joe Blow was happy, they closed the bug report as "fixed."
Therefore, no tests were ever required.
It was ... impressive? I think? I couldn't mentally process what I was seeing at the time. But "impressive" is probably the right word. After all, the system worked.
But ... that logic doesn't work if you chase down the implications. And sadly I was both too shocked and too young to press my coworker for details. (He was a cool older fellow who seemed as amused with the craziness.)
Eventually I became a pentester at Matasano. During my one-year stint, I was parachuted into around 70 codebases. I got to see first-hand that Scottrade wasn't an outlier; they were the average. Most companies have similar WTFs, and the codebases are just as onerous.
The world is held together with duct-tape. That's why pentesting is so crucial.
> thinking about the customer, understanding the right technologies, designing for scale, etc.
If they were asked to design something for a customer then they're fulfilling what is asked - e.g., the bare minimum.
And this is how we get the 737 Max.
The point is the ask for the 737 Max wasn't a "plane that doesn't work." It was for a plane that does work. That was the deliverable. It wasn't met. That is not doing the bare minimum.
You could also apply this weaselly BS to Boeing. Their "ask", if there was one, was to make a plane that customers would buy and that would pass FAA review. Not "a plane that works".
No one's contract just says "make an X that works", even though that's what everyone wants. Because an X that works is hard to define. Fundamentally the system depends on honor and trust. Which is why we despise losers that just try to get paid and fuck the customer.
This isn't rocket science. Think.
If the CDC signed off on it, it isn't deloitte's fault. Sorry.
All you SV elitists do it, you just want to hoard the wealth for yourselves. It's not just deloitte, all the SVs do the same damn thing.
A good internal developer will notice that requirements are missing, or could be improved, and has the incentive to bring it up with the project manager and get it fixed.
That incentive does not necessarily exist for a consultancy developer, because as you say that means their employer loses money.
My conclusion: Use consultans for very well defined and testable problems that don’t require too much business-specific domain knowledge.
The car shop spends months of your time obsessing about the dimensions of the spoiler. Then the car is out of service for as much time and you start wondering what they have been doing.
Then the car arrives and there is a bunch of paint missing around the spoiler, the spoiler is completely unpainted an it falls off when you accelerate to 100mph.
So you ask, "WTF?" and they tell you that none of these things were specified in the contract.
That's more or less how it worked.
I have been contracted for a number of projects and I always assume some level of standard that my client would want me to meet.
For example, around 2003 I have been contracted by a publishing house to restore couple of their servers. Their employees made a mess they were not able to get out of, completely mangled Linux distributions, took out drives out of RAID5 and did not know what to do with them, etc. Drives had files with only copies of texts of books they had so it was extremely valuable to them.
So obviously I did what ANY sane person would do:
- I have discussed with responsible employees what exactly happened,
- We went out on a little trip to buy backup drives
- We made backups of every single drive involved in the mess (I keep saying we because I involved the guys in the recovery so they know how it works)
- We put together the drives and they worked
- We fixed OS-es which they mangled
- We tested that everything works
- We have discussed why they need some procedures and especially backup procedures
- I have showed them how to construct couple of these procedures so that they know how to do that.
I didn't have to do all that. I was already being paid handsomely for the recovery and would not suffer (other than lost pay) if I failed recovery. But I also recognize how important this data was for the company and so I took necessary precautions proportional to how important this.
If spoiler work fails instantly you can say it wasn't done properly, which is true. You then withhold payment, or sue.
Not that complicated. Not worth a damn novel post from you either.
https://federalnewsnetwork.com/pay/2020/01/more-top-career-e...
>But for a GS-15, step 10 in Washington, his or her salary should total roughly $185,509 in 2020. But due to the federal pay ceiling, that same GS-15, step 10 in Washington will make the limit of $170,800 this year.
And that's the absolute Max, after you go to the trouble of getting a security clearance, which you just don't need in the private sector.
Even government contracting involves so much red tape, more innovative engineers would probably just rather not do it.
They don't need industry-leading researchers, they need people who can build a maintainable business systems.
It's infinitely easier to make on 170k happen in the private sector, then as a federal employee. Unless the government wants to create like a special ops tech department which doesn't use the government pay grades, more money can be made in the private sector.
I'm only mid-career or so and I'm already at around 200. But I don't, at least on paper have the qualifications to architect out a whole system. And while we all want to act like we could whip up this entire thing with VueJs in a weekend, pretty sure there's some God forsaken legacy system it needs to hook into.
Since most engineers don't want to work with legacy systems for less than market pay, the engineers who end up working on this stuff tend to not be all that great.
That seems like a much better solution rather than rolling your own.
For example, I'd rate access to Blue Cross Blue Shield Federal Employee Program health insurance for you + spouse for life and kids through 26 as worth 2-3k per month, and invaluable if you or a dependent has a serious health condition.
Can you imagine how much better it would feel to work on important and needed systems than working on the 100th food delivery app or yet another social media facism incubator?
I work for Ad Hoc (one of these companies) on software projects at the VA, and I'm happy to chat with anyone who's interested to hear more about this kind of work. I left a cushy Google job to work on important and needed systems, and it does feel so much better!
One weird thing we've discovered working on VA.gov for the last five years is that we on the contractor side actually have retained a lot more institutional knowledge than the VA has. It's a problem! We think that knowledge should be on the government side, but the structure isn't there yet: USDS and 18F have rotational term limits that keep people moving through, and at the VA (not sure about other agencies) they're just in the last couple years building out an organization to do that long-term product ownership and institutional knowledge retention, even if implementation teams come through. It's moving in the right direction but it's slow, large-organization change, with a lot of extra slowness that's unique to government.
The top 2-5 layers of management tend to be the types who want solutions, and they don't care how it works, and they don't really want to be bothered with the details.
To the extent they expend thought at all on systems, they tend to be focused primarily on questions like, "why is all this stuff so confusing?" "who do I blame when this thing goes wrong?"
Where I work, they go through cycles; a higher-up will say "Hey let's build a staff of in-house developers." And they'll do that for 2 or 3 years, until someone reads an article in CIO magazine about how outsourcing is superior. Then they'll start unfairly purging and firing developers, writing code is now "bad", configuration is "Good." This goes on for several years, until the cycle starts over. So as a developer, after you survive your first purge, you begin to see that invisibility is the only way to survive.
It is difficult to explain but people tend to use the word "bureaucracy" as a catch-all for several dozen discrete problems that crop up in different combinations depending on what level of govt you're operating.
Typical examples might be :
-a developer not having the authority to work on a system
-a developer being ordered to work on a system, but not being given permissions to the necessary tools (even if they're available), or they may prohibit certain common-sense tools, or they may force you to use certain tools.
-a developer being ordered to work on a system where the design is severely irrational, maybe impossible, and but he or she has no authority within the hierarchy to push back on requirements
-a developer being ordered to work on a system that some powerful person wants to see fail
Imagine the "client from hell" that many of us have dealt with before, who doesn't know what they want. Now imagine that person is like 3 levels higher than you, calling the shots, and you're not even allowed to speak to them, much less push back against their crazy expectations.
TBH, I'm surprised we've never seen a proposal of the form "Federal employee salaries are not subject to income tax."
With a single line of legislation, that makes all federal job offers immediately 10-35% more competitive, without actually increasing salaries directly.
Once you get that as precedent, I could see it being used aggressively as a social tool: "We can't pay a public school teacher more than 60k, but we can pay them 60k tax-free, which is a little more competitive with a 75k private-sector position", for example.
Approximately 50% of American politicians are anti-government as an ideology, and use the argument of "cutting overpaid government salaries" as a plank in their economic policy, so having well-paid gov't staff is just "waste" in their minds.
As far as teachers go, they are not employed by the federal government. In most states they are county-level employees and the counties obviously do not have authority to dictate federal income tax levels.
It is already outrageous that government employees are exempt from social security taxes.
>In 2018, one-quarter of state and local government employees—approximately 6.5 million workers—were not covered by Social Security on their current job
I find this offensive because one of the main functions of SS is to subsidize the retirement of low earners at the expense of higher earners. As a private citizen, I don't get to opt out of this social obligation just because my 401k gets a better return.
Not only that, but as a tax payer, I will be on the hook for bailing out the public employee pension funds when they go belly up.
I came from a country where government tries to do all these things in house. It was bad. Paying premium isn't a big problem.
I work at one of the contractors in the space (Ad Hoc [1]) and get to work with people from USDS every day. Like others have commented here, the government salary ranges are a problem for them to hire enough. Our last COO had been an administrator at OPM, and would say things like "on the government side, I couldn't ever hire the kinds of software developers that we're able to here on the contractor side."
[1] https://adhoc.team/ -- we're hiring, full-time remote!
How much does it cost to get a qualified person to put up with all that though?
Now, you don't always have a choice, the expertise is sometimes there, or you need one of the big four to do a prior audit before a real audit from another big four. But be prepared to have a team of OK people who will spend a lot of time producing slides and organize unproductive meetings with lots of other people (including yours) to give out a false sense of "we're making progress here". You'll have to do the PM'ing and be extremely strict with what you want from them, and you'll have to be ready to collaborate heavily. What I mean by this is: don't wait for them to ask questions, you'll have to do most of the work of onboarding them, almost like they're your coworker except that they don't have connections with the folks they need to work with, they don't have access to internal stuff, they don't know how to get there.
There, they put you through a six week course in which you become a developer, and you’re then hired out for £800 a day as a technical expert.
Source: literally half the people I went to university with.
Fond memories of sitting at the kitchen table and trying to coach my flat mate through unfucking the air ground traffic control system that they were writing for BAA in VB5, in 2005. No idea if it ever went into production.
"There was no requirement that the whole thing should work. We can put in a change request and bill separately".
Also - the "no one has ever been fired for hiring IBM" syndrome among the higher management.
I hope we see more govts adopting bake-offs for buying IT.
I recall reading a 300+ page PDF RFP for a DMV information management system. The state wanted to convert their legacy VB5(!) system to a modern solution and preferred a COTS solution. I am fairly certain there are no COTS software solutions for a statewide DMV that also meets the legislative requirements of the state.
It's completely believable that Deloitte might have built a handful of boondoggles like that and dressed it up as a "Salesforce" project.
- Many fed agencies have little tech savvy or ability to evaluate solutions
- Feds can have complex, mandated processes for software procurement, development and deployment that must be followed. This drives up the cost and limits the companies that choose to bid.
- There’s not much of a “get sh*t done” attitude. There’s little consequence for the individuals that managed a failure on both sides. Usually “oh well, another day at the office!” is the norm for many federal agencies/contractors.
- buzzword chasing tech ignorant administrators that consulting firms readily take advantage of - “yes! But does it have machine learning!?”
- there’s many fiefs at these agencies that must be navigated: “Oh! Wow - you spent 6 months building and it’s time to deploy? The security team really hates CYZ tech and we’ll have to escalate this to the director”
- so many g-d stakeholder to please and woo: from managers to administrators to political appointees and congresspeople. The relationship cost can be staggering.
These stereotypes aren’t always the case - some govt groups can actually execute really well. But many of these points dominate sadly.
This right here! The "cyber" folks always throw a wrench in the works at the last moment. They rule by FUD and also have literally ZERO idea how stuff works. There's an old saying about teaching, but these days, and especially in government contracting, it applies to "cybersecurity."
"Those who can, do. Those who can't go into security."
Surely their track record should see them permanently disqualified?
Citation needed, see Oracle, many companies have grown that large on the backs of taxpayers by employing the best "Sales" techniques in the biz.
On the other side, if you get $10,000 of value from a banana, both vendors are viable options. If there is any reason at all preventing you from adopting the $1 banana, you're still better off buying the $1,000 banana than you are scorning it for its price.
But they can all be a bloated, over budget, late mess
Now what I don't understand, is why wasn't the contract written so the state could refuse broken software, or to get a partial refund if it's doesn't meet the specifications.
Someone got wined and dined pretty well. And government needs better lawyers when dealing with private contractors.
The bureaucracy is expanding to meet the needs of the expanding bureaucracy.
Unfortunately this corrupt behavior is deeply rooted and condoned and expected and there doesn't seem to be much motivation to come up with a solution.
Wonder why the government doesn't just staff up a tech wing themselves instead of outsourcing. There is so much to do and its much more efficient to just create an tech department that serves all branches of government.
Maybe, maybe not.
Though rare, there _are_ private success stories, such as healthcare.gov 2.0 [1] and Palantir, though the latter gets in hot water for some tangential reasons. But maybe figure out how these happened, and scale that.
[1] https://www.theatlantic.com/technology/archive/2015/07/the-s...
- The government often grotesquely overpays for mediocre product delivered by vendors without holding them accountable in a meaningful way
- The government process drives up costs to an insane level independent of implementor good faith efforts and strongly selects for companies good at the process rather than execution
These are two different problems, and both of them manifest with any large contract. I would make the observation that it is easier to hide mediocre product and execution when you can hide behind a grotesquely inefficient and mediocre government process.
What transpired here should never happen. But I know why and how it happened.
A harmless example is movie / tv show plots not making sense - there's nobody willing to call out bullshit like obvious plot holes. This is how GoT went from a masterpiece of scheming characters driven by their motivation, making decisions based on information they had and dealing with the consequences to "she kinda forgot".
A dangerous example is the government being full of people who are too incompetent, stupid or useless to get a real job. Sometimes the number of them in the system is so large that competent people can't get anything done to fix things.
We need to do something to fix this. Positions of decision making power should require an IQ and/or knowledge test.
Democracy has, as a general idea, been a way to solve this problem by creating a counterbalance that should prevent important decisionmakers from being overly selfish and at least show they are being effective and altruistic. But that system isnt really applicable for smale scale power dynamics. Unless efficiency, or in some cases having just common sense, is not monitored by anyone aka. there are no consequences for making poor decisions sooner or later no one will pay attention and they will be doing whatever they _think_ is best (or easiest) but without reality checks.
But how to evaluate performance when dealing with abstract things? At least in the case of GoT the ratings did deteriorate so one could point out a problem in the system (albeit way too late, IMO s6 was when it should have been fixed). In other cases no one can see the damage they are doing (especially if they have never been really doing anything) so how to even tell it needs to be fixed? I think best cure to this problem is a great culture where being efficient is praised and rewarded. But changing already terrible culture, well, that's tough...
(Interesting film on this matter is Ikiru by Kurosawa)
Add government and a big consulting firm like Deloitte in health IT, and you'll be ready for a world of pain. Here [0] is a good paper on the topic.
Basically, no, it's not just a meme. American managerial culture (or even Anglosphere at large, as alleged there) really does suck. And no amount of crafty mechanism design gets around the fact that least exploitable system seems to be one where the civil service isn't maimed and the work is done in-house.
Basically government should fund some free software for various general purposes (or just do a decent UBI), and then any "consulting" should basically be public sector and free software sector exchanging best practices.
The higher end academic consulting (ARPA) is legit[1] and should still exist, but it shouldn't be defense specific, and like drug bounties basically should exist to smooth over the "research...development...product" gap --- development is severely sort-changed current since outside of DARPA and ARPA-E so much research that doesn't contribute to someone's rent-seeking agenda just lies wasted.
[1]: I suppose should say I once was an intern at https://galois.com/
Has there been any research into what needs to happen to align government administrators incentives with the population? So much waste - why isn’t the government forcing Deloitte to fix?
I have no familiarity with government contracts, but as far as I know with Big Tech contracts they’re definitely written to favor the Big Tech company in these types of scenarios. The government should take a page from that playbook.
I worked with the architect tirelessly for almost 3 years, trying to implement various changes but at its core the new system was highly flawed and suffered from various team changes and reboots. The UI/UX “expert” was not a developer and had very little technical background, and was therefore unfamiliar with basic HTML and CSS concepts. For some reason they let end-users dictate style and behavior aspects of the application. For example, the legacy system consisted of green-screen terminals and all the associated keystrokes (think ancient POS systems you still see in some retail stores). The vocal minority of these end-users demanded that we replicate this interface exactly instead of building it from the ground up as a modern best-practices UI using the latest web idioms and paradigms.
It gets worse. At some point someone important said Javascript is evil, and we were to avoid using it as much as possible, despite the application using several embedded and third-party Javascript libraries.
I ended up leaving that project for a slightly less insane one, but the pay was good. Anyway, I heard about a year later that the entire project (a 10-year effort that cost tens of millions of dollars) was abandoned. They continued using their greenscreens but I heard they hired a company to build an interface that reads these screens and presents a modern HTML facade over there screens that operates them kinda like a proxy.
You can’t make this stuff up.
I remember watching my friend who was a manager at a Domino's back in the day place an order for a pizza with their old POS system that was driven by key strokes, and it was ridiculously fast and easy. A pizza was a list of characters that described it.
Fast forward a decade, and I was tapping away at a touch-screen POS system for a smaller pizza chain, and it was woefully slow in comparison for even the most basic pizzas, and more error prone because Anchovies and Artichokes were right beside each other.
If you ever have a user-base where time=money, they will want hotkeys for everything.
> “You think about the industries that have been transformed by technology—someone said, How do we get a pizza to your house faster? That’s a competitive advantage,” he says. “That has not happened in American health care.”
The pizza industry is not a exactly standardized industry with public infrastructure either... the failure of VAMS (and healthcare.gov before it, and the unemployment systems, etc) shows that single government solution is not a fix for broken procurement and project management.
Then, whenever you hear another multi-million dollar catastrophe, add it to the list. The brand damage will accrue - as it should - over time.
A specific example that comes to mind was a project a few years back where Deloitte stuck 4 junior consultants along with 5 senior consultants and spent a couple million dollars for 12 months and were unable to produce even one business scenario in the system of choice. They did however produce reams of documents showing how it should work. The software vendor was called back onsite and I happened to be in the right place at the right time. A team of 3 of us ended up doing full implementation in 5 months. That included about 20 scenarios from front to back office along with foreign business.
Very little technical knowledge overall. It was a disaster.
I'm currently also at a federal research institution thats millions of dollars over budget and years behind on a software project, and there is NO PRESSURE on the management because federal grant money printer go brrrr.
If time is money, this is peoples lives. Who holds these people accountable?
https://www.washingtonpost.com/news/the-switch/wp/2014/08/11...
Various discussions over the years:
IDK what the structural solution to this is, but it is horrible that Deloitte will walk away from this unscathed. In fact, there doesn't seem to be any financial incentive for them to deliver usable software.
Imagine screwing over a client that does this much business with you.
It’s insane how far those unfamiliar with software will take the mythical man month. Nonono throwing money and people at your problem is absolutely not guaranteed to solve it better.
My offer is of course a joke, well unless they want to talk specifics and I agree it’s reasonable to take on myself. Or just plain reasonable.
I don't know if it's ever been conclusively proven, but it's been alleged for decades that the major defense contractors deliberately structure their big projects into subcontracts that can employ people in as many Congressional districts as possible. There are allegedly people in 300+ districts working on the F-35 program, for example.
Sometimes they hire junior programmers who would have the latest stuff in their resumes but the kids are into copying and pasting stuff from internet not knowing exactly what the code does and they don't have enough experience producing clean and bug free code themselves. Other times the top/down design produces code that lacks elegance and resembles what truck drivers would produce with too many people working on it and no one responsible for the whole thing.
Have seen that before. Not sure about this case.
Wonder if anyone will pay me $44M to set up a redirect loop?
I expected a dump, but to be honest, SORMAS looks pretty good. It's already in use by Nigeria and Ghana, and it's on Github. It's Java EE with Postgres on Ubuntu, so safe and sane tech choices. They even have a play version: https://sormas.helmholtz-hzi.de/sormas-ui/login that also looks ok. I'm thoroughly impressed.
This case it was a government client, and one could argue there is no singular point of authority/ responsibility that may have caught on to the troubles early on. However I have also it happen in private companies where 350-500$/hr 'consultants't do not provide anything more than code lifted from code samples or community tech forums.
They now use my tiny little business's [1] solution instead. I've been on the market since 2014 and they found me through my web site. They called on a Friday "we need the system on Monday". Signed the contract in the evening, set everything up over the weekend, and now they're loving it. Sometimes, I think these big organizations are just so overloaded with bureaucracy and politics that a little guy like me can compete on pure product quality.
[1]: https://www.indiehackers.com/interview/terminerinnerung-56e9...
The other one was hired on a position, asked to do something completely different he was not qualified for, the work hours were around 70-80 per week (no overtime paid), zero management implication so he left after a few months. They went in panic mode, offered him double the pay and less work hours, he refused to stay there so they become aggressive.
They use to pay above the market for whatever they can hire, but the chaos inside is making any competent people leave asap, so they pay a lot to retain whatever they can, not the experts. It is just a bubble with nothing than hot air inside.
I’m no longer with Deloitte, but I saw that they recently sold off their life sciences department to their only competitor- Model N [1]. The Exalink platform was born out of work being done for the life sciences department. I thought it was an interesting development that Deloitte sold them off since when I was still there it was being peddled as this great in-house cloud-agnostic system that could do whatever you wanted it to do (spoiler: it couldn’t). Anyways, maybe it is completely unrelated?
1: https://www.nasdaq.com/articles/model-n-modn-acquires-deloit...
So you quickly see large numbers of leads, directors, managing directors, managing partners, and partners show up into meetings with 1 or 2 SMEs to make a show of "managing the project", whose only purpose in life is to suck the oxygen out of the room.
And I mean literally suck the oxygen out of a room. The meeting rooms are tiny--cuz real estate isn't billable!--so when you get more people than chairs in a room like that, the CO2 levels literally start to put you to sleep.
I spent a year and a half at Deloitte, building Augmented Reality projects. Every single project I worked, I was only brought in during the last 10% of the project time. The time was always not enough to do a good job. The one project that I cared about--incidentally also the only one that got any traction with any actual clients--got canned during a restructuring from working within Deloitte's "government" division to it's "industry" division. Except we always worked blended across industries and the app was targeted at industry, not gov. To this day, I really think the entire point was to just waste money, and my app getting traction edged too dangerously close to one day having to show efficacy.
I'm now the head of VR (yeah, I never actually wanted to do AR, and didn't even get hired for it, but that's where Deloitte stuck me) at a small company that does foreign language instruction for gov. A year and a half in and we've got an exciting project that has tested well with some limited number of students and is soon deploying company-wide. Barring some natural disaster, I'm extremely happy to never change jobs again.
When those groups talk about innovation and disruption, I often run.
The next time your government awards a tech project to tech illiterate smooth talkers, keep this in the back pocket.
Software is often judged based on these UI/UX nuances, but the complexity of building a competent front-end system remains an undervalued part of the process.
The front-end team is usually (1) brought in too late in the game to provide effective feedback and (2) not given enough time to build.
When there's not front-end representation at early design reviews or at initial discussions on how APIs should work, then it's easy to forget things like form error handling.
Designs don't account for all possible states, services/APIs are in flux, until the very end.
Everything else is late, and then all of it has to get slapped together with duct tape and shipped in a few weeks. Performance and browser support be damned.
Everytime I’ve gone to a doctor, they’re using a different kind of software portal. Many claim its an expensive software the hospital bought but doesn’t do what they want so they write things on paper instead.
This is one of my fears as a dev. I have dev problems that I know how to fix but I’m blind to the universe of other problems that I could potentially fix and deliver huge value. I’d really love to learn more about other people’s niche problems that can be solved by software.
Deloitte just coded the spec, cashed their check and walked away. Deloitte probably outsourced development overseas.
I know some smaller government contractors in the D.C area who would have done it for a third as much and done some of the CDC's work for them when they saw that it wasn't done.
That can be translated into a new corrupt administration is replacing the old corrupt administration and will spend an additional 44 million on a new website built by their own politically connected friends.
Screen scrape data from legacy terminal apps, export into SQL middleware, have the new app interact with the SQL middleware via modern APIs. All legacy data gets batch updated back into legacy systems, all new data gets stored and accessed in middleware. Eventually, the middleware becomes the primary data source and the entity can save on the electric bill.
Sometimes I wonder if flat earthers are right... we indeed live in clown world. A sucker born every minute.
A write up on why it’s so expensive from a standpoint of a developer. https://maxwelljordan.medium.com/why-healthcare-in-us-is-so-...
Deloitte’s name seems to come up a lot when there is a multi-dozen million dollar web site that is falling over. I wonder if it was a former Deloitte person who wrote the Parler backend because it seems to be the same sort of massive footprint that falls over when the wind blows type setup.
https://www.wpri.com/target-12/we-are-very-sorry-deloitte-ap...
Meh.
https://en.wikipedia.org/wiki/Internet_protocol_suite
To be fair... it's also from about 50 years ago. Hmm...
Or alternatively spend a few billion making the government more business savy. It will probably save you money.
Then again maybe not. That will be another few billion going to Deloitte.
Thanks, I hate it. I use Firefox. This 50% of my time on the internet.
Oh and all of this talk about 95% of the work being understanding the problem or integrating with legacy government systems, gimme a break. Every time I’ve seen someone purport to integrate with existing government systems it’s always been a buggy disaster and hideous from the user perspective.
This is the same kind of rot that infects a lot of enterprise software. Addiction to hyper complexity, too many layers, too many points of failure.
Just how complicated does a vaccine management system need to be? If you’re adding more and more requirements, it’s not software that’s the problem, it’s the requirements that the vaccine distribution is being made more complicated than it is needs to be.
I think at this point, a random distribution would be more effective than the complicated rollout they’re trying.
a long time ago i work for a mid size company with a small division that sell specialized software build with Delphi.
its a desktop app. when i got hired by the company, they outsourced it to Capgemini. after 3 years, they still can't get the app convert to .Net. (that's the goal) by the time i got out the company. they changed to another more local company instead of big outsource firm like Capgemini. they still haven't got it all work out and the new company they hired throw away all the code done by Capgemini. the whole project was estimate to be 2 million by Capgemini.
Looks like the CEO of Deloitte and the director of CDC played a few rounds of golf together before...
a) Poor/lack of understanding of who the actual users are e.g. these are systems that will be used by both the young and old people, especially older people in the early stages since they are the ones at the front of the vaccine line; worst case, the system should have been optimized for them.
b) bullet a then leads to poor requirements e.g. given that most libraries are shut down (lots of elderly people get help with online stuff at community centers/libraries), requirements should have included less rigidity (on errors) or a module that is fun and friendly that walks people through it, system should be designed with 'plain' mobile first in mind, and account for a person being both a patient and an admin, etc
Humanity can make an mRNA vaccine in 6 months. That's literally (almost) instructing your cells to make a particular protein.
But one of the largest consulting company in the world cannot develop a freaking website.
I thought this line from the article was interesting:
> Online sign-ups are especially challenging for older people, perhaps the worst group to beta-test a new system.
I think I understand the author's intent when saying "worst group", but in this case I can't imagine a better group. They're literally the first people who will need to be using this system! If you beta-test your product with the people who will be able to use it despite all of its flaws, is that really testing?
Anyway yeah, just upset at this comment given that I have family members struggling with all this vaccine appointment software.
To make a sweeping generalization which doesn't give enough credit to some amazing groups, the Federal government does not have people who are capable of managing complex IT projects. The government has outsourced so much work that it no longer has personnel who are capable of knowing what is going on. Federal workers are competent in managing contracts, but not actual work. So when you arrive at a new and complex project (like widely distributing a vaccine for an unforeseen global pandemic), they don't know what to do. How do you write that in a contract and enforce it?
That doesn't even begin to get in to the requirements around what it takes to launch a production IT system while meeting government cyber security requirements/processes. A lot of comments here are lamenting how easy this is. Let me set it straight: you cannot launch a random Django/Rails MVP in the government. They don’t operate that way, for reasons good and bad. The fact that Deloitte had something in production in under a year, no matter how buggy, is a minor miracle.
First, I would create extremely fine grained but easy to check eligibility tiers. Going by date of birth would be the most basic solution. At any given time, vaccinations are open to people born on or before a given date. [0]
Then, I would set up real mass vaccination sites. None of this almost-all-vaccines-come-from-big-medical providers nonsense we have in most of California. The CA scheme is fundamentally inequitable -- I got an email from a medical system that told me that they would email me when it was my turn if I had seen a primary care provider in the last three years or had two specialist appointments in the last three years. This fundamentally discriminates against people without adequate insurance who haven't seen a doctor in a major medical system recently.
Mass vaccination sites would be open to anyone with a date of birth <= one threshold on a walk/drive-in basis with no appointment and with a different threshold for appointments. Optionally there would be another threshold for people who want to hang around and see if they can get an extra shot at the end of the day. I would adjust the thresholds as needed to keep occupancy at the level I want it. This requires minimal IT infrastructure.
I would let major clinics administer vaccines with whatever appointment system they want, with some rules. First, the clinics may only vaccinate people who have been eligible at the mass vaccination sites for some amount of time, say two weeks. So if you want your chance first, you go to a mass vaccination site -- clinics are just for convenience. Second, the clinics only get more vaccines when they are administering an adequate amount of the vaccines they have already received.
I would track people who have been vaccinated with a database with no integrity constraints whatsoever. Each site submits a trivial machine readable form with a list of people they have vaccinated. This list gets imported to a database. Dealing with all the inevitable issues is not permitted to block actually giving vaccines. [1]
Finally, everyone who gets their first dose gets a printed (or hand-written!) card with the date of their next vaccine and a reference number that can be used to look up the card in the database. This reference number is (date, site, line at site, sequence number) -- these are supposed to match up with the end-of-day logs, but they require no central coordination. [2]
If you have a card for a second shot, it is valid at any site once it's the right time.
Damnit, this whole scheme could be done entirely without computers, and that's how it should be. It doesn't even need two-way communication -- there could literally be a section of the newspaper saying "if you were born before m/d/y, come to Dodger Stadium or Levi Field or ... and get your vaccine, no appointment needed. If you were born before m2/d2/y2, you may call this number of visit this website to make an appointment." If the whole IT infrastructure goes down, then it will be annoying but everything will continue to work.
[0] There are many possible improvements, but they should be kept simple. For example, people born on or before a given date or people with a doctor's note explaining that they have some risk factor.
[1] Pay some contract development firm to develop a nice app that scans an ID card, does all the data entry, and prints out a receipt. The app has one primary requirement: it does not print a receipt until the server acknowledges receipt of the request and commits the transaction. But make sure that, even if the app fails completely, everyone at the clinic knows how to use a piece of paper and a pen to do the same job. Or a laptop running Excel or LibreOffice -- there is no shortage of laptops around. Or a Google Sheet, etc.
[2] Of course you can cheat, but most people won't. And the logs will mostly reveal the cheaters.
Sometimes people get so caught up with new technology that they forgot about old technology, and technology predates computers. Believe it or not, it wasn’t that long ago that you could go to stores, buy things, eat at restaurants, and even fill up your fuel tank without directly interacting with any form of computer. Heck, it hasn’t been that long since you could fly on Southwest Airlines with little reusable plastic boarding passes that the agents would hang out in exchange for a paper ticket. These systems were imperfect, but they worked and were extremely reliable.
It doesn't even need to sit around unused. They should be using it every year for flu shot distribution or coordinating other types of vaccines which are normally in abundant supply.
...stuff like this drives me mad. You see it everywhere. I worked in financial services, and (on my weaker days) it was enraging to see someone pay three or four times the price to get a product that was not only bad, it was literally worthless...life is weird.
https://www.ctvnews.ca/politics/ottawa-gives-accounting-firm...
One of the factors cited in awarding the contract was Deloitte's "experience" doing the same thing in the USA.
http://www.canadian-accountant.com/content/business/why-delo...
Considering that this came out of nowhere, I was impressed how quickly and easily I got a vaccine. Big thanks to the Army who ran the vaccination site.
https://www.theregister.com/2019/04/23/hertz_accenture_lawsu...
(I am invested in PLTR)
Also, fuck Google's monopoly.
(I'm a long-time IE user.)
Still makes me wonder why though, most projects I've worked on have have to support IE, and though it annoys developers because they cannot use the hotHewThingN api, mostly its fine.