Hacker News Reader

Comment by swanson | Hacker News Reader

Let's assume a hefty markup for the long sales/BD time for a government contract like this: $336k * 0.60 = $200k

Since this deals with national security, could probably justify higher rate resources staffing (security clearance required?). Ballpark $250/hr @ 8 hr/day so $200k buys you 100 days of capacity.

Project breakdown:

* UI/UX Design: 10 days (lots of stakeholders+approvals needed, maybe field testing with TSA agents, accessibility audit)

* Development: 15 days (this is very conservative but might be some business rules that aren't being shared -- remote monitoring, analytics, auditing, static analysis, etc)

* Project Management: 25 days

* QA: 15 days (multiple devices, high perceived security exposure)

* Third-party security audit/pen testing: 5 days

* Technical writer for documentation and training materials: 5 days

Total: 75 days of capacity spent.

Math doesn't work out exactly and some of these numbers are pulled from thin air, but it seems like it's at least in the right ballpark.

Do you folks really know people that are doing government contract work (for the TSA no less) for less than $100k?

View on HN

> Do you folks really know people that are doing government contract work (for the TSA no less) for less than $100k?

We have done recently. (Not TSA - not even US govt...)

And, in spite of our "mid 5 figures" price for the iOS+Android apps for a fairly simple app - the whole program of work the government department included this in, and hence the media reporting of "what the app cost" came to 1.3million dollars.

Now I get people asking me "How on earth did it cost over a million dollars for that app? Surely that's practically fraud!"

If the only requirement is "an arrow that randomly points left or right when you tap the screen" - I suspect _anybody_ who reads HN could whip up a provably-correct solution in under 30 mins just using html and javascript. I _strongly_ suspect there's a +lot_ of other requirements and costs we don't understand here. (But cynically, a lot of those costs could well be high paid consultants writing Powerpoint decks justifying why the only requirement should be "an arrow that randomly points left or right when you tap the screen". That'll be $325k thanks - here's our recommended outsourcing partner who can deliver the app for $11k.)

Have you considered that maybe it's the government's fault that the media reported it in this way because - as in this example - the government intentionally obscured both how much the app cost and what was actually being supplied for the figure they stated.

Maybe, but then again the media has zero incentive to report this truthfully - "bad government wasting taxpayer money" is what sells.

To be quite honest, "bad government wasting taxpayer money" is more than newsworthy.

To your other point, we have in the past seen the media reporting these types of things in incomplete ways for fear of irritating their sources and losing reporting access.

> To be quite honest, "bad government wasting taxpayer money" is more than newsworthy.

If its true. In contemporary journalism the truth doesn't matter. Things can have perfectly reasonable explanations and be working as intended, and the only thing required to turn them into scandal is to omit those explanations and add a clickbait headline instead.

There's plenty of examples for that in Europe in the way journalists paint perfectly reasonable EU directives as utter nonsense handed down by stupid bureaucrats.

Things can also have perfectly reasonable explanations and be working as intended and still be an example of waste and scandalous. Just because you can see how it happened and why it happened doesn't mean you have to agree that the why and how are necessarily good reasons.

"Perfectly reasonable explanations / working as intended" and "example of waste and scandalous" are mutually exclusive IMO. You're right in principle, but what I see in practice is something completely different. Reasonable situations can be - and are - painted as ridiculous with proper application of journalistic freedom. It'd be better if people were thinking more critically about what they read, but outrage seems to be a pretty good mindhack the media learned to exploit. So I believe it is their moral responsibility not to abuse the gullibility of the readers.

It's also likely that government accounting departments only report costs at the project level and not at the line-item level -- it is common practice to just report on CapEx vs. OpEx because there are so many variations as to how things can be bought and sold, especially on technology projects (no idea if CapEx and OpEx make sense in the context of government work since they're primarily for tax purposes, but you know what I mean).

I doubt any company in America can tell you exactly what an app cost them to build. They can tell you what the product cost to develop, but the app itself is kind of beside the point: you don't get an app without the up-front work of determining the requirements, so why bother to separate the costs? Some projects have complex requirements and simple implementations, others have simple requirements and very complex implementations. The project cost is what everyone is concerned with anyway; and while the project manager should have the line-item information for the project, it's often not rolled up higher than that because the reconciliation would be nearly impossible.

> And, in spite of our "mid 5 figures" price for the iOS+Android apps for a fairly simple app - the whole program of work the government department included this in, and hence the media reporting of "what the app cost" came to 1.3million dollars.

You're absolutely right. They didn't buy an app; they bought a solution (of which the app was a small part).

Because it's not just the iPads; it's setting up the management structure for the iPads. How to procure them, locking them down so agents can't install Candy Crush, modifying existing IT processes to be able to manage device breakage/reloads/provisioning, etc...

None of this stuff is all that difficult, but nothing in business is free. So while any moron could write the app itself in half an hour, designing an end-to-end system that serves the needs of the customer (can be operated by someone with a GED; doesn't break down often; has simple workarounds for confusing scenarios; etc.) can cost a lot more than that. The software is just a piece of it.

Also, at the beginning of this process, did they know that what they wanted as an end product was an iPad running an app that made an arrow point left/right?

Or did they have a problem they wanted solved, and specing out a way to do it, was just one part of the job? Because often, the output of the contracts I work on, is the just the final step of testing out many possible solutions or version, and actually shipping the one the client finally agreed to.

Except these are iPad apps. Why the hell would you be doing that with HTML/Javascript?

Ever heard of UIWebView?

Doesn't change the question. Why would you be implementing this in HTML instead of doing it as a regular app?

Because if - as suggested - the _only_ requirement is "an arrow that randomly points left or right every time you tap the screen", the solutions is so simple that it _can_ be done in HTML/JS - quite probably written, tested and delivered for less than the yearly Apple Dev subscription required to do it as a native app.

If I had a friend who needed that functionality and had a decent reason but no budget, say for an art project or a non profit event, I'd certainly offer to build them an HTML/JS thing for nothing. I'd think twice about starting an XCode project, dealing with App Store submission requirements, needing to deal with people asking how they install it on their Android tablets, etc.

At least for me - sometimes the stupidly simple HTML version of a thing is a perfectly good solution. (For the TSA, there are without doubt more requirements than the stupidly oversimplified one proposed here...)

So you're going to make it a web page. How are you going to secure it? How are you going to make sure the website can't go down? How are you going to make sure that the only thing that device is running is the web browser, and that the only page it's allowed is the one to your app?

For my friend's art project? I'm not gonna care about any of that. (Not _quite_ true, I'll "care" about the webserver part by making it an HTML5 offline app - mostly to remove any chance that webserver problems make my phone ring while I'm asleep or out drinking).

You're inventing requirements which, while no doubt genuinely part of the TSA's requirements, go way beyond my single simple requirement as discussed.

"For my friend's art project?"

Your friend's art project is not what we're discussing.

"You're inventing requirements which, while no doubt genuinely part of the TSA's requirements, go way beyond my single simple requirement as discussed."

I am not; as you yourself admitted, these are likely part of the TSA requirements, which is what we're discussing.

Sure, except this subthread starts out with a response from me to which you're objecting saying:

'If the only requirement is "an arrow that randomly points left or right when you tap the screen" - I suspect _anybody_ who reads HN could whip up a provably-correct solution in under 30 mins just using html and javascript.'

In the context of the proposal to do this in HTML/Javascript, I've been explicitly discussing a obviously oversimplified single requirement - not the "likely" requirements for the TSA project about which we can only make assumptions, since we've not seen them.

Sorry if we've been talking at cross purposes here.

Honestly... this doesn't sound that expensive.

I think those who don't work at a tiny four-person startup but don't generally touch the money should spend a bit of time at this site: https://tobytripp.github.io/meeting-ticker/ But rather than using it as a meeting ticker, use it to show how quick a team of 5 or 10 reasonably-well-paid people burn through the money over the course of even a day or three. Business is generally a lot more expensive than I think people often realize. I've sat through "all hands" meetings that clock in distressingly close to $350K by that calculator's reckoning (though rather a lot of engineering is really only half-listening and still working...). If you think about the expense all the time you can end up paralyzed but if you never acknowledge it you can end up with very skewed priorities.

To make this line of thought appealing to those who are probably the ones reading this, when your manager asks you for that stupid-ass feature that's going to take two or three days to implement, you've actually got a really big stick if you work the math and give them the numeric figure on what that's actually going to cost. Managers themselves often don't really think about this properly either. You can help them. If you happen to do it, shall we say, a bit "selectively", well that's their own darned fault for not thinking that way themselves more often....

At a few places I've worked, I asked for average salary figures for the engineering team by function so that I could produce weekly reports that tallied up the cost of various projects based on the time people reported they'd actually spend.

What I found bizarre was that each time I've had to explain why I needed the numbers, and people were surprised at receiving a report of what the staff budget went. You'd think that would be expected, not something unusual. But permanent staff is not treated as a cost during day to day work, even though their time is a limited resource.

It's quite fascinating to see how quickly priorities change when the cost breakdowns make the rounds and someone higher up starts asking critical questions about why we spent $50k of staff time on feature X that no customer had asked for.

You have me wondering now.

A big problem I've always had is getting hardware to improve productivity. Most of the time the management chain can't grasp why it's worth a $3k machine and maintenance costs to save each dev an hour a week in development time.

I think this goes to just how often people I management like the title and the power but don't understand that they're supposed to be in charge of strategy. I don't think most of them would know strategy if it walked up and bit them.

I've found that the time itself is less valuable than articulating the workflow changes it can bring about. Saving an hour a week is great, but if you can save 30 seconds at the right time, it can have a much bigger impact.

For example, if I can spend a day tweaking my project's build scripts and CI workflows to get it from taking 30 minutes to build/test/deploy down to 30 seconds, I've dramatically altered my team's workflow.

Also, there are very few reasons to buy your own hardware anymore -- cloud services are better at almost everything unless it's a low-latency operation (or you're in Australia - the speed of light is still too slow, sorry mate). Obviously there are exceptions, but with the scale of GPGPU build-out all the big public clouds are deploying, the benefits of anything else are rapidly disappearing.

Most managers aren't swayed by the argument that 1 hour = ($annual_salary / 2000). They know that if you give people an extra 5 minutes a day, they'll probably just spend it drinking coffee or socializing. Those certainly aren't bad things for employees to do, but unless your team is already running well beyond capacity, giving them more time to do things doesn't really improve productivity. Improving process (by, say, removing a manual step through automation) really gives you more bang for your buck.

> Also, there are very few reasons to buy your own hardware anymore -- cloud services are better at almost everything unless it's a low-latency operation

Cost. I've yet to see a cloud service that get anywhere near as cheap as owning your own hardware (fully loaded with financing costs, full staff costs etc.) for anything that runs more than ~8 hours a day.

Cloud services have plenty of uses, especially for brief capacity spikes. Need to run an experiment for a few days that requires 20 extra servers? Go nuts. But even with reserved instances, if most of your servers run 24/7, you're throwing money out the window.

(And you don't need to own hardware - you can go for rented dedicated managed servers and still get ~80% of the savings)

A little measurement goes a long way. It's remarkable, but there's a lot of inertia even in otherwise successful organizations.

And keep in mind salaries are just a fraction of the costs of staff time. Payroll taxes plus benefits can add up to almost 20% of base salary and that's not counting occupancy and equipment costs.

Bottom line: people cost a lot. It's smart to think carefully through the ramifications of planning decisions.

Your analysis sounds pretty spot-on.

It's government, so you'll likely have a 508 compliance audit (accessibility) as part of the project as well. Figure a work-week of that.

Security clearances are almost a definite "yes". Extensive background checks at a minimum -- and we're talking three days of paperwork, fingerprinting, calls to your third grade teacher, etc.

Great point on the accessibility audit, added that to UI/UX section.

Exactly. I wouldn't be surprised if things on the airport have to meet weird regulations, and that the IBM will have to take some responsibility as a result of possible damages.

This proves that similar absurdities are common practice within large corporations, and breaks down nicely how they are justified. I doesn't, imo, prove the price isn't absurd.

The point about field testing is fair, but shouldn't account for a huge chunk of the price. The invisible-to-the-public features point is also fair, but there's no way to estimate that cost of those.

The rest of the points, though, while fairly reflecting the way enterprise works, just show how utterly dysfunctional the process can be, and imo bolster, rather than refute, the position in the original article.

I'm not seeking to refute anything. I agree that it sucks that software costs are so expensive and that taxpayer money was spent on this, but if you don't try to understand the process that is broken I don't think there is much chance to fix it.

"That is literally a hello world ipad app, how could it possible cost $300k?!?" -- I attempted to provide an answer that goes a bit beyond "shrug govt" or "shrug consultants".

It's like the $10,000 dollar hammer, but this is far worse.

With the hammer, government apologists claimed it was to "secretly move money for DoD"... which really means it was the DoD and military just giving themselves a shit ton of money.

The US is very corrupt. The DoD dodged a bullet when 9/11 happened because that was the day they were going to be put under investigation.

Sadly enough, nothing happened at all, and this type of political graft is very common among the military and their politicians.

A recent case for example is the head of University of California who does not even belong there and is completely ill qualified to lead a college. Her only real experience is as a _spy chief_ for the military and DHS.

This type of political graft resulting from military connections is going to be far more common and invade further into civilian infrastructures. It's not like the NSA's complete treasonous behavior in spying on the US citizens they're supposed to protect isn't bad enough... that isn't even the tip of the iceberg, I'd wager. Before that of course was the CIA funneling drugs into cities to fund their own pockets and programs. Or even the joke of the "war on drugs" which all it was meant to do was control the profit flow of illegal street drugs.

These styles of military programs, also known as soft power projection, directed at the US population and citizens is getting far more common and invasive. These days, it is almost considered the norm.

> It's like the $10,000 dollar hammer, but this is far worse.

> With the hammer, government apologists claimed it was to "secretly move money for DoD"... which really means it was the DoD and military just giving themselves a shit ton of money.

No, it was a $15 dollar hammer with $420 of project R&D costs averaged out and added to it: http://www.govexec.com/federal-news/1998/12/the-myth-of-the-...

The $435 price was then inflated to $600 by the media and finally $10,000 by you.

Likely your security / approval estimate is low [edited]- typically you can write an app like this quickly but the ATO / certification and the BAs and PMs filling out paperwork and sitting in meetings will add weeks / months.

Also I agree the overhead of current T&M contracts likely doesn't make contracts under 100K worth pursuing especially given the huge amount of competition for some of this work and (unless an incumbent) a pretty low win probability.

Fingers crossed the TechFAR and some of the micro auction steps being taken make this a more dynamic market.

EDIT2: Here's the spending on that Contract by task order (as up to date as USASpending can be anyway) https://www.usaspending.gov/Pages/AdvancedSearch.aspx?k=CIO5...

EDIT: If the re-compete was done under OASIS then maybe we can get a lot of information out of this line by line transaction from USASpending: https://www.usaspending.gov/Transparency/Pages/AwardSummary....

PM would be higher than that. There'd also be a Manager above the PM scheduled some days as well.

And the security, analytics, and monitoring are probably all smaller specifics days for individual experts.

Point is though, you're thinking the right way on it. Very reasonable.

Good point, added a couple more line items.

Your math is (relatively) sound but I don't think you understand how trivial the work required for this would be.

It's pretty universally known that engineers are horrible at estimating their tasks but this particular app is so simple as to be impossible to overestimate. It literally would take less than a day for a (half)-competent engineer (and maybe 1 designer/artist) to do the work for this (I won't give you the task breakdown but I'll say the task list is short).

So the answer is probably more in line with the other associated costs as you mentioned. Another just as likely (and as simple) answer; they charged some know nothing bureaucrat a ton because that's how it works in DC.

Yeah clearly this ui http://elliott.org/wp-content/uploads/IMG_6317.jpg required 10 days to design.

Why would this require more than a few days? Simple UI, then just generate a secure random number and draw an arrow. Bam. Give me 200k.

I mean the largest expense would almost certainly be third-party pen testing and QA. And why would it take a technical writer 5 days to document this? It has 1 function.

You need to be on site with a Secret clearance, a DUNS number, and about three months of paperwork, never mind the cost of the proposal hours to get on the multi award BPA to have the opportunity to win this task order for the work which was 1000 pages and half a year of capture and messaging. The market has a huge cost to enter which makes it sound insane to anyone that's ever posted something to Fiverr

Fair enough. I imagine that the paperwork and app verification is where most of the cost was generated.

Sometimes bureaucratic inefficiency just really gets under my skin.

It is a good and understandable instinct. It is the bane of my existence, which is why I've also devoted a solid chunk of my life to unscrewing it. :)

That's one screenshot, are you prepared to submit a legally binding quote for the project with that little information? Are there more screens? I don't know. Did the client require multiple designs and iterations -- each stamped with the approval of several stake-holders? I don't know.

Judging the final output without any consideration for the process is akin to asking why it took a week to fix that hairy bug. It was only one line of code that changed, how could it possible take you all week to fix it?

Sure, since, in government contracts, "legally binding quote" doesn't mean you actually start losing money or have a judgment issued against you that you're personally liable for if there are time or budget overruns.

One change may take weeks to sign off on - it's not just one person giving their OK, it's many layers of project/contract managers giving their OK. Each of whom has to account for their time and each of whom has to be paid for that same time.

There are processes that have to be followed, no matter how ridiculous they are, because they exist.

Don't forget you have to write your arrow direction choices to a database, so that we can prove that over time it was random. And that on August 18th at 10:27, it showed "left" when the person-of-interest (we don't call them suspects anymore) went through the line. This would require a couple of reports be written against the database.

UI/UX 10 days?

Yeah right.. The recoloring of the arrow itself I'm sure took 10 days...

I'm working with a bank right now trying to just get them a UI for an authentication page. We've struggled 2 weeks trying to figure out what the hell they want for the FAQ page... Between the design-spec they provided and where we are now, I'm not sure why they have designers in the first place..

Our first product was according to their spec, which they scrapped instantly and stated they expected X,Y,Z instead, which weren't in the spec (Page should have an accordion, the accordion should have buttons, all buttons should have certain looks and sizes. The spec didn't even have borders, much less an accordion and no buttons in sight..) and when we produced whatever they stated they wanted they threw out half of it and flipped things around (now the buttons shouldn't be arrows, they should be + signs that become - signs when clicked on...)

I'd hate to think what the UI/UX process is trying to get through multiple layers of governmental bodies....

That plus we know little about the actual scope of this project. Sure it looks like a randomly jumping arrow - but does it have a backend? Does it have all the 'settings' and 'technician modes' that usually bloat projects x10 than the MSS scope?

And unless it was a dedicated government services vendor, all those people needed to submit background checks, get security clearance and be US citizens or at least permanent residents all of which costs a bit extra.

I'm going to file this analysis as 'Breakdown of the cost of bureaucracy'. Considering that the process could have been solved with a common sense email that might have identified an existing free app.

great points.

You forgot the mandatory Oracle license the tender no doubt called for.

We do ton of govt work for state agencies, transportation agencies, local govt, etc. That price is messed up.

And of course I get downvoted. Nevermind that 70% of my MRR comes from govt. Whatever.

Reply on news.ycombinator.com