Hacker News Reader

Comment by JumpCrisscross | Hacker News Reader

> Kiewit, the Nebraska-based construction firm that has the main contract to rebuild the main spillway and emergency spillway at Oroville, the nation’s tallest dam, estimated in its winning bid in April that the work would cost at least $275 million. But the price tag has now grown to at least $500 million

Out of curiosity, what was the runner-up bid? Is there no mechanism to re-open bidding when the circumstances change? What prevents me from bidding $1 and then busting it to $1 billion after winning?

View on HN

I'm not saying the economic aspect is very good, but I would make 2 points: they are more or less on time, and nobody got hurt. That probably desserve a shower of money.

Those people were using the biggest dry ice making machine in the world to try to pour not too shitty a concrete on schedule, next to a blasting zone, under a leaking gate (yeah, the gate is leaking it just stopped leaking because the water is lower than the gate now), under high voltage power lines, without anyone getting hurt.

Those are good points, the Crystal Springs Dam earthquake retrofit in San Mateo County is almost fifteen years into the scheduled five year earthquake retrofit. :)

I thought the dam retrofit itself was done and the remaining work is just road work.

You are right but the work is ten years overdue from the first estimate on the first sign they put up next to the Crystal Springs trailhead where the road is closed to traffic. Eventually they covered up the last two digits on the year number sign last time I checked with replaceable numbers like the days without accident sign in the nuclear power plant that is Homer Simpson's workplace.

I find it interesting the default commentary to this sort of event - ballooning budgets on underbid projects - assumes that bad faith on the part of the bidder is the problem, and that if only we could punish these few bad apples - punish those scheming, underbidding contractors, then the problem would disappear.

It seems unlikely that bad faith is the primary cause. Yes, there are really contractors and owners who do act in bad faith, like the Trump organization, but these are unusual, and highly publicized. Punishing dishonesty seems like the easiest problem to solve, and it's easy to rally around for a quick and cheap victory. But I have seen these run-up bids for decades and no amount of punishment has ever seen to stem the flow of these events; it seems as if every city and state is infected with this problem. Either we are in a system where bad apples are so numerous that punishing a few has no meaningful effect on incentives; or we are in a system where bad apples are so rare that we punish good apples mistakenly, and can't figure out the real cause for our poor planning. Until we move on from the assumption that a few bad apples are to blame, we likely won't be able to discover the real underlying problem.

Thank you for saying this. I think there is a deeply distrustful aspect to how the public views these contracts, probably because a few bad actors (as you noted) have in fact screwed the people from time to time. Also people tend to associate these bid contracts with politicians in power who they may dislike for one reason or another therefore further tainting their opinion of the bid system.

The distrust of government and corporations in our society is one of the biggest problems facing us as we go forward in the 21st century. We need to figure out how to restore trust.

> if only we could punish these few bad apples - punish those scheming, underbidding contractors, then the problem would disappear

I'm criticizing a system that incentives optimistic initial bids that are later revised up. A bad system doesn't need bad actors to produce bad outcomes. A $275 million bid ballooning to $500 million without any competitive pressure is a bad outcome.

American infrastructure costs are higher than production factors, e.g. land, material and labour cost differentials, explain [1]. Our auction mechanics are called out by experts as a big part of the problem.

[1] http://marroninstitute.nyu.edu/content/blog/is-u.s.-infrastr...

How do you specify the contract so the unknown unknowns can be priced into the bid and compared to each other. Given that resolving those unknowns costs about as much as the just doing the actual work.

You temporarily re-open bidding when big price changes are requested. See my other comment [1].

[1] https://news.ycombinator.com/item?id=15524807

That sounds like a recipe for delays and finger pointing along with massive overruns on the later parts "to fix what those other guys didn't do our way."

If you want separate bidding for separate phases, bid it out that way.

I suspect this would just make all projects much more expensive - bidding for a large project costs a lot of money and can require large teams of people working for extended periods of time. All of that gets paid for out of the projects that are won.

If you now introduce scenarios where projects can have an unknown number of bidding processes all costs will go up to cover the work necessary for multiple companies to prepare bids.

It would just make things much much worse.

It is definitely a system issue. Our contractual/financial system just doesn't work well when confronted with large complex projects that have enough variance and unknown unknowns. Since they really push to have everything committed to up front, with really basic rules around how to handle change and variance.

Let alone that we basically award this work one estimated number and the ability to get through a paper work hurdle.

If you start making those rules complex it becomes hard to compare bids to each other, since your bids are likely to become non-transitive based on the scenarios.

Cost plus is the normal way this is side stepped, but unless you have a system that is mostly ethical actors that can break down. And if you have all ethical actors pretty much any system will work...

Cost plus percentage gives a very clear incentive to raise costs; that's the only way you can get more profit. If it's so widespread that might be one reason why all our infrastructure is so much more expensive.

Why not just have major contractors required to post a bond when they place a bid, and if they have cost overruns they pull those from the bond (ie, their own money)?

> Cost plus percentage gives a very clear incentive to raise costs; that's the only way you can get more profit

Case in point: SpaceX challenging the launch industry's cost-plus culture with publicly-posted fixed prices [1].

[1] http://www.spacex.com/about/capabilities

Case in point #2: the new Bay Bridge.

Case in point #0: the Space Shuttle (STS).

Sorry, you're citing the Space Shuttle [1] as a successful public procurement process?

[1] https://www.nasa.gov/pdf/382045main_19%20-%2020090730.11.STS...

No, my cites relate to the parent comment:

  Cost plus percentage gives a very clear incentive to raise costs

Compare the original winning bid for the STS with the actual costs; I think they rose more than tenfold (and the per-flight cost rose more than that), IIRC.

And had the competing team won, the Challenger disaster never would have happened, because the solid-rocket boosters would have been manufactured as units and transported to the Cape by barge rather than having to be broken into small enough components to fit on rail cars.

Oh for sure, I should probably add a non-profit driven caveat to the actors in the system.

I am not sure if you can implement a system that would work well when your actors are profit driven work that needs to be done can only be done exclusively by one entity at a time, and as a bonus that extra profit gives you more feedback into how the system operates. There aren't a lot of natural negative feedback to keep things under control in that system.

Especially if your goals are to reduce overhead.

Even better: the more rules and processes you put in place to try to avoid/punish this, the harder it is to get the contract in the first place. Eventually, most companies decide it's not worth jumping through all the hoops you've put in place to avoid corruption, and you end up with a handful of entities which get every contract and face no meaningful competition to try to keep the bids down.

This is already a problem. A company I worked for submitted a bid for an IT infrastructure project several years ago. We were rejected for not having completed a similar project for a government agency before.

Our bid was 1/3rd of the "winning" bid.

Problem is easy to identify - We are using waterfall for massive scale urban projects, and getting the known associated problems. Add to that massive corruption and cronyism.

The fix is not clear. Can't just drop in agile, every attempt to legislate away corruption fails.

The bids aren't in bad faith. They are just over-rationalized and under-justified until the number is small enough to be in the winnable range. Middle Management talks out of both sides of their mouth, demanding lower estimates, while demanding integrity of results (wink wink).

They don't really want integrity, they want to get a number that won't get them yelled at by their boss. The integrity bit is a pass the buck clause.

So the estimator is in the position of trying as hard as they can to lower the estimate. "Well, if we assume xyz unreasonable assumption, then we can lower the amount by this amount."

And then there are "risk" shenanigans. You itemize all the risk items, declare an arbitrary probability of happening. Then management rolls those up, and declares that they are not going to fully fund the risk and instead "carry" it forward.

Fully funding basically would mean multiplying the likelihood of occurring by the cost, and adding all the items up. Which is already iffy as the probabilities aren't based on much except a w.a.g.

So you win the bid, the risks are realized, and weren't funded. blow out.

Standard practice. And because everyone does it, the honest bidder is the loser. Losers go out of business. So the whole bidding practice is self selecting for cheats.

If you say, "this is unethical, I'm going to do my best to give a realistic bid", you lose.

Should be easy to fix, if the contract for work includes a "penalty for being wrong for circumstances completely within the bidders control that they should have seen".

Austin government did this for its MoPac toll express lanes and got them for way cheaper than what they should have paid: https://www.bizjournals.com/austin/news/2017/09/27/big-settl...

However, the project ran over for two years longer than it should have (it was supposed to be completed in September 2015, and they are only just now wrapping it up). So I guess no one really won there aside from the Austin transportation budget guys, except the populace hates them now for traffic jamming the place up for two years longer than they said they would.

Isn't completion something that is at least in part in the bidders control. Did they employ all available personnel and work 24 hours?

Sort of, when they could. In this case there were penalties for going longer on the project. Main problems ended up being weather (heavy rains) and lack of knowledge of the soil - many changes were made to the project because the contractor (C2HM) did not do a proper soil survey where they were going to be digging.

There's an interesting game theory problem here. If you stick the bidder with the cost of overruns, they will bid high -- and be overpaid if the risks don't eventuate.

So, the consensus seems to be Governments would rather get low bids that don't price in unknowns. If unknowns happen, they can dicker over the increase. But if you award a contract that's too fat, it's difficult to claw back.

If the work goes exactly to plan, the contract will come in on budget. But if problems occur, the price usually goes up. Our current system seems to be optimized for lower overall project cost at completion, with the side effect of making it less useful as a budgeting tool.

>What prevents me from bidding $1 and then busting it to $1 billion after winning?

Isn't this why most publicly financed projects have massive cost overruns? Bidding realistically will lose you the contract; bidding low and then running overbudget may have consequences, but they are not as severe as not getting the contract in the first place.

This doesn't happen nearly as much outside the US.

Are you posting that from inside the USA or from outside of them?

I guess what I'm actually asking is, which country is that promised land? (I have a few candidates in mind myself, but I consider them to be the exception rather than the norm and I don't even know if they are truly as good as I suspect)

Most countries have much cheaper subways. None are worse then the US.

That isn't going to solve this overrun. They got on site, dug down and discovered that they were going to need a lot more concrete.

When they said "we need $275 million," we used competitive pressure to validate that claim. They had the lowest bid and so fairly won. Now they're saying "we need $225 million more." Yet we demand no market validation.

Note: I'm not saying the $225 million is unfair. This is a complicated project with lots of unknowns. But refusing to test 9-figure quote adjustments is bizarre.

They bid on a job that was only 30% specified. The scope of work eventually doubled when the site was fully analyzed and the price tag almost doubled. Nothing to see here.

There is work that is 100% documented in the proposal and in those cases it’s reasonable to hold the winning bidder’s feet to the fire, but this isn’t one of those.

In the absence of time pressure they could have let a contract to fully specify the work, then one to do it. But if they’d done that there might not have been a dam by the time work started.

> Nothing to see here....In the absence of time pressure they could have let a contract to fully specify the work, then one to do it.

I agree that in this specific case, had the contractor come back and said "it will be $1 billion more," it would have been worth paying to avoid the destruction of the Central Valley.

The broader discussion is that this is avoidable. Two models abound in the private sector. For less time-sensitive situations, one has changeover processes [1]. For time-sensitive work, companies avoid like death being totally dependent on a single company.

The latter would require large changes in the way contracting is done in America. But it's not like nobody else in the world faces time-sensitive repairs. The defensiveness we, as Americans, have towards our system which shows clear and apparent pricing flaws is surprising.

[1] https://news.ycombinator.com/item?id=15524807

The standard contracts handle this pretty well; it's not a surprise that surprises happen.

In commercial projects, this often triggers an arbitration about the overrun. You get experts to challenge the fairness of the price, hopefully using market data. And, of course, there are the courts. In worst case, the buyer can cancel and/or rebid. Not 'competitive,' but adversarial.

The bidding process is for sure a very problematic one for every project. But in this case, I assume that the bidding wasn't a fixed package, but rather a price per certain action, e.g. for a ton of RCC poured etc. As the bidding happened before the plans were finalized, that would be the only honest way to project the costs. And as we know, the work executed was way more extensive then assumed during the bidding process. There was much more RCC used - they couldn't know where exactly the bedrock was before they dug it out, and more of the upper spillway replaced with a final solution. So some of the costs which would have occurred next year have been part of this years budget. Considering those changes, it is no surprise, that the costs were overrun, actually the overrun is reasonably modest compared to other projects where there was time for a detailed planning.

Typical for California, and in fact experience tells me to doubt that this will be the last increase.

http://www.wnyc.org/story/316201-brief-history-64-billion-ba...

On time at only double the original cost is a bargain by Californian standards.

As you say, better to wait till its done before getting too excited.

Pretty common strategy in bidding. Undercut the second bidder to get the contract, then run over budget to make a profit.

The problem as I see it is you don't have to actively run it over budget in bad faith.

Even as an ethical actor you can only price what is specified. However, you can then use your knowledge about likely the real variances that will be needed and risk profile to place a bid that takes this into account to remove margin or discount the bid price.

How would you re-open bidding and enact a switchover? They're already time crunched. There would be big costs in money and time from hauling out all the existing equipment from the first bidder, and bringing in the new staff and equipment from the new bidder.

> How would you re-open bidding and enact a switchover?

When the price change is announced, anyone from the original pool has [30] days to match. First to match immediately wins the work from the existing contractor. Loser pays to move their equipment out; winner should have included that cost when they agreed to match.

This system works for two reasons. First, the first bidder will be cautious about requesting increases if it means risking losing the work. Second, the change-over cost, borne by both the outgoing loser and incoming winner, imposes just enough friction to grant the first winner an incumbency bias. This bias helps keep bidding in the original auction competitive.

That just wouldn't work for large civil projects for so many reasons.

It also seems like you are only trying to address the problem of people introducing needless variances as a way increasing profit.

Which is not the case being discussed here and I don't think it is major reason for cost overruns. They haven't requested an increase, they have found out that to satisfy the engineering requirements of the project that more work has to be done.

> Which...I don't think it is major reason for cost overruns

It's commonly called out as one of the driving reasons behind the horrible economics of American civil engineering [1].

> They haven't requested an increase, they have found out that to satisfy the engineering requirements of the project that more work has to be done

It's both. The change may be merited. If so, it should win--again--in a temporary re-auction. But refusing to market test an 82% cost increase is game theoretically begging to get screwed on pricing. Given American taxpayers get screwed on pricing for public-sector civil engineering (after accounting for land, labour and materials cost differences), I think it's a fair discussion to have.

[1] http://marroninstitute.nyu.edu/content/blog/is-u.s.-infrastr...

It is most definitely a fair discussion to have, and the bidding/financing/contractual system for these projects really does need to change to have better incentives.

I just don't think your incentives would produce better outcomes, from my perspective it would make things worse.

I will make no attempt to defend the specific recommendations I mooted. There is better work, in any case, in the academic domain on fixing our flawed civil-engineering auction processes.

The problems anyone campaigning for auction reform runs into are three-fold. One, it's a boring problem with boring solutions. Two, there are vested interests. And three, Americans suffer from a just-world bias when it comes to infrastructure [1]. The latter is apparent even in this thread. It's difficult to find solutions when people vigorously defend a clearly-flawed system.

[1] https://en.wikipedia.org/wiki/Just-world_hypothesis

For sure. I don't know how you overcome the inertia of our societies structure to head towards something that would likely be better for the majority of citizens. Which is a problem in many more domains that just civil works auctions.

I can see how the recommendation you mooted would improve incentives in some procurement domains. Large civil works isn't one of them however.

This demonstrates a total and complete lack of understanding of how large scale construction projects like this work.

You really believe the main issue is "getting the old equipment out" and "the new equipment in"?

Your plan sounds like an amazing way to create a tremendous amount of safety risk, shoddy work, or a project that runs forever.

> You really believe the main issue is "getting the old equipment out" and "the new equipment in"?

No, I was responding to a specific complaint made upthread.

> Your plan sounds like an amazing way to create a tremendous amount of safety risk, shoddy work, or a project that runs forever

Do you have any basis for this? The change-over requirement for midstream cost adjustments (above a threshold) works well enough for e.g. aerospace subcontractors. It adds delays in the short-term but promotes better behavior in the long.

> It adds delays in the short-term

A delay in this project is a safety risk. They are trying to have a working spillway by Nov 1, so if unexpected rains cause another extreme inrush into the lake they can handle it by releasing >100,000 cfs from the spillway. Without that ability, they are risking failure of the emergency spillway and the flooding[1][2] of a large part of the central valley.

[1] https://www.youtube.com/watch?v=yVcGUvE3BBc

[2] https://www.youtube.com/watch?v=4p0Hc56vJ2Y

You're correct in observing that change-over processes are less-frequently implemented for time-sensitive work. I was making a broader point around our public-sector low-bid auction process.

For this type of work, the best process would avoid appointing a single contractor. Unfortunately, that would also require more expertise in the public officials allocating work, a concentration we find politically toxic in America.

Reply on news.ycombinator.com