Oroville Dam: Cost to repair spillways nearly doubles in price to $500M | Hacker News Reader

Oroville Dam: Cost to repair spillways nearly doubles in price to $500M

mercurynews.com

102 pointsDrScump111 comments

I'm not surprised at the cost, given the nature of the project. Filling in the massive canyons that were cut when the spillway failed has taken an incredible amount of RCC (roller compacted concrete) and man-hours of difficult,dangerous labor. Last I heard the project is still zero injuries, which is really impressive.

The blancolirio channel that nraynaud mentioned has a lot of very impressive reporting and flyover footage, and the CA DWR itself has also been posting regular drone footage[1], which I highly recommend for a sense of scale. The spillway is a lot larger than it looks like from most of the overview photos. The closeup drone footage that includes the repair crew shows how massive the structure really is. It's supposed to be able to handle 270,000 cfs outflow (the outflow that damaged the old spillway was "only" ~100,000 cfs).

I'd also like to point out this[2] video from blancolirio, which shows just how close the emergency spillway was to being undercut. Yah, this repair is going to be expensive, but it's good to see that proper repairs are being done. Some things are worth the money; the central valley flooding would be a lot worse.

[1] https://www.youtube.com/user/calwater/videos

[2] https://www.youtube.com/watch?v=oU4AGuQ5gMo

If no one gets injured on a $500M project, safety standards are almost certainly too stringent.

I feel like this comment is callous, but unfairly downvoted.

All projects contain risk. Getting out of bed and taking a shower involves risk, you can slip and fall. The larger the project, the larger the sample size and the more statistical considerations for risk should be managed.

Modern western society has greatly lost all tolerance of risk. One difference that struck me for risk culture, compared to the past, is old footage of the 1955 Le Mans disaster: after the catastrophic crash, bystanders themselves helped carry bodies away from the scene, and clean up wreckage. This calmness and social damage tolerance in the process of a disaster is very different from a modern "evacuate and let the professionals handle it" attitude, and I think this is an example of a clear shift in social tolerance of risk.

I wonder if the Apollo project would have worked today, instead of in the 1960s. At the scale of an infrastructure project of half a billion dollars, we're not talking about something so big that zero injuries safety standards are bad. However, as you get larger projects, it is statistically impossible to keep risk at 0; I'm not sure how willing modern society is to accept that injuries on large projects is inevitable.

Elsewhere in this thread, user JumpCrisscross comments about comparing the cost overrun to the second bid. Following up that comment, the top response is something along the lines of "nobody was injured, so showering the contractor with money is ok". This attitude is prevalent in modern society, and it doesn't really get questioned. Let's pose a hypothetical scenario: If a person has their leg broken, let's say a platform caused someone to fall and hurt their back. Would that amount of damage be worth a $250million additional cost? Think of the amount of social good you can do with $250million. At what level does society overvalue the reduction of risk? This is an important question to ask.

Let's not even talk about minor injuries. Let's say 1 person died. If you walk up to a random person, tell them that they have the opportunity to die; BUT their immediate family would receive $125 million dollars, and $125million would be donated to the charity of their choice. I suspect a very large percentage of people would actually be willing to take that offer.

And yet, giving an extra $250million to a contractor to build something is considered "worth it" to the general public if no injuries were reported. At which point does society overvalue reducing risk?

>I wonder if the Apollo project would have worked today, instead of in the 1960s.

George Mueller (who was head of NASA during the Apollo program) said that it would be impossible to repeat Apollo today without it being a classified program as Congress has passed so many laws and regulations in the meantime.

Source: https://dominiccummings.com/2017/06/12/the-unrecognised-simp...

I honestly think this is just reflection of how the value of a human life has changed in the last few decades. The thought of spending millions of dollars you don't have to save a newborn with a serious heart defect is praiseworthy. The thought of bankrupting yourself and your loved ones for a chance at another year or two on the earth is normal.

What I can't say is whether this is a good thing or a bad thing. There's a part of me that recoils at the thought of putting a price on a life. Then there's another part of me that recoils at the selfishness of bankrupting my wife should I get cancer. That same selfishness recoils at the thought of bankrupting myself should she get cancer.

I'm not looking forward to those decisions.

Risk tolerance has definitely gone down, but what’s the basis for assuming that the level of tolerance was right in the past, and today it’s too low?

Historically, the people investing in safety were not the people who were exposed to the risk. That created perverse incentives, where $1 in safety improvements that would save an average of $10 in human health or life would be rejected because the people who would pay the $1 would not be the ones paying the $10.

That’s still true, of course, but we at least attempt to make people do the right thing.

Regulators today aren’t afraid to assign a dollar value to a human life and reject safety improvements that don’t have a positive ROI. The standard value currently is something like $8 million.

I suspect that the parent wasn’t downvoted for callously making a good point. They were downvoted for baselessly assuming that the correct level of safety tolerance would result in injuries or deaths on this project, and doing so callously.

So what value would you put on someone being killed or maimed to save some money? Because that is exactly the calculation you are proposing should happen.

And that is the exact same calculation the local transportation agency makes every time their is a car accident with fatalities on a major undivided highway / road. Roads become known as dangerous (eg Blood Alley, officially the 126 is Southern California), and clearly the govt could step in and install some Jersey barriers. Yet they don't, because implicitly the cost of X lives per year is not worth it to society.

But I hope whoever does the calculations for you doesn't decide to sell safety barriers from roads because nobody died on them.

I was objecting to the idea that having nobody dying on a project being a bad thing because it implies overspending on safety.

A quite a bit less than $250million, which is how much is being paid.

In fact, people would even volunteer to die for "just" $100million. Just see this reddit thread: https://www.reddit.com/r/AskReddit/comments/4jqfka/what_woul...

This is EXACTLY what I mean by "risk management for large projects". For a small $10k project? A small percentage cost decrease for increased risk of death isn't worth it. For a multi-billion dollar project? All of the sudden, the incentives change drastically. How many people would be willing to work on a project that is a tiny amount more risky, but knowing that their families will be handsomely compensated in case anything happens? If you told me that my family will receive $100mil guaranteed if I die/get maimed, there's very little I'm not willing to do.

Also, to note: $100,000,000 is enough to save around 30,000 children from an excruciating death by malaria.

TL;DR 1 average human life generally isn't worth throwing away $250million. I will literally volunteer my own life on this fact.

> A quite a bit less than $250million, which is how much is being paid.

That idea was made up in this thread, it doesn't reflect reality.

The cost overrun was explained in the article as due to only scoping out 30% of the project as part of making the bid, and as such underestimating how much concrete they would need:

> Jeff Petersen, project manager for Kiewit, said that once construction workers got on the site they discovered they had to dig much deeper to get down to bedrock than they had expected. That meant they are having to pour almost twice as much concrete this year — 870,000 cubic yards — as the company and the Department of Water Resources had planned.

But how do you know that it is a simple trade off between money and safety standards? It's been noted on HN that the US seems particularly bad at large infrastructure projects compared to other developed countries and I'm struggling to believe that's because the US has wildly better safety standards.

I don't. That's moving the goalposts.

Who cares if Oroville Dam is in the USA or not. The same thing would still apply if it was in the UK. I'm merely saying the modern public should reconsider spending $250million extra, instead of applauding it simply because no injuries happened.

I agreed with some of your more abstract posts on this topic, but I really can't buy into that assumption that just being a little less fearful would reduce costs by half. You wave that second bid around as if they had actually done the same work, at the lower price, with a known quantity of bloodshed. But that is only a hypothetical, That second bid might have had even bigger cost overruns, we don't know. The winning bid might actually have terrible safety standards while being lucky, we don't know. There is no simple, isolated safety/cost tradeoff that you can dial in at will and get the expected results. On the contrary, a contractor that is sloppy with safety will often also be sloppy with other things, causing costs to increase, not to sink.

I agree, I'm only arguing the hypothetical.

I would highly doubt that $250mil went into slower progress for the sake of safety. Definitely, there were other costs.

You're supposed to only consider the abstract idea I presented. That's why I changed from $250mil numbers to $100mil halfway through, although I didn't make it fully clear to divorce the concrete numbers from the abstract cultural ideas.

I'm going to make a guess here and assume that you are not working construction. And if that's so I would highly recommend you join a construction crew for a couple of years and then come back here with your idiotic approach to workplace safety. It's all about preventable accidents, and your assumption that we could save 250M on a project by compromising workplace safety is both wrong and the beginning of a slippery slope that ends with maximum savings and no workplace safety at all. After all, it's not the life of you and your family that is at risk here.

Fuck those construction workers, fuck those miners and who cares about factory workers. After all they're disposable. /s

Workplace safety is there for a reason: to stop white collar criminals from putting a price on human lives and then to optimize for the lowest they can get away with.

I feel you're missing the point, I can't tell if it's deliberate.

Suppose you wanted to reduce risks for computer programmers - everyone gets a personal doctor who sits with them to monitor their vitals, state of the art monitoring device, daily blood tests, screenings for all sorts of diseases. Each person gets a fitness coach, dietician, sleep coach, whatever. Programmers with injuries get oxygen therapy and physios on hand. Basically you treat them like world class racehorses (!). Whatever, hopefully you see the point.

Would that be overkill? No? Well then you don't think programmers matter at all and want to see them all die in industrial accidents. /s

If we can accept at any point that we should, maybe not buy the most expensive chair/table/keyboard/screens, even if there's a chance of injury (eyesight, cardio problems from 20 years of sitting, cut from a sharp edge on a filling cabinet, etc.) then you're agreeing with the general concept of the OP IMO.

Now, in public procurement, should we pay $1000 per seat for a currently $100 app to ensure no programmer ever gets a thrombosis/carpal problem/etc.?

So you are saying you believe that if a project completes with no deaths or injuries then too much money was spent on it?

I'm saying that as size of project (lim -> $infinity), (probability death or injury) approaches 1.

To be willing to throw your own life away for $250M is one thing, but to impose that choice on others, so that the government or some large contractor company gets to save money, smells of an utilitarian view of society and is frankly immoral.

I love it when deskjockeys start putting a price on other people's lives.

Construction is dangerous, if you are willing to compromise on that then you'll see that you can do even better when you drop that price. And before you know it we're back in the 1930's again.

The simple fact is that there is a trade-off between the effective safety measures put in place, and the costs of doing so. And at some point you cross a line between reasonable and unreasonable expense. Dollars may be imaginary, but they trade for real resources, and those resources exchange for opportunity costs elsewhere, including lives that might be saved or improved by other routes.

I'll but the question back to you: how much are you willing to pay to save one additional life? $1 million? $1 billion? $1 trillion? $100 trillion? (That last is more than total gross world production.)

I've got my own considerable problems with numerous elements of evaluating costs and benefits in economics, many of which I've been trying to articulate for over three decades and am only just now starting to get a handle on. I would not argue, for example, that just straight earning potential is an adequate basis. I also don't think that various proposed "market bid" rates for how much individuals are willing to accept under very specific and limited conditions are particularly useful.

But even accounting for community, social, and disruption losses, somewhere between tens and hundreds of millions is likely a reasonable balance.

That's fine as long as the ones deciding on the balance have to take a large share of the risk of injury. Be the first to work in the same conditions you expose others to. Otherwise, you're just a hypocrite and a monster.

Your rhetoric doesn't match the world.

Part of society involves distributing decisionmaking and risks. And no, not always equally. I don't find the argument that those most adept at accurately and fairly assessing risks are necessarily those who can accomplish the tasks involved.

To put this in a different field: I'd not expect my airline pilot to be the best qualified accident investigator, nor systems engineer, nor aircraft fabricator. And I as passenger bear the consequence of their combined work.

Reality bekons, if you care to heed its call.

'No taxation without representation' is not based on the presumption that those being taxed are the best at determining the value of taxation, or even of electing their own representatives. Those who do not bear the cost of their own decisions are a priori unqualified to make those decisions as they are subject to adverse incentives.

Well, yeah. They can opt to not pay the "risk tax", by simply not going down that career path. Not every construction job is rebuilding a high risk dam spillway. Not every aerospace engineer job involves stepping into a 100% oxygen flammable Apollo rocket chamber.

The understanding is that 1. society should be allowed to consider increased risk, and 2. in the event something bad happens, the victims should be compensated.

Literally nothing I said stated about society and risk stated that dangerous jobs are only for low level construction workers. You're just using silly ad hominem ("desk jockey") and emotional appeals (poor vs rich!) without considering the point.

In fact, one of my examples of risk was the Apollo project, where the most notable 3 casualties (Apollo 1's Virgil Grissom, Edward White II, and Roger Chaffee) were highly paid astronauts with engineering backgrounds. I'm not discriminating against construction workers, I'm making a statement on acceptable risk in society in general.

Finding a balance on putting price on human life is simply a necessity. I have a friend who's a petrol engineer working dangerous jobs at a high salary, and it's a risk he's willing to take. However, the modern general public is culturally shifting to people like you who literally hear the words "possible risk" and the frontal cortex of the brain shuts down and start ignoring the costs.

You've put an interesting perspective on the safety/cost balance. I find it ironic that the replier is effectively demonstrating your point: people have an emotional attachment to safety, but not opportunity cost, regardless of the math. This is perhaps a form of "one death is a tragedy, ten thousand deaths are a statistic."

Considering that even 100 years ago, such a project would have figuratively (and sometimes literally) mixed a good bit of blood into the mortar, yeah.

Having worked a little bit of industrial construction in a past life, it's amazing anything gets done at all, if you do follow all the rules. And even then all it takes is one slipup to get somebody hurt.

In that case, what's the "correct" number of deaths that safety standards should target per $1M spent?

The statistical value of a human life in the US is ~$10M.

https://en.wikipedia.org/wiki/Value_of_life

Serious injuries are some fraction of that, e.g., ~$1M for a limb. The correct tradeoff between safety and cost obviously depends on the marginal cost to increase safety, which is known to the project leaders but not us, but I'd be surprised if the optimal number wasn't between 5% and 25% of total cost. 5% of $500M is $25M, so for a project of this size I'd expect, as a lower bound, at least one death and a dozen serious injuries.

To be clear: by not making this trade off and by spending huge amounts of money to ensure such tiny injury rates, we are wasting resources that could be spent on saving lives at the rate of $10M/life. Indeed, this number is the key threshold by which the EPA decides which environmental interventions to fund, or the NHTSA decides which traffic safety measure to take.

Are you volunteering?

> Kiewit, the Nebraska-based construction firm that has the main contract to rebuild the main spillway and emergency spillway at Oroville, the nation’s tallest dam, estimated in its winning bid in April that the work would cost at least $275 million. But the price tag has now grown to at least $500 million

Out of curiosity, what was the runner-up bid? Is there no mechanism to re-open bidding when the circumstances change? What prevents me from bidding $1 and then busting it to $1 billion after winning?

I'm not saying the economic aspect is very good, but I would make 2 points: they are more or less on time, and nobody got hurt. That probably desserve a shower of money.

Those people were using the biggest dry ice making machine in the world to try to pour not too shitty a concrete on schedule, next to a blasting zone, under a leaking gate (yeah, the gate is leaking it just stopped leaking because the water is lower than the gate now), under high voltage power lines, without anyone getting hurt.

Those are good points, the Crystal Springs Dam earthquake retrofit in San Mateo County is almost fifteen years into the scheduled five year earthquake retrofit. :)

I thought the dam retrofit itself was done and the remaining work is just road work.

You are right but the work is ten years overdue from the first estimate on the first sign they put up next to the Crystal Springs trailhead where the road is closed to traffic. Eventually they covered up the last two digits on the year number sign last time I checked with replaceable numbers like the days without accident sign in the nuclear power plant that is Homer Simpson's workplace.

I find it interesting the default commentary to this sort of event - ballooning budgets on underbid projects - assumes that bad faith on the part of the bidder is the problem, and that if only we could punish these few bad apples - punish those scheming, underbidding contractors, then the problem would disappear.

It seems unlikely that bad faith is the primary cause. Yes, there are really contractors and owners who do act in bad faith, like the Trump organization, but these are unusual, and highly publicized. Punishing dishonesty seems like the easiest problem to solve, and it's easy to rally around for a quick and cheap victory. But I have seen these run-up bids for decades and no amount of punishment has ever seen to stem the flow of these events; it seems as if every city and state is infected with this problem. Either we are in a system where bad apples are so numerous that punishing a few has no meaningful effect on incentives; or we are in a system where bad apples are so rare that we punish good apples mistakenly, and can't figure out the real cause for our poor planning. Until we move on from the assumption that a few bad apples are to blame, we likely won't be able to discover the real underlying problem.

There's an interesting game theory problem here. If you stick the bidder with the cost of overruns, they will bid high -- and be overpaid if the risks don't eventuate.

So, the consensus seems to be Governments would rather get low bids that don't price in unknowns. If unknowns happen, they can dicker over the increase. But if you award a contract that's too fat, it's difficult to claw back.

If the work goes exactly to plan, the contract will come in on budget. But if problems occur, the price usually goes up. Our current system seems to be optimized for lower overall project cost at completion, with the side effect of making it less useful as a budgeting tool.

The bids aren't in bad faith. They are just over-rationalized and under-justified until the number is small enough to be in the winnable range. Middle Management talks out of both sides of their mouth, demanding lower estimates, while demanding integrity of results (wink wink).

They don't really want integrity, they want to get a number that won't get them yelled at by their boss. The integrity bit is a pass the buck clause.

So the estimator is in the position of trying as hard as they can to lower the estimate. "Well, if we assume xyz unreasonable assumption, then we can lower the amount by this amount."

And then there are "risk" shenanigans. You itemize all the risk items, declare an arbitrary probability of happening. Then management rolls those up, and declares that they are not going to fully fund the risk and instead "carry" it forward.

Fully funding basically would mean multiplying the likelihood of occurring by the cost, and adding all the items up. Which is already iffy as the probabilities aren't based on much except a w.a.g.

So you win the bid, the risks are realized, and weren't funded. blow out.

Standard practice. And because everyone does it, the honest bidder is the loser. Losers go out of business. So the whole bidding practice is self selecting for cheats.

If you say, "this is unethical, I'm going to do my best to give a realistic bid", you lose.

Should be easy to fix, if the contract for work includes a "penalty for being wrong for circumstances completely within the bidders control that they should have seen".

Austin government did this for its MoPac toll express lanes and got them for way cheaper than what they should have paid: https://www.bizjournals.com/austin/news/2017/09/27/big-settl...

However, the project ran over for two years longer than it should have (it was supposed to be completed in September 2015, and they are only just now wrapping it up). So I guess no one really won there aside from the Austin transportation budget guys, except the populace hates them now for traffic jamming the place up for two years longer than they said they would.

Isn't completion something that is at least in part in the bidders control. Did they employ all available personnel and work 24 hours?

Sort of, when they could. In this case there were penalties for going longer on the project. Main problems ended up being weather (heavy rains) and lack of knowledge of the soil - many changes were made to the project because the contractor (C2HM) did not do a proper soil survey where they were going to be digging.

It is definitely a system issue. Our contractual/financial system just doesn't work well when confronted with large complex projects that have enough variance and unknown unknowns. Since they really push to have everything committed to up front, with really basic rules around how to handle change and variance.

Let alone that we basically award this work one estimated number and the ability to get through a paper work hurdle.

If you start making those rules complex it becomes hard to compare bids to each other, since your bids are likely to become non-transitive based on the scenarios.

Cost plus is the normal way this is side stepped, but unless you have a system that is mostly ethical actors that can break down. And if you have all ethical actors pretty much any system will work...

Cost plus percentage gives a very clear incentive to raise costs; that's the only way you can get more profit. If it's so widespread that might be one reason why all our infrastructure is so much more expensive.

Why not just have major contractors required to post a bond when they place a bid, and if they have cost overruns they pull those from the bond (ie, their own money)?

> Cost plus percentage gives a very clear incentive to raise costs; that's the only way you can get more profit

Case in point: SpaceX challenging the launch industry's cost-plus culture with publicly-posted fixed prices [1].

[1] http://www.spacex.com/about/capabilities

Case in point #2: the new Bay Bridge.

Case in point #0: the Space Shuttle (STS).

Sorry, you're citing the Space Shuttle [1] as a successful public procurement process?

[1] https://www.nasa.gov/pdf/382045main_19%20-%2020090730.11.STS...

No, my cites relate to the parent comment:

  Cost plus percentage gives a very clear incentive to raise costs

Compare the original winning bid for the STS with the actual costs; I think they rose more than tenfold (and the per-flight cost rose more than that), IIRC.

And had the competing team won, the Challenger disaster never would have happened, because the solid-rocket boosters would have been manufactured as units and transported to the Cape by barge rather than having to be broken into small enough components to fit on rail cars.

Oh for sure, I should probably add a non-profit driven caveat to the actors in the system.

I am not sure if you can implement a system that would work well when your actors are profit driven work that needs to be done can only be done exclusively by one entity at a time, and as a bonus that extra profit gives you more feedback into how the system operates. There aren't a lot of natural negative feedback to keep things under control in that system.

Especially if your goals are to reduce overhead.

> if only we could punish these few bad apples - punish those scheming, underbidding contractors, then the problem would disappear

I'm criticizing a system that incentives optimistic initial bids that are later revised up. A bad system doesn't need bad actors to produce bad outcomes. A $275 million bid ballooning to $500 million without any competitive pressure is a bad outcome.

American infrastructure costs are higher than production factors, e.g. land, material and labour cost differentials, explain [1]. Our auction mechanics are called out by experts as a big part of the problem.

[1] http://marroninstitute.nyu.edu/content/blog/is-u.s.-infrastr...

How do you specify the contract so the unknown unknowns can be priced into the bid and compared to each other. Given that resolving those unknowns costs about as much as the just doing the actual work.

You temporarily re-open bidding when big price changes are requested. See my other comment [1].

[1] https://news.ycombinator.com/item?id=15524807

That sounds like a recipe for delays and finger pointing along with massive overruns on the later parts "to fix what those other guys didn't do our way."

If you want separate bidding for separate phases, bid it out that way.

I suspect this would just make all projects much more expensive - bidding for a large project costs a lot of money and can require large teams of people working for extended periods of time. All of that gets paid for out of the projects that are won.

If you now introduce scenarios where projects can have an unknown number of bidding processes all costs will go up to cover the work necessary for multiple companies to prepare bids.

It would just make things much much worse.

Even better: the more rules and processes you put in place to try to avoid/punish this, the harder it is to get the contract in the first place. Eventually, most companies decide it's not worth jumping through all the hoops you've put in place to avoid corruption, and you end up with a handful of entities which get every contract and face no meaningful competition to try to keep the bids down.

This is already a problem. A company I worked for submitted a bid for an IT infrastructure project several years ago. We were rejected for not having completed a similar project for a government agency before.

Our bid was 1/3rd of the "winning" bid.

Thank you for saying this. I think there is a deeply distrustful aspect to how the public views these contracts, probably because a few bad actors (as you noted) have in fact screwed the people from time to time. Also people tend to associate these bid contracts with politicians in power who they may dislike for one reason or another therefore further tainting their opinion of the bid system.

The distrust of government and corporations in our society is one of the biggest problems facing us as we go forward in the 21st century. We need to figure out how to restore trust.

Problem is easy to identify - We are using waterfall for massive scale urban projects, and getting the known associated problems. Add to that massive corruption and cronyism.

The fix is not clear. Can't just drop in agile, every attempt to legislate away corruption fails.

>What prevents me from bidding $1 and then busting it to $1 billion after winning?

Isn't this why most publicly financed projects have massive cost overruns? Bidding realistically will lose you the contract; bidding low and then running overbudget may have consequences, but they are not as severe as not getting the contract in the first place.

This doesn't happen nearly as much outside the US.

Are you posting that from inside the USA or from outside of them?

I guess what I'm actually asking is, which country is that promised land? (I have a few candidates in mind myself, but I consider them to be the exception rather than the norm and I don't even know if they are truly as good as I suspect)

Most countries have much cheaper subways. None are worse then the US.

That isn't going to solve this overrun. They got on site, dug down and discovered that they were going to need a lot more concrete.

When they said "we need $275 million," we used competitive pressure to validate that claim. They had the lowest bid and so fairly won. Now they're saying "we need $225 million more." Yet we demand no market validation.

Note: I'm not saying the $225 million is unfair. This is a complicated project with lots of unknowns. But refusing to test 9-figure quote adjustments is bizarre.

They bid on a job that was only 30% specified. The scope of work eventually doubled when the site was fully analyzed and the price tag almost doubled. Nothing to see here.

There is work that is 100% documented in the proposal and in those cases it’s reasonable to hold the winning bidder’s feet to the fire, but this isn’t one of those.

In the absence of time pressure they could have let a contract to fully specify the work, then one to do it. But if they’d done that there might not have been a dam by the time work started.

> Nothing to see here....In the absence of time pressure they could have let a contract to fully specify the work, then one to do it.

I agree that in this specific case, had the contractor come back and said "it will be $1 billion more," it would have been worth paying to avoid the destruction of the Central Valley.

The broader discussion is that this is avoidable. Two models abound in the private sector. For less time-sensitive situations, one has changeover processes [1]. For time-sensitive work, companies avoid like death being totally dependent on a single company.

The latter would require large changes in the way contracting is done in America. But it's not like nobody else in the world faces time-sensitive repairs. The defensiveness we, as Americans, have towards our system which shows clear and apparent pricing flaws is surprising.

[1] https://news.ycombinator.com/item?id=15524807

The standard contracts handle this pretty well; it's not a surprise that surprises happen.

In commercial projects, this often triggers an arbitration about the overrun. You get experts to challenge the fairness of the price, hopefully using market data. And, of course, there are the courts. In worst case, the buyer can cancel and/or rebid. Not 'competitive,' but adversarial.

The bidding process is for sure a very problematic one for every project. But in this case, I assume that the bidding wasn't a fixed package, but rather a price per certain action, e.g. for a ton of RCC poured etc. As the bidding happened before the plans were finalized, that would be the only honest way to project the costs. And as we know, the work executed was way more extensive then assumed during the bidding process. There was much more RCC used - they couldn't know where exactly the bedrock was before they dug it out, and more of the upper spillway replaced with a final solution. So some of the costs which would have occurred next year have been part of this years budget. Considering those changes, it is no surprise, that the costs were overrun, actually the overrun is reasonably modest compared to other projects where there was time for a detailed planning.

Typical for California, and in fact experience tells me to doubt that this will be the last increase.

http://www.wnyc.org/story/316201-brief-history-64-billion-ba...

On time at only double the original cost is a bargain by Californian standards.

As you say, better to wait till its done before getting too excited.

Pretty common strategy in bidding. Undercut the second bidder to get the contract, then run over budget to make a profit.

The problem as I see it is you don't have to actively run it over budget in bad faith.

Even as an ethical actor you can only price what is specified. However, you can then use your knowledge about likely the real variances that will be needed and risk profile to place a bid that takes this into account to remove margin or discount the bid price.

How would you re-open bidding and enact a switchover? They're already time crunched. There would be big costs in money and time from hauling out all the existing equipment from the first bidder, and bringing in the new staff and equipment from the new bidder.

> How would you re-open bidding and enact a switchover?

When the price change is announced, anyone from the original pool has [30] days to match. First to match immediately wins the work from the existing contractor. Loser pays to move their equipment out; winner should have included that cost when they agreed to match.

This system works for two reasons. First, the first bidder will be cautious about requesting increases if it means risking losing the work. Second, the change-over cost, borne by both the outgoing loser and incoming winner, imposes just enough friction to grant the first winner an incumbency bias. This bias helps keep bidding in the original auction competitive.

That just wouldn't work for large civil projects for so many reasons.

It also seems like you are only trying to address the problem of people introducing needless variances as a way increasing profit.

Which is not the case being discussed here and I don't think it is major reason for cost overruns. They haven't requested an increase, they have found out that to satisfy the engineering requirements of the project that more work has to be done.

> Which...I don't think it is major reason for cost overruns

It's commonly called out as one of the driving reasons behind the horrible economics of American civil engineering [1].

> They haven't requested an increase, they have found out that to satisfy the engineering requirements of the project that more work has to be done

It's both. The change may be merited. If so, it should win--again--in a temporary re-auction. But refusing to market test an 82% cost increase is game theoretically begging to get screwed on pricing. Given American taxpayers get screwed on pricing for public-sector civil engineering (after accounting for land, labour and materials cost differences), I think it's a fair discussion to have.

[1] http://marroninstitute.nyu.edu/content/blog/is-u.s.-infrastr...

It is most definitely a fair discussion to have, and the bidding/financing/contractual system for these projects really does need to change to have better incentives.

I just don't think your incentives would produce better outcomes, from my perspective it would make things worse.

I will make no attempt to defend the specific recommendations I mooted. There is better work, in any case, in the academic domain on fixing our flawed civil-engineering auction processes.

The problems anyone campaigning for auction reform runs into are three-fold. One, it's a boring problem with boring solutions. Two, there are vested interests. And three, Americans suffer from a just-world bias when it comes to infrastructure [1]. The latter is apparent even in this thread. It's difficult to find solutions when people vigorously defend a clearly-flawed system.

[1] https://en.wikipedia.org/wiki/Just-world_hypothesis

For sure. I don't know how you overcome the inertia of our societies structure to head towards something that would likely be better for the majority of citizens. Which is a problem in many more domains that just civil works auctions.

I can see how the recommendation you mooted would improve incentives in some procurement domains. Large civil works isn't one of them however.

This demonstrates a total and complete lack of understanding of how large scale construction projects like this work.

You really believe the main issue is "getting the old equipment out" and "the new equipment in"?

Your plan sounds like an amazing way to create a tremendous amount of safety risk, shoddy work, or a project that runs forever.

> You really believe the main issue is "getting the old equipment out" and "the new equipment in"?

No, I was responding to a specific complaint made upthread.

> Your plan sounds like an amazing way to create a tremendous amount of safety risk, shoddy work, or a project that runs forever

Do you have any basis for this? The change-over requirement for midstream cost adjustments (above a threshold) works well enough for e.g. aerospace subcontractors. It adds delays in the short-term but promotes better behavior in the long.

> It adds delays in the short-term

A delay in this project is a safety risk. They are trying to have a working spillway by Nov 1, so if unexpected rains cause another extreme inrush into the lake they can handle it by releasing >100,000 cfs from the spillway. Without that ability, they are risking failure of the emergency spillway and the flooding[1][2] of a large part of the central valley.

[1] https://www.youtube.com/watch?v=yVcGUvE3BBc

[2] https://www.youtube.com/watch?v=4p0Hc56vJ2Y

You're correct in observing that change-over processes are less-frequently implemented for time-sensitive work. I was making a broader point around our public-sector low-bid auction process.

For this type of work, the best process would avoid appointing a single contractor. Unfortunately, that would also require more expertise in the public officials allocating work, a concentration we find politically toxic in America.

For the people interested the blancolirio YouTube channel does an incredible job reporting on the dam.

https://www.youtube.com/user/blancolirio

Highly recommended. He is much much better than any news channel / newspaper article you'll read about the topic. He's pretty awesome.

He keeps referencing a "plunge pool" half way down the new slipway. Does anyone happen to know what the purpose of the plunge pool is?

I think it's the natural depression that eroded where the water first comes off the top half of the chute.

I watched everything and I confirm, it’s what was the crack, and it doesn’t exist anymore, it’s filled in. At about the same place the are putting an aeration feature to mix air in the water and make it less destructive to the concrete downstream.

It's simply where the water from the broken spillway plunged down into the hillside.

They talk about two plunge pools (upper and lower), basically two different depressions in the stable bedrock. They do not have a purpose, but are rather just the result of the disaster.

I'll second that. I've been checking in on his videos every once in a while ever since the original Oroville dam crisis.

The total construction cost of the Oroville dam, ending in 1968 was $480 million. Inflation-adjusted, that's $3.5 billion in 2017. $500 million just to repair the spillway seems excessive...

See table 8 in http://www.water.ca.gov/swpao/docs/bulletins/bulletin132/Bul...

Yet there's no way you'd ever build Oroville today for 3.5bn.

It seems inflation in large scale civil engineering has gone up at least a factor of ten more than general inflation.

Modern safety standards in construction are much more stringent than late-60s safety standards in construction.

And modern requirement for additional administrative roles and salaries, both directly and indirectly linked to that business, have probably more to do with inflation...

My folks live in the nearby area not downstream. That’s what happens when preventative maintenance is ignored by reactive bureaucrats. Pennywise, ...

Here’s one of the YT channels that’s been following the progress, filming and combining DWR footage. https://www.youtube.com/user/blancolirio

I feel like this thread is ignoring the main point, which is that preventative maintenance would have been cheaper.

Also, when this thing started falling, they could have stopped it earlier, which would have resulted in cheaper repair, but no...

Sure we would not have awesome drone footages, and I like the discussions on worker's risk and contractor down-biding, but at the end of the day it's a lot of money for something preventable. Same can be said of worker's riks, health care, environment, retirement.

All in all, pretty impressive engineering project, correcting engineering defects dating back more than 50 years.

500 million? Wow. Can't they just lay down a big pipe and decrease the water level that way? Seems to me like just about any solution would be cheaper than 500 million...

The spillway is a sort of a big half-pipe.

Largest syphon tube in the World?

Lots of patterns here similar to large IT overruns.

They should do an ICO to raise money to fix it. Citizens shall pay for their water in ETH to the token holders. Kidding, but in 20 years I might not be.

cost of not repairing is still higher

Well... obviously... otherwise why would they be doing it?

Yes. However don't let that kind of thing open for exploit. People tend to like these kinds of disasters as opportunity to make hefty profits.

And of course the cost of proper maintenance would have been much less.

Why is this on HN?

Whenever non-tech posts are put up on HN, they're ruthlessly hounded. What does this article have to do with tech that absolves it? The fact that it comes from San Jose's newspaper?

It is a tech related post, especially an engineering related process. And beyond the importance of the Oroville dam for California, the repairs are an interesting case in project management. They have a fix deadline (Nov. 1) and not meeting the deadline was not an option. And it looks like they are making the deadline, what an achievement!

That's quite a stretch, though, don't you realize?

The H in HN stands for "hacker". I fail to see the connection to any "hacking" here in the article other than maybe "hacking bureaucracy to make a dam", but you must see the incredible lengths to which one must stretch the notion of hacker-related ideas to apply to this article.

There are other non-SWE-related topics that certainly qualify, like overcoming natural barriers in engineering a la the Iceland forest article that was posted a couple days ago.

This news article in OP is little more than a status update about something that affects California's water infrastructure and nothing else.

Non-tech posts aren't ruthlessly hounded. There are many non-tech posts and the intellectually interesting ones are warmly welcome. People who think otherwise need to read the HN guidelines—the mandate of the site is in the first paragraph:

https://news.ycombinator.com/newsguidelines.html

Reply on news.ycombinator.com