The Danger of Popcorn Polymer: Incident at the TPC Group Chemical Plant [video] | Hacker News Reader

The Danger of Popcorn Polymer: Incident at the TPC Group Chemical Plant [video]

youtube.com

191 pointsoatmeal183 comments

Hey a story from the USCSB!

Fun anecdote the company I worked a few years at was investigated and audited by them. We were handling a lot of polyisobutylene type chemicals and had a few small fires and near misses in the plant. (Its used in diesel detergents.). We reached out to them before anything serious went wrong based on our environmental engineers recommendation

Anywho the uscsb did a spectacular job writing up a whole book of protocols and recommendations that made the plant safer and quieter. They even helped coordinate annual response events from the local PD and FD and helped with our annual report to the city.

Stuff like that is really heartwarming… org identifies they aren’t safe enough and reaches out to an agency with experience in the area, then the agency coming in and helping make stuff safer instead of just writing a bunch of stuff up, charging some fines, and then leaving.

Building inspection is actually like this in my city. If they get a tip about unpermitted work, they just bring the permit application, will waive a few fees so it costs $7.50, and they electronically send it to the office wheee someone approves it immediately. This also enables the owner-builder to request they come by for a pre inspection or for advice. They actually want to keep buildings safe and most of the inspectors were formerly working in the construction industry and hated working with bad inspectors.

That needed some serious political effort to pull off. In many places, fine and fee income is a fixed part of the budget, so it incentivises everyone to optimize for "most amount of fines" instead for "best experience for builders and public safety".

This mentality sounds similar to Peels principles of policing. Get people on side and work to keep them there, ignore minor transgressions when there is a larger goal etc. The devil is in the detail though and achieving compliance is far from easy.

https://en.m.wikipedia.org/wiki/Peelian_principles

It is my understanding that the get most of their revenue (>95%) from fees charged to larger contractors and developers who all have access to a self serve portal (you just need a contractor license # which is not hard to get). As they build up a history with the city, they do not require as many inspections throughout the construction process (but they are on the hook to fix it if anything is found after the fact). The bare minimum for new construction would be the slab pour or pier/beam foundation setup, electrical, plumbing, framing, and final. For a reno, it’s just final.

For owner-builders there was a series of homeowners insurance claim denials in the 1990s and 2000s because unpermitted work was observed by the adjuster. This resulted in the former occupant simply abandoned the house, leaving a subpar structure, contributing to blight, and lower property values. The city building department made very easy and inexpensive for these people and to get a permit. For minor work, $30 out the door and for major work $87 (assuming you don’t get the discount). This fee includes two free inspections (pre and final) from professionals at the building department, usually same day.

I bought a fixer-upper in 2017, did a full rewire, repiped water supply lines, gas to electric range conversion, trenched 30ft to my detached garage (for 240V/125A power, data, and water), and installed a mini split system. They got to know me well and just instructed me to do a video walk through as they did not have any problems with my work prior. This was only possible because they care more about public safety than fines.

One theory I have is that policy dictates that they cannot issue a fine right away, they must leave a notice of some sort that they owner must respond to (usually by getting a permit, sometimes by arguing it’s not required) or will face a fine. They just cut out the junk in between by helping them the permit right there, which ends up being a good enough experience that most people just get the permit next time.

I wish there were good companies like that in software, but honestly it feels like it's a lot of consultants that only add more workload and headache over the span of years, or ones that just throw some recommendations from books like Clean Code or Google's SWE book or worse, SIG your way.

Mind you, I'm a consultant myself so in many ways it feels like we're the ones that are supposed to do that kinda thing. Supposed to, in practice we put down overcomplicated solutions that the company in question cannot do on their own, locking them into consultancy / freelancers just to keep things afloat.

I’m a consultant, but I’m confused by what you said. What do you mean “put down”?

Fwiw, generally speaking we try to train and mentor our clients so that they don’t need to depend on third party contract work.

I don't blame consultants for this. I blame clients. I can't even count the number of times I've brought solutions to clients that were 'too expensive' for them, only to be asked to make it less expensive... and less expensive. Until the solution is so cheap, it's more expensive than the original solution (Tech Debt).

I generally show two costs for any multi-solution estimate these days:

1. The upfront cost (in time)

2. The maintenance cost (in +/-% of all maintenance)

I generally inform my clients they want to keep maintenance costs <100% (meaning adding any new features requires refactoring due to tech-debt). Most clients generally want to keep it around 20% (meaning 1 out of 5 tickets requires refactoring code before implementation) but are willing to go up to 50% for a strong push.

I also provide a dashboard where they can see the "maintenance cost" in real-time (essentially refactor + bugfix PRs divided by feature PRs where a divide by 0 is 100% -- aka, maintenance mode).

All of the videos on that channel are great - it’s one of my favourites on YouTube. Some of them show horrible accidents ultimately caused by extremely silly process issues (including nobody being responsible for cleaning in a factory, or failure to label the input pipes for a factory, or everybody in charge of monitoring a system skipping out on work).

Also, it’s impressive how dramatic the quality increase is between the old videos and the new ones.

Or in this case, failure to flush dead leg pipes where popcorn polymer was known to form.

> extremely silly process issues

None of these were silly. It takes an extraordinary amount of hard work and collective intelligence to fix these issues.

> failure to label the input pipes for a factory

Labels are not the end solution, no one reads labels unless they want something, anyone in IT should know this. (It's a great dark pattern trick)

That video was a great example video of "Murphy's law" the engineer principle, not the joke. The problem is the solution, different attachments for different fluids goes against the safety of standardization.

I've seen USCSB give advice I would consider incorrect or in question, they suggested locking compound gates. You can be directed to leave them unlocked to allow emergency evacuation in person or by car.

I don't know what's better there, but USCSB needed to also address evacuation safety if they make statements like that.

This stuff is hard and has to be applied to accidents that have not happened. It's a 'Bullshit jobs' mega-machine.

[edit] I have the say the 30 second intro to the "Explosion at the Husky Superior Refinery" https://www.youtube.com/watch?v=sFhkzK7jkKg gave me goosebumps.

The brain finds amusement when it detects a discrepancy between expectation and result.

https://youtu.be/Tflm9mttAAI

I think this one especially is simple enough of an error, and the consequences are so disastrous, that it’s reasonable to find it silly.

> Labels are not the end solution, no one reads labels unless they want something, anyone in IT should know this. (It's a great dark pattern trick)

That's pretty solvable in IT, just assign full responsibility for fixing the problem to whoever messes up via ignoring the labels.

Hard when ignoring the labels kills them before they can be punitively assigned the task I guess.

Hence why I said "in IT"? Did you misread my comment?

Obviously in a chemical plant this would be futile in a lot of scenarios. And so much more complex proactive measures are the norm.

I guess I did! I thought you were saying it as a way of recommending it outside of IT.

The USCSB didn't just recommend labels in that case (though that likely would have prevented the incident since it was caused by someone unfamiliar with the facility who might have read them). They also recommended making sure the connectors for the inlets were different so it would be physically impossible for the error to occur.

That said, it's quite silly indeed that the inlets for two incompatible chemicals that would release toxic fumes if mixed were 18 inches apart, unlabeled, used the exact same connector, and were locked behind the same gate. I don't think it takes extraordinary collective intelligence to see how this could become a problem.

The report for anyone who wants to know more: https://youtu.be/Tflm9mttAAI

I don't recall the compound gate thing but I'd appreciate a link if you have one.

> mixed were 18 inches apart, unlabeled, used the exact same connector.

They were labeled.

Same connector - Standardized connectors are safer. Every chemical can't have a different connecter. If the truck swaps connectors in and out per location you are mostly back to square one.

The connectors were locked, that's the differential they choose. Except one wasn't, which is the same issue as the truck swapping connectors. It shouldn't have been possible to have two connectors unlocked at once.

> were locked behind the same gate.

Multiple gates creates other issues.

> 18 inches apart

This the problem, multiple gates a distance apart are an issue, but probably the solution. It also allows clear signage on the gate rather than than the 7 characters you can see on the connector label. I count 5 - 7 connections, they can't all be behind a gate, so there is no hard rule.

> compound gate thing

https://www.youtube.com/watch?v=jh2HWT8gPeY

They actually want a 'written' policy to stop people coming in. People ram raid gates to get into compounds so they can't be flimsy. In an emergency power could be out and you can be in a car or running while on fire.

> They were labeled

Were they? It says they had labels some distance down the pipe, but they weren't labeled at the point where the hoses connected where someone could actually read it.

> Every chemical can't have a different connecter.

Every chemical doesn't have to have a different connector. Maybe just ones that will cause a dangerous reaction if mixed.

> I count 5 - 7 connections, they can't all be behind a gate, so there is no hard rule.

Sure. Again, a good rule of thumb would be to separate the ones that could cause rapid death.

> They actually want a 'written' policy to stop people coming in.

So no fire code issue then.

> Every chemical doesn't have to have a different connector. Maybe just ones that will cause a dangerous reaction if mixed.

The issue is that they're buying chemicals from vendors so they don't necessarily have control of what connectors are on the truck. If you have Bleach in connector A and acid in connector B, you better hope they don't send an acid truck with an A connector.

I think having a way to close the fill lines remotely would have at least seriously downgraded the problem.

I doubt every chemical facility uses the same style connector, so at some point in time this information would have had to be communicated to the chemical supplier anyway. And sure, the supplier could just happen to send the truck with the exact right connector for the other chemical (though that wouldn't be a problem if they just moved the inlets apart), but that's a lot less likely than a screw up when the connectors are the same. I'm not saying (and I doubt the CSB is either) that any of this would eliminate the possibility of incidents. The point is that they'd become much less likely.

The inlet valves had separate locks on them, and presumably the plant worker was supposed to unlock the correct one for loading - but on that day both were left open for some reason.

Yup, so I guess I'd add that to the list - careless management of the padlocks for the valves, in addition to the gate, labeling, connector, and placement issues.

Oh my god they locked away their safety equipment too?

The one where a box factory blew up because of a chemical safety issue was a real eye opener. Who knew that if you make cardboard boxes, you have to be aware that sometimes municipal water has too much oxygen in it and it will cause your machinery to randomly blow up, destroy the neighborhood, and kill 4 people one day? I had no idea.

https://www.youtube.com/watch?v=rGVbGNdx7g0

That’s a pretty charitable summary of the issue of pressure vessel safety. I’m just a hobbyist with tools in my garage and I’ve avoided buying an air compressor for air powered tools because I don’t want to have a pressure vessel to maintain. If I know this, industrial operations definitely should.

Sure, pressure vessel rusts, pressure vessel explodes is pretty understandable, but it's surprising that you need a pressure vessel to make boxes.

Eh, they needed steam to make the cardboard. Sounds reasonable.

A lot of the risk scales with size. I have a 2.5 gallon air compressor that does everything I need. Even if it has a blow out, it’s just not large enough to be risky.

You're obviously right that the risk scales with size, but I'm still suspicious of pressure cookers. Yeah, it won't destroy the house, but I could still be seriously injured by it if the pressure release valve was clogged and I wasn't paying close enough attention.

Pressure cookers are a bit different. They have a large movable opening that's the weak point. When that fails, it tends to send the lid sailing (with 15ps across the diameter).

Air compressors seem to end up with cracks. They'll release air, but don't have the tendency to create a projectile.

> I’ve avoided buying an air compressor for air powered tools because I don’t want to have a pressure vessel to maintain.

You shouldn't worry about that at all. My fathers old machine shop ran on an air compressor manufactured in 1948 and the tank never ruptured in its 50+ years of service. Never mind the myriad of little compressors who's tanks are all that's left after the cheap compressor gave up the ghost (they make nice portable tanks or buffers or toss another compressor on).

Air pressure is bad enough, but any time you've got water above 100c and/or 1 atmosphere of pressure, you're at risk of a steam explosion, which is far, far deadlier. Water expands to about 1800 times its volume in the process of flashing off to steam, and it can displace all the air in a room, leading to people inhaling steam, and being burned from the inside.

Steam explosions are terrifying. Boilers are extremely hazardous things to operate without proper training and supervision.

You may really enjoy this book[1]. Very short chapters with pithy yet conversational analysis of a wide variety of disasters, including things like "pumps were labeled perfectly clearly, but due to one being installed at a later date and other being removed, they were numbered in a line going 1, 2, 3, 5, 4, 6".

0: Earlier edition of same book with awesome cover to get people to understand it's not a super boring textbook at all (in fact it's really not a textbook): http://libgen.rs/book/index.php?md5=83EEA4069C89A895711DEFAD...

1: Latest edition: http://libgen.rs/book/index.php?md5=DC1C71A6B692001AA8FAFACA...

Amazing videos that make me go ARGH. I remember when a silicone plant blew up in the NW Chicago suburbs, and sure enough they had a video, ready to make me facepalm: https://www.youtube.com/watch?v=8j8EprZP4IE

I would gladly pay federal taxes if more government safety agencies put out high quality streaming content like the USCSB!

That said, there is a facepalm moment in each video when you realize the agency has no regulatory authority and can only make recommendations that may or may not translate into voluntary compliance actions.

It's powerful to have investigative powers not be hard-tied to regulatory powers (see NTSB / FAA... in fact, I believe that was the actual model chosen for USCSB).

I'm totally not in the field, so I have no idea how often OSHA/EPA follow up on USCSB recommendations, and if OSHA and EPA's remit are sufficient to be able to enforce/mandate the majority of USCSB recommendations.

For example in the video USCSB identifies that TPC had internally recommended and green-lit procedures for flushing out of use lines to prevent popcorn polymer formation, but did not actually implement the action.

I know in medical device land, this type of thing (say someone internally recognizes a risk to device safety in the manufacturing process, a mitigation is accepted and greenlit, but never actually implemented) would be subject to both quality audits (so by someone checking up on your ISO status), and potentially as part of FDA follow up audits. I don't know what the situation in this field would be like.

> It's powerful to have investigative powers not be hard-tied to regulatory powers

The idea being that they'll have purer incentives to get the truth? And maybe get better cooperation? I'm sensing the outline of how this arrangement is "powerful", but having trouble with the full picture and details. And per your sibling comment, it might still be good to have a formal process for enforcers to have a look at their results...

- they're able to investigate whether the regulators did their job properly in regulating, while being organisationally at least somewhat distant from the regulator they're looking at (obviously they tend to maintain close relationships with the appropriate regulators but at least they're not literally in the same building)

- people will be more willing to talk to investigators if they know that their evidence won't be used against them in court or used for a licence suspension

- you can grant non-regulatory investigators greater powers to compel testimony and documentary evidence, perhaps even if it's self-incriminating (I can't speak for the US but it's common here in Australia where we don't have a 5th Amendment entrenched guarantee), without running into as many ethical issues as when you give coercive information-gathering powers to regulators - you ban such compelled testimony from being used as evidence in court or in regulatory proceedings

- you don't need as high a standard of proof to make an investigatory finding as if you're taking punitive action against a specific individual

- the overall purpose of the investigation is aimed at systemic safety, rather than getting a successful prosecution

None of this prevents the regulators from running a parallel investigation with the aim of regulatory punishment - and indeed, two sets of independent eyes on complex scenarios is good too.

Too bad they don’t hand out jail sentences. Kind of amazing considering the amazing negligence at play is some of these cases.

That's not their job, and if it was they'd be far less effective at maintaining safety in chemical processes.

Assigning blame usually results in more arse-covering and indirection than disaster reduction. See also: NTSB.

I was really enjoying their videos around the end of 2016, and then something happened and it seemed like the money they requested was not going to be put in the government budget. I'm glad to see they are back doing this work.

Just in case anyone else was confused, 'popcorn polymer' refers to a popcorn-like porous polymer that can form in petrochemical processing, not a polymer that forms during popcorn production.

I was assuming it was something similar to acrymilide formation in french fry cooking, and very confused how such a process would end up on the USCSB page.

Absolutely wonderful video in any case

I'm curious about this popcorn polymer. I went looking, and found some papers on how it forms, but no pictures of what it looks like, or videos of it forming.

EDIT I looked again and found some tiny pictures:

https://www.shilohsciences.com/blog/popcorn-polymer-in-steam... (at the end - this is a great article)

https://www.hydrocarbonengineering.com/special-reports/14072...

https://epsc.be/epsc_media/Learning+Sheets/2020/20_05+EPSC+L...

This comment covers how the popcorn forms and produces localized pressure in a dead leg pipe:

https://news.ycombinator.com/item?id=36797295

I thought this would be about this popcorn ceiling that gets mentioned every once in a while. Wasn't that stuff toxic or asbestos or something?

I think the narrator isn't a CSB employee, and is just hired for these videos.

I would be interested in seeing some sort of parody video made about commercial/industrial deep friers. e.g. '2018 Destin Florida McDonald's Fryer Ice Immersion Incident' Could even hire the same animation company.

Sadly they dropped the new intro (https://www.youtube.com/watch?v=sFhkzK7jkKg ) they debuted on the last video, and goes really hard. Don't get me wrong, I love the production value on classic CSB videos too, but they truly went the extra mile there.

Agreed, I loved the intro.

That is absolutely _magnificent_

Even though that into was so comically and excessively American I was sad to see it go.

Holy shit that video has a higher budget than my country's entire years film catalogue

Thanks for the link, that was so great. I was in such anticipation of a "classic but inaccurate" Bald Eagle scream that I genuinely fist-pumped when it actually happened.

> "classic but inaccurate" Bald Eagle scream

Specifically, a Red-tailed hawk scream. Ex: https://youtu.be/33DWqRyAAUw

If anyone is curious about the "Butadiene Product Stewardship Guidance Manual" referenced in the video, it's free: https://www.americanchemistry.com/industry-groups/olefins/re...

Very sad time when the trump administration and Republicans wanted to ELIMINATE the USCSB.

They do great work.

They can eliminate it, but the industrial accidents and deaths are on their hands then.

if they ignore them, then they clearly never happened!

I watched the whole video, and missed any discussion of obvious safety mechanisms (beside the recommendation for remotely operated isolation valves). How about monitoring mechanisms. Every pipe leg should have a remotely sensed pressure gauge. If there is a significant pressure difference at different ends of a pipe (more than expected of a pressure drop during flow), not explained by a valve being closed in the middle of a pipe, that is a blockage.

There must be surely be ways of detecting popcorn polymer buildup without opening the pipe. Some kind of acoustic reflection or whatever. If we can detect a stud behind drywall with a five dollar device, surely we can tell whether there is popcorn polymer inside a pipe?

It seems in this case the pressure would have to be measured in very small section increments. I imagine it's like a clog forming in one particular section that could expand locally, while the rest of the pipe leg might not show an increase in pressure.

Maybe just have flow meters to detect dead legs. If a particular pipe section doesn't indicate any flow over some time period, assume it's a hazard condition and raise and alarm.

Another idea might be to find an ingredient which doesn't affect the product but will react when polymer forms, such that it can serve as an indicator that somewhere in the system there is a significant amount of polymer formed.

There is the question of why does pressure rise that much? It seems that there is no pressure control behind the pumps. So that is to say, they use powerful pumps in some kind of open-loop operation, where the pressure is controlled by the lack of impedance to flow.

It's as if, in electronics, we used a power supply with an insanely high unloaded voltage, and then just relied on that voltage dropping to an acceptable level due to system load. Then if the load is too low due to an open circuit, the voltage rises, and fries something that is powered from the same node.

Why not control all the pumps so that the maximum pressure that can exist in the system is well within the capacity of the pipes and cisterns, even if everything plugs up.

I don't think that can be it: this was a dead leg, so a clog wouldn't increase pressure in the system elsewhere.

Instead I think it has to be that popcorn polymer expands as it forms, directly putting pressure on the pipe in contact with it.

This channel is one of the few I have bookmarked.

In case you ever needed to know why zoning laws are useful and why you would never want to live anywhere near industry, explore their videos.

Sometimes I worry about the job related health problems that come from sitting in a chair all day, but would trade it any day for the horrific injuries that happen every one of these USCSB videos.

I don't fully understand this incident report.

A blocked pipe alone doesn't cause it to burst. It has to be combined with something that causes the pressure to increase, like perhaps a breakdown product that produces gasses?

Popcorn polymer implies, to me at least, a polymer that expands in size like popcorn.

I assume that with exposure to oxygen, the product expands into this polymer.

Here is the PDF report from the CSB:

https://www.csb.gov/assets/1/17/tpc_factual_update_10-29-202...

Bottom of page 7 it explains the mechanism. Essentially butadiene peroxides react with the butadiene to form the polymer 'seed'. The polymer is not stable and expands, increasing in volume.

The quality of this review is honestly breathtaking. It’s great work, but now I’m left wondering if maybe they should shovel some of their review production budget into hiring more agents

Given their lack of enforcement powers, this video does far more benefit to their goals. Making workers aware of potential situations like this can help put more pressure on businesses to improve their overall safely standards.

Dont discount the reach of well made videos.

What’s more likely? A company reading through a 200 page report or watching a 15 min well done video?

^ Their reports may have a small audience, but their videos have much more mainstream appeal and raise public awareness. It's a seed that a teenager may watch today, and remember in 10, 20, 50 years when they work at a factory and they're like "Hold up, this is bad because I remember a video I watched back in the day". Public awareness is super important.

Plus, the grandparent comment seems to imply it's an either / or; both is good. There's always money for both. You can hire more agents AND keep publishing videos.

Many many years ago my dad, with his fresh ChemE degree, was a junior engineer at a chemical plant near Houston. Probably the kind of role that would have been responsible for ensuring that things are working correctly and things like popcorn polymer (or the 1940s equivalent) weren't building up.

He found the job so dull that he quit and left the field completely.

What are the insurance payouts like when plants like these are destroyed? Could those be manipulated to balance the risk motives for omitting or delaying mitigations when warnings signs appear? Or would that just encourage poor record keeping and falsifications?

Ideally, they want the plant running as much as possible. I think that's what motivated them to delay mitigations.

I imagine that in the future, when climate change gets so bad, those types of petro-chemical facilities will become easy target, and seen how complex and dangerous they are, it will be very difficult to keep them safe and functioning.

I have become a huge fan of this YouTube channel (and agency!), the USCSB.

The lack of remote-operated safety valves shocked me the most.

Insanity in 2020.

How many millions of valves? If the internet goes down?

Why would the internet be involved?

Remotely operated valves always have manual controls in case of network failure.

I imagine you would want them on some wires network to reduce EMI noise in these all metal environments.

remote operated is also remotely vulnerable.

https://hackernoon.com/6-major-scada-attacks-that-happened-a...

getting some eliza vibes from the talking head at 6:15

Reply on news.ycombinator.com