Comment by jrockway | Hacker News Reader

Comment by jrockway | Hacker News Reader

Parent Full thread

I doubt that everyone on the team missed the fact that electronics get hot outdoors in the summer. Rather, they probably chose to launch a product that breaks when it gets hot, because it's an easy problem to fix later. Now that something is out in the field, they can debug their network, satellites, RF hardware, installation instructions, management applications, and figure out what kind of customers want to buy this thing -- all in parallel while some tiny EE team redoes the BOM to use better-spec'd components. Everyone at the company now has a ton of work to do to produce the next version, taking into account real-world feedback as they do it. Would it be worth blocking all those projects another 3 months just so that some guy in Arizona gets more Internet uptime on the hottest day of the year? Probably not. It sucks to invest in an early product and have it break on you, but it's also probably not possible to have a company that releases a totally flawless product on day 1. If you demand perfection, what you are really demanding is for the product to never exist.

(It's also completely possible that the parts simply aren't available. You may have noticed that there's been a pandemic going on for the last year or so, and that's hurt the availability of many electronic components, among other things.)

This is exactly the whole "agile" vs traditional/waterfall mentality, scooters before motorcars, etc., put into practice. Hardware is hard.

> It's also completely possible that the parts simply aren't available

Fun fact: they don't make (commercially available) GPUs, NICs, pretty much any PC components rated below 0°F. If you need to design a system which needs to sit in an aircraft in Deadhorse, Alaska, you either pay out the nose for some serious engineering, or you just roll the dice. We rolled, and it turned out fine. That won't always be the case.

What Spacex is doing is smart, not sloppy.

> We rolled, and it turned out fine.

Who is “we”?

Company I worked for. We built an aerial multispectral imaging system for monitoring seals, polar bears, and other mammal populations. It's 3 high power rugged PCs, semi-affixed in a shock mount case in the tail of a Twin Otter. There's no hangars available at the airbases in e.g. Kotzebue, Deadhorse, Utqiagvik, so it sits on the tarmac and experiences some intense temperature swings.

It's made it 2 winters and dozens of missions so far.

The customer is NOAA and they really don't have oodles of cash to spend on this thing, so it's mostly commodity hardware. Initially we designed this whole crazy insulated case to keep it from getting too cold, but the aircraft engineers nixed it due to fire hazards. The airframe isn't airtight so it gets most of the brunt of the diurnal swing.

Sounds like a fun job!

It was brilliant fun and also the most stressful period of my life, due to it being a bit far out of our core competencies at the time, delays in getting the camera hardware we had to interface with, limited staffing, and a comedy of other errors. Also the PM gets insanely airsick. As a result of all that, I got to go up to Kotzebue, AK with the PM to do the integration testing.

So there I am, hacking like mad in a terminal in the back of this airplane over sea ice, trying to squash various bugs. I blame my love of travel and reading while flying for my resilience to motion sickness.

Stressful AF. So worth it.

You could also add a heater.

And, fwiw, that's exactly what SpaceX did to make Dishy tolerant of low temperatures (and to help it clear snow/ice from the dish). There's just no equivalent solution for high temperatures.

I used to think it was just a matter of product cost to do active cooling... really the problem isn't the temperature it's the dew point! Which I guess could probably be solved with enough hardware, planning, and cost as well... but a ridiculous amount compared to adding an electric heater for the opposite problem.

Come to think of it does anyone know of a consumer device that utilized active cooling (as in would get colder than ambient not just fans or whatnot)?

a fridge

I mean electronic device, things in a fridge/freezer generally don't mind condensation or deposition.

A pump, a home depot bucket in shade, some duct tape and tubing would probably be sufficient.

Thermoelectric coolers can cool below ambient. You can get them for CPUs, I saw one on a Linus YouTube video.

https://en.wikipedia.org/wiki/Thermoelectric_cooling

Sort of. Thermoelectric coolers transfer heat from a "cold side" to a "hot side", and generate a lot of additional heat in the process. They can be part of a cooling solution, but you still need active cooling (like a heat sink and fan) for the hot side -- they don't just "generate cold" in the kind of way that a resistive heater can generate heat.

Which is what Google does in at least one of its data centers.

Is that for condensation control or because you're hitting the low end of your temp range?

It’s some data center in Nordic region, and during winter it gets so cold the servers can’t heat the place.

It'll be incase of an emergency shutdown.

Can't have the servers freezing when turned off.

Along the same lines, most hardware is designed under the assumption that you aren't significantly above sea level. Tons of medium-high voltage electronics have trouble with arcing once it turns on at 6000+ ft.

Considering most of humanity lives within an hour's drive of the ocean, it's not a bad assumption.

I'm interested why you are being downvoted, as you bring up a good point that aligns with what kortex was saying in their post.

edit: ahh, now I see it's because you're factually incorrect. It's only 1/3rd of humans that live within 100km of oceanic coasts.

https://science.nasa.gov/earth-science/oceanography/living-o...

arcing involves ionizing air, right? I would have expected that the breakdown voltage of air increases with altitude, due to decreasing density of air.

I would guess that it could be due to having more ions in air due to higher radiation levels.

It's actually the opposite (to a point, I think). Higher mean free path means more time to accelerate, then hit and ionize something else. That's how fluorescent tubes maintain discharge a meter long at measly voltages.

Radiation doesn't help either, as sibling commented - it creates ions making the initial breakdown easier, though it shouldn't make a difference for a continuous flow that involves innumerable ions.

Here ya go:

https://www.youtube.com/watch?v=MIqK45nxcTo

He even explained it with balls and magnets!

Wide temperature (-40C to +85C) DIMM and SSDs are available.. Companies like ATP and Apacer carry them. Intel's Xeon D series are meant for wide temps.

-40C is very close to 0F (-32C) claimed above.

Update: -40F is -40C, oops, my bad, I don’t understand Fahrenheit. ¯\_(ツ)_/¯

Um, -40C is -40F.

-40° Celsius is actually equal to -40° Fahrenheit, it's the temperature at which the two meet. 0° Fahrenheit is about -18° Celsius, did you perhaps multiply 0 by 5/9 then subtract 32 instead of the other way around?

The starlink dish is rated to -30C so it’s not that bad on the freezing side.

I wouldn't want to roll the dice when it comes to aircraft

The hardware is an imaging system (see other comment). It does not impact flight avionics in any way, and is on an isolated power source. The flight engineers wouldn't let us roll the dice on anything mission critical. They did make us redo a bunch of wiring harnesses to use non-smoke-producing wires, for example.

"Hardware is hard", I would say Manufacturing is hard.

That expensive part is NRE and only making a few tens of them, no production efficiency. With the volumes spacex will be hitting, efficiencies will certainly be found and NRE exceptionally better amortized.

No, it’s not. Ignoring perfectly predictable issues because you can’t even learn from recent history (millions of sat tv dishes in the nineties) or from industry (literally millions of cellular antennas) is not “smart.”

Also your example is a poor one. Engineering a systems solution for 0F is far easier than engineering for outdoors in the sun. Heat removal is a primary consideration and a 0F environment makes that far easier. In 0F your issue isn’t the cold, it’s the lack of humidity - again, something everyone who has more than two years in HW knows.

There is tremendous value in procrastinating that is still overlooked even in 2021. Fixing stuff and making decisions "just in time" means reaping value now and in the meantime.

" Would it be worth blocking all those projects another 3 months just so that some guy in Arizona gets more Internet uptime on the hottest day of the year?"

Umm, yes?

A few hundred thousand guys from less overheated areas might politely disagree.

Yeah and with the data they gather, they might be able to work out geographically appropriate versions. Eg add a smaller AIO water cooler to a hot climate area version, or some other hardening. Maybe it makes sense to apply to all versions? Can't know until you have some data. Sounds like a great and sensible beta program to me. Avoids premature optimisation, lets you see what actually goes wrong (where, and in what conditions) without a massive effort up front.

For fuck sakes people this is ambient temperature, it does not require beta testing, it requires someone to get off their ass and look at over a century of temperature data that organisations like nasa have been collecting.

I see people here hate the idea of startups or engineers being responsible for any kind of reliable service even though lives depend on it, and as this "SV spirit" infiltrates more of our life even coffee machines don't work without an internet connection, a subscribtion and software update.

Completely prosperous

prosperous

I think that's the idea...

Why do you think the parts spec'd for this use case are available? The humble STM32 microcontroller has ONE YEAR lead times right now. If you're launching a hardware product right now, the supply chain dictates the design, not the other way around.

Reply on news.ycombinator.com