Hacker News Reader

Comment by blhack | Hacker News Reader

Starlink is in beta. The entire purpose of the beta is to find stuff like this. They found something, and I'm sure that the next iteration of this before it goes to mass market will account for it.

You're just seeing the system work.

View on HN

Speaking from experience in this same industry, this is the wrong way to look at it. They significant amount of engineering goes in to designing the ground systems such that it can tolerate all different types of environments. Currently with Starlink you are only seeing probably less than 5% of the environments that it would be rolled out in.

This is mainly due to how they are targeting RDOF areas to appease the government from winning all that money. Heat is one thing, but freezing temperatures are also very difficult to deal with. A deicer is common on high-end antennas to melt ice off in subzero temperatures.

I would be surprised if SpaceX changes the design at this point. To manufacture it antennas in the quantities that they need, they would have had to order a lot of the parts 6 to 9 months ahead of time in large quantities, especially given that they lose money on the terminal cost currently.

Preemptive thermal shutdown at this temp doesn't mean the units aren't designed to handle much higher. It means at this stage in the beta program they aren't ready to push the envelope with customer deployed hardware.

I always back off from the maximum limits for the gear I design then after years in the field and proven reliability you can remotely update it and push things a bit. And that's in production. This is beta

You don't know that. It's highly likely that they know it will do damage above those temps and are doing this to prevent damage. We will see in a year or so.

Responding with 'you don't know that' to a comment with no specific assertions, followed by 'its highly likely...' is a good example of 'irony'. We need more of those on US centric forums as a public education service. Thanks.

That's not how it works. It's well known that overheating electronics for an extended period of time causes them to fail. Especially ones that are not rated for large temp ranges. So to speculate that it's probably not a real issue and that they could just change it later needs more evidence.

They put the limit there in the first place, so why did they choose that temperature? Keep in mind these terminals need to last 5-10 years in these conditions. The motorized antenna is another part that is known to fail in extreme weather. Let's revisit this in a couple years when they have a decent amount of customers.

This shouldn't even be considered "pushing the envelope" it's not that hot.

Not that hot? It's 122°, 134° is the highest ever recorded on Earth, it's pretty hot.

122F air temperature or 122F device temperature? Intel CPUs shut themselves down at 212F.

In the shade.

The critical components are in the shade of the dish.

It has hit 122 deg in the shade in California both of the past two summers (not yet this year). That's in the shade. Put equipment on a dark roof in full sunshine and your dish becomes a solar cooker. Maybe they could sell it that way, and claim autonomous switching between two useful modes.

Fahrenheit not Celcius

That’s the air temperature. Dark objects in the sun heat up a lot faster.

We have a thermometer with a black plastic sensor (why?!) in our balcony. When the sun hits it in mid-morning, it gets up to 120°F, even when the actual ambient temperature is more like 65°F.

The dishes reportedly use around 100W. To safely dissipate such a power at higher temps, the design would need visible features (such as active cooling). Can't be done by just pushing software update.

Dial down throughput and reliability of the connection once enough people have bought into the original hype? Downgrading the performance seemed to be the popular solution for battery issues on phones not too long ago.

Things standing on pavement in direct sun can reach 50°C even without any added internal heat.

Phones aren't supposed to be used in such situations at all, so it's hard to carry over design lessons from there. Plus the radio frequencies used are new, unlike phones.

I guess users would have to strap on heatsinks and fans?

This. I don’t understand why people give these companies credit for being able to solve problems when they straight up ignored industry experience.

It’s like.. hey guys, the rest of the world has been designing satellite dishes for 40 years that addressed these problems including very low cost stuff from the satellite TV era. Why didn’t SpaceX learn from the rest of the industry instead of having to rediscover it?

> Why didn’t SpaceX learn from the rest of the industry instead of having to rediscover it?

Elon Musk famously works by „first principles“, which can indeed be roughly translated to „discover everything yourself“. The advantage is that it gives a clean slate for approaches considered impractical when the books were written.

> The advantage is that it gives a clean slate for approaches considered impractical when the books were written.

I'm not sure when that has ever worked out like that though. SpaceX built on a lot of existing aerospace knowledge. Tesla went through a lot of manufacturing hell trying to automate what turned out to be pretty unautomatable parts of manufacturing a car. As far as I know, whatever you think of the EV, the car manufacturing process Tesla has isn't uniquely good.

I guess Musk still wanted to find out anyways. His goal isn't merely building cars the simplest way possible. The guy is an educated physicist, he knows how stuff is usually done.

He has a BS in physics and never worked in the field a day in his life. No one would call someone with merely a BS a “well educated physicist.” This kind of thinking is cult-like attestations of faith.

To give them some benefit of doubt, all the industry experience is with geostationary sats. There is AFAIK little experience with tracking multiple quickly moving satellites using one antenna array.

122F = 50C

Means the antenna is not even outdoor rated

It could just be a cutoff that's set to low, this event does not tell that much by itself. It may turn out that the antenna fries itself at 60C and that's why it's turning off at 50. Or perhaps it could easily handle much more but it was set at that temperature for some reason.

SpaceX is known for using less than ideal products during test phases and working in parallel for the improved/final versions. it depends on what they are trying to test. and in this phase they are trying to test the network and will use the cheapest dishes that can do for this purpose.

I mean they practically used a water tower to test their starship engines. while in parallel they worked for rocket design just to test the first flight/landing and yet in parallel another deisgn for orbital flight tests etc. It seems a waste but this aproach has allowed them to be years ahead of ULA in rocket design. In this case I think they have demonstrated how they work and it's fair to assume that these problems are not engineering miscalculations but deliberate "go with the fastest thing you can test" decisions.

> SpaceX is known for using less than ideal products during test phases and working in parallel for the improved/final versions

That works for building rockets one at a time. It works less so for mass producing consumer electronics. Lines cost a lot of money to set up, and economies of scale exist.

Current dish (manufactured and supplied in April 2021):

  {
    "dishGetStatus": {
      "deviceInfo": {
        "id": "ut01000000-00000000-0000e586",
        "hardwareVersion": "rev1_pre_production",
        "softwareVersion": "db6b5d22-a642-43d6-bbec-97a816e3dbd6.uterm.release",

"rev1_pre_production" suggests the hardware will be subject to revision.

But also that it hasn’t been iterated on yet, after 8 months.

The reason I don't believe this is they are required to start providing service per the FCC by the end of this year. The antenna lead time is long enough to where they can't change the design now unless it was already in the pipeline.

If your engineering team had to wait for a product failure to realize an item that's going to be outdoors in the sun is going to get hot, what on earth else are they missing?

They put a non-automotive grade LCD in Teslas. I assume because they thought they knew better. I suspect hubris is at play more than full-on incompetence.

Just curious, what are the differences between normal LCD's and automotive-grade LCD's?

Automotive-grade LCDs don't burn out and melt at the temperatures commonly found in ... well, automotive environments. Like your car sitting in a tarmac car park all day in sunny weather with the windows closed.

Tell that to my 2012 Suzuki SX4 radio LCD, it won't work when its warm out.

I've seen my 2017 Mazda LCD go into protection mode a few times. It also dims before that stage.

Certified operating temperature range would be one.

One thing that was shocking to me when I looked this up myself is the temperature range of a car dashboard where the LCD is mounted. It can get ludicrously hot in the summer if the car is parked under the sun[1], like 80-90 C (180-200 F) hot.

[1] https://www.clickorlando.com/news/2019/09/26/heres-how-hot-t...

SpaceX and Tesla are separate companies with different employees. "They" did not put anything in Teslas.

Separate companies with the same CEO, (some) shared board members, joint projects, cross sales of services and equipment, loans of employees, etc. Surely there's significant culture bleed.

I wish more of the SpaceX culture bled into Tesla then.

Move faster, break more things? Tesla delivers far more refined products than SpaceX

Yes, break more things on testing grounds. Keep iterating until the thing works well.

SpaceX, paradoxically, has easier time experimenting and partially failing, because they mostly haul machines. Every Tesla car, OTOH, has a human pilot.

The put a non-automotive grade LCD in the original Model S because it was that LCD or nothing.

The automotive screens didn’t exist in the size they needed and they weren’t at a stage in their company where they could design their own.

They took a risk to achieve the product vision that they were determined to bring into the world given the technical constraints of the age. I’d say overall it paid off.

IMO it wasn’t arrogance.

Yeah, and their typical failure mode, yellowing around the edges, still looked better than a lot of other vehicles from its era.

I don't like their handling of it from a warranty perspective, but I'm also not sure that their initial decision was actually the wrong one.

I agree, it wasn't arrogance.

Deliberately going forward with something in spite of knowing for certain that it's not fit for purpose is called engineering malfeasance, not arrogance.

Making practical tradeoffs to get what's important is engineering reality.

"The ends justify the means," is an epitaph on countless headstones.

Edit: Here's some food for thought. Tesla was basically forced to admit the display issue because it was so pervasive, and they basically got lucky that their display didn't cause other problems or bigger problems.

Just how lucky was Tesla? We know they're willing to take a gamble. Where else did they cut corners or skimp that we just never found out about? What else might have gone wrong? Automobiles are one of the most dangerous machines on the planet. We've also heard horror stories from former employees about working conditions, too.

People are Tesla fans because they're excited by the image. By electric cars. By the idea. But they're letting their desire to have that image in their head untarnished means they're not going to be willing to accept genuine criticism of genuine failings. Just like Apple, and Linux, and so many other products, it's generated fans who live up to fanaticism.

This is a stupid opinion. Its a display screen. Any engineer would look at the failure modes (and it was an engineer, not Elon himself) and go "we might have problems, but given the use case it's not really going to be that bad".

Elon companies have plenty of issues but the cadre of people who are insistent every issue is an unforgivable catastrophe are ridiculous.

The display screen in my 2004 Prius had the touchscreen suddenly stop working as well. Was an issue with thermal cycling with that particular part model as well. These things happen.

People certainly get worked up about how Elon companies don't fit the traditional mold. But I think there are some good reasons why that is. Mostly this comes down to the "we'll just fix it later" mentality, but without an actual process for fixing real user devices when the "engineering tradeoffs" inevitably produce a broken experience long before an item of that quality should fail.

the second issue is that the severity of failure is not always taken into account. In your example of the Prius radio, there are several important differences IMO. The radio of a prius has much more limited scope than that of a tesla. It is a pain if it breaks, but you can replace it with something aftermarket and everything will be fine. While it is still possible to drive the car with a non-functioning touch screen, a lot more features will be missing, including climate control and the reversing camera. Additionally these units consistently failed early and Tesla initially didn't fix them under warranty.

A similar issue (failure of the display due to worn out eMMC after 5-6y) triggered a recall after pressure from the NHTSA. https://www.theverge.com/2021/2/2/22262072/tesla-display-fai...

Sure, fine. But the MCU still crashes on my (very yellowed screened) unit regularly, randomly. Right now, that doesn't cause me to crash: autopilot stays engaged. The car continues to function. There's even a low-level beep when autopilot is disengaged, even though all other audio fails. Blinkers still work. Etc.

But the new car has removed the shifter stalk, and moved the controls to the screen. Now, if the screen reboots, I can't place the car into park or reverse. With the shifter, I can push and hold the parking button to engage the emergency brakes on the car. Can't do that if the screen is rebooting. The blinkers are now capacitive touch buttons on the steering wheel. While I imagine they'll stay low level and decoupled from the MCU, I sure hope that's the case.

The yoke is possibly the stupidest thing Tesla has done, ever. So many people are going to end up with completely shattered arms as they get into accidents and have the airbags deployed while they're turning and have their arms crossed. All for what? To look cool to a 13-year-old? Idiotic. And I say this as someone with a MS on order for September which I'm desperately hoping I'll be able to get with the regular wheel. Still bitter about the stalks, though.

There are park and reverse buttons on the center console too. (not just the screen)

> Making practical tradeoffs to get what's important is engineering reality.

Not being aware that objects placed outdoors and exposed directly to sunlight easily reach temperatures above 80C is either a collosal design error or sheer incompetence.

Let's put it this way: for anyone to draw the line at 50C they needed to do at least a review of environmental constraints and operational ceilings, and the most rudimentary of cursory reviews (i.e., googling and looking at the first search hits) is enough to show that white objects placed outdoors at direct sunlight easily get to 80C.

It's a heckin' internet uplink dish, not a mission-critical piece of engineering lives depend on (if Starlink is your only way to call emergency services, you ought to have a radio backup).

I'm not taking about the dish in my comment, I'm talking about Teslas shortcuts.

But it doesn't really help if your poor engineering only causes your company to fail due to recall, or gets you shut down for fraud or negligence, or your product is so bad that customers entirely leave your market. That's still not something to champion.

The correct order is:

1) Start a pioneering car company in the 21st century with many incumbent car manufacturers, dealerships and regulations

2) Give business advice on (1)

> Deliberately going forward with something in spite of knowing for certain that it's not fit for purpose is called engineering malfeasance, not arrogance.

An argument can be made for both. Malfeasance in that it was a significant and potentially catastrophic engineering tradeoff due to unknown failure modes, and arrogant in that the decision valued company vision over potential consumer safety risks as well as over the expertise of the component manufacturers.

+1. This is something software engineers (of which I am one) often miss-understand about other engineering disciplines. You can totally ship software that you know will run into problems down the line, because you can reasonably expect to solve those challenges and ship updates before they come into play. But when you sell someone a durable good that is expected to last 10-20 years (eg a car) there are a whole host of things that go into designing that thing that are not ok to fix later.

The result is that these engineering disciplines have practices that are much more rigorous. If you think about it, a recall is essentially a hardware patch, and they are typically highly embarrassing and costly.

There is also the whole separate aspect of professional licensure which certain engineers get. Licensed engineers are expected to hold their work to certain standards regardless of the business needs of the company they work for. Now i'm not saying what Tesla did falls in this category, but it certainly looks like designing a product below the standard expected for that product. (what most people would call shoddy engineering).

In software we often get away with this, because we can put in the work later on, and also because people often have insanely low expectations of how well software will work.

> This is something software engineers (of which I am one) often miss-understand about other engineering disciplines. You can totally ship software that you know will run into problems down the line ... because people often have insanely low expectations of how well software will work.

I feel like this is the self fulfilling prophecy. We definitely didn’t use to ship software we knew would break, because we literally shipped it on disks, and delivering updates was hard.

The internet has been good for many many things. Software reliability hasn’t been one of them.

You can still replace parts in physical products. For example, the yellowing screen of the Tesla could have been replaced no questions asked with an automotive version once available. It is just so more expensive due to logistics and assembly.

Selling something you know breaks within a short amount of time in the lifetime of a car and then not wanting to replace it is fraud.

It wasn't designed for a car, but they did run additional tests on it.

> he put a non-automotive grade LCD in the original Model S because it was that LCD or nothing.

No. It was not no LCD or nothing. It was either an LCD that would melt in the heat of a car on a summer day, or a smaller LCD that wouldn’t.

This is just one of many decisions traced directly to Elon that fly in the face of established best practices. Lest we forget, this is a man who made the factory less safe, resulting in higher than expected workplace injuries, because he didn’t like yellow stripes and beeping forklifts. And of course the intentional misnaming of “autopilot”, and the resulting body count of the wide consumer release of its “beta” software.

So yes, it is arrogance.

https://nymag.com/intelligencer/2018/04/tesla-workers-gettin...

It wasn't an unknown that the screen might yellow in high temperatures. A smaller LCD would have been a different product, and certainly an inferior product given a (diagonal) 17 inch non-automotive screen would have had to have been an (at most) 10 inch screen. It was a tradeoff, not arrogance (maybe arrogant for them not to know that jonathankoren would disapprove of it 8 years later).

> Lest we forget, this is a man who made the factory less safe, resulting in higher than expected workplace injuries, because he didn’t like yellow stripes and beeping forklifts

I can't find any news on forklift-related incidents at Tesla. If you mean reducing forklifts all together, Amazon has also designed their warehouses around reducing forklift usage - forklifts are extremely hazardous regardless of tape and beeping.

They didn’t need that size, they could have gotten by fine with a smaller size just like every other car company.

This is just the wrong angle to look at this. The big screen was one of the features that was talked about the most. It was genius marketing, and also part of why the car felt like the future.

a lot of things can feel like the future when you don’t have to support them through their lifetime.

And they could have "gotten by" with a diesel engine just like every other car company. that's a useless comment to make. We are tired with getting by. we need companies like Tesla/SpaceX that are willing to push the envelope with simingly stupid and useless ideas. After some time people will wander how past generations have gotten by without what they have just like now most of us can't go by without access to smart phones whereas 15 years ago everyone just went by fine.

I wonder if they are doing that for other components too, or just the LCD?

IIRC there were issues with the flash memory in the media unit (afaik that means the whole giant center screen) where it would fail to boot after a while because the flash chips were failing. They actually issued a recall for that one at least.

I don't know, so this is a genuine question, but is that a safety-critical part in that case? If it's not it's still not ideal, you want to make sure that your parts will happily survive the car's environment, but it's not as bad.

> Safety concerns revolve around cars losing their display for rear-facing cameras (now required by law) and consumers complaining it’s impossible to defog windshields with the center console inoperable. [0]

> Another complaint said the failure disabled safety monitors associated with Tesla’s driver-assistance system Autopilot. [1]

It looks like losing access to the touchscreen does actually effect safety components, and driver's abilities to use them. (Though I'm not clear on to what extent - driver's might just need to crack out the manual to see how else to defog, for example.)

[0] https://www.thetruthaboutcars.com/2020/06/tesla-investigated...

[1] https://www.reuters.com/article/us-tesla-safety/u-s-probes-t...

Safety critical? Who knows, but probably depending on how you want to define "safety critical." Tesla runs practically every user control through the touch screen. Climate control, lighting, wipers. Did they design with reasonable fail-safes? I wouldn't want to bet my life on it.

I doubt that everyone on the team missed the fact that electronics get hot outdoors in the summer. Rather, they probably chose to launch a product that breaks when it gets hot, because it's an easy problem to fix later. Now that something is out in the field, they can debug their network, satellites, RF hardware, installation instructions, management applications, and figure out what kind of customers want to buy this thing -- all in parallel while some tiny EE team redoes the BOM to use better-spec'd components. Everyone at the company now has a ton of work to do to produce the next version, taking into account real-world feedback as they do it. Would it be worth blocking all those projects another 3 months just so that some guy in Arizona gets more Internet uptime on the hottest day of the year? Probably not. It sucks to invest in an early product and have it break on you, but it's also probably not possible to have a company that releases a totally flawless product on day 1. If you demand perfection, what you are really demanding is for the product to never exist.

(It's also completely possible that the parts simply aren't available. You may have noticed that there's been a pandemic going on for the last year or so, and that's hurt the availability of many electronic components, among other things.)

This is exactly the whole "agile" vs traditional/waterfall mentality, scooters before motorcars, etc., put into practice. Hardware is hard.

> It's also completely possible that the parts simply aren't available

Fun fact: they don't make (commercially available) GPUs, NICs, pretty much any PC components rated below 0°F. If you need to design a system which needs to sit in an aircraft in Deadhorse, Alaska, you either pay out the nose for some serious engineering, or you just roll the dice. We rolled, and it turned out fine. That won't always be the case.

What Spacex is doing is smart, not sloppy.

> We rolled, and it turned out fine.

Who is “we”?

Company I worked for. We built an aerial multispectral imaging system for monitoring seals, polar bears, and other mammal populations. It's 3 high power rugged PCs, semi-affixed in a shock mount case in the tail of a Twin Otter. There's no hangars available at the airbases in e.g. Kotzebue, Deadhorse, Utqiagvik, so it sits on the tarmac and experiences some intense temperature swings.

It's made it 2 winters and dozens of missions so far.

The customer is NOAA and they really don't have oodles of cash to spend on this thing, so it's mostly commodity hardware. Initially we designed this whole crazy insulated case to keep it from getting too cold, but the aircraft engineers nixed it due to fire hazards. The airframe isn't airtight so it gets most of the brunt of the diurnal swing.

Sounds like a fun job!

It was brilliant fun and also the most stressful period of my life, due to it being a bit far out of our core competencies at the time, delays in getting the camera hardware we had to interface with, limited staffing, and a comedy of other errors. Also the PM gets insanely airsick. As a result of all that, I got to go up to Kotzebue, AK with the PM to do the integration testing.

So there I am, hacking like mad in a terminal in the back of this airplane over sea ice, trying to squash various bugs. I blame my love of travel and reading while flying for my resilience to motion sickness.

Stressful AF. So worth it.

You could also add a heater.

And, fwiw, that's exactly what SpaceX did to make Dishy tolerant of low temperatures (and to help it clear snow/ice from the dish). There's just no equivalent solution for high temperatures.

I used to think it was just a matter of product cost to do active cooling... really the problem isn't the temperature it's the dew point! Which I guess could probably be solved with enough hardware, planning, and cost as well... but a ridiculous amount compared to adding an electric heater for the opposite problem.

Come to think of it does anyone know of a consumer device that utilized active cooling (as in would get colder than ambient not just fans or whatnot)?

a fridge

I mean electronic device, things in a fridge/freezer generally don't mind condensation or deposition.

A pump, a home depot bucket in shade, some duct tape and tubing would probably be sufficient.

Thermoelectric coolers can cool below ambient. You can get them for CPUs, I saw one on a Linus YouTube video.

https://en.wikipedia.org/wiki/Thermoelectric_cooling

Sort of. Thermoelectric coolers transfer heat from a "cold side" to a "hot side", and generate a lot of additional heat in the process. They can be part of a cooling solution, but you still need active cooling (like a heat sink and fan) for the hot side -- they don't just "generate cold" in the kind of way that a resistive heater can generate heat.

Which is what Google does in at least one of its data centers.

Is that for condensation control or because you're hitting the low end of your temp range?

It’s some data center in Nordic region, and during winter it gets so cold the servers can’t heat the place.

It'll be incase of an emergency shutdown.

Can't have the servers freezing when turned off.

Along the same lines, most hardware is designed under the assumption that you aren't significantly above sea level. Tons of medium-high voltage electronics have trouble with arcing once it turns on at 6000+ ft.

Considering most of humanity lives within an hour's drive of the ocean, it's not a bad assumption.

I'm interested why you are being downvoted, as you bring up a good point that aligns with what kortex was saying in their post.

edit: ahh, now I see it's because you're factually incorrect. It's only 1/3rd of humans that live within 100km of oceanic coasts.

https://science.nasa.gov/earth-science/oceanography/living-o...

arcing involves ionizing air, right? I would have expected that the breakdown voltage of air increases with altitude, due to decreasing density of air.

I would guess that it could be due to having more ions in air due to higher radiation levels.

It's actually the opposite (to a point, I think). Higher mean free path means more time to accelerate, then hit and ionize something else. That's how fluorescent tubes maintain discharge a meter long at measly voltages.

Radiation doesn't help either, as sibling commented - it creates ions making the initial breakdown easier, though it shouldn't make a difference for a continuous flow that involves innumerable ions.

Here ya go:

https://www.youtube.com/watch?v=MIqK45nxcTo

He even explained it with balls and magnets!

Wide temperature (-40C to +85C) DIMM and SSDs are available.. Companies like ATP and Apacer carry them. Intel's Xeon D series are meant for wide temps.

-40C is very close to 0F (-32C) claimed above.

Update: -40F is -40C, oops, my bad, I don’t understand Fahrenheit. ¯\_(ツ)_/¯

Um, -40C is -40F.

-40° Celsius is actually equal to -40° Fahrenheit, it's the temperature at which the two meet. 0° Fahrenheit is about -18° Celsius, did you perhaps multiply 0 by 5/9 then subtract 32 instead of the other way around?

The starlink dish is rated to -30C so it’s not that bad on the freezing side.

I wouldn't want to roll the dice when it comes to aircraft

The hardware is an imaging system (see other comment). It does not impact flight avionics in any way, and is on an isolated power source. The flight engineers wouldn't let us roll the dice on anything mission critical. They did make us redo a bunch of wiring harnesses to use non-smoke-producing wires, for example.

"Hardware is hard", I would say Manufacturing is hard.

That expensive part is NRE and only making a few tens of them, no production efficiency. With the volumes spacex will be hitting, efficiencies will certainly be found and NRE exceptionally better amortized.

No, it’s not. Ignoring perfectly predictable issues because you can’t even learn from recent history (millions of sat tv dishes in the nineties) or from industry (literally millions of cellular antennas) is not “smart.”

Also your example is a poor one. Engineering a systems solution for 0F is far easier than engineering for outdoors in the sun. Heat removal is a primary consideration and a 0F environment makes that far easier. In 0F your issue isn’t the cold, it’s the lack of humidity - again, something everyone who has more than two years in HW knows.

" Would it be worth blocking all those projects another 3 months just so that some guy in Arizona gets more Internet uptime on the hottest day of the year?"

Umm, yes?

A few hundred thousand guys from less overheated areas might politely disagree.

Yeah and with the data they gather, they might be able to work out geographically appropriate versions. Eg add a smaller AIO water cooler to a hot climate area version, or some other hardening. Maybe it makes sense to apply to all versions? Can't know until you have some data. Sounds like a great and sensible beta program to me. Avoids premature optimisation, lets you see what actually goes wrong (where, and in what conditions) without a massive effort up front.

For fuck sakes people this is ambient temperature, it does not require beta testing, it requires someone to get off their ass and look at over a century of temperature data that organisations like nasa have been collecting.

I see people here hate the idea of startups or engineers being responsible for any kind of reliable service even though lives depend on it, and as this "SV spirit" infiltrates more of our life even coffee machines don't work without an internet connection, a subscribtion and software update.

Completely prosperous

prosperous

I think that's the idea...

Why do you think the parts spec'd for this use case are available? The humble STM32 microcontroller has ONE YEAR lead times right now. If you're launching a hardware product right now, the supply chain dictates the design, not the other way around.

There is tremendous value in procrastinating that is still overlooked even in 2021. Fixing stuff and making decisions "just in time" means reaping value now and in the meantime.

There is an interesting article about this.

https://arstechnica.com/information-technology/2021/06/starl...

> Officially, SpaceX has said that "Dishy McFlatface" is certified to operate from 22° below zero up to 104° Fahrenheit. Temperatures reached about 120° yesterday in Martin's town of Topock,

> the Starlink app provided an error message saying, "Offline: Thermal shutdown." The dish "overheated" and "Starlink will reconnect after cooling down," the error message said.

This is a problem when Arizona was above 100 degrees for 144 days out of the year last year.

I mean, the units didn't fail randomly, they shutdown.

They're beta, so it could be that the thermistor was placed poorly in a redesign, that they intended 122°C instead of °F, that they shipped with a dummy value used in an older fw during testing, etc.

All in all, a little embarrassing but not that significant, assuming this is certifiably not a scam.

I dont get the °C vs F one? with 122C youre covered for 122F. 122C = 251.6F and 122F = 50C

The device detects overheating and shuts down orderly.

If they planned to shut it down at 122C (which is indeed close to melting the solder) but by mistake made it shut down at 122F (not that much for most electronics), it would be a completely understandable blooper. It doesn't mean that the hardware isn't capable of operating at 122F.

122F is exactly 50C, so it's probably just a nice round number that was chosen for the shutdown temperature.

The stated temperature range tops out at 104F so I doubt it's a units issue, it's just not designed to stand up to temperatures a good portion of the US reaches over the summer.

Your missing optimization engineers who come behind the original engineers and optimize design with a focus on production cost over performance.

This often leads to some of the most spectacular failures, and unforseen issues.

Or someone fucked up

According to the article the dish literally reported to the user that it had overheated, and then it worked once it had cooled down.

Maybe not the greatest behavior, but clearly they already accounted for this.

Do aircraft engineers get a pass?

> the Bombardier CRJ aircraft used on some shorter routes have a maximum operating temperature of 118 degrees.

https://www.npr.org/sections/thetwo-way/2017/06/20/533662790...

Is that plane temperature, or air temperature in the shade? Probably the latter?

In the case of Starlink's dish, it's not clear which temperature they mean.

You may think this is a smart example, but you're mostly illustrating that you know nothing about aviation engineering.

The temperature limit for takeoff performance has nothing to do with the maximum temperature for electronics or other components. It's about air density decreasing which reduces the engine thrust as well as the maximum lift from the wings. So the warmer it gets the lower the maximum weight you can take off with and the higher the speed required before lift-off. At some point that's not safe anymore so the manufacturers only publish verified performance data up to a maximum temperature. You can't legally fly above that.

I think this particular issue might not be related to the safe flight envelope, although that was the hot take at the time... I should have added more context.

About a year or so after the article I linked above was published, Bombardier obtained ISA+40C certification for the CRJ700 and CRJ900 from both the FAA and TC. For the flights in and out of Phoenix, AZ, this meant that those aircraft have since been able to legally fly in temps of up to 123.8F. I don’t know the details of the certification upgrade, but my understanding is that there were no changes to either the aircraft or weight restrictions.

I don’t personally know anyone at Bombardier to know exactly why they didn’t certify the aircraft for ISA+40C to begin with, but given that there were apparently no changes to either the aircraft or the weight restrictions, this seemed very much a case of “your engineering team had to wait ... to realize an item that's going to be outdoors in the sun is going to get hot”.

No. ISA+40 isn't a static temperature, it would be lower in Denver than at sea level. So in no way is that a warm electronics kind of problem.

We use ISA differences (the +40 you wrote) in flight performance calculations specifically. Not in any "aircraft too warm / cold" kind of limits, since those would be absolute numbers not relative to ISA.

They didn't wait. That's why it shuts itself down.

The link is mostly a honeypot for Elon Musk haters. I wonder if they have bothered looking at the operating temperatures of other satellite Internet dishes?

A simple search will net you heating and cooling devices for satellite dishes. If you face more extreme temperatures, you need only go buy a heating and cooling unit.

http://www.dishcamo.com/dish_heater.htm

Surprisingly, the heating element built in appears to be pretty solid, or at least that's been the responses on /r/starlink.

You're definitely right that people are hating on it. Most of the people I know in real life that knocked it have access to land based internet. Once you get out into rural areas where 4g is the basis for your internet, it's amazing.

And it's not like 4g hotspots and old cell phones don't overheat either.

Yes, my iPhone went into thermal shutdown sitting by the pool on a 90 degree day but I don’t see a hacker news discussion on it…

Your iphones primary environment isn't outside under the sun for half of every day of the year though either.

I have yet to see cellphone towers shut down because the summer is too hot.

I have yet to hear about the starlink spacecraft going into thermal shutdown in Arizona.

You should work in network operation. I have seen BS go down for so many reasons…..

Heh. But whatever it was, the impact on the end user was low. When I lived in Miami there would be cellphone trouble on weekend nights, never in the midday heat!

You get an actual 4g router, not a phone, and they actually worknwell

It's far easier to build a heating unit than a cooling unit.

Thermodynamics FTW

Well, in general cooling something down is much more complicated than heating it up.

For one you need a sophisticated reverse heat engine, for the other a piece of wire with high enough resistance.

> For one you need a sophisticated reverse heat engine

This is a very obtuse way of saying “refrigerator”, which is a type of heat pump.. a technology over a hundred years old. The simplest refrigerators/heat pumps are pretty simple — https://en.m.wikipedia.org/wiki/Absorption_refrigerator

That's a subsystem with three major sections, a working fluid, and its own heat rejection requirements. A resistive heater contains two parts - a properly placed wire and a switch.

Still say “sophisticated reverse heat engine” is a needlessly verbose way of saying refrigerator or heat pump. Also how sophisticated is something that is mass produced for devices sold under $100 in every big box retailer.

A refrigerator for a kitchen has drastically different constraints. The outside temp will always be something between 15-30 °C, the inside temperature will usually stay relatively constant unless you dump hot stuff in there.

Cooling for active outdoor technology has extremely variable outside temperatures (basically anything from -20 to 60 °C, or more if you take direct sunshine into account) and a constant generation of heat to be taken away.

Still I can't really think about any mass produced consumer item that includes a heat pump and is not either part of building/car cooling/heeting or a refrigerator itself.

Like mobile base stations of course have AC units and some amateur astronomers use peltier cooled sensors, but I would hardly call these mass produced consumer items.

Many of these folks are used to internet going out because the wind is blowing too hard, or it's raining, or snowing, or a cow is stepping on the fiber link.

https://www.datacenterdynamics.com/en/news/how-cows-caused-s...

People are always surprised when I describe how bad the internet connection I have is here in the UK. 3meg down 0.2 up. If it rains or is too windy the connection dies. I’ve spoke to bt and openreach hundreds of times but it never gets any better. I’m the last house on the end of a copper line and there is no interest in fixing it. Four years ago the village was supposed to get fibre, recently “delayed” again. I rely on the 3G signal, which isn’t much better tbh.

My Starlink arrives next week :) I’m so exited to be able to finally download some photos I put on google drive a couple of years ago.

Yeah, British internet can be fairly inconsistent. I'm in a rural-ish area and can get 100 megs down on a fibre-to-the-house connection, but when I lived in an actual town I would be lucky to get 2 megs down on a good day. I literally ended up doing everything over 4G back then because the fixed internet connection was just abysmally bad and Openreach never cared enough to fix it.

The more you hype something, the more susceptible it is to criticism. It's not necessarily that people are Musk haters (although some are I'm sure), it's that it's been hyped to hell, by Musk and others, so problems will be hyped also.

You can't have a lot of positive attention without a corresponding amount of negative attention when something goes wrong.

Yeah, it’s probably a mix.

https://www.highspeedsat.com/dw7000.htm

-30ºC - +55ºC

The previous link was only for heating dishes. I wonder if there are any products out there for cooling dishes. There are products out there for Dish TV receivers but Starlink has circuits in the dish itself. Seems more like the dw7000 above.

That's a de-icing system. Starlink dishes have the opposite problem, they get too hot because they have to dissipate 100 watts electric energy plus the heat from the sun and the nearby rooftop.

OK, maritime outfitters will sell you a climatized radome, which will set you back a few grand. Proper engineering costs money (can't have radar fail in the middle of the Strait of Malacca), and that's why Elon relies on his army of fanboys, Elon's outrageous conduct is part of marketing.

> OK, maritime outfitters will sell you a climatized radome, which will set you back a few grand.

To be fair, isn't the maritime market a bit like the aviation market, where a pack of 6 disposable AA batteries potted in a pack will cost you $43? (https://www.youtube.com/watch?v=d1YjGLwfYSQ&t=37)

On land with a good supply of fresh water (so, not in drought regions) I think you could probably cool a dish for a few dollars of parts with an open-loop cooling system. I'm not sure how much water you would actually need though, it might be an unconscionable waste in the regions that are prone to these sort of temperatures in the first place.

To be fair, isn't the maritime market a bit like the aviation market, where a pack of 6 disposable AA batteries potted in a pack will cost you $43?

Fair point, a portable AC sets you back 150 dollars or so, so with outdoor proofing and all, you can build a passable cooling system for 250 dollars. That would be the proper engineering solution, but it wouldn't look cool, like the steering yoke in the recent Tesla.

What sort of magical solution is there though for the price of the dish? If the ambient temperature is 50 degrees C, you need cooling. It looks like the Hughes dish is a little better at 55 degrees C. Looks like the expensive solution is that randome irrespective of the particular dish. The cheap solutions seem to be myriad. Relocating it, putting it under shade, sprinklers, fans.

An Elon Musk hater here:

Starlink provides an excellent example of a deeper problem I see with new technology vs. traditional infrastructure. We know how to do broadband in rural places. We do it all over the planet. We simply tax income and spend it on infrastructure.

Elon Musk (and the likes) see an opportunity to disrupt this simple answer and make money selling a different unproven solution that has the potential to be at best as good. For me this is bad news. If politicians get sold by the likes of Elon Musk, it will stand in the way of traditional proven infrastructure and—in the case of starlink—fast and affordable broadband connection for rural communities.

It’s hard to fault SpaceX for realizing a market opportunity here. If they can bring decent Internet to rural areas who don’t currently have broadband access, that’s just mutually beneficial for SpaceX and those customers.

When (if) those customers ever get broadband access by additional infrastructure investment, SpaceX will need to find a way to provide better value. I’m not sure I understand how this is a bad thing.

After investing[0] billions per year in traditional communications companies, we still have people where the only way they can get service is by laying the fiber themselves. States have also been trying to invest in expanding fiber, but traditional companies have been just pocketing the money and blocking anyone else from improving things (eg: google fiber).

Starlink seems to be a very cheap option in comparison, even if it has LoS and overheating problems.

[0]: https://arstechnica.com/tag/rural-broadband/

Other countries seem to be able to get fast internet connection with traditional infrastructure to small towns and communities. Why not America?

>We know how to do broadband in rural places. We do it all over the planet. We simply tax income and spend it on infrastructure.

The problem is that this view means you need politicians who care enough about the issue and many simply don't. I like there being a solution that eliminates a lot of the political factors that cause endless dither and delay over building traditional infrastructure for rural broadband, and for that matter my opinion of Elon Musk is still higher than most of my country's politicians when it comes to technological policy.

There's also another important use-case for things like Starlink which is people who can't be reached by traditional infrastructure. These are people who live in remote parts of the world where governments won't consider investing in infrastructure on grounds of cost, and people who live a nomadic lifestyle (liveaboard sailors for example, and van dwellers depending on where they go). I don't think they should be cut off from the internet because of an overly sceptical attitude towards new approaches.

Stepping back, low Earth orbit satellite constellations appear to be a better solution than fiber for these remote locations. Then the debate becomes whether it should be a public, private, or private but publicly funded program.

Amazon and OneWeb are also looking to build constellations. It isn’t a StarLink question.

Why are you showing links to heaters? You ask if other manufacturers have this problem, then you provide links to the exact opposite problem which is freezing temperatures. The answer to your original question is no, other manufacturers do not shut down anywhere near this temperature.

I replied to my own comment with another manufacturer with this problem. You are right that my link was to deicing but another comment said that randomes are a maritime solution to both heating and cooling.

Oh, and apparently there are heatsinks you can buy for satellite dishes.

https://www.alibaba.com/product-detail/Tokyosat-Radiator-Alu...

Radomes cost over $10k at these frequencies, so that's not a solution.

The _service_ is in Beta, but they are still selling a product to consumers, which means they are expected to follow normal engineering practies. This type of mistake is simply failing to meet best practices. I think it is roughly the hardware equivalent of using 3DES encryption, storing passwords in plain text, or allowing SQL injection. When you sell a device to consumers one of the considerations you have to consider is the operating range of that device especially if that device is used outdoors. Operating ranges are somewhat standardized, and most components will specify their operating range in the datasheet precisely because this is one of those fundamental considerations (like power requirements).

Examples: [1] The least stringent automotive standard I could find (AEC-Q100 grade 3) expects components to operate from -40°C (-40°F) - 85°C (185°F). [2] A pretty basic article on designing outdoor products suggests at least -20°C(-4°F) - 70°C (158°F).

This is a standard part of the design flow precisely so that you don't make a mistake as monumental as shipping an outdoor device that can't withstand the weather at the location you shipped it to.

[1] https://community.cypress.com/t5/Knowledge-Base-Articles/Mea... [2] https://suntsu.com/2017/09/15/the-great-outdoors-designing-p...

The reception hardware is the front end of an amazing aerospace engineering effort. Honestly, I am not worried at all, since these issues will get ironed out in new iterations of the hardware.

I am more interested in seeing the service achieve streaming latencies required for video games and trading. The day that happens, I have no need to live near a city anymore.

Sorry HW engineer here and I strongly disagree. You don’t ship hardware like this.

122F is only 50C. Most components can handle 85C so shutting down at 50C means the thermal rise is >35C. This is simply poor design.

Directv dishes are only rated for 140F or 60C[0]. Pretty close, but it clearly makes a difference in Arizona.

0: https://www.att.com/support/smallbusiness/article/smb-direct....

That is not pretty close at all. You don't get ambient temperatures anywhere near 140F.

10C difference can be life or death

> The entire purpose of the beta is to find stuff like this.

No, it really isn't.

The point of a test is to verify that the system works as designed, and in case it doesn't then there's planned room to iterate out unexpected issues.

If a system designed to be operated on the outdoors is not designed to operate with temperatures above 50C then this is a collosal screwups and gross oversight, as this sort of stuff is well-known for decades.

https://web.mit.edu/parmstr/Public/NRCan/CanBldgDigests/cbd0...

122° Fahrenheit is not "just" hot, a lot of things will have problems operating normally at that degree of heat.

Like in the area where I life a satellite dish reaching 50°C would be the rear exception. And if the alternative is not having proper internet at all other days, Startlink would still be the better option, even if there are a very few days where it fails during the afternoon of the day.

Eh, while 122F would probably be a rare temperature in most areas, judged by weather reporting, I think if you consider hyper-local effects it might not be too rare. If a dish were positioned over or near tarmac, it would experience weather several degrees hotter than the weather channel was reporting for that area. You'd also want to avoid placing it near the radiators of air conditioning systems, and other things like that.

Just something to keep in mind during installation I suppose, not insurmountable problems by any means.

I'm not sure, it seems like 110+F is pretty common nowadays in parts of the US. Adding a dozen degrees if the dish is in direct sunlight seems pretty realistic.

(not saying things won't break at that heat, just that it's not unrealistically high)

This is not something to find in a beta. This is a built-in feature and operating temperatures are not picked at random (they should not be, at least)

It’s like when Tesla finally realized their imaging cameras weren’t worth crap during the winter with road crust all over them. Proving the moderately-self driving capabilities moot.

Ideal conditions during development may not replicate real-world use.

If you ever work on hardware, please talk to people who actually know what they are doing.

Anyone with a six week internship worth of experience would have tested this in an environment chamber.

It’s bad enough when idiot “first principals” MEs and EEs toss this stuff over and I have to deal with it, but it doesn’t fucking reach customers with faults that bad.

You can bet the social media teams of Verizon, Comcast, and the rest of their ilk will be pushing this article through social networks hard.

> You're just seeing the system work.

This sounds reasonable, and you make excellent points. But the important fact we all have to remember in these discussions is not the details, but that SpaceX is run by Elon Musk.

Whoa. What guy? This thermal issue is critical.

And it's a failure in a beta, with a sample size of one. As the poster upthread points out, this is the kind of failure you want to find in a beta test of a new product.

But right right... I forgot. Musk.

I couldn't agree more. But it seems some people are reading my comment literally.

(HN really, really doesn't like sarcasm. But this particular story just absolutely drips with irrational hate and really deserves some. Good grief, new product has correctible flaw!)

Always worth remembering that HN has an international audience.

English is terrible enough before we start butchering it by playing with intent.

(Said with love and encouragement; I've forgotten about not using sarcasm here too).

Reply on news.ycombinator.com