Hacker News Reader

Comment by qzw | Hacker News Reader

pushes up glasses I would watch the heck out of a Twitch stream of their debugging/brainstorming sessions. I always loved the movie Apollo 13, especially the technical troubleshooting parts.

View on HN

Henry S F Cooper Jr.'s book The Evening Star describes some of the remote debugging and other problem solving that was necessary when the Magellan probe experienced computer problems while orbiting Venus. It's been a few decades since I read it, but it was pretty detailed and rather exciting.

This sort of comment is why I still read HN.

And the people downvoting this comment is why I will stop reading HN.

What did that comment actually bring to the conversation?

Occasionally, very occasionally, it’s nice to read somebody just expressing enthusiasm instead of just posting a clever counter argument. It’s like a spice that you only want a little of but that’s still nice

Does that spice go bad if it turns gray?

"this"

Eh, I can understand both sides of the fence. 'this is why I read HN' is nothing but a slightly more verbal '+1', but as you stated it does humanize HN and makes it feel more social.

In terms of downvotes, what really irks me and what I often see is people posting factually correct information, but still being sent into faded oblivion because some sect of the community's worldview doesn't agree with the facts.

I don't understand this viewpoint. Information being factually correct is a low bar. I have a lot of factually correct information that is irrelevant or misleading, or that I could state in a way that drags down the level of discourse more than it illuminates truth. Factually correct information is usually involved in tu quoque fallacies, or used to goad people into drawing false, non-sequitur conclusions. The Hacker News guidelines lay out a list of expectations for comments that go beyond factual correctness.

If someone uses factually correct information to make a comment thread worse, I can see how downvotes could be justified.

The fact (heh) that people instantly rammed the downvote button to get what I said to -2 ironically proves my point. Oh well.

It encourages others to post more these comments.

Since comment scores was removed this is the only way to signal this to others besides the original commenter.

That said it should not be overused. If it annoys someone I guess they should downvote it but I don't think there is a need to reflexively downvote every time someone adds a friendly meta comment.

(And if people start gaming it for karma farming I guess it should be downvoted relentlessly until that stops :-)

It didn't necessarily bring anything to this one conversation. It did, however, communicate that "this is the type of information that that person finds valuable on Hacker News". And knowing what other people in your social group like to hear/discuss is an important part of keeping that group vibrant and wonderful.

So no, it's probably not as useful as the comment it was referring too, but it was useful (to some of us) as it pertains to the community as a whole.

Conversation.

It's arguably even worse than just commenting "This." At least that is small enough you can scan over it and barely even register its existence. But this fedora-tipping "Thank you kind sir this is the type of Internet Content I enjoy!" doesn't even afford you that luxury.

I really dislike the downvote function because it reinforces self-censoring. And I completely loathe the implementation of it, you need xxxx upvotes to downvote posts....I have no words.

Well, in opposing it I especially read the faded comments and upvote any of those that are not completely abhorrent.

Take that, ycombinator.

>I especially read the faded comments and upvote

This seems to happen a lot more frequently here than anywhere else.

I'm not really sure what that says, other than people still read comments that are faded. Also that people shouldn't worry about self-censoring.

I don't have a problem with downvotes or the karma needed to do it, but

I do sometimes wish it were possible to reply to a dead comment, especially if you vouch for it and it's still dead.

Sometimes they're worth defending, or is relevant in a non-obvious way, and sometimes the comment itself is discussion worthy, as it relates to the topic, even if it's wrong or seems trollish.

How could it have been reworded to avoid the "fedora-tipping" connotation?

I'm being sincere here since I also appreciate book recommendations and I get probably half my book recommendations from HN.

I think the point is to instead of using the keyboard, use the mouse to click the up arrow, and leave it at that. (I know how tempting it is to reply quickly to something, I have the urge to just post whats going on my mind right away unfiltered. So I am very forgiving, but not everyone is)

That only tells the person who owns the comment that you appreciate it, with comment scores you're correct that almost all "I like this" comments are wrong, without comment scores then they become useful again.

Worth considering that comment scores were hidden for a reason. Exposing that information to everyone, as opposed to just the comment author, does not necessarily improve the discussion.

This sort of comment is why I still read HN!

This youtube series of video, follow a group that restored an Apollo Guidance Computer that a collector basically pulled from the trash.

https://www.youtube.com/watch?v=2KSahAoOLdU&list=PL-_93BVApb...

My favorite part is when they needed the version of the software that was used for the moon landing but they only had the source code for a previous version (scanned from giant binder) and the hash value of the version of the landing. By a series of educated guesses, by reading memos and by analysis of the source code they modified the old code the exact way so it gave them the correct hash, confirming that they correctly and exactly recreated the original code.

It's being a while and I go from memory, I might have some details wrong. See this video for this story. https://www.youtube.com/watch?v=-JTa1RQxU04

If you want to see debugging a computer in space, check out Apollo 13's sequel, Apollo 14. The moon landing is being held up by shorted-out switch that's causing the LM to abort the landing, and it's up to the programmers back home to figure out how to work around it in time to allow the landing.

Apollo 13 was the story of a 'successful failure', while Apollo 14 shows how hard work and creative thinking can turn failure into success.

Don Eyles' book Sunburst and Luminary has a chapter on this, and Don was primarily responsible for the Apollo 14 workaround. The book is also generally just a fantastic account of what it was like to develop software for the Apollo Guidance Computer.

Also about living through the sexual revolution. It's a really interesting book, but as much of a memoir as a technical book.

Scott Manley to the rescue: "The Computer Hack That Saved Apollo 14" https://www.youtube.com/watch?v=wSSmNUl9Snw

Is there actually an Apollo 14 movie? I can find Apollo 18 but not a Apollo 14 feature movie.

I saw this but it's a short documentary and may not be what you meant: https://www.amazon.com/Apollo-14-Complete-Downlink-Edition/d...

There's the HBO series "From the Earth to the Moon", which covers this with the Apollo 14 mission. Highly recommended.

Ahh, yes; I've seen the series multiple times - agreed that it's great, especially for those who enjoyed Apollo 13. I wasn't sure if there was a different Apollo-14 movie OP/Trothamel was referring to...

No, or at least not that I know of. I was having a bit of fun by declaring Apollo 14 (the mission) the sequel to Apollo 13 (the mission).

Was this the scenario where there was a false positive warning light but they had no way to test if it was truly a false alarm? I remember attending a talk by an Apollo engineer who convinced the control room that the switch design had a propensity to a short and it really came down to a probability-based judgement call

Honestly that movie is a lot of why I got into math and engineering.

Jim Lovell was undeniably a badass but I watched that movie and thought the heroes were the ones reading telemetry off a computer screen and using their slide rule to figure out what to do. I hope Hidden Figures does that for another generation.

Not was. Is!

I noticed on the last season of the Expanse that Luna headquarters was named after him and bothered to look him up. Dude's 93 and still kicking!

It's amazing how well the astronaut medical screening worked. Unless they get killed in the line of duty, these guys are all living incredibly long.

Like John Aaron, the steely eyed missile man!

This would be cool for earth satellites.

On the other hand watching a stream involving something on Mars, let alone voyager, would be pretty boring!

Send: ls

Ok let's take a 20 minute break.

From my work experience its like this: 1. Assemble commands to run 2. Run the commands and see results in the 15 minute window 3. See if you can do more commanding in the minutes you have left 4. Make a new plan, wait for next pass, and goto 1

For LEO satellites, that usually means you have 2 blocks of 3 13 minute passes, when the groundstation is in The Netherlands. For a Svalbard groundstation, you get a lot more, but still 13 minute or less passess.

I've worked on something like this, just a lot more mundane. We had Linux PCs strapped to the ceiling of various locations, mostly malls, together with a camera and projector to produce an interactive display on the floor. I had a couple of times where somebody would be onsite and the projector would be off or the display would be mangled. And it takes quite a while to get a lift to get up to the box (if it would even be allowed at that time of day), there was no network at that time, and all they had was a wireless IR keyboard that occasionally dropped keypresses.

Imagine dictating shell commands, over the phone, to a salesperson who has no idea what half the characters are that you're asking him to type, and the only output signal I could come up with was ejecting the CD tray, which was just visible from the ground...

(Note that the goal wasn't usually to fix things on the spot, it was more to triage things like whether we needed to have a replacement projector on hand, which was a big deal.)

Job Posting: NASA programmer, needs at least 1 wpm typing speed and experience with compiling large projects.

The amount of preparation is much, much more than every other "accessible" installation. Typos are the worst to recover from, backspace usually doesnt exist. As I've sent commands to our Linux-running satellites, its usually prepending your commands with ctrl-c characters and a couple newlines at the end, just to make sure it runs and nothing is left in the buffer. There is also a possibility that commands get executed multiple times, and there are usually limits in transmission speed, processing speed, and frame length. Sending a lot of characters over a terminal can cause characters to be eaten, creating typos you can't see, affecting the commanding immensely.

Isn't there some way to ensure that what you typed is what is being executed? Dropping characters from the terminal sounds terrifying.

I don't know enough about ssh and terminals to know if it's possible to type "12345" and see "12345" echoed back to me but really what the remote session sees is "1245".

Yes, terminals usually echo back the characters. In our case this would be buffered and we could request the buffer. But that would still take some operations. Best way, usually, is to send a bunch of commands in a way you ensure proper order of execution (eg write a file, check checksum of file, execute file), and make sure you can pull the logs afterwards.

Nowadays, links and systems get easier to work with, and you can sometimes have a literal TTY open to the system, like Reactor Hello World has ( https://reaktorspace.com/reaktor-hello-world/ ). However, this is over S-band, which is a 2Mbit/s link, so overhead for a stable TTY (or ethernet connection) is a lot less than using UHF/VHF.

Very fascinating! You haven't happened to written a blog post or something on this, have you? I am sure HN would love reading about it.

Sorry, I did not. There are plenty of stories on the internet about cubesats, they get launched by universities even :)

I'm surprised theres no error correction in your uplink. Crazy.

One of the more interesting projects I ever had to complete was while awaiting clearance when I first started working for the NRO years back in ground processing algorithms (no longer work there, by the way). One of the long time guys gave us an assignment to implement the BCH error-correcting code, and then iteratively optimize it until we were able to implement the algorithm for computing the code published in a company proprietary whitepaper that was more efficient than any of the publicly published algorithms. That was just everything that is fun about programming. The actual production implementation corrected up to 8 bits per block, though I can't remember the block size any more.

Ground processing was a totally separate program and contract from mission control, though, so my team only ever received data from the satellites. We never sent data to them.

It depends on the API. If your API is "put this data over uart to the TTY", and the uart of the device is overloaded and drops characters... Or maybe mangles characters due to bitflips. Or what have you. Its all possible!

Best programmer. -1 wpm.

I'm so fast that I can do -127wpm. Only in certain software though.

Only needs a keyboard with one key.

A telegraph key: https://en.m.wikipedia.org/wiki/Telegraph_key

Time to bring back flowchart templates.

If you ever worked on a busy mainframe your compile jobs could easily be queued for 20-30 minutes. Made you much more careful to check for typos and do test runs of the code "in your head" before submitting.

In case you haven't seen it: https://apolloinrealtime.org/13/

Hey there. I'm the author of the website. Thanks for the post. Happy to answer any questions.

Wow - so here I am an hour later... Thanks for that link!

What a fantastic website. Reminds me of Encarta 97

Thank you for sharing, this is amazing

Fantastic site. Thanks for sharing.

Fun weirdness of even limited multilingualness: For some reason my brain first parsed this as "a pollo in real time" - or, from Spanish, "a chicken in real time".

It's worth noting that the people in that movie were WAY more loud and emotional than the real NASA engineers and operators.

You can see how NASA people react to tough situations by watching the videos of mission control during the Challenger and Columbia disasters. No shouting. No arguments. Just cool professionalism and restrained emotions.

They have a job to do, and they do it well even under stress. "Steely-eyed missile men/women" indeed.

>the people in that movie were WAY more loud and emotional than the real NASA engineers and operators.

There are plenty of NASA engineers and leaders who lose their cool. I’m only saying that so people don’t overly lionize them in a way that prevents them from pursuing a similar job because they feel they are somehow cut from a different cloth.

Everybody knows that when presented with the irrefutable evidence that the Challenger o-rings would fail, they more or less just let the astronauts die. Definitely cut from same cloth as any other org.

That’s not quite accurate. It wasn’t that there was “irrefutable” evidence that the o-rings would fail, it was there wasn’t data that they would, or wouldn’t, fail.

“The O-rings were never tested in extreme cold.”[1]

There wasn’t data which led to discussions about uncertainty, but that shouldn’t be conflated with irrefutable evidence of failure.

The obviousness of it (like many engineering failures) was only apparent in hindsight.

“Evidence, in retrospect, points to a long period of time, especially based on post-flight inspections when the joint design weakness was ‘sending a message’ and the true potential of this message was not perceived and reacted to.”[2]

“Not perceived” isn’t compatible with “irrefutable evidence that it would fail”.

[1] https://www.space.com/31732-space-shuttle-challenger-disaste...

[2]https://www.govinfo.gov/content/pkg/GPO-CRPT-99hrpt1016/pdf/...

I dunno. That sort of thing is exactly my job, except the remote devices are still on earth somewheres. What you'd see is me sitting in a library drinking coffee and looking at source code and schematics until I had an answer that matched the evidence.

Satisfying, but not exactly must watch tv.

> Satisfying, but not exactly must watch tv.

What's in your head could be though. That's my pet theory on the movie Hackers, what we're seeing on the computer screens isn't what's actually there, it's the characters' mental constructs visualized.

you just killed any future dramatic space troubleshooting film scenes for me

Debugging on a computer that is down is not a very exciting process.

In the movie Hero[0], two kung fu masters fight a battle purely in their minds. And when the mental fight was over, they only execute one physical move to finish the battle.

Think of this as the computer equivalent of that scene.

[0] https://www.imdb.com/title/tt0299977/?ref_=fn_al_tt_1

In case anyone wants to watch a clip of the fight:

https://youtu.be/AeeoEpmyb2Y

_Hero_'s always been one of my favorites. A lot of kung fu movies try to strike a balance between aesthetics and realism - I really enjoy a movie that picks one (in this case the former) and goes all in on it. It's got a fight that takes place entirely on the surface of a lake, and another that takes places in a forest of falling leaves that change color several times throughout the scene. It's an incredibly beautiful movie.

Both: Dance around for 10 minutes trying to out-physics each other

Guy 1: Why don't I just poke him with the pointy bit

Great scene though, makes me want to watch the whole movie.

Yeah, this is strictly artistic kung fu, which is basically high-mortality ballet. There are also many "realistic" martial arts films, if that's your thing. I enjoy both styles, depending on mood.

Agreed. Wuxia to be specific, which is kinda like fantasy kung fu and has a long tradition.

It includes powers like becoming weightless, killing with a single movement, flying, etc.

Like the battle between Sherlock and Moriarty in Sherlock Holmes: A Game of Shadows?

Yes. A lot of Western action movies owe their inspiration to Chinese movies (and I suppose, viceversa). In this case Hero (2002) -- or a similar movie, since I doubt it invented this trope -- is likely an inspiration for A Game of Shadows (2011).

HK movies clearly inspired the action movies of the 90s.

What a wonderful visual representation of the notion that, "a battle is won before it begins."

Which is a Tsun-tzu reference presumably, he says don't enter a battle unless you have 'already' won (through preparation, numerical supremacy, etc.).

Ugh, such a good movie... If you haven't seen it, do yourself a favor and go see it.

This sounds amazing!

No, but "Debugging on a computer that is down... in space," does sound more interesting, right?

You have a computer that you can only interact with over a radio link, and need to make it start working again with only what you know about how the system is built and a limited set of remote commands. Sounds like something I'd get obsessed with solving.

There is a game in here, somewhere, somehow.

Paging Zachtronics (https://www.zachtronics.com/).

Speak for yourself

I did something like that when trying to boot my brand new root server in Finland a few weeks ago (tried ~50 times while having UEFI enabled plus mdadm raid1 on GPT partitions, never worked, asked support to disable UEFI, worked).

Confirming that not being able to ping/connect to it during the failed attempts was absolutely not exciting :)

I'll settle for the post-mortem

pg down

Sigh.

pg down

Sigh

...

Repeat for hours.

especially with extremely long response times.

I think what you're really saying is you'd watch the movie version of this.

And considering there are no life or death stakes, it still wouldn't be as exciting as Apollo 13.

That might not be true though.

This guy took about 30 hours of video of him porting an 80s version of unix to the ESP8266. Warts and all -- live!

I've started to watch it and it's fascinating!

https://www.youtube.com/watch?v=cDHcGY7EzUM&t=62s

You could have a whole channel with different teams debugging satellite technology and if you're bored, it would probably be quite interesting. The bigger problem is most likely concerns about IP and secret protocols and so on.

"And now Bob will log into TeleSat123 via SSH." <We see bob type in root / password123>

"Oups, uh..gosh we'll just go to a commercial break!"

I guess the last thing you want when you're debugging something during your work is for the whole word to watch over your shoulder.

I am halfway through the second season of For All Mankind, it's a series based on an accelerated (versus the decelerated) series of events around the time of the Apollo missions. I think you may enjoy it!

I have no idea if it would be more or less exciting than a tech company warroom.

Me, earlier this week:

Did that work... no. Well, what about... THIS... still no. 3 hours later... clear the cache?!? Aww crap

Someone is going to suggest unplugging it and plugging it in again, i'm sure

Reply on news.ycombinator.com