Comment by MBCook | Hacker News Reader

Comment by MBCook | Hacker News Reader

I used to work at a small company. We had a few remote embedded devices that did work and sent data back to the mothership over the internet. Their firmware could be remotely updated, but we were always very careful.

Well one day a mistake was finally made. Some of the devices went into a sort of loop. They’d start running the important process, something would go wrong, and they’d just retry every few minutes.

We caught the issue almost instantly since we were watching the deploy, and were able to stop updates before any other devices picked it up. But those that already got it were down.

We could ask the devices to send us the output of a command remotely, but it was too limited to be able to send back an error log. We didn’t have time to send back like 255 characters at a time or whatever, we needed to get it fixed ASAP.

And that’s when the genius stupid hack was suggested. While we couldn’t send up a full log file, we could certainly send up something the length of a URL. So what if we sent down a little command to stick the log on PasteBin and send up the URL to us?

Worked like a charm. We could identify what was going wrong and fix it in just a few minutes. After some quick (but VERY thorough) testing everything was fixed.

How you could have enough control over the machine to reroute the error log to (what I assume was) a Pastebin api, while also lacking access to any of the files on the machine? In my mind you'd be required to ssh into the machine to upload, and if you're ssh'd in, why not just cat the log?

Good question! We couldn’t SSH in, which is too bad this would all be trivial. We had no direct access to the boxes, they were often behind firewalls. In fact that was the suggested placement for security reasons. They weren’t full servers, just little embedded things.

We had a little HTTP API that it was always talking to. It would call the API to send data back to us or just check in regularly, and we would return to it a little bit of status stuff like the current time to keep clocks in sync, and a list of which “commands” they need to run.

Mostly the commands were things like “your calibration data is out of data, pull an update“ or “a firmware update is available“.

But one of them let us run arbitrary shell commands. The system was very limited. I wasn’t a developer directly on the project but I think it was just our custom software plus busy box and a handful of other things our normal shell scripts used. I assume it had been added after some previous incident.

I believe the basic idea was that during troubleshooting you could tell a box to return the output of “cat /etc/resolv.conf” or something else that we hadn’t preplanned for without having to send someone into the field. But since it was only for small things like that it couldn’t return a full file.

Luckily one of the commands was either curl or wget. So we could send down “curl -whatever /log/path https://pastebin/upload” or whatever it was. I don’t remember if we signed up for a pastebin account so we knew where it would show up or if we had it return URL to us in the output of the curl command.

This suggestion was literally a joke. We were all beating our heads against the wall trying to help and someone just said “why don’t we just stick it on pastebin“ out of frustration, and the developer on the project realized we had what we needed to do that and it would work.

> In my mind you'd be required to ssh into the machine to upload, and if you're ssh'd in, why not just cat the log?

Ssh on remote IoT class devices is works. The problem is rarely ssh, the problem is always some form of key management plus NATs in-between.

If you've got a few thousand devices in the field, public key management can become a a real pain, especially when you want to revoke keys.

I’ve worked at a company where our remote access was over a super slow modem line but the machine did have access to the internet.

I was doing some proxy soak testing for a company once where we had to run the tests from the server room but there was no non-proxy connectivity from that room to where we were monitoring the tests. Simple solution: output the progress to Dropbox, watch the same file upstairs. Bit of delay, sure, but better than having no idea how things are going until the 30-60min test is done (and no, we weren't allowed to sit in the server room watching it.)

Your company had remote embedded devices but didn't keep one "locally" for debugging issues?

There's a zillion things that you can't necessarily test for locally. When you have a fleet of IoT devices deployed in other people's environments, there's literally no way to test everything.

Your question comes down to "Why didn't you just deploy bug free code using perfect processes? That's what I always do."

I mean, cmon.

I never said they should test for everything.

OP's description suggests it was an error that's common to all the deployed instances that received the update, rather than some specific combination of environment and that deployment.

It would have allowed them to run the same deployment locally and use physical access (serial, a display, sd card, whatever) to capture the error log.

What they came up with is clever but it's very surprising that they needed it, especially given that they have very limited remote access to the units that are in the wild.

No, they said some.

Emphasis is mine:

> We caught the issue almost instantly since we were watching the deploy, and were able to stop updates before any other devices picked it up. But those that already got it were down.

You’re correct. I think every box that got it started having problems (or at least most), the only reason any were still up is that updates were scattered in case of think kind of incident (and to avoid hammering out poor little server).

After all this time I don’t remember what the bug was. We did have boxes locally that we tested on of course. But somehow this got out.

It might’ve been something that only showed up under certain configurations. It might’ve been something that just should have been caught under normal circumstances and wasn’t for some reason. It may have been something that worked fine tested off-hardware until some bug in packaging things slightly changed it and broke a script. Or it could’ve been a case of “this is trivial, I know what I’m doing, it will be fine“.

We were a very small operation so I’m not going to say that we had an amazing QC process. It may have been a very human mistake.

> I know what I’m doing, it will be fine

I don't think anyone is truly a programmer until they've learnt the hard way the outcome of "what could go wrong"!

Thanks for the update, happy holidays mate!

Sure! I’ve learned that lesson the hard way a couple of times myself.

Reply on news.ycombinator.com