Comment by esaym | Hacker News Reader

Comment by esaym | Hacker News Reader

Parent Full thread

>Technology robbed workers’ of what had been highly valued physical knowledge about a job: the precise way to jimmy a stuck gear, the sound a machine makes when something’s about to break.

There's an experience that lives rent free in my mind as well. As a teen, I worked at a bait shop that was on the Gulf coast. They sold mostly live shrimp but had other live stuff for bait as well. Salt water was pumped from the nearby innercoastal canal by two heavy duty swimming pool type pumps. From there, the water was split through several different pipes and manifolds and dumped into the live bait tanks. The overflow ran back out to the canal.

There was this one customer I was getting bait for and he meandered around the bait room randomly (customers weren't supposed to be in there but I didn't feel like saying anything). He ended up way in the back of the room where the pumps were and I could see him touching them. Figuring I needed to get him out of there politely I walked back there pretending to check on stuff. Before I could say anything, he says "You will need a new pump soon". Um "What?" I say. "Put your hand on this one". So I did, I felt nothing. "Now put your hand on this one". And that one had a really high frequency vibration coming through it. "The bearings are shot, it will lock up soon".

I was impressed. Granted for two more years I'd regularly "feel" the pump again, the vibrations were slowly getting worse. But then a hurricane came and flooded everything anyway so we got new stuff.

But still, anytime I see a high RPM electric motor running, I got to "give it a feel".

People who work closely with machines develop an instinct for how it operates. If you think about it, all mechanical machines are state machines (in the CS sense) and if you’ve had to work with and fix shit long enough your brain commits those experiences to long term memory.

Or more likely state machines are a highly simplified view of reality... Working with machines is a combination of probabilities, intuitions, experience, chance... Most work with machines is not 1 or 0.

Analog state machines are the older kind. Examples: gearbox, ignition switch, Big Red Button. They don't follow or respect the laws of digital state machines, but are reasoned about in a broadly similar way.

This is really obvious if you ride bicycles & do your own maintenance. You can quickly identify weird sounds/patterns with problems, because it’s such a simple machine.

I agree, and I'd say good computer guys also make good mechanics. The core skills are the same. There's a guy called M539 on YT who's the best mechanic I know and just started out doing it as a hobby aside his IT job.

It reminds me of an old story - The Handyman’s Invoice

https://www.snopes.com/fact-check/know-where-man/

I like this. It holds true for looking at code to some extent - what people call "code smell". But the main difference would be that failure modes in CS tend to be binary - a piece of code produces the expected result or it doesn't. And unless you do something to the environment, it doesn't wear down over time. Whereas your car's brakes don't go out all at once (hopefully), and yet it's guaranteed to happen eventually without maintenance, so a mechanic's eye is trained more toward what's currently going wrong with a known system than what might go wrong with a novel system.

> And unless you do something to the environment, it doesn't wear down over time.

Disagree here (at least in spirit?) - code smell can also point to maintainability issues - if the codebase is continually changed some code is more likely to break than others.

I've never heard/read/thought of "code smell" as anything BUT referring to maintainability issues.

When you're debugging a concrete issue and have a hunch what might be the cause, that's not a "code smell."

Code can smell and be perfectly maintainable. Sometimes it's even more maintainable, because it's so unnecessarily replicated that the patterns become obvious. You could even argue that the best precision code is the hardest to grok and maintain in the long run.

A code smell is a heuristic to detect maintainability issues, so while a heuristic can fail, if it fails consistently, you're using a bad heuristic.

Sounds like you're basing your heuristic on dogma rather than experience.

I'm kinda coming from a do-it-once, do-it-right background. Usually code I write just runs untouched for 10 years. So I'd classify any breaking changes to the codebase a change in the environment. But either way, something breaks or it doesn't. It's not like some non-obvious wear and tear is going to happen over a year or two on its own, if no human messes with it.

> But the main difference would be that failure modes in CS tend to be binary - a piece of code produces the expected result or it doesn't.

I think if you’re talking about CS and code you’re right - it’s a very pure and deterministic system. But if you consider software engineering at scale, you do get scenarios where failure states emerge in a gradual fashion, like the OP example of a worn mechanism. An example would be monitoring error rates on a large fleet of servers. You expect a certain background level of failures, but when the rate starts to rise you might be observing the beginning of some kind of cascading failure mode. The errors might be manageable up to a point, but it isn’t always clear when the system will tip beyond that point. Post-mortems of cloud platform outages often mention these kinds of failures, because they typically arise from complex interactions between different layers which put the system into a state which nobody anticipated.

I guess my response to that would be that any change of platform - like scaling up, replication, etc - is an alteration of the environment, similar to changing the geometry of your car's wheels by changing their size, getting different shocks or raising/lowering it. Then you could expect failure states to emerge, but that's when you're looking for unpredictable behavior. I think I was just trying to make the point that in a static system, code you write that works keeps working, whereas mechanical things break down. Emergent problems in mechanical systems come from normal wear, not from scaling.

Yes. The point is that software is not static. I work on mobile app that doesn't really have the scaling issue, but it's not that uncommon that a new OS update or phone model has new bugs that needs to be worked around or changes some undocumented behavior that the app accidentally depended on. It's not that difficult from mechanical wear.

In a previous job in early 2000s they did have some legacy software that was originally written for VAX/VMS and run on an emulator. Even that environment wasn't stable enough so they maintained some real VAX hardware just in case. And as far as I know, the hardware was physically breaking down.

"code smells" are almost never about reliability.

There is a reason I won't listen to music or wear noise cancelling headphones in the workshop. I wear ear protection, that dulls the damaging frequencies, but I want to hear when the drill is biting or about to skip, I want to know if the table saw is struggling or if the cutter head on the router is biting a little too deep.

Several forestry workers I know used the same argument: while working with a brush cutter, they prefer not to use a FM headset because it would distract them away from also hearing what the machine's engine is doing.

I personally have also felt less concentrated and more mentally tired while trying to simultaneously cut brush and listen to music (let alone an intellectually challenging podcast). As a tinnitus sufferer, I use both ear plugs + best possible earmuffs to reduce the dangerous frequencies as much as I can. Works fine, and I can still hear the machine, too.

Bench Dog protectors with the built-in ear plugs, plus 3M muffs. I have pretty bad tinnitus from my musician days and spending too long listening to the drummer pound as hard as he can "because he just wants to be heard."

When I hit a nail in some salvage wood, or the moisture was a little higher than I estimated, I want to know about it before it becomes a problem. You can feel the slip, and hear the fence creak on the saw long before you get a kickback. Only ever once had a kickback because someone distracted me as I was cutting.

The rule in the workshop is: You don't talk or call to the guy operating any kind of machine or power tool, even handheld. You get in clear line of sight, and wait for him to acknowledge you. You don't signal or wave or point.

A friend working at a photocopy shop could hear a paper jam from one high-speed copier when taking a break across the street. They'd be in the shop, opening the machine covers, before it came to a complete stop.

Well-running equipment almost always has a characteristic sound and rhythm. Variations from that become quite apparent.

I hear my computer’s processor when it’s processing. It’s like a modem sound, but high-pitch.

All of us used to rely on the HDD distinct sound to know when the games were finished loading ;)

ahah same. I think we hear the coils that are used to lower the voltage for the CPU. They are modulated based on the current demand.

My Dell XPS13 does that and it’s super annoying. It’s some kind of cross talk with the speaker amplifier because when I mute the sound or use headphones it goes away.

When I was a kid our BBC Micro used to do the same, you could “hear it thinking”.

Yep, my XPS does too. I believe it's called "coil whine" and is caused by surface-mount inductors whose windings aren't fully encased in resin or whatever material they use.

One I remember is knowing if Windows 95 was going to lockup and crash just by hearing the hard disk start to thrash - with SSDs swapping to death is silent.

And with many of these the difference between a journeyman and a master is the journeyman is saying “it’s never done that” whilst the master is already running.

Macbooks are the worst, they don't even HDD led, or even "switched on" led!. Quite often, I can't tell if it's on already and coming back from sleep, or do I need to press "Power" button again. What a frustrating design.

I was fighting the removal of temperature and RPM gauge on EV busses for this very reason. Obviously I was overruled.

You want to know what your stuff is doing and the reasoning behind the suggestion was litteraly "lets do it like Apple".

I'm not sure I understand this complaint in detail?

If a Macbook is asleep, it will just wake up, you'll see the keyboard light, there is no need to press the power button.

If there is a need to press the power button it is because the machine is off. Don't worry, it will make a distinctive and iconic sound once it is not off and display, you guessed it, a white apple logo on the screen.

What circumstances lead to your confusion about its state? I can't relate!

The machine can wake up for many seconds (I'm running Windows 10 on my Macboo Pro). During that time, there's no communication on the state of the machine at at all (the keyboard doesn't light up for me). Is it still asleep? During wake up? Or maybe powered off completely for some reason? If it's powered off, then I should press the "Power" button... But what if it isn't? - will pressing it kill the wake up process? Who knows! So much frustration caused by removing a single LED.

I think this is one of the cases where Apple doing hardware and software shines through. They’ve made macOS resume quickly from sleep, to the point where I’ve never really thought about/noticed the problem you’re describing. So the LED really isn’t needed for 99% of the users, the same way that you don’t have a power LED on iPhones or iPads. On recent models the login screen is there before you’ve fully opened the lid.

(It’s interesting in itself to track the history of power/sleep lights on MacBooks, since as far as I can remember they only turned on in the case where the internal display is off, or when the laptop was sleeping (pulsing light). Jony Ive even talked about it in the Objectified documentary: https://vimeo.com/7827217 )

My private laptop is fine, but my corporate laptop with a lot of bloatware sometimes takes a few seconds to wake up. From hibernation sometimes it takes 10+ seconds from hitting the space button to see the something is happening.

There are startups today that automate this process with machine learning. They record the vibrations from your equipment and notify you when something changes or is likely to fail given historic patterns for similar equipment.

Not just startups[1]. AWS, among others I'm sure, have at least attempted to turn this man's intuition into a little yellow box and an algorithm. Scaling to many thousands of machines at once, instead of just the ones he happens to touch.

[1] https://aws.amazon.com/monitron/

They are selling the delusion that factories wont need qualified mechanical engineers to maintain the machines and interpret the vibration data.

Many, many, many startups - this seems to be a very popular idea!

Ah, selling ML to the predictive maintainance market!

Now I’m commuting I’m fed up of how noisy the tube is compared to the newer trains, was BREL just crap?

I’ve taken to using an audio spectrograph app on my phone to watch the white noise, and the band of resonances.

Quite interesting in probably far too geeky away, I’ve not been called up for suspicious activity whilst innocuous I’m not sure how I’d be able to explain it.

I’ve also been thinking about tracking the sound of the bath filling to gauge how much is in it, just to track not to use. I guess air pressure, and water temperature play key parts.

Reminds me of the classic joke: A surgeon is at home, and the plumber comes to fix his sink. The plumber works for 15 minutes, the sink is fixed, and he hands the surgeon his invoice.

“$850?” the surgeon exclaims, in shock. “I’m a surgeon and I don’t make that much!”

“Yeah,” says the plumber, wistfully. “I didn’t make that much when I was a surgeon, either.”

My grandfather was a car mechanic after he come out the navy and always had a stethoscope in his tool-box. He'd listen around the engine before diagnosing crankshaft or valve maladies like some kind of auto-doctor. Come to think of it, machine listening techniques could be deployed in a modern mech-engineering to predict physical failures in advance given how easy it is to embed a piezo crystal and and TPU core.

> machine listening techniques could be deployed in a modern mech-engineering to predict physical failures in advance

It has been done for decades.

Yes, I remember reading something about its use on bridges and giant steel structues to listen for bad welds when the wind blows.

I guess for auto stuff there's so many embedded sensors in a modern engine there's little need these days.

There’s still plenty of bearings and pumps that wear or make noise without setting a code or having the computer tell you what to do.

They don’t use TPU cores, but most of not all modern commercial helicopters and aircraft have vibration monitoring and reporting. I would assume the same is the case in many other industrial systems as well.

We NEED a multifaceted version of this for computers, relevant to their internal state (and maybe HD errors). Something that would tell of meme use, mem fragmentation, cpu use and spikiness, locality of cpu, disk behaviour, working set size and so much more in a visceral form. I mean more than on-screen info.

(there's an old story of a guy sitting on ancient, giant disk drives to tune his program but not so much else I know of).

Why is there an apostrophe after "workers", and why has it caused me to read this six times?

It may have been caused by an incomplete editing decision. If you remove a few words the sentence becomes grammatical, if clumsy. Compare:

> Technology robbed workers’ of what had been highly valued physical knowledge about a job: [...]

...with...

> Technology robbed workers’ highly valued physical knowledge about a job: [...]

Maybe the original sentence was the second one and was edited for clarity, but they forgot to remove the apostrophe.

Reply on news.ycombinator.com