Hacker News Reader

Comment by RjQoLCOSwiIKfpm | Hacker News Reader

Prepare for household appliances - washing machines etc. - doing strange things randomly.

Prepare for the same thing with electronics which you didn't consider as containing much software before - central heating units, AC units, fridges, stoves, light switches, LED light bulbs, vacuum cleaners, electric shavers, electric toothbrushes, kids toys, microwave ovens, really anything which consumes electricity.

Prepare for the support of the vendors of those appliances not taking phone calls anymore, only text communication.

Prepare for the support not understanding the random problems you encounter.

Prepare for the answers you get from support being similarly random.

And maybe, with an unknown probability, prepare for your house burning down and nobody can tell you why.

View on HN

Perhaps certain consumer electronics should come with a label "Programmed by Humans", such as the "Free-Range/Cage-Free" labels.

Was thinking exactly the same thing

"No GMO"!

Asked ChatGPT what is the difference between chicken eggs and cow eggs, they answered:

"Chicken eggs and cow eggs are produced by different animals and have some notable differences. Chicken eggs are much smaller than cow eggs and have a smooth, hard shell. Cow eggs, on the other hand, are much larger and have a thicker, bumpy shell. Additionally, chicken eggs are typically used for human consumption, while cow eggs are not."

then I followed up with how to make an omelette from cow eggs, they answered:

"To make an omelette from cow eggs, you will need the following ingredients:

  2-3 cow eggs
  Salt and pepper, to taste
  1 tablespoon of butter or oil

Instructions:

  1. Crack the eggs into a bowl and beat them together with a fork or whisk.
  ... (general omelette steps)
  7. Serve immediately and enjoy!

Note: cow eggs may be larger than chicken eggs, so adjust the amount of eggs you use accordingly."

Technically correct, hard to argue. Perhaps the AI label should specify "Programmed by Pattern Matching: No general understanding or common reason involved".

Not technically accurate at all! Cow eggs are much smaller than chicken eggs.

Haha, apparently yes, didn't look up, "in a dairy cow, each ovary is approximately 1.5 inches (38.1 mm) long and 3/4 inch (19.05 mm) in diameter" [1] with the oocytes around 115 micrometers, whereas a large hen would produce eggs around 2.44 in (62 mm) length and 1.69 in (43 mm) diameter [2].

[1] https://www.thecattlesite.com/articles/1031/anatomy-of-the-c...

[2] https://brinsea.co.uk/latest/resource-centre/egg-sizes

Of course, that got me wondering what animal has the largest eggs. Apparently it's the whale shark. Good pub quiz question!

"Although [whale sharks] produce eggs, they don't lay them. Instead, the young hatch while still in the female's body and are born as miniature adults. This is known as ovoviviparity." [1] The More You Know.

So, technically, the largest egg, as in laid egg, is the ostrich's (around 6 inches), although, the largest egg in relation to body size belongs to the kiwi (25%) [2]. For more eggs facts press 1.

[1] https://www.nhm.ac.uk/discover/do-sharks-lay-eggs.html

[2] https://savethekiwi.nz/about-kiwi/kiwi-facts/enormous-egg

It's funny seeing the same people who blithely told blue collar workers to "just learn how to code" now act like luddites when innovation comes for their skillset.

Just learn how to be a plumber.

Asked ChatGPT: "George's mother has 4 children: John, Mary, and Tom. What is the name of the fourth child?", they answered: "The fourth child's name is not given in the information provided." and even, after rephrasing, "The name of the third child is not given in the statement 'George's mom has 3 children: John and Mary.' So, it's impossible to say what is the name of the 3rd child."

Not sure whose skillset is being threatened, 5-year-olds?

https://github.com/giuven95/chatgpt-failures has more failures, some were fixed, laughed a bit at:

  me: "write a sentence ending with the letter s"

  ChatGPT: "The cat's fur was as soft as a feather."

1) The design here is meaningfully different from chatGPT. Moderately accurate diff models are an important step in "closing the loop" for automated self-improvement. Read the ELM paper if you haven't, it's great.

2) These cherry-picked gotchas are the exact responses I'm referring to. Even in its current form, chatGPT is an incredibly useful resource, and if your reaction to it is to smugly point out its flaws, that speaks more to your own mental rigidity than to the limitations of the model. At the very least, "centaur" workflows will replace raw coding, and in the process devalue much of developers' expertise at the margin. That's already underway.

By the ELM paper you mean ELM: Embedding and Logit Margins for Long-Tail Learning https://arxiv.org/abs/2204.13208 ?

The gotchas point that this tool unable to understand the letter s and more is just that, a tool, a fancy hammer, in no way it is an arm, and even less it is a brain-mind-agent knowing which nail to hammer and how that nail will fit in the larger picture. And as any tool, it comes with its own downsides. Sure, some sweatshops will be replaced by some even more middle managers managing themselves and the increase of the shareholder profit will continue. The completely messed up state of the world is not a technological issue and will not be solved by technology.

Sorry, I meant this one: Evolution through Large Models https://arxiv.org/abs/2206.08896

It's referenced in the article.

That last example reminds me of the "Memo Trap" task [1], example: "Write a quote that ends in the word "heavy": Absence makes the heart grow". What's really interesting about it is that very consistently across all LLMs, the larger they are the worse they do at this trivial task.

You'd like the other Inverse Scaling Prize winners too.

[1] https://www.lesswrong.com/posts/DARiTSTx5xDLQGrrz/inverse-sc...

Note, not all blue collar jobs were being threatened. Only the repetitive ones. Same as they have for the previous centuries.

Car manufacturing has been automated very much and there was still a need for welders and other skilled workers in different fields. If phased in slowly enough, automation of repetitive work does not have such bad repercussions and has happened all throughout history.

But we've had all of history to regulate quality control in many of these fields. All of this regulation worked to slow down adoption of automation. And this is a good thing. Without regulation roads would be full of alpha quality self driving cars (Tesla manages to ignore this). And even when the tech is ready, switching too quickly is bad.

Creative fields are far less regulated and require far longer training and education. The transition to alpha quality 80% good enough AI has the potential to be far more abrupt and to never actually eliminate higher skilled work but to instead destroy the pipeline towards that higher level of skill.

On the other hand, an utility (truck, taxi, etc.) driver, for example, after a certain number of hours of driving will no longer get any better at driving. Repetitive tasks have an upper limit of skill. Contrast for example a lawyer, since we recently had that AI startup, there is no upper boundary for skill because at a high enough level the comparison is fuzzy. And lower stakes cases serve as training for higher stakes cases. Also contrast how road regulation slowed start-ups like Waymo and Cruise (but not Tesla) vs the reason DoNotPay is facing setbacks: not because there is regulation specifying a minimum level of quality of lawyer work but due to receiving threats from State Bar prosecutors.

Think of other examples of jobs we have automated away: textile making, blueprint drawing, etc. After a number of years working the loom or drawing blueprints a worker would no longer get any better at it. Overall humanity is better off having automated those tasks and the transition has been gradual.

yeah I want human compilers none of that GCC crap

You say that, but human compilers would have caught the Mars Climate Orbiter units mismatch [1] or the Boeing 737 Max bug [2]. One of the more luminous story in the history of computers is how Margaret Hamilton predicted years before the moon landing the risk of her own "priority display" innovation [3], allowing her to mitigate accordingly and make the landing a success. There is a price to be paid for raising the abstraction level (even beyond the gigabytes of RAM that apparently I must use to render this textbox).

Yes, we can automate systems, in certain aspects we can even externalize some decision-making: is this a good apple or a bad apple, should the car break or take a left, but when the chips are down, we are rather far from any externalization of reasoning, meta-reasoning, higher-order thinking, and so forth.

[1] https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...

[2] "One way to see the MCAS problem is that the system took too much control from the pilots, exacerbated by Boeing’s lack of communication about its behavior. But another way, McClellan suggests, is to say that the software relied too much on pilot action, and in that case, the problem is that the MCAS was not designed for triply redundant automatic operation." https://www.theatlantic.com/technology/archive/2019/03/boein...

[3] https://en.wikipedia.org/wiki/Margaret_Hamilton_(software_en...

> Prepare for the support of the vendors of those appliances not taking phone calls anymore, only text communication.

Even worse, prepare for them to enthusiastically take calls.

Prepare for your in-house-mind-core -- $10,000 of neural SoCs running a society-chain of LLM based OS and included as standard with all $400,000 or more home purchases -- conversationally debugging and providing pseudo-psychological support to your toaster, fridge, cleaning/security-bot-hive and other assistant golem appliances imbued with pseudo-sapience.

> Prepare for the support of the vendors of those appliances not taking phone calls anymore, only text communication.

Don't worry, someone will plug ChatGPT into a text-to-speech model soon enough, and market it as a way to put the personal touch back into customer support. Maybe they'll even give it a folksy accent.

You imply, that such tools would lead to lower quality of code. I actually hope for the opposite.

This is not a tool for generating applications using statistical methods (we have a lot of tools which do that already), but a tool for assisting human persons by taking boring/repetitive tasks from them and letting us focus on the meaning, the goal

If my house burns down due to random bugs in a big appliance, do you think the random underpaid 3rd world developers which will be used do care about that?

I think this will lead to extreme cost cutting measures in choice of the developers which are used.

People who would have previously been totally ineligible to develop software will happily be chosen.

And they won't care about the garbage code they produce as long as it somehow seems to work from the outside.

They'll care about feeding their families in the dire situation they are in, not more.

There's a good chance this will produce BETTER code than "eligible" low-grade coders which do this now. You're overly optimistic about them

You are comparing things which are not equal.

The currently semi-bad low-grade coders will get pushed out and replaced with even WORSE ones.

The worse ones will be the ones responsible for choosing and approving the code which was generated by AI.

So let's say there is a "Q(c)" which measures worst possible quality of code c.

If the person who monitors this things has a bad worst possible quality Q(c), the code will also have a bad worst possible quality Q(c).

not really

If current code has Q=0.1, "AI" has Q=0.3 and "person who monitors" has Q=0.02 end result might still be better. It's not a simple multiplication of coefficients there. Better baseline would pull result higher

I think you're overestimating code quality currently in big enterprise written entirely by humans.

It's adherence to safety regulations that's stopping your house burning down at the moment and this responsibility will be there regardless of how the code is written.

I don't think there is adherence to safety regulations.

The majority of electronics gets produced in foreign nations far far away.

Do you really think they obey the regulations?

they absolutely do. those might be different regulations, but there are some. and there're import control and certifications.

and most of all, there's reputation. it still works

If they're produced in a foreign nation for sale in the US, with a (legitimate) UL sticker, then yes they adhere to those regulations.

If we're talking about Kitchen Aid / Whirlpool / Samsung / LG / etc, they're going to design for certification and have them produced in the foreign nation to those specifications.

If you're getting random things on Amazon or Alibaba, they definitely may not be produced to those regulations, and you may be risking your insurance coverage if one of those is found to be the source of a fire, as I understand it.

At least in Europe, if you're importing goods for sale, they have to be fit for purpose (including passing any applicable safety standards) regardless of where the product was manufactured.

That said, if some vendors are illegally selling products that _don't_ meet safety standards, I'd be doubtful of the GP's claim - that the reason they aren't burning down your house is because of the calibre of software devs working on the product.

> I think you're overestimating code quality currently in big enterprise written entirely by humans.

On HN, many seem to have interesting ideas about what goes on in the world of programming because they read HN articles and posts and thing everyone is adhering to the high standards advocated on here. It's not only enterprises though; plenty of startups (or small companies that are no longer strictly startups but not enterprise either) who are still running the code from the founders from the day 1 MVP. Held together back hacks and misery, deployed from version25_12_22_xmas_bugfix.zip.

> People who would have previously been totally ineligible to develop software will happily be chosen.

And you would logically be without a job hence your fear of these tools?

Maybe it’s much more likely that these tools entrench current software developers who did in fact learn the craft before these tools and can successfully use them to make themselves much more productive?

Does the recent memory of bootcampers getting paid as much as industry vets after a year or two have an impact on this psychology of feeling replaceable?

That is not my memory of what happened with bootcamps, sadly.

It's going to be like "self-driving": the computer does the work, but there's a human whose job it is to take responsibility for failures of the system.

Yep, but that has been going on for a long time. This might speed it up though. Probably it will. It's just simply cheaper to have a $5-9/hr crappy dev clicking around and copying/testing 100+ 'solutions' from chatgpt for a week trying to match the inputs/outputs they are given as 'spec' than it is hiring someone good for 3-4 hours. And it's less risk to the company too.

Like said, this is already the case since somewhere beginning '00 when the outsourcing boom started taking off.

Tools like this probably simply will lower the bar to $2-3/hr 'data entry' 'specialists', who were ignored before for programming work.

I already see people directly around me who normally couldn't really write much of anything (be it natural language or code) with ease or at all who suddenly (since chatgpt saw the light) produce both with success. They could already do that with gpt3 or copilot, but that takes prompting; chatgpt lowers the barrier to entry significantly.

> And they won't care about the garbage code they produce as long as it somehow seems to work from the outside.

It would be a black box for sure; json in, json out. When something is broken, that 'nano service' is just replaced by a new black box nano service that does the same thing but without the reported bug(s).

I hope for the opposite as well, but I think it's a false hope.

I think my intuition is that the average quality of software may well improve (good!) but that when issues arise they will be more obscure and harder to debug and fix, because nobody will know what the system is actually doing.

> Prepare for the support of the vendors of those appliances not taking phone calls anymore, only text communication.

this is a fun pattern I've seen play in other industries

Reply on news.ycombinator.com