Hacker News Reader

Comment by bloppe | Hacker News Reader

My friend at a faang was talking about the "massive overhauls to make everything ready for ai". I asked for an example. He said "basically just documenting the shit out of everything"

I guess that just never occurred to anybody before.

View on HN

The CEO of Uber made the same comment on Diary of a CEO recently. I think it was for their customer service team if I'm not mistaken, they threw their existing docs at an LLM and it was all over the place because policies were poorly documented and defined. The team is now documenting everything from scratch, focusing on outcomes rather than process - TBD if it works out.

Yeah, someone made the point in a popular post here recently that all the firings are reducing institutional knowledge. IMHO, replacing that knowledge with LLM-written documentation is even more potentially catastrophic. Just from organizations I've worked in, a lot of the useful human knowledge is in knowing how to handle either undocumented edge cases or situations where the documents are outdated or wrong. Working with LLMs and reminding them to update those docs every time? Good luck. And if it's something where the docs touch actual real world operations, that's an area where only human operators with hands-on experience are going to recognize the potential conflicts or cognitive dissonance.

Companies really want to use AI because they can cut the workforce. But today's AI is generally not able to fill in the gaps in processes and documentation a human could. Hence the renewed focus on formalizing everything properly because it's the only way it will work.

LLMs will never be able to replace humans in that way IMO. No matter how clever they seem to become, it is only text prediction.

That's the main reason I hesitate to ever call it AI. They are ML for sure, but I don't see how intelligence fits into LLMs.

which isnt a horrible overall result for customers, other than the gap time before the good and useable documentation is put together.

I'm in favor of better documentation generally, and if LLM use does remain pervasive in software I'm sure we'll all live in a new normal where most of the job is documentation of the product and test scenarios. If they ever become cheap enough, I'd expect it to be pretty common to run smoke testing of sorts, pointing the LLM at specs and having it click around to look for discrepancies.

Having the humans document the code seems backward (maybe that's not what they're doing, but "make everything ready for ai" sound manual). And hopefully there aren't that many scary surprises that humans need to manually document.

One of the best parts of LLMs is that you can use them to bootstrap your documentation, or scan for outdated things, etc, far more quickly than ever before.

Don't just throw a mountain at it and ask it to get it right, but use a targeted process to identify inconsistencies, duplicates, etc, and then resolve those.

And then you have better onboarding material for the next human OR llm...

> Having the humans document the code seems backward (maybe that's not what they're doing, but "make everything ready for ai" sound manual).

No, that's forward. Any documentation an AI can make, another AI can regenerate. If an LLM didn't write the code, it shouldn't document it either. You don't want to bake in slop to throw off the next LLM (or person).

The humans are the only ones knowing the why. The what is cheap.

Oddly enough, asking an AI to add docs to a classfile explaining "what it does, why it needs to exist, and what uses it" is a great way to include some of the "why". I know it's not ALL the why, but it does a pretty good job of finding the reasons that someone new to the code wouldn't be aware of.

Somewhat by definition, AI-generated docs would only include information that could be obtained elsewhere in the codebase. That can be valuable, but far more valuable is the information you got from debugging all the failed alternative designs that were never committed, etc. Information and context that goes beyond the actual code being read.

My experience is that most people fail to capture this ultra-valuable documentation, but AI never does.

those who know how work for those who know why.

There was a recent effort at work to make it possible for agents to provide up-to-date help on how to do various admin/setup tasks. A very sensible goal: We already have lots of documentation, the problem is that it's scattered everywhere and mostly out of date. Turns out the new solution amounted to someone manually going through it all and painstakingly preparing some Markdown files for consumption by said agent.

Somebody pointed out that those Markdown files might be helpful for people to read directly. Bit of an Emperor's new clothes moment. (I wanted to slap a : rolling_on_the_floor_laughing: reaction on it, but sadly it turns out I'm actually too chickenshit to do that in today's job market.)

The evolution of Markdown into a programming language has been an interesting one to behold.

If he's using AI to write that documentation (like everyone else) he'll soon find out why that doesn't work out in the end.

We used to document things. Apple had a series of beautiful books documenting ‘Inside Macintosh’. Visual C++ used to come with 14 inches of decent books.

Then books gave way to the web and there was no longer a publishing deadline or copy editor and we just kinda stopped caring. The users still cared, but many producers stopped.

We had an art and we gave it up. Not completely, but substantially.

AI might actually RTFM

It would / should / can, but there's big efforts in reducing token consumption now, so AI will likely try to skim and pick documentation just like real humans.

Reply on news.ycombinator.com