1,016 karma · joined August 31, 2014
When you say "cross-entropy loss" people without stats background go to Wikipedia, take a glance, and adjust their mental model to "inscrutable magic".
Thinking of the main difference as the trade-off in how much CPU, memory and storage is allowed is not really wrong.
The part that is wrong is to think of gzip as a method that might reach similar complexity or generalization. And more importantly, to ignore the advanced way how training data gets curated or generated for (instructed, chain-of-thought) LLMs. But even then. The mental model that the LLM's goal is text compression is not wrong. The question to ask next is what kind of text it is expecting to compress.
Or like a machine learning lab claiming SOTA on a benchmark, beating a well-known method that they re-implemented, possibly with bugs, on their private dataset, for millions of compute. But you don't get the source to check, and they don't release any intermediate results or ablation experiments. Aka, from the outside you can't distinguish it from corporate marketing.
Compare that to being asked for advice, to explain directly something you know or have done, with the expectation of the person asking to learn something. Like asking why some decision was made in the past, based on what information and context, instead of confidently saying that it was done wrong. Both will eventually get you the same information.
Or a login form that gets hidden after login, but clears the username and password only when you click "login back in". (Bonus points if the backend also enforces a 5min session timeout "for security".)
AI can mirror this style of writing, but it can't mirror the clarity of thought behind it.
The author is: https://sigmoid.social/@pynicolas@mastodon.social
It's my favourite puzzle about meditation. You have to approach it sideways, but if you really try that it won't work. So you try hard not to that instead, which won't work either. It's kind of funny how focusing on your breath tends to work, but... don't hold that thought.
It's a contradiction, and that's okay. Get comfortable with it ;-)
I think it's the other way around. The human soul requires the art. Or rather it craves the process of creating art, more than art-the-trade-item. When image generation was new, someone raged: "They want to build a genie to grant them wishes, and their wish is that nobody ever has to make art again."
> As for proving that you wrote it...yeah that could be tricky.
On social media, when people share their drawings they often share a photo with their tools visible (pencil, eraser) instead of a clean scan. It sends a message like "Look! I made this!". Or they post a time-lapse. Or details about their process, like here: https://www.viruscomix.com/makingofpartfive.html
The point of this is not mainly to "proof" anything, I think, but more a community signal that they care about their craft and are curious how others work. You can probably generate a convincing fake process with AI now, but... I wonder if anyone is willing to go that far.
Instead of wasting some resources on corruption and bad judgement, we now waste them by forcing people to chase the wrong goal. In other words: "If you rely on incentives, you undermine values." (Barry Schwartz)
So I don't disagree with you, but I think we went too far into that direction. We have allow people to use their better judgement sometimes.
Imagine a world where those only get done for a business incentive by people with this hard-reality business mindset. Or not done at all, reinvented at every place, no transferable knowledge for developers.
After a video tutorial I always concluded that I had a few really clumsily ways to work around my lack of knowledge. Some workflows really need an expert demonstration, or you won't even know what is possible. (I do pause the videos to look up the docu.)
Not the most common way to make money from videos, for sure, but it's not "nobody", and they need hosting.