503 karma · joined March 4, 2017
It's absolutely true that children learn (and even generate) language grammar from a ridiculously small number of samples compared to LLMs.
But could the availability of a world model, in the form of other sensory inputs, contribute to that capacity? Younger children who haven't fully mastered correct grammar are still able to communicate more sensibly than earlier LLMs, whereas the earlier LLMs tend toward more grammatically correct gibberish. What if the missing secret sauce to better LLM training is figuring out how to wire, say, image recognition into the training process?
Whether there's any fundamental truth doesn't much matter, everyone wants to bend the idea to fit their own agenda.
My own efforts don't quite earn me "the big bucks" that I read about at companies I'll actually never work for. Just a respectable east coast income that I shouldn't complain too much about. Don't know that I'd be happier anywhere else anyway. The bills get paid while I'm making the world a better place.
On a bad day, I'd say we do it because we're suckers for punishment. It's in our nature.
I relate to this so much. I'd offer an open letter to some poor anonymous soul who inherits one of my dumpster fires. Because I'll certainly never be given the chance to revisit anything.
Yep, my shit's broken. It was written long ago, under different constraints, for a different audience. I made the best trade-offs I could. I documented them! Though you'll probably never find my writings, and if you do trip over them, you won't understand them yet. Today, I'd scrap most of it if given the opportunity. But to whomever is now taking on the rewrite, you know that little nugget at the core which perplexes you so? The one that doesn't fit your mental model of any type of programming you've touched in your career? That there is the core value proposition that makes it all possible, and if you throw that out, you've doomed yourself to learning my wisdom the hard way. If you're at all up to the task, then the next few years of your life and the remainder of your days at this company will be spent fixing the same problems by reading the same papers and obtaining the same obscure domain knowledge. No, it can't be abstracted away. Yes, it's fundamental. No, no one else will really understand it either when you try to explain it, because no one else without the accountability you now have will sink enough of their time into building the mental model you've had to construct. Good luck.
These early MR references use the term to mean a spectrum from "Augmented Reality" to "Augmented Virtuality". The AV term never really caught on, and the Wikipedia link for AV [1] actually just redirects to their page for MR [2]. I can't think of many notable examples of AV at the moment, though something like "Mario Kart Live: Home_Circuit" [3], as mentioned on the Wikipedia page for MR [2], might fit the bill, in the sense that reality is being used to augment the game. Uses of haptic feedback in VR might also count as AV.
This usage of MR has nothing to do with how Microsoft began using the term. It's almost as if they embraced an academic term for the sake of marketing, extended the meaning, and successfully extinguished the original academic definition.
[1] https://en.wikipedia.org/wiki/Augmented_virtuality
[2] https://en.wikipedia.org/wiki/Mixed_reality#Augmented_virtua...
[3] https://en.wikipedia.org/wiki/Mario_Kart_Live:_Home_Circuit
Moreover, "Mixed Reality" strikes me as marketing-speak. Compare the search trends for AR [1] vs MR [2]. While both terms have been in use for a long time, AR was always more prevalent, with a notable rise in interest around 2009. My first encounter with the term MR was in marketing materials for Microsoft's HoloLens circa 2015. Now compare the search trends for MR [2] with HoloLens [3]. Searches for MR don't begin to trend upward until after the introduction of HoloLens, where Microsoft's marketing materials tried to make a big deal of said distinction and the rest of the world started to run with it. As a supporting argument for the original broader usage of the term AR, I'll cite discussions of a 2007 anime series, Dennō Coil [4], which explored various examples of fantastical AR usage. Probably just a coincidence, but the search trends for Dennō Coil [4] do precede the search trends for AR [1] rather closely.
Naturally, all the usual caveats about word meanings changing over time apply, but this one is a pet peeve of mine.
[1] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
[2] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
[3] https://trends.google.com/trends/explore?date=all&geo=US&q=H...
[4] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
Ability to BS and multiply, more or less. Feels like a waste of a question even for the one asking it!
> Yet some people get flabbergasted . . . since they think too much about specifics of each of the numbers they use and how wrong they might be. Basically, they have a hard time of letting go of the inaccuracies :)
Having a hard time letting go of inaccuracies seems justified if you're in a more technical profession. Too often there's good engineering reasons for not wanting to excuse loose reasoning with bad guesses. On the software dev side, there are also too many cases where those kinds of fudged numbers are inevitably used against us.
It amounts to saying that anything that benefits me is good and anything to my detriment is bad. Sure, there's a consistency to that. However, if that's the foundation of one's positions, it leads to all manner of other logical inconsistencies and hypocrisies.
The law isn't there to protect my feelings, so whether I mind or not is irrelevant. Artists have had to deal with shifting art markets for as long as art has been a profession.
> What if your clients tell you they bought the AI generated art instead of yours?
I'd be sad and out of a source of income. Much the same way I would be if my clients hired another similar but cheaper artist. The law doesn't guarantee me a livelihood.
I can appreciate that there are all kinds of potential "intellectual property" issues with the current glut of AI models, but the level of misunderstanding in some affected communities is concerning.
In an engineer-led environment without a project management class calling the shots, maybe I'd feel different, but whoever is ordering the backlog is always going to prioritize features and fires, because that's what drives their own career advancement.
The inevitable question is "well why wasn't it written correctly to begin with?" even if the reason, as is typical, was pressure from product managers in the first place.
Where it will never be picked up, because rework tickets never get prioritized over feature work until something breaks.
And when some future engineer comes along and asks themselves why the hell this code is the way it is, they'll have to git-blame, dig through commits, find PRs for commits, hopefully find referenced ticket numbers, read those, and play archeologist to try to reconstruct whether there's good reason for the existing code's being from those indirect signals. Whereas a simple "TODO" could just tell said future engineer what they ought to know.
I sincerely appreciate a well-written TODO, and I make every effort to write good TODOs for others.
Water "wants" to flow downhill. Gasses "want" to expand to fill their container. Genes "wanting" to replicate drive animals literally wanting to reproduce, and the incidental awareness of that drive in some species comes down to a certain molecular arrangement brought about by said genes. The genes are data, the minds are data, and the natural pressure is that that data which succeeds in reproducing will tend to keep reproducing. So in the original sense of the notion of memes as proposed by Dawkins, yes, information "wants" to be free, as that is its tendency. The only other option is that said data ultimately dies out.