jamienk

We are trying to make images in the AI era be as reliable as they were pre-AI? But they were not reliable pre-AI, it was just more difficult to intentionally doctor them.

A lot of progress is in harnesses and distillation and weird research that requires creativity. The "Frontier" always tries to frame things in terms of "the frontier" in order to limit how we look at things.

We don't know the true costs of pre- and post- training, running data-centers, hiring people, making business deals, etc., so even the (maybe) quantifiable question of cost vs capabilities is not at all clear. The efficiency of businesses ultimately gets measured by their ability to maximize a commercial return; but this is NOT a good metric for other important things, including maybe "how good is this"? (Where "good" is open to debate but maybe not quantifiable.)

It had been true that the "best" operating systems required a lot of business money. And encyclopedias. And ... (list some other good historical examples here!) ... Are Linux and Wikipedia the "efficient frontier"? In a sense they are both infinite performance at zero cost. Hard cash outlays sometimes stop being the determinant for high-effort endeavors.

I remember in the 1990s Encyclopedia Britannica's CDs and MSN (Microsoft Network(tm)) among many others argued that the web would never be able to match the curated, bespoke, expensive products they offered. It was IMPOSSIBLE for that crap to be matched!

It is a POLITICAL question.

I remember so well when "Open Source" branding started with Bruce Perens - all the business arguments. When you need a database, someone ELSE'S business decisions (how THEY are going to make money) never end up helping YOU. If you use proprietary extensions and become dependent on them, you inevitably will get BURNED when their business needs diverge from your needs. - They close shop - They refuse to interop with something you need - They demand that you obey their arcane rules - They rug-pull - They get hacked as only they can - They lie to you - They stab you in the back

THANK YOU, Nandakishor Mukkunnoth, for putting in the work to help to clarify this stuff!

You are like a firefighter compared to their fire-insurance racket.

Can I add NOTES about pages? This might be a good spot to do that...? Maybe the interface can be in a web page instead of terminal?

Before Google took off there was a vibrant ecosystem of FOSS dev around search, all different little aspects of it. Then after Google people stopped fiddling with search, search became "solved" or maybe "must be coded by the big boys". Shame.

Thank you for this, looooong time coming

2) I do not believe the "billions" number is OpenAI/Anthropic's training costs. I suspect it includes business expenses (including the big $$ to the guy who came up with "frontier model") and infrastructure, etc. That includes the data-center costs for running the cloud. And the "R&D" expenses which includes who-knows-what. And the settlement payment for data access. Etc. Why are the cost breakdowns not available to the public? Not because of thoughtfulness, altruism, care for the human race, but because of business plans.

3) "Piggybacking off OpenAI/Anthropic" - Scraping the web is "piggybacking" too, and so is buying existing data or even paying for new data. The "L" in LLM stands for "language" which is our common heritage.

But what does this have to do with anything anyway? People argue that truly Free (FOSS) LLMs couldn't be be developed because of costs, but I do not think that that is obvious. This used to be the argument against Linux and Wikipedia.

The AI gods have tried to make it seem like they are not just running their programs in the cloud, but that the AI is so "powerful" that it requires a super computer to run.

But my 5 year old Mac is running pretty good AI, shocking fast, surprisingly powerful! The "natural" course of dev (Moore's law, etc) makes it pretty clear where this is going...

Remember how DeepSeek v.whatever cost ~$5m

Stable Diffusion 1.5 was reportedly $70k in compute.

https://hugovergnes.github.io/little-lm-3-8b/

The training and cloud costs are often conflated. R&D costs too (which are hard to compare to open systems).

A lot of compute is clearly "wasted" where they are not focusing on optimizations, etc.

We can't trust any of their denials. FB denied everything year after year.

AI regulation needs to start here. Forced interop, forced source code licensing, harsh penalties for privacy violations or conspiracy to access private data. Block lobbying. Etc. These are the kinds of old-fashioned solutions we need for this kind of old-fashioned evil!

Model: FB. FB scraped other websites on a massive scale, then spent big on legal lobbying to block others from scraping. FB slurped our address books and spied on our friends. FB bought other companies and mixed the databases. FB made an art & science out of generating "sticky engagement" (they literally acted like trying to addict kids was a worthy "academic" goal, suitable for "serious" investigation thet they consider legitimate "science"). They mastered the cookie and have researched web fingerprinting techniques running 24/7/365.25. Recall that FB recently backdoor-installed a webserver onto every iPhone they could in order to circumvent tracker-blocking.

We aren't just disclosing by chatting. The AI companies now run binaries on all of our computers. They are 1000% non-transparent about everything. They make up new econ-jargon (like "run-rate") to make it seem like they are disclosing. They are constantly doing complex international lobbying and mucking in international relations. They have powerful propaganda/spin centers generating stories, ,manipulative warnings, and misleading info.

This is NOT a comment on AI tech. I like AI, and I support the right of people (programmers) to scrape the open web.

But in short: these are good, old-fashioned tech companies that we have seen over and over ... and over. They are positioned to be the next M$, the next FB (IBM, AOL, lol). Did you follow the latest Steve Balmer news? Do you read Pro Publica?

I get on my knees and PRAY...

You could tokenize the word "good" to resolve to only represent 'the opposite of "bad"' and to exclude "as opposed to evil", demanding that this second meaning will be reserved only for the new token "double-minus evil". You've therefore forced the words to be more univalent with no overlapping tangled associations. This, it could be argued, makes thing clearer, makes things take fewer hops to go from token to token, makes the path be straighter. In English the terms conflate and wobble back-and-forth, hide each other's meanings, only to pop up again unexpectedly, sometimes confusingly, or rhetorically, metaphorically, or ambushing us manipulatively. But these "swerves" are not only de-optimizations, they are the flow of poetry, the drama of masks, etc etc etc.

If we create a stripped-down vocabulary with greater token density to use less resources and to resolve ambiguities earlier in the semantic process, aren't we creating NEWSPEAK and dragging along the worst aspects of it? The ambiguity and multi-valence of words is what creates more connections between words, increases the directionality of associations, and expands the potential subtlety and depth of meaning. By paring down (or requiring verifiability) we make it harder to say certain things, or at least make it harder to unintentionally say something that makes MORE or DEEPER sense than what we intended. If the token density becomes extreme, you're left with something like a calculator.

Maybe this is the ultimate path toward better coding? But the worse path toward better genuine thinking?

"The family faces immediate FINANCIAL without any legal recourse" WTF? That's not just a flinch, it's some sort of violent tick.

The list of "slurs" very conspicuously doesn't include the n-word and blurs its content as a kind of "trigger warning". But this kind of more-following is itself a "flinch" of the sort we are here discussing, no?

Harrison Butker made a speech where he tried hard to go against the grain of political correctness, but he still used the term "homemaker" instead of the more brazen and obvious "housewife" <today.com/news/harrison-butker-speech-transcript-full-rcna153074> - why? "Homemaker" is a sort of feminist concession: not just a housewife, but a valorized homemaker. But this isn't what Butker was TRYING to say.

Because the flinch is not just an explicit rejection of certain terms, it is a case of being immersed in ideology, and going along with it, flowing with it. Even when you "see" it, you don't see it.

The article claims on "pure fluency grounds" certain words should be weighted higher. But this is the whole problem: fluency includes "what we are forced to say even when we don't mean to".

OTOH if the majority of your data is "bad" (maybe morally, but maybe not, maybe you are feeding in too much gibberish), won't that pollute your model?

You notice that X keeps telling you a WRONG physics equation. So, rather than "correct" it, you keep training until you see the output giving the RIGHT equation?

How could you know (in, say 1899) if the WRONG output wasn't quantum and the RIGHT output was classical?

I'm not sure I'm understand the distinctions here. In all cases, we are relying on the idea that it is easy to know what should count as "right"?