920 karma · joined May 20, 2023
It serves nobody except CloudFlare and hardware companies when one side set up blockers and the other side spend money putting VPN SDKs in consumer TVs.
I am also curious how the (Russian?) paywall bypass mirror archive.is is doing given that they are probably subject to similar amounts of traffic.
On the technical side, there's a lot of grunt work that's completely unrelated to machine learning but underpins modern AI. E.g. high performance computing and numerical techniques have zero relevance to day to day LLM research but makes or breaks the implementation. There are better reading list for those but you need to at least have a vague understanding of what numerical methods or a math kernel is before starting to lecture others on the economics of ML scaling and hardware. For stuff like data cleaning and scraping, it's a well known gray area field so you aren't gonna find too many for dummies guide on it. Can't have honest discussions on hacker news either because the people here get their panties in a twist over data scraping despite many of them doing it with zero hesitation or compunction if it comes up in a jira ticket. Proxy farms are the unsung hero of LLM engineering.
And regarding the data question the other commenters are asking — you scrape everything you can (oh look I used an em dash, wanna run me through the cover-your-ass Pangram?). Anna's archive, The Pile, the various Huggingface data sets and aggregates, Common Crawl. You pay proxy farms like Bright Data to run residential and mobile gray area proxies and VPN and CloudFlare bypasses to do more scraping. I see a lot of HN users up in arms on the front page thread about LG TVs having VPN SDKs within them etc. And then five minutes later they will go back to their frontier LLMs to print their pay cheque. Fucking hypocrites of the highest order. It's oh noes how bad this tech is in my house but we will happily benefit from it.
Then you have a data cleaning team deduplicate and clean and annotate the data (with or without help of more AI)
Certain RL specific datasets for supervised fine-tuning and RLHF like coding and git commits and chat needs to be curated by hand depending on your use case.
The level of discourse on AI has fallen tremendously on HN if 5 years into the AI revolution people are still wondering why datasets aren't being released. They aren't being released because they are a fucking snapshot of the internet for fuck's sake. There are a few "sanctuary" nations where AI data scraping is somewhat legally unenforced but the United States is not one of them so stop asking why data isn't released on a Bay Area website. Use your head for once. Too many React and YouTube influencers and the brains have been rotted.
Out of all the so called "AI engineers" here pontificating about "alignment" and "AI safety" and "Recursive Self Improvement", I wonder how many can even formulate or describe what an ELBO is. I wonder how many product managers here yapping about "recalibrating their priors" actually know what a prior is. I truly wonder why LLMs seem so magical to people when it's only a few steps removed from the same neural networks people have been using since 2015, at least architecturally (except scaled up by a few magnitudes).
Get your head out of Roko's Basilisk's agentic ass and maybe actually read the technical reports and papers for once.
(One important paper post Attention is All You Need is the DeepSeek paper where they used RL to bootstrap the "thinking" chain of thought token chains. IIRC it's the DeepSeek R2 paper. That's one of the most important papers for understanding LLMs beyond basic ML neural networks. If you need a quick way to get up to speed, read that one).
Nice in theory but in practice they still need a ton of training data. The bitter lesson is very bitter. Tesla's end to end FSD uses occupancy networks and brake stabbing still hasn't been fully solved.
Since when was Automatic a public listed company?
I am thinking it's probably an AI lab that misconfigured their data scraper (made it too agentic) and it ended up looking like a DDoS.
The new generation of scrapers are all agentic and self healing. (As an example see YC's https://parse.bot)
One of the most studied issues in Catholic evangelism is the Chinese rites controversy which is also one of the historical reasons why religions are suppressed in China (aside from ideological policies ascribed to communism).
https://en.wikipedia.org/wiki/Chinese_Rites_controversy
(That and the Taiping rebellion which was estimated to have killed as many people as World War 2 on the higher end. It was led by a failed scholar claiming to be alleged brother of Jesus. Most of the deaths were due to starvation. Not quite Netflix material but interesting plot nonetheless)
My personal criticism of data centers is mostly regulatory failure. The tech companies are choosing random ass states and stealing land from poor farmers instead of the traditional data center states like Virginia mostly because of tax breaks. The local government is just as if not more responsible for the issues with data centers. It's easy however to just point fingers at big tech instead of he underlying market failures in a lot of the states. The traditional data center states almost always have a supply of clean energy and water with a proper regulatory framework.
I know for some types of ML analysis, a separate model is already used to analyze the weights.
Are we talking about experimental laboratory ones here? What happens when the Alibaba players start getting into the game? They have plenty of humanoid robots.