Hacker News Reader

Comment by zitterbewegung | Hacker News Reader

AI has always had the concept of a “winter” or bust. [1] As in computer science in general we have had people shift concentration on what can make the most and that can occur during a PhD. The thing about the AI winter this time is that ML and Deep learning and LLMs advanced so quickly that a large bust in AI didn’t really occur. Also, I have heard about people getting the AI jobs and then complaining it’s all about cleaning data and nothing about applying new AI techniques to solve a big challenging problem. For CS in general and IT I can sort of see this as a business cycle and LLMs doing some replacement of jobs or it could just be a business cycle.

[1] https://en.wikipedia.org/wiki/AI_winter?wprov=sfti1#

View on HN

  > Deep learning and LLMs advanced so quickly that a large bust in AI didn’t really occur.

I think the problem is we railroad too much. It’s LLMs/large models or bust. It can be hard to even publish if you aren’t using one (e.g. build on a pertained model). The problem is putting all our eggs in one basket when we should diversify. The big tech companies hired a lot of people to freely research but research just narrowed. We ignored the limitations that many discussed for a long time and we could have solved them by now if we just spent a small portion of the time and money we have on railroad topics.

What I’ve seen is research become very product focused. Fine for industry research but we have to also have the more academic research. Academic is supposed to do the low TRL (1-4) while industry does higher (like 4-6). But when it’s all mid or high you got nothing in the pipeline. Even if the railroad will get us there there’s always other ways and ways to increase efficiency.

This is a good time to have this conversation as we haven’t finished CVPR reviews. So if you’re a reviewer, an AC, or meta, remember this when evaluating. Evaluate beyond benchmarks and remember the context of the compute power people have. If you require more than an A100 node you only allow industry. Most universities don’t even have a full node without industry support.

https://www.nasa.gov/directorates/somd/space-communications-...

Somebody from Nvidia donated some money to our university to buy a few H100s but it’s tied up in procurement hell between the CS/DS department and IT for almost a year now

I had gotten a contract with an unspecified California energy lab because they wanted to continue my summer internship research and it after 2 terms it still wasn’t solidified… so we moved on. People say government is slow but it wasn’t the government in my case… All these administrators drive up the cost but I’ve failed to see the value add

east bay or peninsula?

The one with the bigger computer ;)

Other machine learning techniques are still being researched . Deep learning / lLMs are just more popular in general due to being in vogue and the public can understand GenAI and what it does.

No one says they aren’t being researched. But there sure is an extremely disproportionate amount of time and money being put into a few things. Try publishing something in like classification or detection using a pretrained huge model or without lots of compute (more than 1 node of A100s). You’ll just get asked for more datasets and why your 10M parameter model isn’t better than a 1B model even if it’s better than their 100M model. There’s always reason go criticize and so there’s a compete arms race which is unnecessary

Problem is that people controlling the budgets have tunnel vision and little patience. They see a direct line from research to product, they see the line hitting revenue generating milestones. Nobody wants to hear that research is uncertain, that dead-ends happen, that having everyone researching a single topic is not wise, that you can't predict what findings you will get.

If you want to to "solve a big challenging problem" somebody has to do the data cleaning. As Edison put it: genius is 1% inspiration and 99% perspiration.

I think you two are talking about different things. The AI winter as I understand it refers to slowing technological advancement in “expert systems” for example. But technological development is moving faster than ever.

This is different. It’s a weird combination of huge amounts of capital chasing a very specific idea of next token prediction, and a slowdown in SWE hiring that may be related. It’s the first time “AI” as an industry has eaten itself.

From what I read, AI "winters" have not just had less progress but also have had less research activity in general: during them, AI was seen as a dead end that didn't provide value, which translates to less funding, which translates to less people doing research on it (or introducing AI to the research they wanted to do already).

The AI winter refers to the collapse in government and industry funding. It came as a result of a lot of hype like expert systems not panning out but it was about the money that dried up. This time “it’s different” but VCs are just as fickle as grant reviewers and executives - that funding could dry up at any moment if one of the big AI companies has a down round or goes bankrupt altogether. I don’t think a second AI winter is likely because of how useful LLMs have proven this time around, but we’re probably going to have a correction sooner or later.

I don’t think the hiring slowdown and AI are related. Some companies are using rhetoric about AI to save face but the collapse in the job market was due to an end to ZIRP.

The down rounds have started for those companies that either haven’t found product market fit, or aren’t in contention on leaderboards. Especially those companies that are a OpenAI wrapper and can be easily replaced by a new, more broadly applicable, foundation model or a new modality model.

https://www.businessinsider.com/ai-down-rounds-rise-valuatio...

Reply on news.ycombinator.com