Same, I have one living document that is constantly being updated with TODOs, questions, notes, but once they are done or irrelevant, I delete them. I am actually surprised how many people here use append-only approaches.
I don't think the author meant that you have to be the world leading expert at any topic. You can be pretty average, but you need to give it your personal twist. He is warning against very generic abstract talks that can be replaced by reading a man page.
There is none. Filevine is not even an "AI" company. They are a pretty standard SaaS that has some AI features nowadays. But the hive mind needs its food, and AI bad as we all know.
I think it's cultural. Managers today do daily stand-ups, one-on-ones, retrospectives, syncs, and all kinds of meetings. They are heavily invested in the day-to-day operations of the team. The societal expectation for this role is that they are hands-on, and when a problem arises, they will immediately do some shuffling or reshuffling to address whatever problem is at hand. In a sense, this is the outcome of agile-like methodologies spreading in the industry. If this is the tool we are teaching managers to use, of course it's the tool they are going to use.
Another reason why social media matters is that people actually spend their free time in all those feeds. On the other hand, using LLMs is much more oriented towards specific utilities. Having 1B users who visit you once a day to ask for an email proofread will not make you profitable.
I can see both perspectives. If you are a skilled hiree, this seems like a waste of time. But if you are hiring online, you will inevitably get a lot of terrible candidates and you need to filter them out. If you are a small team, you can't spend weeks interviewing randos with little to none coding experience for a SE role. The problem is that online hiring is full of noise, but both sides suffer from the expenses this creates.
If I would get a cent every time I solved a difficult problem in a project by pulling out difflib and shocking the team with it, I would have two cents. It's not a lot, but it's amusing that it happened two times already.
What is often missing from the discussion is the expected lifecycle of the product. Using Electron for a simple app might be a good idea, if it is a proof-of-concept, or an app that will be used sparsely by few people. But if you use it for the built-in calculator in your OS, the trade-offs are suddenly completely different.
The number of runs you can afford are not enough to perform Bayesian optimization. Count how many different options they explored in the text and take a guess how many samples you need to start modeling the hyperparameter space.
Really impressive writeup. In your opinion, how long will this stay up to date? The field is constantly evolving, do you plan to keep updating this document?
There is this homogenization happening in AI. No matter what their original mission was, all the AI companies are now building AI-powered gimmicks hoping to stumble upon something profitable. The investors are waiting...
I don't think AI has anything to do with this. For a while, investors were using headcount as a metric for growth, even though startups often showed very poor utilization of their new hires. Hiring was done for the sake of hiring. Trying to achieve more with less is simply a reasonable response to a market where money is much more expensive than before. Many tech companies were unreasonably ineffective with their manpower, this is a much bigger problem now.
I think that z-order is used to increase speed of loading texture from RAM. But this is not an issue in ML. You usually have all your model weights directly loaded into your GPU memory and you do not need caching for your inputs. At the same time, the entire stack for ML is heavily optimized for other memory layouts already.
They raised billions of dollars, and they're obviously spending it to develop just about anything they can think of. What's rather concerning for them is that the core AI seems to have pretty much stagnated by now.
What about the fact that it has 45 pages with exactly one comparison with transformer architecture (Figure 7)? This is just a fluff piece for a company trying to raise funding.
Fine-tuning is a good technique to have in a toolbox, but in reality, it is feasible only in some use cases. On one hand, many NLP tasks are already easy enough for LLMs to have near perfect accuracy and fine tuning is not needed. On the other hand, really complex tasks are really difficult to fine-tune and clevem data collection might be pretty expensive. Fine-tuning can help with the use cases somewhere in the middle, not too simple, not too complex, feasible for data collection, etc.
We should completely strip all this talk from AI as a field (and get rid of that name as well). It just causes endless confusion, especially for general audience. In the end, the whole shtick with LLMs is that we train matrices to predict next tokens. You can explain this entire concept without invoking AGI, Roko's basilisk, the nature of human consciousness, and all the other mumbo jumbo that tries so hard to make this field what it is not.
Fully agreed. The author also can't decide whether AI is a Ponzi scheme, a bubble, or a cargo cult; so let’s just use them all! It's just buzzwords without any real analysis beyond what is generally known about the field.
> The way I think of these transformations, but happy to be corrected, is more a matter of adding information rather than modifying
This is very much the case considering the residual connections within the model. The final representation can be expressed as a sum of representations from N layers, where the N-th representation is a function of N-1-th.
What a great write-up, kudos to the author! I’ve been in the field since 2014, so this really feels like reliving my career. I think one paradigm shift that isn’t fully represented in the article is what we now call “genAI.” Sure, we had all kinds of language models (BERTs, word embeddings, etc.), but in the end, most people used them to build customized classifiers or regression models. Nobody was thinking about “solving” tasks by asking oracle-like models questions in natural language. That was considered completely impossible with our technology even in 2018/19. Some people studied language models, but that definitely wasn’t their primary use case; they were mainly used to support tasks like speech-to-text, grammar correction, or similar applications.
With GPT-3 and later ChatGPT, there was a very fundamental shift in how people think about approaching NLP problems. Many of the techniques and methods became outdated and you could suddenly do things that were not feasible before.