heavyarms

I think this book [1] makes a similar argument, but in more concise and direct terms. The gist of the argument is theoretical physicists have wasted too much time and effort trying to come up with elegant math and grand theories that sound plausible but can't be tested or, when they can, fail the test. And rather than acknowledging that, the field as a whole keeps digging in without making much progress.

[1] https://www.goodreads.com/book/show/36341728-lost-in-math

I would agree with you to a certain extent, but I still think there is a big missing component to make this a reality. Larger and more accurate general language models are great, but to enable use cases other than categorization, translation, summarization, etc., there will almost certainly have to a be contextualized knowledge graph layer. This is basically what I assume you mean when you say "the ability to create, synthesize, and understand content." The way I see it, transformer-based general language models will be the first and last step of a NLP system. I other words, they will do the raw processing of the input and will do the "take this output and put it in natural language based on this context" portion. Automating the part in the middle, the part that actually understands what the text means and can do logic with it, there's still no progress on that AFAIK.

This is similar to how computer vision models work great in most cases, but to build a self-driving car you still need all the other components that do path planning, predicting what the cars around you will do based on the state of the environment, etc.

I'm not sure about the whole "build it yourself" part, but the "Bessemer process" for making steel was a simple innovation improving on something that existed but made it so much more affordable that many experts think it was a significant contributor to a "second" industrial revolution.

https://en.wikipedia.org/wiki/Second_Industrial_Revolution

That was my first take as well. This is explicitly called out in the Conclusions section in paper.

"Furthermore, immigrant parents were “under-placed” in the income distribution, thus allowing their children who were native English speakers and educated in the US more scope for upward mobility."

I don't see numbers on how much of the total "upward mobility" can be attributed to any of the individual factors.

Concerns like this are not appreciated enough in the data science/ML community. And it's not just the size and resource consumption of the final model. For any real enterprise/business solution, the best ML model is usually not the one that has the highest benchmark scores, but the one that delivers the greatest value and can put online quickly by integrating with the existing systems and software in place.

You really have to start with performance and use case considerations from the beginning. Before you even try to train a model you have to know how scalable it is to load/process the inputs and what you do with the output. For example, in a NLP use case for text categorization or conversational agents, do you have to load historical data like customer notes, emails, etc., that are sitting in a production SQL instance and have to be queried with complicated joins and where clauses? How performant is the current API for doing that? Will you have to run preprocessing on the raw inputs each time? Does it make sense to have a preprocessed copy of the data in a different data source? If so, how frequently should that data be synced? Depending on the answers to any of those questions what looks like a great model that works in a Jupyter Notebook suddenly becomes either not feasible or maybe too expensive to justify.

I understand that a data scientist or ML engineer can't also be a cloud infrastructure expert or software architect who understands how all of the pieces are connected and how performant/expensive certain options are. But it makes no sense to start training a model without at least having an answer for what data will be needed during inference, how fast it has to be to work in real time, how much traffic the model will get and how much that compute might cost in the cloud, etc.