Hacker News Reader

I find the implicit assumption a bit funny that the only reason OP might be asking this is for career reasons rather than say, curiosity, the joy of learning, love of knowledge for its own sake.

That's exactly correct; I currently want to learn this more so in a hobby/personal interest fashion. I think the field is quite competitive already and I'm not fooling myself about doing novel model-architecture work professionally.

Not surprising though. I'm sure this is just my stereotype, but I swear that most times such assumption comes along with interesting words which I don't understand at all (in this context) like "tools" "integration" "landscape" etc.

And that he isn't in the "PhD, elite academics" crowd.

He can grow into all of that.

I wouldn't say that "the opportunity all seems to be on the front end". Specifically for stable diffusion, there are a lot of different ways to use the model. I think we're just starting to scratch the surface of what SD can do, so there is some value in tinkering with different ways to use and apply the model.

Example 1: have a look through here: http://synapticpaint.com/dreambooth/browse/ for some examples of dreambooth models people have created

Example 2: you can merge different dreambooth models together to varying degrees of success (the idea being, you train model A on subject A, model B on subject B, and now you want to generate pictures of A and B together). My understanding is that this doesn't work too well at the moment, but it's possible that a different interpolation algorithm can yield better results.

I do agree with the general sentiment that you wouldn't necessarily be training your own models or creating your own architecture, just want to provide the perspective that understanding the AI side is valuable because it can lead to different capabilities and products.

I'd say the "opportunity" is primarily in creating business models with these new AIs. The AI field is going to be innovating on them regardless what any individual chooses to do. Discussions about viable applications of these new capabilities are scant, beyond reducing existing technical artist head counts I see little to no discussion about new capabilities and new applications not possible before. Sure, we're developers, but we're also supposed to be entrepreneurs and this lack of creative discussion about what can be done that was not possible before is curious in itself.

Guess based on historical anecdata: I think that is because it looks like the current gen AI:s will help automating lot of stuff, but wont enable anything new. I would say we are at the analogous point where ’computer’ no longer meant a chain of humans with calculators. First the automation needs to become entrenched, then new innovations can emerge.

Can I future quote your "current gen AI's will help automating lot of stuff, but won't enable anything new" when that "something new" tears a new economic hole in our global economy?

Personally, I see this tech capable of destroying and recreating the advertising industry completely via inserting everyday consumers into ad media, depicting them as happy consumers of a product they've not used yet - while celebrity spokespeople, appearing as their personal friend, inform them how much the celebrity idolizes them for using said product. This is an obvious non-subtle application. There will be many, many more.

Current gen is not ready for that. It's ready soon enough for sure. When I said "gen" I meant "not currently" but with this speed of development I'm not ready to bet if the scenario you described is 3 months of 5 years away.

http://synapticpaint.com/dreambooth/browse/

Can you explain the value of these models, please?

This is a serious question. My eyes just see one horrendously ugly (and dystopian) eyesore after another.

I agree that the current produced images by say SD, are more of a curiosity than true art. Give it a year or so, I would say, and we will change our mind. Remember the early VR models? Not comparable with the quality you have now in real time. Only AI seems to increase in quality of output at a much faster pace.

This comment does not quite make sense: "The new trend here is very strongly Large Language Models (LLM)." Is every problem a language problem? No, of course. Is every problem going to be solved by an LLM? What about problems that require unique data sources that no LLM will ever be trained on?

And this: "If you are spending time with Jupiter notebooks I would say you are probably completely wasting your time" And how do you suggests one performs data analysis on any problem that's not an LLM -- data analysis of any kind, such as "is the model I am trying to build a front-end for even works for my problem?"

I don‘t like how this comments sounds …

Also you‘re wrong. Look at what the OP wrote and then look at how the latest models are actually built and you would see at least 2/3 of their knowledge is relevant.

There is a lot of transferable knowledge to gain from learning this stuff properly, even if you don’t expect to do core AI work in a commercial setting. Optimization, function fitting, probability/statistics, GPU programming…

My impression is that the field is more disciplined in terms of knowledge now than it was ~8 years ago - the fundamentals are better understood and more clearly expressed in literature.

Also there are still plenty of topics on which the new techniques can probably be fruitfully applied, especially if you have some domain knowledge that the math/CS PhDs don’t have.

For OP - I’m in a similar situation and have been going through Kevin Murphy’s “Probabilistic Machine Learning”, which is pretty massive and dense but also very lucid.

> My impression is that the field is more disciplined in terms of knowledge now than it was ~8 years ago - the fundamentals are better understood and more clearly expressed in literature.

Is that really true? That's not my impression at all (though to be fair I haven't been keeping up with current research as much as I used to). My understanding is that there is still hardly any knowledge on what deep learning models (and large language models in particular) actually learn. The loss surfaces are opaque, one still doesn't know why local minima reached by gradient descent tend to generalize fairly well for these models. The latent representations that generative language models learn is, with the exception of the occasional paper that finds some superficial correlations, hardly investigated at all and overall extremely poorly understood.

Very much interested in any references that contradict that sentiment.

Maybe I'm biased specifically because of the book I mentioned. For me it's providing a theoretical basis for many things that earlier I learned in a hand-wavy way (e.g. way back I took Hinton's NN course and Ng's ML course, and learned about gradient descent, momentum, regularization, training/validation loss etc, and now with this book for the first time I feel like I get the bigger picture in the context of optimization/stats).

The previous version of this book was from 2012 though and I'm not 100% sure how much of the material in the current edition is new (there is definitely a _lot_ more deep learning stuff in it).

So yeah it could be that my impression is wrong, or that I made the scope at which it applies sound bigger than it is.

I've only read (parts of) the Murphy book from 2012, I assume you're reading this one: https://probml.github.io/pml-book/toc1.pdf?

Almost all of the content that the new book covers, with the exception of the third part on deep learning, is about theory that was almost exclusively invented before 2012. Classical ML (non deep learning) is actually very rigorous compared to modern ML. There exist good theorems (statistical learning theory) for most of the classical models I'm aware of.

Yes that’s the one. Good to know, thanks for making me aware!

I disagree with this comment, and anyone reading it should take it with a big grain of salt. Let's go back to 2016 and replace "LLM" with "Reinforcement Learning". Everyone thought every problem could be solve by RL because it's a looser restriction on the problem space. But then RL failed to deliver real world benefits beyond some very specific circumstances (well defined games), and supervised learning is/was still king for 99% of problems.

Yes, LLMs are amazing but they won't be winning every single Kaggle competition, displacing every other ML algorithms in every setting.

Sure enough LLMs are not going to win every kaggle competition. But... I am fairly certain that transformers may. Embed all categorical values, scale continuous features by embeddings, and run it through a graph neural network, with high probability it will beat nearly everything.

Transformers require a lot of data to converge. There’s a reason tree models are still king of kaggle even though transformers have been around for 5 years now.

To me, the vast majority of people in the field seem to hold on to the idea that different technology suits different problems.

Also, are you aware that one of the most prominent AI tools of this month (ChatGPT) was obtained with RL!

Applied AI/ML is still a great career field, especially if you have a knowledge of physics.

Hi, would you mind elaborating, or throwing some links my way regarding the physics connection in particular? I am a physics postdoc that has been interested in the field for some time (this becoming less of a unique characteristic as the technology develops and hype cycles peak). I was motivated by Stuart Russell's Reith Lectures in 2021 to pursue the field outright and had been working towards this, but am becoming increasingly aware that seeking direct research involvement in the field is a bit quixotic starting from where I am.

I was motivated by Stuart Russell's Reith Lectures in 2021 to pursue the field outright

For anybody intrigued, here are those Reith Lectures:

https://www.bbc.co.uk/programmes/m001216k/episodes/player

https://wikipedia.org/wiki/Reith_Lectures

Thanks! The key point for me was that Russell put forward a strong case that the moral approach to the problems of artificial intelligence was to involve oneself with the evolution of these technologies where you can (8000hours style) and seek (in whatever limited capacity one can) to guide them away from the excesses and damages of the invisible hand (and the human condition it is a proxy for).

This would have been influential to me when I was a teenager and nihilistic about accelerating technology in the hands of distinctly-not-developing humankind (reading all about the Manhattan-project forebears to AI), but after devoting a decade to science and being fascinated yet boycotting of AI research and it's implications, I think it has ended up a bit too late for a change of heart and direction to have as much positive impact.

For those fortunate enough to be in a position of even trivial influence over the cutting-edge, and whose moral sophistication can therefore matter more than the next person, I hope you don't take the implications of whatever agency you may have lightly!

Anyone with even that kind of "trivial influence" can be absolutely sure than an "agency" is constantly tracking them.

I wouldn't be surprised if AI conferences have a minimum of 10 spooks from every major western intelligence agency in attendence.

One connection I've seen is energy-based models. I suppose you could try applying ML to things like fluid dynamics?

In general, it is thought that physics knowledge transfers better to ML than math because it's less abstract and physicists are more likely to be used to dealing with large datasets and software.

Thanks for the insight! I have some friends that do computational fluid dynamics, with particle physics being similarly numerical, and was looking at physics-informed ML for my own particular area in quantum physics in a recent grant application in the hopes for funding to close the gap a bit myself. What is so powerful about ML and related statistical techniques is their versatility and genericity, so a project that can be benefited by that region of statistics tends not to be too far away. I will look into energy-based models, too.

His comment is quite good, I do work with physics informed ML for cfd and other dynamical systems (temperature, hydrology etc.), there is just a ton of opportunity and funding for this type of work in research. Coming from a typical physics education, where you’re learning quantum and Astro, and realizing that 90% of the physics funding from government is in the earth sciences and the related physics was eye opening. I felt shortchanged by my physics education not even including fluids etc.

It was the same here - fluid dynamics was an elective at my university as well (one I took, but still not core syllabus). I guess amount of funding for a domain depends strongly on impact, and in the earth sciences output is much more immediately tangible than uncovering another supremely true but at-the-time inapplicable pattern of physical behaviour in the quantum, or context to humanity in the astronomical or cosmological domains.

check my reply to the other comment

Great! Thank you for taking the time. With that information I can look into a more local equivalent.

Could you elaborate? I’m in the AI field myself but as a software engineer. But have a PhD in physics

https://www.usajobs.gov/Search/Results?k=physical%20scientis...

If you're already in the field you'd likely have to take a pay cut, but the federal government is always interested in research physical scientists and will snatch up anyone who knows the physics and also knows ML.

One thing I'd like to add is that you do not have the computing power to generate a large language or vision model. Period. Unless you have hundreds of thousands of dollars for compute time you are just not going to do anything interesting with model building and AI.

Upgrading existing systems with AI is probably where it's at using existing models like stable diffusion, GPT-3 or some of the smaller downloadable language models if the task is very simple and the economics of using GPT-3 don't make sense.

Not OP but thanks for this response, as someone on the front end with a passing interest in AI this helped me recalibrate my thinking on this.

Is LLM even applicable to many things though? If we want more nuanced and contextual vision, does LLM help?

Multimodal models are also a red hot area. The interesting thing might be to combine specialized instances of LLMs and other models together, where each runs a specific subtask or event processing loop.

I think it'd be fun to use vision models to pick up the interesting parts of an image, describe only the high-yield context as English text, and put them in a sort of Unix pipeline or node graph to connect it to other models that can input/output text. With fine-tuned or prompt-engineered LLMs as the intelligent centerpiece.

Fine tuning models is where the future will be, it doesn’t necessarily mean LLMs but I suspect that LLMs will shortly become multi modal and at that point they will be the ultimate models to fine tune for a given task

Thanks for the comment.

>Fine tuning models is where the future will be

Who would request fine tuning? Businesses?

Are you suggesting to build some sort of data set broker?

Businesses who have their own data fine tune trained models (this is already a thing). So you need model repositories (like hugging face) not dataset brokers.

The LLM component is almost correct, but not exactly. The whole point is the transformer architecture, which really is just graph neural networks. Once you start seeing things through these lenses, the possibilities are endless and it starts making sense.

LLM is the major trend. Focus entirely on that and the tools landscape and how to integrate it and apply it.

Rather than jump on the same horse that everyone is jumping on, maybe one should start looking at where language models fail -- and from the very nature of how they are conceived and what they are made of -- will most likely never be good at.

Most failures of models are usually 1-2 research cycles of being updated, the amount of brain power dedicated to this problems is astounding

There is a huge opportunity on the backend for ML Engineers, running these models in production is expensive.

Yes this. Be a subject matter expert who knows how to take the technology and apply to some problem in a novel way. Don't be the method maker because you will always be behind. How do I know you will always be behind? If you weren't behind then you wouldn't ask questions like OP.

Nearly everyone in a field starts out behind.

MS and PhD students start out behind and they’ll spend most of their time during the next 2-5 years on irrelevant things (e.g. 95%+ of their graduate coursework will be behind the state of the art; most PhDs will focus on a niche project that fails to have major impact or relevance by the time they graduate).

It sounds like OP just wants to learn out of general interest, which is fine. But others shouldn’t be discouraged. A sufficiently-dedicated person with talent and a strong classic ML foundation can catch up reasonably fast, to the point of getting their foot in the door professionally.

Exactly this. Having a MS/PHD does not guarantee that you are on the forefront of a field. When I started my MS program, I was behind, and when I finished, I was behind.

What would you say about someone new to all AI starting by working through the Neural Networks from Scratch book? https://nnfs.io/

Any resources you can suggest to get this:

"knowledge of the emerging landscape combined with a broad understanding of different tools and how they are applied."

This is a good reply, but the OP might be generally interested in the fundamentals of ML