Hacker News Reader

Comment by dr_dshiv | Hacker News Reader

Lots of great insight. Here’s one:

“Given the long timelines of a PhD program, the vast majority of early ML researchers were self-taught crossovers from other fields. This created the conditions for excellent interdisciplinary work to happen. This transitional anomaly is unfortunately mistaken by most people to be an inherent property of machine learning to upturn existing fields. It is not.

Today, the vast majority of new ML researcher hires are freshly minted PhDs, who have only ever studied problems from the ML point of view. I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML.”

View on HN

As somebody who has crossed the line between ML and chemistry many times, I would love to see: more ML researchers who know chemistry, more chemistry researchers who know ML, and best of all, fully cross-disciplinary researchers who are both masters of chemistry and ML, as those are the ones who move the field farthest, fastest.

You could probably fit all the people who fit the last criteria in the same room (chemistry side is probably the bottleneck, especially drugs which is a effectively a specialization).

Present. I think there exist many of us. Chemistry is a very wide field though, so not sure if organic synthesis vs theoretical chemistry vs physical chemistry vs biochemistry will end up more useful to help tackle drug discovery problems or other chemistry applications. Same with ML I suppose; even though the specialties are less concrete nowadays, the breadth of publications has far exceeded the breadth of modern chemistry.

Society is not structured to encourage this. Getting a job sooner is more lucrative. Any breakthough you make having studied for a couple of decades is property of a corporation not you.

> Society is not structured to encourage this.

I wish more people understood this and the value that it would add to society. For some reason most people understand how roads and transit improve commerce, but fail to understand how education and social support for the individual does as well.

Sport incentives work out nicely for getting a supply of top talent. Big money through sponsorship/ads/tickets, fairly formulaic system to get better at a sport (top coaches/teams, get fitter, practice a lot), an easy way to identify progress (wins games, scores goals).

But for sciences, someone could do great work for 20 years and produce obscure papers that might stay obscure, or may unlock the mystery of the universe or help us meet the world's energy demands without pollution! So the pay off if magnitudes higher than sport, but magnitudes less probable. Someone will run 100m fast, guaranteed, but will we solve science problems X Y Z and when?

Would employees of non-sports industries accept the weird-ass ways that sports contracts work? Major League Pharma Research is certainly intriguing

To my understanding it'd need to be set-up so the proposition of R&D is granted by unbiased parties. I believe its been linked on HN before, where it describes what system would be best to incentivize research? It stated the dangers of funding the research of only the popular (and contemporary) topics, which tilts R&D away from discovery and more towards 'lets try to yield xyz, and if it fails, lets not release it'. Someone who is motivated to grant something must be completely separate from the publication, most likely those who researched it and many other scientists, & unlike peer review try not to trim any of it out

I agree with you but does anyone even recognize the last category outside blue-sky research? People have a tendency to bin other people into buckets. Being a master at 2 things means you can’t be easily placed in a typical team structure.

> I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML

I can confirm. We regularly look for people to write some computational physics code, and recently for people using ML to solve solid state physics problems. It’s way easier to bring a good physicist or chemist to a decent CS level (either ML or HPC) than the other way around.

This is because software developers are too good at automation themselves out of a job.

It's also because nobody goes to get a phd in solid state physics for the money or career prospects, at least not in the last decade. So it's a small and self selected group.

We love automation. There is just too much to do, and the field is unbounded. More automation means more papers and more interesting science.

It's the same reason analysts come from math rather than economy degrees.

You can teach a mathematician what he needs to know about finance, you can hardly do the opposite.

That’s the opposite of what OP is observing. It’s easier to teach a domain expert “good enough” quantitative and technical skills than to teach a pure quant “good enough” domain expertise (and the corresponding intuition)

I think you’re reading this backwards, or perhaps it was edited? Mathematics is surely the more pure discipline compared with economics, so physics : CS :: mathematics : finance is the right ordering.

You just decided that one of those expertises is the "domain" while the other is not. There's no reason at all to pick one of them.

and it's easier yet to find one of each that are willing to collaborate.

In my experience, people with degrees in math, physics, chemistry, biology/medicine, astronomy, computer science have higher/stronger intellectual rigor that can be re-applied to other fields, e.g., finance. Also, those areas of study are much harder than economics. Partly, it is a self-selection process. Yes, there are some with economics degrees whom are very bright, but they probably could have majored in any of the sciences mentioned earlier.

The story I’ve heard is the economics undergrads can’t get into economics grad school. This is just a rumor but the sentiment is that undergrads get taught a watered down version of economics theory. Economics theory is potentially very technical and includes game theory and proofs. Even in CS, undergrads take intro theory courses and “bottom out” in their math skills, even though grad-level CS gets much more difficult. Therefore, I’d imagine the primary determinant of this rigor phenomenon is the GPA inflation of the major.

This is somewhat true - economics undergrads do get into grad school, though math+econ or an econ degree with lots of math is better. You can just the check the difference between the textbooks for undergrad and graduate economics, eg Principles of Economics by Mankiw (undergrad) and Mas Collel (grad).

What sort of problems are you trying to solve?

Lots of things, but mostly things like designing software architectures for massively parallel codes implementing fancy physics or coming up with ML models to predict complicated properties (things like wear, radiation, or corrosion resistance for which we just do not have any comprehensive model) or explore humongous problem spaces (things like predicting simple properties of very complex materials with 5 to 8 elements and complex microstructures like superalloys).

Neither the Physics nor the CS are cutting edge (that’s why we do not necessarily reject people with limited experience in Physics, though they need to show motivation and abilities to learn); what is is the combination of both.

We’d like to be as close to the cutting edge on ML as possible, though, because it’s a significant competitive advantage to be able to use fancy new techniques before our friendly competitors. But as I said it seems to be easier to train a Physics or Chemistry undergrad to get some feeling about how ML works than to train a CS undergrad to have some intuition about the Physics. And intuition is critical to detect when models hallucinate and get off the rails.

Ha well if you’re looking for a solid state physics PhD I’m on the market ;)

Not right now, unfortunately. Where are you located? We can discuss if you need any advice (username at me.com).

New Orleans, Louisiana.

Damn, my contact at LSU has moved. I know there’s a good group at UT Knoxville where they do great stuff with Oak Ridge’s neutron source but that’s not specifically ML. There are adverts regularly for post-docs at Los Alamos; they can be found on lanl.jobs . They work with various universities depending on the group and might point you to something.

I did email you, thanks!

I'm yet to see an ML PhD be required to learn chemistry to a similar extent that chemists would need to doing ML (especially at research level)

That's because application and research are quite different. If one does a PhD in ML they learn how to research ML. Someone with a PhD in chemistry learns how to research chemistry, they only need to apply ML to that research

Back when O'Reilly was still hosting events (sigh), at one of their AI conferences, someone from Google gave a talk about differences between research/academic AI and applied AI. I think she had a PhD in the field herself but basically she made the argument that someone who is just looking to more or less apply existing tools to business or other problems mostly doesn't need a lot of the math-heavy theory you'll get in a PhD program. You do need to understand limitations etc. of tools and techniques. But that's different from the kind of novel investigation that's needed to get a PhD.

>>math-heavy theory you'll get in a PhD program

Lol. With the exception of niche groups in compressed sensing, math doesn't get too hard. Furthermore, ML isn't math driven in the sense people are trying things and somebody tries to come up with the explanation after the fact.

Well I think the issue is more of if you’re Genentech and you need ML people and can’t afford to pay them you’re probably better off retraining chemistry PhDs.

I think you missed my point. Genentech, AFAIK, was not doing research on machine learning as in the principles of how machine learning works and how to make it better. They do biotech research which uses applied machine learning. You don't need a PhD in ML to apply things that are already known

As a PhD student working on core ML methods with applications in chemistry, I second this. During my PhD, I read very few papers by chemists that were exciting from a ML perspective. Some work very well, but the chemists don't even seem to always understand why they made the right choice for a specific problem.

I don't claim that the opposite is easy either. Chemistry is really difficult, and I understand very little.

Genentech has several ML groups that do mostly applied work, but some do fairly deep research into the model design itself, rather than just applying off-the-shelf systems. For example, they acquired Prescient Design which builds fairly sophisticated protein models (https://nips.cc/Conferences/2022/ScheduleMultitrack?event=59...) and one of the coauthors is the head of Genentech Research (which itself is very similar to Google Research/Brain/DeepMind), and came from the Broad Institute having done ML for decades ('before it was cool').

They have a few other groups as well (https://nips.cc/Conferences/2022/ScheduleMultitrack?event=60... and https://neurips.cc/Conferences/2022/ScheduleMultitrack?event... and https://neurips.cc/Conferences/2022/ScheduleMultitrack?event...).

I can't say I know anybody there who is doing what I would describe as truly pure research into ML; it's not in the DNA of the company (so to speak) to do that.

You can get an ML PhD doing applied ML.

Sadly a lot of foundational ML research works for single-label image classification and not much else. ImageNet is a niche problem and way too much ML research is over-indexed on it. If you can make your problem look like ImageNet, you're going to do OK, but if not you effectively need to re-invent the wheel...

What you wrote was true until 4-5 years ago. These days almost all foundational ML research is about generative models.

It's still problematic.

The HEAR benchmark is a great eye-opener. They have basically three classes of audio tasks, and find very different models excel in each, with the best overall models being kinda-mindless ensembles of the ones that do well on particular problems.

So if you've got something that works well for text... it'll take a couple years and maybe an entire new branch of research (diffusion!) to work well for image generation. I have no idea what generative models for chemistry will look like, but will happily bet that it takes some significant specialized effort.

I see two things currently happening in ML:

1. The era of ML benchmarks is ending. New models have to be and will be evaluated the same way human experts are evaluated.

2. Foundational models are becoming multi modal. There will be no separation of text and image generation. Sure, different methods will be used for each, but the learned representations of visual and textual objects in models like Stable Diffusion already live in the same conceptual space.

I don’t think there will be specialized generative models for chemistry two years from now. There will be GPT-5 (and similar competitors) which will be used to perform all kinds of research, including chemistry.

I think this is a bit over confident, though...

For example, AlphaFold just fundamentally isn't a language model, but is fundamentally useful. We'll still need these models that Do Stuff in many areas, and that will still involve benchmarks... Even if we're able to ask GPT-N+1 to design the next version of the model for us.

What if they don’t need “ML people”? Computational biology has been a thing for a while.

Well they had a whole suite of presentations at NeurIPS that suggests they hired a bunch.

They could afford them then…

https://www.gene.com/scientists/our-scientists/prescient-des...

I’m pretty sure the author is implying that the new crop of ML PhDs are just not a smart group of people - at least the level of intelligence required to do truly transformative things with ML in any field.

I think what you’re saying is a commonly found attitude that relates to this topic: it’s pretty limiting to think a cursory knowledge of a field is sufficient to go change it. That’s likely why most “use ML to solve x” projects fail when some like AlphaFold succeed because the ML engineers truly understood the fundamental tenets of the topic and exploited it.

    AlphaFold succeed because the ML engineers truly understood the fundamental tenets of the topic and exploited it

What makes you think it wasn't chemists or biologists who learned enough ML to solve the problem?

Because the chemists and biologists have been doing that for decades without success.

maybe he just say do not expect people resolve problems outside of their discipline.

I don’t understand what you mean. Here’s how many applied ML papers work: create a new dataset for a novel problem, download a PyTorch model, point model at dataset directory. Is it novel? By construction. Is the ML technique novel? No.

I actually meant exactly this. People apply ML to their domains and say thay that this is similar to having a chemist with a PhD.

Not so dramatic, but I have a traditional CS background and got into ML much later in my career.

Last year my company formalized the process to hire ML Engineers. The interview is the same format as the software engineer round, but with an extra theoretical ML round.

I've observed two distinct groups of candidates in the process. One are recent grads with PhDs in ML. Other are people from diverse backgrounds that happened to start working in ML in their current job. The first group tends to excel in the ML theory part, but flunk the leetcode style coding questions. The second group tends to do better in coding, but do worse in the theory part. This is exactly what the process has been designed to do.

I am very put off from looking for ML Engineer positions in other companies if they follow a similar process. I know I would fail the interview for my current job. They could get away with it for a while, but I doubt it's sustainability or desirability in the long term.

Loved this argument as well.

With respect to more mature research fields the entry point to ML is much lower.

Hence I always recommended people to have major in Physics, Chemistry, Biology etc but look for projects in these fields that could benefit in ML (I have a number of them about Physics).

So that argument was not novel.

But the fact that pure ML PhDs will have significantly lower multidisciplinary knowledge is a good one. It could be compensated by the fact that ML is growing fast and all kinds of people join the ride, but still.

>I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML.

Perhaps this is selection bias. Among all the chemists, the ones who will dabble in ML will likely be the chemists with the highest ML related aptitude. In contrast, a ML expert on a chemist project is more likely not being internally driven to explore it but instead has been assigned the work, which means that there is less bias in selection and thus they have less chemistry aptitude.

> "[...] I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML.”

That's good ol' academic gatekeeping for ya, available wherever PhD's are found.

There’s more to it than that.

CS is unusually easy to learn on your own. You can mess around, build intuition, and check your progress—-all on your own and in your pyjamas. It’s easy to roll things back if you make a mistake, and hard to do lasting damage. There are tons of useful resources, often freely available. Thus, you can get to an intermediate level quickly and cheaply.

Wet-lab fields have none of that. Hands-on experience and mentorship is hard for beginners to get outside of school. There are a few introductory things online, but what’s the Andrew Ng MOOC for pchem?

On the flip side, software development and engineering rigor is largely absent in academics, as has been discussed previously here on HN. This is enough of an issue to make replicating research even when the code and data are provided, but it's an even bigger issue trying to turn academic code into a product.

This is more about ML than CS. ML is fundamentally about developing general-purpose algorithms applicable to a wide range of problems. If your job is using ML to solve problems in chemistry, it's more about chemistry than ML, and a chemistry background is more important than an ML background. It's unlikely that you have to develop novel ML methods for the problems you are facing.

I've seen the opposite in bioinformatics. While dedicated bioinformatics programs are now common, you still see many CS / mathematics / statistics / physics / EE people moving to bioinformatics after bachelor's / masters's / PhD / postdoc. In some bioinformatics jobs, you often have to solve new computational problems, and it's easier to teach enough biology to people with a methodological background than the other way around.

I've seen both in bioinformatics, because the field is so wide now.

1) Bioinformatics as tool-building, algorithm-dev: you're right, you don't need to know much biology there if the problem is defined well.

2) Bioinformatics as a tool to answer biological questions: here I've seen ML-background people really struggle, either developing stuff that's not useful or reinventing-the-wheel-but-now-it's-deep-learning. I've seen ML people present their fancy plant disease image detector which turned out to be pretty good at spotting 'yellow' - very good at training accuracy and benchmarks, does not add anything to what people in the field are doing.

In regards to 2), it sounds a bit directionless to be proposing stuff people don't need? Isn't that more of a problem of selecting relevant problems to solve, and getting supervision on your ideas?

Yeah that's the problem! The ML people and the biology or agriculture people don't talk, they're not in the same building. A biologist might see the ML-person's work only after it's published.

That's because academia is ass backwards with it's attempt at gatekeeping high knowledge/low skill fields.

Chemistry is a centuries old discipline, that people study undergrad a full four years before getting a PhD in the field of chemistry.

ML is a, practically speaking, 15 year old field that PhDs often begin to study after a couple of AI courses in undergrade and a specific track in grad school (while they study other parts of CS as part of their early graduate CS work).

There's just way less context in ML than Chemistry.

I don't think that's really true, but you don't need the full context in ML in order to apply it to your field of interest. Just like you don't need the full context in Physics to apply it to an engineering problem.

Some of the most successful ML researchers have read several decades of research papers. It's not uncommon to see references to papers from the 70s or 90s.

Edit: No doubt that the relevant parts of stats, random matrix theory, and ML is a newer field than Physics or Chemistry, though.

DL is a 15 year old field, ML is far older than that.

That’s why I said practically speaking. And realistically I don’t think you can even generously consider anything prior to perceptrons.

ML is so much more than DL / NNs. I recommend that you open a book like Bishop or ESL, the overwhelming majority of ML has been non DL going back to the 70s.

Most of that I learned doing non-ML stats work. But regardless, my point was the history of it is far less than chemistry. Let’s say ML started at the turn of the 1990s to be generous. Still much younger than chemistry.

> realistically I don’t think you can even generously consider anything prior to perceptrons

I mean, sure, but for the practical reason that perceptrons (proposed 1943, implemented 1958) are about as old as digital computers, and ML as a field is tied up pretty strongly with the existence of computers.

I wasn’t comparing ML to digital computers though — but rather chemistry. Theory of computation and algorithms (bedrocks of CS) far predate digital computers.

> I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML.

Haha, I've seen that for so many topics. "It's much easier for someone used to circuit switched phone networks to learn IP than the other way around", says the person who started with circuit switched.

I just thought "dude, you're literally the worst at IP networking that I've ever met. Your misunderstandings are dug into everything I've seen you do with IP".

Ironically, in my field I'd argue this isn't necessarily true - so long as the ML PhD sees value in that field.

Part of this is because public health as an undergraduate discipline is extremely new, so the field is used to having to teach Folks From Elsewhere about our field, rather than the fields built on the assumption of a large foundation of undergraduate coursework.

Interdisciplinary and intersectional skillsets are critically valuable.

Separation of concerns especially at the beginning innovation stages can be more of an inhibitor than accelerator of success. Scaling and growth is another thing.

A similar pattern existed in the late 90s with web developers coming from many different industries with their domain knowledge and domain insight.

The code and frameworks were early but the insights of what was a problem most pressing to solve.

The ML folks pretty much nailed protein folding though. Where any of them phds in molecular bio or chemistry?

Do you have any anecdata on topologists?

Topologists will occasionally come to work with their clothes on inside out and insist nothing is wrong.

Topologists can't tell the difference between a coffee cup and a donut, and will occasionally make a mess trying to dunk their donut in their coffee.

obligatory XKCD: https://xkcd.com/793/

That's a great xkcd, but there are 2 upsides to this arrogant approach. First, arrogance is a nerd-snipe maximizer. Second, there is a small chance you're absolutely right, and you've just obviated a whole field from first principles. It doesn't happen often, but when it does happen and there is no clout like "emporer's new clothes" clout.

EDIT: The downside, of course, is that you appear arrogant, and people won't like you. This can hurt your reputation because it is apparently anti-social behavior on several levels. I think its fair to call it a little bit of an intellectual punk rock move that is probably better left to the young. It's an interesting emotional anchor to mapping a new field, though.

Not laughing! /s (physicist here)

Actually most applied physicists like myself go down that path cause we're pretty efficient, lazy folk & skip through as fast as possible--I call it the principle of maximum laziness.

> Second, there is a small chance you're absolutely right, and you've just obviated a whole field from first principles.

Mostly when I read about things like this happening, it's happening to a formerly intractable problem in mathematics. Do you have examples outside of math?

Alfred Wegener as the initial author on the theory of plate tectonics comes to mind. He was a trained meteorologist who observed the similarities between geological formations between the South American east coast and African west coast. He was lucky, in that his father in-law was a prominent geologist and helped him defend this thesis.

Oh yeah, revolutionary insights are very important for the advancement of knowledge and the elimination of wrong ideas. But as you wrote, this was the work of a thesis, not a random commenter from another field.

The only time I’ve seen this happen is AlphaFold. Every other time some math or software guy came into biology it worked exactly like in xkcd.

That's a good one.

The biggest innovation of this decade, LLMs was created by Math/CS not linguistics

Biology has already been transformed by mathematics, statistics, and CS for decades. These days, if something like in that xkcd happens, those people are probably not even familiar with the relevant parts of math / stats / CS.

What exactly has it transformed? Nothing has fundamentally changed in biology, you still have to run western blots and give mice cancer. A great example of “math” solving biology is super resolution - which was just a dud.

Genomics, for example. The entire field could not exist without extensive algorithmic research done over several decades. Even today, many of the key people in the field have a background in CS and mathematics.

Theoretical ecology is a bit more old-school answer. Many mathematicians have been involved in that field.

I am not sure I have seen any meaningfully brilliant breakthroughs in those fields though.

I take that back. One can never forget the brilliance of HiC by Erez. But other than that TBH all other “mathematical” or computational breakthroughs I’ve seen are just at best meticulous application of obvious math and computational algorithms to biological problems (and may I say poorly? Thinking back to the microarray nightmare years).

I guess that depends on what you count as a breakthrough. From my perspective, HiC is a small detail, and even AlphaFold is just the latest improvement to an existing process. A true breakthrough would be something like ancient DNA, which gave the humanity a new tool for studying the past.

Pretty much everything in genomics depends on shotgun sequencing, which in turn depends on CS. The algorithms used for assembling something useful out of the sequence reads are highly non-obvious. In fact, most developments in string algorithms since the 80s have been driven by the needs of DNA sequencing. (Information retrieval used to be another contender, but word tokens turned out to be a better tool for that field.)

LLMs weren't invented by Naom Chomsky but Math/CS guy with a simple "Attention is all you need"

Creating a good chatbot was never a goal of linguistics, and I don't see what insights into human language have been derived from LLMs.

https://xkcd.com/1831/

An introductory book on neural networks, written a few decades ago, starts with a short history of the field at that time.

This included a phase where some physicists began having opinions about the subject, anticipating they might quickly find a model for the brain, with fields, or other physics-like paradigms.

Their expectations were that with their superior understanding of all things fundamental, they would rush in, and rush out. Leaving the stunned machine learning researchers dazzled, frazzled, and asking "Who was that masked physicist?!?"

Except their ideas went nowhere.

I can't find the book, but this story made for a memorable foreword.

Reply on news.ycombinator.com