Hacker News Reader

Comment by fmajid | Hacker News Reader

When I was last interviewing candidates for a data scientist position, I can't count how many physicists applied for the job when they didn't even understand what overfitting was.

There is a definite sense of "physics is hard, therefore everything else must be easy" groupthink, combined with Dunning-Kruger syndrome.

View on HN

That's sad because overfitting predates ML by a long shot, and most experimental physicists of my generation (PhD 1990) knew what it was, even if we didn't use that term.

In my view it's not so much that "everything else must be easy," but that physicists are expected to take a multidisciplinary approach to our work. In the process of getting my thesis project to work, I had to teach myself electronics and programming, not to mention a variety of mechanical skills such as machining and vacuum plumbing.

Naturally, the job market situation encourages physicists to spread our wings beyond physics research, and to claim that we can do things. ;-)

Most physicists are pretty good at math. This actually gives us a head start in quantitative branches of other disciplines such as engineering. Today, when the engineers at my workplace run into a hard quantitative problem, they go looking for a physicist. If math is what makes a subject seem hard to most people, you might discover a few physicists trying to sneak in.

> (PhD 1990)

The title of the post is extremely misleading.

The author of this post graduated from an undergraduate program in 2018. By "transitioned from academia", the author means "graduated with a B.S. (after a few research-y summer internships) and got a job".

That first job is always an accomplishment, but in this cae was definitely not a "transition from academia" in the usual sense.

As a result, the post generally contains irrelevant advice for people actually transitioning (i.e., professors, post-docs, and late-stage graduate students).

That's your chance for a cheap hire? They're probably not going to get hired anywhere else and you can see in-interview if they are able to understand quickly! There's not much value for junior positions to "know concept X", so what you want to know is the derivative...

I'm really curious now. How were you testing their understanding of overfitting? Like what was the setup/questions?

That's really interesting because in my undergrad (which I hold a BS in Physics) overfitting and error were big topics in all labs. In fact, most physicists I know complain about overfitting in current ML models and how people are making these models and just blindly accepting results.

Sometimes I wish physicists or mathematicians or whoever will actually try these engineering or whatever problems. There may be certain aspects of them that your training will make you very able to handle, but these problems are actually very hard and you need a lot of their specific knowledge for them.

Was physicist. Did engineering problems. Didn’t have much trouble (especially compared to those with more specialized educations in the relevant fields).

To be fair, OP seems to have actually successfully transitioned, so they don't seem like the other physicists you seem to have a glut of.

If you are hiring physicist to do Machine Learning, then you are doing it all wrong. Physicist are excellent at mathematical modeling, especially those in their 50s+. Data science from my experience is vastly inferior to mathematical modeling. The OP that wrote the article seems young, and doesn't seem like they've seen much. In the world of quantitative finance, physicists and mathematicians reign supreme. These guys get paid millions in salary and bonuses, because they earn 10x that for the firms. Their entire jobs are to mathematically model things. There are very successful hedge funds that only hire physicist and mathematicians. The current data science and machine learning is child's play compared to the stuff these quant funds do. Physicist, especially ones that have mathematical maturity are incredible assets, they have a lethal skillset that isn't really there in silicon valley. The majority of data science posts I see on HN are mediocre at best. I think age and experience are highly under valued. The physicists and mathematicians that Draper, Raytheon, DARPA, Renaissance Tech, Black Rock or Bridge Water (I would include SpaceX and Tesla in this list) have are superior to any data scientist that you will find in Silicon Valley. Mathematical modeling is incredibly hard, seasoned physicists are very good at it (I can't emphasize how difficult and long this process can be, nor can I articulate the reward for doing things this way).

> There is a definite sense of "physics is hard, therefore everything else must be easy" groupthink, combined with Dunning-Kruger syndrome.

You are severely misinformed, and you severely lack a breadth of industry experience. Physicists (especially the experienced ones)have something called mathematical maturity. Having mathematical maturity is a skillset, you are able to make precise formulations of some phenomenon you want to understand things about, and prove with rigor that your ideas are correct, or that they converge to what you want. This is a lethal mind to have, it's far superior to machine learning or data science. You aren't relying on heuristics like "overfitting". These concepts aren't well defined, and are very subjective.

I can tell you this with 100% confidence, AlphaGo, Duplex, Siri etc. are all gimmicks, no different from Watson or Deep Blue. They are just applying statistical brute force to narrow domains. I'd love to see a machine learning system that consistently matches the profits that the successful quantitative funds do. I know no tech company can do this, they don't have a fundamental understanding of mathematical analysis, they don't hire the right people. The right people are at DARPA, Draper, Tesla, SpaceX, any organization working on the hard sciences. Go figure; pretty much all of the major important discoveries came from either military or laborites investigating the hard science.

Data science/ML, is great for simple data sets (there is a mathematical definition of this, not heuristic). For many industrial problems like what SAP, or McKinsey does it's perfect. The available cloud compute power and a basic Statistics + CS degree will disrupt these businesses. But do know this: they are severely limited. Machine learning will very quickly start showing its cracks.

I don't disagree per se but I think you're over-selling physicists' capabilities (former academic physicist here, PhD and all) and also using the latest hype-based machine learning cult in the Bay Area as a stand-in for all ML.

Mathematical modeling is essential to well-done data science. I won't let my team (I manage some machine learning engineers and data scientists) get away with what I call "toy modeling", that is parameter tuning and basic model implementation, and claim they're "doing data science." They need to provide a fairly rigorous analysis and basic mathematical models to justify their choices for particular algorithms or reaching for the "I give up" tool of so-called deep-learning. And the second my bosses tell me to just let them "do ML" is the second I resign. I won't be part of diluting the quality of my team's work

Out of curiosity, which industry to you work in?

I do agree that modeling is useful in every industry, but there are some industries where it matters more than others.

I think this level of worship of mathematical models is the exact reason that physicists shouldn’t be employed outside of physics. Most academic communities have over the past several decades adopted a Bayesian point of view for their modeling purposes. The idealized perception of a perfect mathematical model is so antiquated, I would consider it the equivalent of a flat earth theory in modern statistal physics. I say this as a prior mathematician who has since seen the limitations of this perspective.

Its funny that in the same comment you mention statistical physics and claim physicists don't understand statistical modelling. Physicists routinely use statistical modelling in astronomy, in condensed matter physics and in high energy physics. The emergence of the Bayesian point of view was originally spearheaded by a physicist Edward Jaynes and the torch has been carried forward by many other physicists over the decades.

I think Jeffreys, Wald and Savage had all started on advocating Bayesian statistics/probability before Jaynes.

I'd love to see a machine learning system that consistently matches the profits that the successful quantitative funds do.

How about Google Search?

As @sidlls rightly commented you are overselling "mathematical modeling" a bit, up to the point it becomes another black box, near to magical ability. I think it is essentially the capacity to abstract and find the relevant key features of what you want to model. And perhaps equally important is to cross check your initial model by seeing if it behaves reasonably in limit and corner cases, e.g. by putting certain parameters to zero or infinity.

>I can tell you this with 100% confidence, AlphaGo, Duplex, Siri etc. are all gimmicks

I think here you might have went slightly over the cliff. Couldn't you not with the same conviction claim those hedge funds only use toy models from statistical physics or wherever to sell you some Sokal hoax like vapor fund (of course they now the tricks that you or the public will pay when the bubble collapses again)?

I'm a bit split on the interesting tangent that @thoroughaway opened of distrusting mathematical models altogether. When I left physics somehow this field of "social physics" tried to establish itself, and I remember how the old lads in statistical physics viewed this field with great suspicion as not being proper physics. (One of the physicists of that field was consulted prior to the Love parade disaster in Duisburg https://en.wikipedia.org/wiki/Love_Parade_disaster - it's not about blaming him, but about the suspicion maybe also raised by thoroughaway that not everything, especially humans in a panic, should be modeled like Brownian motion or something)

It's worth knowing that computer vision originally worked with manually programmed feature identification using more-or-less standard signal processing techniques, but those techniques were wiped out by neural networks. In some domains ML performs far, far better than any manual model could, simply because in those cases first-principles understanding is impractical to obtain. In fact, most textbooks on optimization start out with some comment to the effect of, "such and such a technique is only good to use when you have no knowledge of your problem domain." That speaks for a lot of problem domains.

Tangential rant: I worked in a computer vision lab around the time ML/neural network was on the rise in the area. The chief scientist there was seriously annoyed by such papers because he got into the field to understand at a deep level what vision is. Neural networks get the job done, but after successfully implementing a neural network you haven't learned as much about how you can infer information about the world out there from images of it, as you do from first principles thinking.

This not 100% true. I am not advocating first principles. I am advocating mathematical modeling. Mathematically modeling vision seems pretty much impossible. I do think there are many theoretical parts of topology that might be useful like homotopy. It could be the case that the "next calculus" needs to be discovered. By that I mean a totally new branch of math that will change mathematics for the next several hundred years (the same way Newton and Leibniz did with the invention of calculus). It took a very long time for us to go from algebra to calculus, in the end calculus ended up being simple. The same may be true here.

Or another scenario, instead of modeling vision, model the brain, mathematically (this is the path I favor)

There are mathematicians trying to model the brain. Unfortunately the field is very new, around 50 years or so. Deep learning and neural networks (in their current form) are temporary. There is a tremendous amount of experimental and quantitative work that was done in the last 50 years that (I believe) will solve vision far better than deep learning.

My major question to you is, why do people use back propagation?

https://www.axios.com/artificial-intelligence-pioneer-says-w...

This is the smartest thing Hinton has said in his career. Back propagation is a pseudoscience, sort of like ether theory was.

Deep learning is a pseudo science. Why back propagation? Why sigmoid function? I know the intuitions behind why these decisions were made. It's all questionable, there is no rigor to it, nor is there experimental evidence for any of this. Pseudoscience.

What will replace back propagation and the sigmoid or the relu function?

The guys modeling the brain have a good idea. Right now is a wonderful time, there are institutes all over the world that have done tremendous amount of work in this domain. There's a wide amount of competing ideas and models. These ideas haven't trickled their way into computer science yet, and remain esoteric. Which one is the right one? I have my "team" already picked, and it will solve audio and vision, and basic problems in language. But the proof is in the pudding.

Will it give us AGI? (Hint: no it won't. Not in our life time at least, the math isn't there yet).

Can you expand more on what direction your "team" is researching towards?

As for over-fitting, I am not in my 50s, and I knew what over-fitting was from my undergrad (or highschool?). The idea is very commonplace in engineering. Numerous times, when encountering a "new term" in a different field, such as overfitting in machine learning, I have found myself thinking in overly complicated ways, because I never thought that what I considered so simple could be the central point of discussion. Perhaps the physicists who could not describe what over-fitting was, were in the same shoes. The other unfortunate scenario is that the interviewers expect certain responses that are just not the generalized things one knows of.

But regarding other matters, I hardly doubt that the sentiment you are suggesting is lost on all.

One point of view is that the self affirmation bias is easy to come these days. The material on basics of machine learning are beyond trivial, but the hype makes money. Even traditional engineering does that to some extent, at least that's what I saw in my education. Same goes for basic finance. I see hype attempts in non-"computer science" engineering research, which amounts to noise. Some writers are oblivious, some are doing it for money. The problem is these days every noise is amplified, and at first I was confusing noise for signal. Particularly at a period when I was practically doing solo research despite having colleagues.

All being said, and I could be too quick or too short on this, large scale computation may automate many aspects of our lives, and funding goes for automation whereas mathematical modeling does not automate things in and of itself.

As for new mathematics, I doubt serious researchers would disagree on looking for more.

I came across this today, you might recognize one of the names, haven't read it and doubt will find the time for a while, but perhaps it could be of interest to you. https://arxiv.org/abs/1608.08225

100% right. The only thing I can add to your comment is that “machine learning” is rapidly beginning to expand to include heavily quantitative model-based work.

What graduate level mathematics courses are relevant for these Wall Street jobs? Your comment has me interested in taking some of these courses.

If you are interested in how mathematical modeling works I recommend any course that covers the contents of this book: https://www.amazon.com/Mathematical-Financial-Derivatives-Sp...

Again, experience beats academia. These courses do a great job at introducing you to how mathematical modeling works. You need to go through an incredible grind (I can't stress this enough, it's a GRIND). I would recommend having a solid foundation in analysis. I can't stress how important this is, it's pretty much your bread and butter. I never took any computational finance courses, I wasn't a physicist, but a pure mathematician.

I will say this: from my experience, there is no such thing as a genius. Anyone can be a great mathematician, and gain this skillset. Often times, these so called geniuses from Ivy league schools are over inflated. You will get stuck, and get stuck for weeks to even months. It takes a very special mindset to push through it and those are the minds that are the most successful. Math is hard, no one is naturally gifted at it. You may have had people in your past that you thought were geniuses. I assure you they weren't. You saw them solve "easy" problems. They weren't anything special. Mindset always trumps innate abilities.

Reply on news.ycombinator.com