high_derivative

He is rather controversial in machine learning circles. I would say he is essentially right about the fact that a lot of ml work is crap. Like most of anything is crap, so that is not a groundbreaking insight and it comes off as somewhat grandstanding to keep saying that.

He has however really turned this into a personal brand which many suspect he is using to advertise his startups. Many people feel he is sort of disingenuous about the actual progress being made in computer vision and NLP, even if there is many ways to go.

To answer your question: You can read a single piece of his and maybe get some value from that, but he keeps rehashing the same points to sh*t over different new work, and there is not much to be learned from that after a while.

Disclaimer: I did not read the above article in full, I am just familiar with Gary Marcus + work in ML industry.

Edit: Keep expanding this but it got me thinking. Earlier in my career, I was also eager for the 'gotchas', for being able to point out why something is crap, exaggerated, cannot do what it promises, and so forth. As I outgrew this impulse to focus more on what is good about some work, I began spotting it on new students a lot. They would come into seminars and smugly crap all over recent papers.

I have since come to the conclusion that it is useful to have realistic internal assessment of progress in your field, but that sharing your harshest criticisms is not necessarily a good way to look smart or make progress.

It often turns out that while feeling clever about realizing limitations of some work, the field just moves forward. The people you just criticized leapfrogged while you were busy thinking about how to one-up them.

hah, I made a bet with myself too that this was a Gary Marcus article. He is really riding that brand..

Again, disagree. I read the comment differently.

There is a certain type of person who thinks because they are smart and good at one thing, e.g. software development, their powers of reason allow them to come to the right conclusions about everything. They are super vocal about this on social media, e.g. typical Twitter mob warriors. Because they think they are intelligent and their views are logical to them, everyone intelligent will agree with them ("the best and brightest will want to work with people like me").

Conversely, people disagreeing must either be dumb or have outright malicious intent (e.g. James Damore). This type of person has difficulty accepting that there can be highly intelligent people who come to entirely different conclusions about various issues. They can be intelligent, but they are also often toxic because they are loud and vocal and lead other people to just withdraw from the work environment, lest to provoke them. I am reading the original comment as the HR trying to avoid hiring such people, not that they don't hire someone who expresses dislike about Burger King's offerings or the latest Marvel movie.

That sounds completely reasonable to me. In particular, this says nothing about the morals of the people you do end up hiring. I am not aware of any research investigating the correlation between doing good and virtue signalling on Twitter, but my guess is that you can do one without the other, and that in fact participating in vocal outrage is a distraction to serving your community in more productive ways (e.g. getting people to vote, or otherwise getting anything done to change what you believe must be changed).

I think that is a fantasy stemming from the idea that everyone who is talented and intelligent must basically agree with you politically. The world does not work like that. Talent comes in many shapes and forms. I know many people, myself included, at FAANG jobs who would love for a less political environment.

I disagree, I don't think it's unfair to say at all. It is simply mismanagement to input resources into a language rework that is dead on arrival, community wise. This is of course par for the course for big tech research orgs where big names get a lot of free rein, but that does not mean it's not a strategic mistake here.

This is simply about focus as an org, and this is the reason why PyTorch is getting so popular.

There seems to be a massive lack of focus and direction in the TF org, too many egos wanting to put their stamp all over the APIs and subsystems (tf.keras anyone?).

TensorFlow eager with autograph or Pytorch solve all differentiation problems as far as researchers and practitioners are concerned.

Exactly how I got into it. At university, realized that despite buying some office pack with a laptop, there was no visio contained. Began with draw.io and never looked back.

Thank you for responding. Well, my point is that in particular the gradient on the likelihood ratio is what trips people up. They ask questions like 'why is this ratio not always 1' or similar. This is why I would say explaining what is going where here is critical, i.e. that we save the prior logp_pi (even though we could recompute it) to treat it as a constant value when computing the ratio/the gradient. That would be, from my perspective, the key pedagogical moment of a PPO tutorial. However his is purely subjective and I agree that one can feel differently about where to put explanations.

Ok, since you took the time to respond, I just want to be constructive as well:

So I don't have a big problem with some of the function definitions which can be compact, as the other comment points out.

The reason I don't like this code is that it does not comment anything on the critical bits. I don't necessarily care about whether you call the input to your matmul 'x' or 'tensor' or 'input' (although consistency is nice).

The thing that would stop be from absorbing and modifying this code is that it does not comment on all the bits that are non obvious to me if I haven't written a Transformer before. For example:

'Same as tf.matrix_band_part(tf.ones([nd, ns]), -1, ns-nd), but doesn't produce garbage on TPUs.' - I will have to ask the colleague what that means. Why not write out what the actual issue is instead of mysteriously hinting at some potential problem?

Code like this "q, k, v = map(split_heads, tf.split(c, 3, axis=2))" will require me re-reading the paper section, then printing out all the tensors to think about what tensor would have which shape at which point. Instead of writing relatively useless linecomments like '#Transformer', I would comment all non-trivial shape modifications with the current layout, and what we are trying to achieve while modifying the layout.

The other issue of my original comment was not specifically on that codebase, but I am sure you would admit that the baselines code was pretty much exactly what I was writing about re: ml scripts. That's not to denigrate its incredible usefulness to the community.

Since you mentioned spinning up, I thought I would add a few comments on that as well:

I think the spinning up code base is good at making the code compact, and terrible at making sense of data flow for beginners. There are a lot of line comments, but they do not actually explain what is conceptually going on but often just repeat short-hands.

For example, look at the PPO implementation: https://github.com/openai/spinningup/blob/master/spinup/algo...

Here, the function is returning pi, logp, and log_p_pi (and v). Do you know how incredibly confusing the distinction between these is for beginners? In particular, there is no explanation why logp_pi even needs to be stored in the buffer.

We could recompute it from the states and stop the gradient when computing the likelihood ratio. A sensible tutorial-level comment here may be something along the lines of computing the likelihood in the same forward pass as computing the action, so we can later use it to compute the likelihood ratio. We could also later re-compute this from the buffered states.

I will stop here but I hope my point comes across, whenever I read code from your repos, there are some good parts (conciseness, cute numerical tricks) but there is as general missing sense of thoughtfulness on what the code is really trying to convey to a reader. It shows in the comments and it shows in the code organisation.

As a final note, I have seen this in many organisations and I do not mean to call you out. There is just this quality degradation that inevitably happens when nobody is incentivised (read: promoted, rewarded) to think about these things for an organisation.

Managers at all levels typically don't because they don't get close enough to the subtle issues on a day to day level. If you are lucky, you get senior individual contributors who still look at code and and raise the bar for the entire org. My genuine recommendation to you is to look for that, because a manager won't do that, and more fresh grads can't do it.

I don't see how what you are saying is contradicting me. The point is that the scientists generally don't view it as their job, whereas the ml engineers/research engineers typically have it as their job to do the software architecture/engineering side.

My professional observation (as ml researcher at big tech):

These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework.

The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti.

And this is the engineers who are meant to clean up research scientist code. Scientists generally don't feel like it's their responsibility to write strong code.

Systems-side teams/orgs have better code, but essentially as soon as you enter the 'ml engineer/research engineer/research scientist' layer, it's doomed.

So can Europeans send a GDRP right-to-be-forgotten request to all of these?

Well this may be a controversial stance, but her baby passed at birth. There was no maternity leave to take any more.

What there was to take maybe was a bereavement leave.

To save some people the read, it's the same leetcode bs performance as anywhere else.

From what I read, cloud is massively expanding headcount in sales, marketing, and support.

What makes this a softbank-crazy investment is:

- the founders were just out of undergrad

- zero background/credibility in either distributed systems or gaming or simulation

So I will guess there was no well-defined R&D roadmap ahead at all. Where should it have come from?

Hah. I have wondered about this particular one a fair bit.

First, it seems there is just not market for their product that could possibly support the valuation in the near future. Second, by virtue of already being so large a company, there is no room for evaluation of new small projects that would change direction. Anything must immediately have enough potential to generate millions in revenue.

Runway is maybe 5 years, so they got until then to produce enough revenue to raise more funding or become profitable. IIRC they sold approximately 50% of the company for the 500M round. Maybe I got this one completely wrong, but I just don't see how this can go well. What an incredibly inefficient allocation of capital for the ecosystem. Imagine giving 100 teams 5M each instead.

The reckoning may be still to come for various other Softbank-backed companies. Will just take a while for them to run out of cash.

For example, Improbable also got 500M from Softbank a few years ago, and makes 500k (!) revenue at a loss of 50M [0] and cost of revenue of 10M. It's insane.

[0] https://uk.finance.yahoo.com/news/softbank-backed-gaming-sta...?

edit: for anyone interested in the details, look at the most recent group accounts file here: https://beta.companieshouse.gov.uk/company/08070525/filing-h...

Neumann's basilisk?

The 4 steps of cancel culture, or socially sanctioned sadism:

1. He said/did thing x. He must lose his livelihood and be annihilated.

2. He did not say thing x, but he said y which is basically thing x.

3. He said neither x or y, but he essentially deserves this anyway because of other things he may have done or said. They may have not offended anyone at the time to cause this level of outrage, but clearly combined with what he may have said recently, his time was up anyway.

4. He may not have deserved what happened to him, but maybe he should have just kept his mouth shut about issue x. If me and my friends accused him, then he probably wasn't innocent.

(5.) Write medium/Quillette/twitter post: I was part of the mob until it came for me.

Yes, that is always a case, but with deep learning, the hype train I feel was uniquely big in terms of salaries and money in the space.

No, it's not. Im an AI researcher at a big tech company. It would have been far wiser financially to start as a software engineer after my Bachelor's.

Also, having a few NEURIPS or ICML papers is not a hiring guarantee any more. It's decidedly not 2015-17 any longer. In particular, I feel bad for the people who started their PhDs with $$$ in their eyes around that time.

Don't get me wrong, they will all be employable, but the field moves so fast, I have my doubts there will be cushy FAANG jobs for everyone capable of playing around with network architectures in a few years. It's a terrible idea to start one based on the hype few years ago.

I found the best response to standups in a prior job was to randomise a delay about ongoing work. Simply talk about what you did 2-5 days ago (depending on urgency of things). This obfuscates your exact productivity and takes away organisational power from micro-managing bosses.

Whenever a manager comes in or there is a meeting to check on your progress, you can slice of something from your buffer to present. For every problem you present (from few days) ago, you can already present the solution.

Or, in other words: Your productivity is some process that you don't want to be observed directly by your manager, because you will suffer negative consequences from them detecting the exact amplitudes. You simply apply non-stationary smoothing to it.

I used the term grad stipend loosely, I was on various scholarships that meant I was not technically an employee or dependent on funding from my adviser. Did not make a difference at all.

Where you are right is that I did allow myself to be treated like that. I could have quite any time, of course.

The salaries at FAANGS are level-driven, so I refer to levels.fyi. I think the perk of being a research scientist is more freedom, building prototypes, not having to do support/oncalls in general, just being able to hand off that type of work.

So I am a machine learning researcher who moved to a FAANG as a research scientist after graduation. My salary is 10x against the grad student stipend. That does not even account for the free food, the healthcare, and other perks. However, I have not adjusted my lifestyle so it does not feel real.

The thing is, even though having 1000x the resources compared to university, that does not really make me happier about the work specifically. It makes some things easier and other things harder.

No, what I really feel is that at work I am not actually treated like a servant any more but like a person. I don't have to work weekends and nights any more. I can take vacations and won't be flooded with emails every.single.day during holidays. I don't have extra unpaid responsibilities that I have zero recourse against.

The thing that I just cannot stop wondering is, why, knowing the perks of industry, advisers still treated us like this. Even though ostensibly trying to keep us, it felt more like squeezing out as much as they could.

Please do not argue like this on hn.

I did not say shitty. Clearly, there are strong benefits including pay, career development, network, benefits. Everything comes at a price. They price of that is the psychological pressure of a highly politicised environment with many subtle layers which can be hard to navigate for many.

My SO works in healthcare and their job is arguably tougher in every possible dimension even though end-to-end qualification time via med school was even longer. Almost every job is compared to FAANG research scientist.

That doesn't mean I wouldn't prefer the same or slightly worse benefits without the open plan and culture.

Lots of HN readers (including myself) have worked at FAANGs.

The cultural characterisation seems completely accurate to me.

Everybody plays multiple games of performative open plan activity, pushing other people to join their projects, all with the friendly face of 'make the world a better place' while counting their RSUs.

It totally weirds me out how defensive people are about your anecdote here. Talking to any clinical psychotherapist dealing with young adults, you will hear lots of stories about weed-induced psychosis and anxiety.

What I wonder is why it matters to people that it's not as safe as they may believe? Cognitive dissonance? We do lots of unsafe things.

How would you know this without entering their personal space?

It may be high time to discuss what AI policy has actually done so far. From what I can tell, not much other than letting social scientists get in on the deep learning gravy train.

Meanwhile, misuses of ML are proliferating without limits, and 'AI policy' is apparently mostly used as a fig-leaf to collect good-will, marketing, and buy a seat at the table for future regulations. As usual, regulations will protect incumbents, so my as-usual cynical read is that OpenAI's policy interests are about protecting its own future interests. From that perspective, the entire GPT-2 stunt was highly effective.

Now depending on your outlook, that may be an argument that we need more people in policy, or fewer. Or different ones.