Machine Learning Research at Apple | Hacker News Reader

Machine Learning Research at Apple

machinelearning.apple.com

258 pointstosh39 comments

Hats off to people like Carlos Guestrin & John Giannandrea, who seem to have pushed a cultural shift through Apple. I didn't think I'd see a site like this a few years ago, based on Apple's historic reputation of strongly discouraging employees from publishing research.

Totally respect what they have achieved, one thing to keep in mind is to structure the incentives so that researchers don't just focus on h-scores and the like but also ship features that can transform user's experiences.

Not to take away from the people who worked hard to shift the culture (which is always hard at a large company), but I saw the same shift happen at Amazon and it came down to a simple fact - it's very hard to recruit and keep the best talent if you don't publish. You don't build ML reputation and the best researchers often don't want to work somewhere where they can't add papers to their resume.

Not sure about this: a lot of interesting work is just hidden because of NDAs, commercial IP, military, etc. You will never see anything made public until necessary and just the minimum amount required.

This is still hugely frustrating to the people working on it. I build high pressure distsys architecture and I've worked on some incredibly cool bleeding edge tech that nobody will ever hear about and I can't even put on my resume because of NDAs. The subject pops up on hn maybe once every 3 months and I'm just dying to talk to other people in that industry but it's just a private group so I enjoy what tidbits I can read, they're always low reply posts, and pay for white papers for most of the rest.

It actually became really important to me over the last 5-10 years. I have no desire to discuss the secret sauce but I've been at companies so secretive that we couldn't even send a github issue up on our personal accounts because some context might bubble up eventually as to what we built and who we are.

I am so, so much happier working on my niche at places where I can interact openly with people from other companies/projects, not in a competitive manner but as engineering peers working to improve some subsystem of our architecture.

I feel like the shift is coming, especially with all of the open standards groups companies are rallying behind. I'm loving that. It's just taking its time to come east of SV.

A big part of maturing as an engineer is realizing that if you’re not talking about something under NDA that’s actually cool, someone else on your team is, and it just makes you worse off.

The rabbit hole goes deeper. Sometimes you’re hired to recruit your friends where you’ll necessarily have to break the NDA to get them excited.

A lot of people look back on R&D work where they did not thrive and quit. I think a big part of it is not realizing what confidentiality really means. They miss out on working with their smart friends, they miss out on getting ideas from other people, they realize there are not enough people they know whom they could trust with confidential stuff. They turn out to be way too square to be doing R&D work, they're hung up on breaking the little rules so how are you going to break the big rules? They're being paid to break rules and they're just afraid to break them. Then when it just comes to doing good work, it’s so important to talk to friends and family for useful feedback especially since you almost always find out from your boss that you’re doing something wrong way too late.

This is especially acute at places like Apple that glorifies confidentiality. They’ve been more successful than ever with their more relaxed attitude towards leaks. It was totally unproductive but it was cargo-culted into places like e.g. Facebook, Snapchat and Samsung that have a hard time recruiting because nobody finds out what they’re going to be working on.

I use NDA talk as a marker of seniority of the person I'm talking to. If the person says they can't discuss this and that due to NDA, they are usually junior people. When you talk to senior people, they know what they can and cannot talk about, and will walk the line carefully, but never ever bring up NDA in any conversation.

Yeah but there are also senior people whose heads are so far up their butts. Definitely a situation at Apple that gets cargo-culted elsewhere.

The line they walk is in the service of their own egos. Senior people are rarely in possession of the sort of detailed knowledge that is actual valuable IP to steal - that's like, in the documents and code. They want to get chauffeured in a Bentley. [1] They want everyone to hold their breath when they talk. And by the way, Valleywag turned out to be spot on about how good Tim Cook would be and how bad Johnny Ive has turned, based entirely on their attitudes and not their past performance, which was a serious refutation of the entire way that R&D org was oriented and is reflective of the positive shifts Apple is doing today.

Senior people just have ideas, which honestly anyone can guess that Apple is working on an head mounted display, or that they are experimenting with a Siri that can see through the HMD's camera, working on their own bank, etc. etc. So what is there to keep secret? Google is literally working on everything all the time, even and especially ideas that have failed in the past, so there is literally nothing of value you can learn from "What is Google working on?" So the actual economic value of the head of R&D's secrets is very low, their job is to go and recruit and in that case they should really be talking quite openly about what excites them.

What are the odds any of them are like Shigeru Miyamoto, whose body guards brought around shrouds because last time he turned one of his day to day activities into a video game, it made billions of dollars? Slim. Some VP at Apple is not Shigeru Miyamoto. If you were Shigeru Miyamoto you'd go start your own thing. And that's really what I mean by ego, what 55 year old, at the peak of their seniority and career, is really as great as their paycheck and ego says they are, if they aren't you know, telling their amazing ideas to everyone and recruiting people to do their thing?

[1] http://valleywag.gawker.com/apple-design-boss-jon-ive-gets-c...

> What are the odds any of them are like Shigeru Miyamoto, whose body guards brought around shrouds because last time he turned one of his day to day activities into a video game, it made billions of dollars?

Sorry, what's all this about?

An aside, and an unnecessary one, but after reading your [1] reference, I cannot say how happy I am that Thiel was able to shut down Gawker. That reference read like a cheap supermarket tabloid, and the internet is better for Gawker having been destroyed.

My point really wasn't about discussing grey-area NDA stuff in an interview or with my family. It was about contributing upstream, publishing my research, and interacting with the communities of projects that we use as components of our systems. I'd ideally like my industry to be more open with its exchange of information, best practices, etc, but it's literally all competitors unless you're a university.

I know exactly what is reasonable to say in a business context. Comfortable walking that line - it is for business purposes and I'm never concerned by my own ability to make judgement calls.

But when talking to a recruiter? Ehhhhh. I am pretty sure I would break the precise terms... so I guess the real question is how vindictive is the holder of the NDA :)

I use NDA talk when I don't particularly trust the person I'm talking to and can't work out what their angle is.

I guess my point is more that as a huge tech company that wants to hire many very good ML researchers, a culture of publishing is necessary

I don't think that's actually true. Lots of stuff gets published that could (and if the bean counters had their say _would_) be "hidden". But NONE of it is published by commercial labs before all the patents are filed. MS Research cranks out an amazing number of patents. As does Google Brain and Deepmind. Then they have this Mexican standoff and "license" patents to each other.

But _some_ select stuff does not get published. I know of at least two examples first hand: one at MS, one at Google. This is usually the case when publishing a paper would help large, direct competitors to partially or fully close staggering competitive gaps. As you can imagine Google doesn't publish a whole lot on the subject of search ranking, for example.

Isn't it just defensive ? Is there any instance of anyone using or licensing an ML patent ? Mostly people like IBM, MS, etc. were doing patents anyway for decades, and they continue to do so. So everyone else has to play the same game. I don't think there has been any big case involving patent infringement over ML/AI.

It's "defensive" only until the company starts sliding financially, at which point it becomes _very_ offensive. That's how IBM got MS into this game: one day IBM lawyers showed up with an invoice at Bill Gates' office.

And a lot of stuff doesn't get published until the system described is already obsolete.

Concur. There is amazing work being done behind gobs of money and NDAs. Some people are motivated by money and accomplishment rather than citation rings.

Amazon also launched their own portal for showcasing their publications. https://www.amazon.science/

I can't imagine too many nerds that would willingly work for Apple.

I've turned down military contractor work despite it paying better than my current job.

Being able to contribute to society goes a long way.

This post makes no sense to me. As one of the so-called FANGs, the data suggests Apple is one of the most desirable jobs out there.

Apple is in FAANG. For some reason (likely related to 2015-17 era SWE compensation) FANG just includes Netflix (with amazon facebook and google) and not Apple.

I mean the actual companies that go into these acronyms are generally arbitrary but AFAIK it was Facebook-Apple-Netflix-Google as companies with rapid growth/decent profitability during that period.

So are cushy military jobs. But some people have morals.

"You don’t become a man until your father dies."

The open stance on R&D that the industry is currently adopting is good for everybody involved. I hope it will become the norm someday.

Journals and other centralized entities that regulate access to science should wane and disappear, the sooner the better.

> Journals and other centralized entities that regulate access to science should wane and disappear, the sooner the better.

To the extent that other mechanisms are in place to to peer review research and curate quality, yes. And yes I'm aware that the current system is far from perfect.

Ian Goodfellow is at Apple but doesn’t appear in any of these papers. Interesting.

Is he on something too secret to reveal? Leading the SDC effort? The Karpathy of Apple?

I think he joined last year, not sure when. Maybe that has something to do with this.

EDIT: He joined in March 2019 as Director of the Special Projects Group.

Built with Next.js: https://mobile.twitter.com/rauchg/status/1286769353027801088

I remember ICML 2015 in Beijing when I was the only one (I suspect there was another guy but apparently he even had to hide his affiliation) from Apple at the conference. Not allowed to officially/publicly to talk about what I do. Went only because of networking and to hire people. Times change slowly there.

Discussed 3 years ago: https://news.ycombinator.com/item?id=14804018

Different content. Back then it was a handful of cherry-picked publications expanded into articles, now it seems like the page is an overt advertisement of Apple’s Machine Learning efforts.

Interesting. I thought that apple mostly did closed research, yet they are publishing in public conference.

I mean Apple would have to be open eventually right? To attract and retain talent, if everyone else is doing it?

It's like Microsoft under Satya Nadella, really understood that Open Source is a way to actually win developers, rather than Steve Ballmers "Developers! Developers!" chant that was just marketing speak.

So if you want to be in the same league as Deep Mind, MSR, FAIR, Google Research, then you gotta play like them.

guys, is this the first time apple is involved with academic papers?

Nope, they've been sharing their research more widely in the last few years. Here's an example from 2018, in a different area of research: https://www.vldb.org/pvldb/vol11/p540-shraer.pdf

No, the site links to papers that go back to 2017.

Reply on news.ycombinator.com