Hacker News Reader

phillyphamon Homelessness rises faster where rent exceeds a third of income (2018)

Serious question. What's wrong with living next to basketball hoops? I live across the street from one in NYC and consider it a big positive to have a park nearby.

phillyphamon Apple, its control over the iPhone, and the internet

Those few hours you spent are probably worth at least 2 years of YouTube Premium if you're a SWE. And you'd be supporting creators.

phillyphamon Why is the university of California dropping the SAT?

It wouldn't surprise me if at least one factor contributing to the higher yield of Asian American students is that they are being discriminated against at Ivy+ schools and have to "settle" for UCs that don't practice affirmation action.

phillyphamon Introduction to Locality-Sensitive Hashing

It was used in https://ai.googleblog.com/2020/01/reformer-efficient-transfo... for faster and more memory-efficient attention.

phillyphamon Deep Graph Library: Easy Deep Learning on Graphs

Traffic prediction https://deepmind.com/blog/article/traffic-prediction-with-ad...

phillyphamon Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks

Edit: This is not meant to criticize the Tensorflow, TPU, or XLA team. They responded quickly to our bugs and made the work possible. I just meant there was some extra organizational overhead.

phillyphamon Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks

I think the insight is not incredibly original and follows naturally from OpenAI's Sparse Transformer. The idea is similar to Longformer. Two teams at Google had a similar insight, hence the high number of authors.

The original implementation only took a couple of months and was primarily motivated by internal Google applications. Natural Questions was the first external benchmark we tried to validate on, which took a few months to find the right setup. All the other datasets, took a few weeks but the effort was done in parallel given the large team.

There was quite a bit of frustration dealing with Tensorflow, TPUs, and the XLA compiler that maybe set us back a few months, too.

phillyphamon Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks

This is something we want to explore. BigBird just replaces the attention mechanism in BERT.

We believe something like BigBird can be complementary to GPT-3. GPT-3 is still limited to 2048 tokens. We'd like to think that we could generate longer, more coherent stories by using more context.

phillyphamon Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks

Not really. The proofs are more of a curiosity really. I think the strong performance on QA takes that require multihop reasoning give some evidence that the model is capable of complex reasoning.

phillyphamon Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks

Right now, BigBird is encoder only so it doesn't generate text. Causal attention with the global memory is a bit weird, but we could probably do it.

GPT-3 is only using a sequence length of 2048. In most of our paper, we use 4096, but we can go much larger 16k+. Of course, we don't nearly have as many parameters as GPT-3, so our generalization may not be as good.

BigBird is just an attention mechanism and could actually be complementary to GPT-3.

phillyphamon Google ‘BigBird’ Achieves SOTA Performance on Long-Context NLP Tasks

I'm one of the authors. I may be able to answer some questions or concerns.

phillyphamon RobotDiAngelo: GPT-2 learning to be antiracist

GPT-2 fine tuned on the works of Robin DiAngelo.

phillyphamon Google interviewing process for software developer role in 2020

I know it isn't perfect, but I personally am a huge fan of Google's interview process. It's a boon to people switching careers because data structures and algorithms can be studied without any professional experience. Behavioral interviews and GitHub portfolios are biased towards those with previous experience and internships, which many of us never have the opportunity to do.

After leaving graduate school (not CS), every startup expected me to magically know how to be a professional software engineer, and I found Google to be the only company willing to hire and teach me. If the goal is to bring together smart people with diverse academic backgrounds, I find that Google's process is rather successful.