For FAANG engineers this will likely mean moving to Vancouver, Zurich or Singapore with their job, salary, rsus and taxes.
106 karma · joined June 22, 2022
Some people love being a slave, but that is not a requirement for having electricity, railways or capitalism. Just look at any other country.
> naive, childish, saffron-tinted glasses.
A brain with less self-righteousness has more space for common sense and logic.
So golden age of India was when the country with a seventh of the world's population won 2 nobel prizes over 5 decades ?
> The British rule also was the largest and most stable unification of India till the modern times. After 1850s there were almost no pockets of military resistance against the British rule.
Mughal and Gupta empires lasted over 3 centuries, Mauryan empire a little under 1.5 centuries. By comparison, east india company rule lasted a century and the British crown's rule less than that. So again completely incorrect.
> The third golden age which no one wants to admit (left or right) is the British Golden age.
There's your hint: if people on both sides of the aisle don't "want to admit" something, maybe it doesn't make sense. Not to mention a slap in the face of billions of Indians.
> The British age declined with WW1 and WW2, and ended with Indian independence.
Thank god for that decline, otherwise Indian taxpayers would have been funding Brexit and the crumbling British economy right now.
> My oversimplified summary has been
This is not a summary, it's a lazy opinion backed by little research.
- Looks like there's just 7 abstracts right now.
- Most of the abstracts are written by the author of the paper, so might not be as unbiased as an actual "community-written" abstract.
- There's no stated guidelines for the "community-written" abstract e.g. should it be less biased than the original abstract, should be shorter than the original, should it be more accessible to a less AI crowd or all of the above.
- There's no way to upvote/downvote some abstracts e.g. the "attention is all you need" paper has two abstracts and one of them is clearly worse than the other.
Out of genuine curiosity, do you have any pointers/evidence to support this. I know that some of the industry leading research labs haven't switched over to DPO yet, in spite of the fact that DPO is significantly faster than RLHF. It might just be organizational inertia, but I do not know. I would be very happy if simpler alternatives like DPO were as good as RLHF or better, but I haven't seen that proof yet.
- 1444x faster for single character prefixes
- 252x faster for two character prefixes
- 55x faster for three character prefixes
- ~20x faster for 4 and 5 character prefixes
- <= 5x faster for longer prefixes
I used to work on a production auto-complete system operating at over 100k peak QPS. For prefixes of length one and two we would not even bother hitting the server, just from a quality perspective, not because of latency/throughput considerations. Btw, up until 3 characters, you could store everything in an in-memory hash map. 20x speedup on length 4 and 5 prefixes is still very impressive, but not quite 1000x speedup either.
It's not just the pace of improvement, but also the marketing spin. I find the strengths of smartphone camera and ILCs pretty complementary. Smartphone cameras work pretty well outdoors where there is enough light. DSLR and mirroless are hard to beat indoors in low light conditions. Coincidentally it is also easier to find your ILC indoors at home when you need it, rather than lugging it around on a hike. When we didn't have kids, we used to spend more time outdoors and so most of our memorable pictures are from a phone. Now that we have restless young kids and are spending more time indoors, almost all of the memorable photos are from a mirrorless camera. But the marketing spin makes it seem like ILCs are completely redundant.
1. Creating needless technical complexity.
2. Creating needless compatibility issues.
3. Distracting the focus.
Hiring 2X more people is not going to result in 2X more value creation, but laying off half the staff is not going to 2X the productivity either.
> rough steps:
> 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $.
Pile seemed quite clean and manageable to me (I was able to preprocess it ~8 hours for a simple task on consumer grade hardware). Is Pile clean and rich enough for LLM training too ?
> 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts.
It seems like the standard instructGPT model itself is based on a 1 billion param GPT model. Wouldn't that fit on a 24GB RTX 3090 ? Might take longer, maybe not enough opportunity for hyper-parameter search, but still possible right ? Or is hyper-parameter search on a thousand machines in parallel the real magic sauce here ?
> 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec...
Sounds like a good opportunity to learn. No pain, no gain :-)
> 4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 to get a sense of the work involved here.
Maybe somebody would open source the equivalent datasets for this soon ? Otherwise the data collection seems prohibitively expensive for somebody trying to do this for fun: contract expert annotators, train them, annotate/reannotate for months ?