Hacker News Reader

Comment by anon1253 | Hacker News Reader

"The cost of training the model from scratch using our code is about $50k."

Still a substantially steep curve for a bootstrapping startup. It's something I continually run into myself. I have somewhat of a weekend project trying to build a search engine but man ... the cost of just the SSDs and GPUs is daunting on a regular salary. As the complexity of these models grows, so does the barrier to entry for a regular joe like me; which is a shame I think. I know in the US it's fairly normal for a data scientist to pull 100k+ / year, but in the Netherlands salaries pretty much stall at 40k (and angel investment in IT/AI is at an all time low). More generally I fear this will become a bit of a sociotechnical issue if complex AI models will be out of reach for entire economies (especially for cases like language because not everyone speaks English and "minor" languages like those in EU countries are a massive market to explore, yet hard to get into).

View on HN

Seriously, you do not need massive amount of money to build search engine. The V0 version of Google was literally a single cheap desktop running whole thing end to end. Average webpage size even today is just 2MB. You can just start with crawling 1B pages which you can comfortably store and index on 8TB HDD (you don't need SSD at this stage). You should make your algorithm work for single user on this 1B page crawl. Don't worry about freshness or speed, just focus on relevance. If things works beautifully, find a VC and show them the goods. If things works out, you might probably end up with $100K seed money. Then you go out and buy dozen good desktops, increase crawl size to may be 15B pages and support few dozen simultaneous users. Now you can go out to big guns, send them a link to your search engine and get next level of funding. At that point you hire real employees and now your job is scaling up to thousands of invite-only users and squeezing out the long tail of performance in terms of relevance. Constantly measure your customer satisfaction and complaints, use gmail-like invite your friends model to incrementally get high-value users who are willing to try out something new. At this point, if your search algo is really much better, you should be able to plot exponential DAU/MAU growth and get funding rounds in $100M range to really scale up.

And they're just "two masters students, with no prior experience in language modeling" with $50k lying around for training a huge model.

The money came from Google: "We would like to thank Google (TensorFlow Research Cloud) for providing the compute for this".

They mentioned they spent $500k (in research credits) on all the experiments to actually find the hyperparameters.

Where did you see that?

Also how do two masters students with no experience in NLP get $50-500k in compute credits? How do I get that deal?

It was in the article last night, they deleted it for some reason after my comment.

One of the authors has a peer reviewed NLP publication [1], the other has several publications in computer vision. I don’t know how they got research credits from Google.

[1] https://arxiv.org/abs/1905.13153

Well that is pretty dubious. That completely changes the metric of how much money and how difficult this costs. (And I definitely believe you because there's several other comments with that same number). Could be a typo? But I feel like that's something you'd say. Then again, these are masters students.

It's $50k per training run. Their main contribution has been finding the optimal hyperparameters, not described in the OpenAI paper. Obviously you need more than one training run to to that.

They didn't, their advisors -- Stefanie Tellex and George Konidaris -- did.

And the answer to that part is, they asked: https://www.tensorflow.org/tfrc

Interesting. I'm doing ML in a niche area so I'm wondering if I can get funding (for my PhD). Obviously there's politics involved, but honestly I just want to try out methods and experiment (typical grad student I guess). So if they have a funding route I'm stoked. Thanks for the info.

In most programs, if you get accepted into the Ph.D. program, you're funded. Obviously, you should check first. Different programs treat these things differently with TA requirements, but most Ph.D. students are funded by an "RA" (research assistant, aka, doing your actual eventual-thesis research...).

For more info, my colleague Mor wrote a very useful doc about the Ph.D. process in CS: https://www.cs.cmu.edu/~harchol/gradschooltalk.pdf

Oh I'm funded. Definitely was told not to go unless I'm fully funded. But $500k for computing resources is bigger than a lot of small term contracts (I'm mostly on research and usually don't have to TA unless there's a hiccup in funding schedules. Gov money...). I definitely see that amount for long term, but this seems like a pretty short project.

Got it. (1) Yes, 500k is a lot, but (2) the TFRC program is making fairly expensive TPUs available at shockingly good prices for researchers willing to do some tire-kicking and feedback-providing. So 500k of TPU time is easier to come by than, say, 500k of generic compute.

There is no need to train the model, they already provide the parameters. These transformer models , like BERT are pretty adaptable to reuse.

There is no need to train this particular model, but adopting this (or any novel model, the field is moving fast) to say Dutch, Italian, Hungarian, Icelandic or whatever still requires training. Luckily for most languages they are provided (at least in the case of BERT, FastText, or regular skipgram). But there is also still quite a bit of leeway in domain specific adoption (for example SciBERT for scientific texts, or legal and financial documents) reddit/wikipedia does carry a bias. Each of which not only requires pretraining the model, but also generating a huge and fairly well formatted corpus. And, although the parameters are usually finetunable, it does break down sometimes on the various sub-word tokenizations used.

Reply on news.ycombinator.com