Ask HN: Is this normal? Long-form transformer take home

They are looking for someone with the level of experience that allows treating the problem as trivial.

That means plugging in the data to the tool they use all the time, adjusting a few parameters, and writing the 1000 words while waiting for the results…that’s possible because they know the tradeoffs they made as they made them.

If it looks like a hard and interesting challenge, that desired candidate isn’t you.

Nothing wrong with that.

They want someone who can knock this class of problems out before lunch on a Tuesday when the boss asks at 10am.

To put things into perspective, someone like that makes $600k+ at FAANG. At a hot new startup they would be making $300k+ base. And these people are in huge demand currently.

If you're expecting this level of compensation I believe a take home task like this is not too much.

That much? For the task that OP is describing? I know that FAANG salaries are high in the upper bands but I think that OP's take-home is doable at a much lower level. Or I'm very underleveled and underpaid.

And of course, the task being doable doesn't make it a good take home, I still think OP should turn it down.

For the person described in brudgers’ comment.

This role pays around 170-190k

How many years of ML/NLP experience do you have? 190k is probably reasonable for someone with less than 2 years. If you have more, definitely try more companies.

A dollar today is worth is lot less than even 1 year ago, just look at the housing prices.

12

What if you're expecting a third of that?

Why are you working for free? I'm not in machine learning, but if I got this in an accounting field I would think it was a scam and not a legitimate job offer. Just my two cents. Best of luck friend.

From the mouths of babes. You get so used to grinding LeetCode and doing take homes weekend after weekend, you stop questioning reality. Maybe it takes someone from an outside field looking in to see it. Thank you for the refreshing perspective..

I've enjoyed reading about programming on this site for a few years, so, no, thank you for providing a peek into your reality.

FWIW, some companies do compensate you for trial work.

For the second part, I’d also train a fastText classifier. It’s cheap to do and can often get very usable results at 1/500th the computing power of a BERT/SBERT model.

Re: the length problem, some ideas:

a) Just use the first part of each doc. For most humans, in most cases, that would be enough to do the classification.

b) Mix head and tail of each doc (i.e. cut out as much of the middle as necessary).

c) Split each doc into as many parts as needed, classify all of them, and then go with the majority vote.

Good luck with the interview!

Your first suggestion seems like the way to go, I read a couple of papers this morning which seemed to confirm that it is a solid strategy. The first one claims that the Longformer architecture is tenable for truly long documents with some minor parameter optimization [0]. The Amazon/MSFT paper claims that just looking at the first part of the doc is superior to most other more complex solutions, like Longformer/Big Bird or hierarchical classification [1]. But here's what's really cool, they concluded that if you still use only 512 tokens, but you don't just look at the first ones, you look at basically an extractive summary of the most salient phrases in the document, it's more accurate. They call it BERT-TextRank, and it's using an extractive summarization algorithm from 2004.

Found a tutorial which at least makes classification with RoBERTa pretty straightforward.

I honestly didn't think this was possible before today but it's always nice to have your preconceptions challenged.

[0] Revisiting Transformer-based Models for Long Document Classification (anonymous ACL submission, published yesterday) https://arxiv.org/abs/2204.06683

[1] Efficient classification of long documents using transformers (Amazon, MSFT, March 12, 2022) https://www.amazon.science/publications/efficient-classifica...

> Here are 150k long-form (6000+ words) documents, and a list of labels.

> write 1000 words about the performance tradeoffs.

Big no. And it shouldn't be a close decisions for you - this is way too much work to do for someone you aren't working for. Who in 2022 is asking for 1000 words as an explanation? And who is actually reading it???

Fascinating stuff, thanks for sharing.

(I kind of wish take homes were less common and more companies were doing LC for data science and ML)

Don’t know ML but this is sounding like a lot of work. I almost always say no to take homes. There’s also a limit to how much the a take home should take. >2h of your time is asking a lot. I philosophically disagree with take home interviews for many reasons.

Yes it is too much. I'd decline to do that much work on an unpaid basis without more commitment from the company.

Regarding jstx1's question of who was going to read the 1000 words: obviously, an NLP text summarization package that will spit out a number from 1 to 5.

Take a pre-trained Bert model and then fine-tune the label classifier part using the documents.

It really is that simple, but I didn't think we were there yet!

Hey man, google pytorch.

Spent a lot of time trying to get PyTorch and CUDA to play with my 3070ti yesterday and finally gave up and made a CoLab notebook. Got a pipeline working and trained a multiclass classifier, but the f1 is in the toilet. Might try a different dataset, then maybe a tensorflow implementation next if that doesn't move the needle. Hard to believe the accuracy could be this bad compared to a linearSVM baseline, must be giving it the wrong x/y labels or something.

More than a couple of hours, no, it’s a scam.

Name and shame