I analyzed the history of “Who is Hiring?” threads | Hacker News Reader

I analyzed the history of “Who is Hiring?” threads

blog.floydhub.com

195 pointsphilipkiely74 comments

A little off topic... but I love this because it gives me an idea.

I've been looking at LinkedIn with some immensely negative feelings for some time now. I've always wished someone would just disrupt the hell out of them, because they don't deserve the near-monopolistic position they're in. They just don't care about their users!

And before someone says, "well [pushes up glasses] their real users are recruiters that they sell $20k/year seats to their search tool". Yeah, as a hiring manager I've tried using it. Awful! :)

And jobs boards are nothing new, I guess, but something feels really nice about how much bigger these "Who is Hiring?" threads are getting on HN. Maybe we really do need a jobs-centric "social network" for industry verticals. Beyond terrible UX and general disrespect, LinkedIn also just doesn't do enough with their data to make it useable. Maybe that's because it's just too hard to do for every job and industry out there.

And maybe that next-gen professionals' social network can be the first open and free one?

Or maybe I'm just wishing too hard for something that can't/won't be done.

> Beyond terrible UX and general disrespect, LinkedIn also just doesn't do enough with their data to make it useable.

From LinkedIn's perspective, that's a feature, not a bug.

LinkedIn has become plagued by self-help-like stories and other self promoting, vague and meaningless write ups, in my experience.

It wasn't always like this. I used to have a reasonably cool feed centered towards technologies and news that are at least partly relevant to me and my industry. No more. It's all 'I gave a homeless person lunch and today they are the #1 duck breeder in Brazil' or some other nonsense.

I don't know when this started or if it is localized, but it renders the 'place to find useful information' aspect of social media completely void. There's no amount of reasonable curating effort that will save a news feed like that.

I try not to be too negative about things but without some serious change I'll use a static LinkedIn profile and visit it for a fraction of a second every week - to check for messages - and nothing more.

I just can't stand the humble brag posts!

I know! As someone with 500+ contacts to high powered professionals I'm inundated with a huge amount of humble brag posts!

*chuckle

You must hate Hacker News then.

Well actually the bragging here often doesn't even get a veneer of humbleness.

I disagree. I don't see much bragging around here. And amazing people hang here, which are often very humble.

And if they brag, there is something to brag about.

Those are called "broetry" posts. I believe LinkedIn implemented some changes to reduce exposure on these, but I still see them regularly.

WHAT WOULD YOU DO

WITH 12 DOLLARS IN YOUR POCKET

AND A WIFE AND KID BACK HOME......

I agree. But since I am young I have a lot of people who are still trying to upgrade to a "real job" after having graduated in the last 1-2 years and these articles are disproportionately shared by those people.

Usually the articles are about how they sent 200 applications and it was all worth it to get that one acceptance, or are re-share spam about some manager taking a chance on a new hire or some shit. I haven't done any work into curating my feed since I rarely scroll through my feed anyway, but I think LinkedIn could do much better at putting industry-related news articles or blogs in there with less perseverance-porn

https://twitter.com/bestoflinkedin?lang=en

A good running collection of the 'best' of LI. Not a great running collection, but a good enough one.

The only thing I really use LinkedIn for at this point is searching for 2nd degree connects / alumni at companies I'd like to work at, and looking up the professional backgrounds of people that I will be meeting with. However, being able to do this is super critical to networking and I almost always end up paying for premium whenever I'm job searching.

The median HN reader is missing the point of LI because it's not for developers. So you think "man, this sucks". You're like a carpenter looking at a welding machine saying "who needs this huge thing?".

We developers have so many social networks to be found at - SO, github, angel.co, random reddit or discord groups.

But others don't, and linkedin provides amazing value for them as a place to advertise their skillset.

I used SO for candidate search. It worked well for SF, but I recently moved to LA to start a team down here and it was just ridiculously... bad. LinkedIn was better, and that's saying something. But I think it's the scale effects of the social network at play here. SO just doesn't have the scale LinkedIn does. Neither does angel.co, though I can't say I've tried to do recruiting on github or reddit (is that something people do?).

What I am saying is that after having used this thing for so long, I want to improve upon it. If it were open, like HN's Who's Hiring threads, I could just run a BigQuery job on it and do a sophisticated search for the candidates I want. I can maybe even do that and open it up for anyone to use, or sell it, whatever! It would provide value for people.

But the thing that sucks about LinkedIn, aside from the spam by their growth-at-all-costs team, aside from the 5 second page load time, forget the terrible caliber of thought leadership... the BIG THING that makes it really hard to use is their data, or what they expose of it. And I'm wondering if it's just because figuring out the true years of experience (or something else) of someone is too hard for all jobs, but I know some tricks I can use to gauge that for an engineer.

Because if I know everyone I'd want to hire is on the network, I would pay a lot of money to get full access on querying that data. But what LinkedIn has built for it is just sad. And the biggest barrier to building an amazing recruiter search tool is the fact that the entire network belongs to this giant corporation that isn't even doing a good job generating value from it!

So let's open it up!

Linkedin has years of experience filtering.

Since you like simple responses I will try my hand at it. :)

The problem is, "filtering" for jobs across multiple industries is harder than doing it for one industry.

When I last used the recruiter tools, I didn't see a way to filter for years of experience in a specific job type (e.g. software engineering) or a set of roles.

There's no way to filter for people who have 10+ years' experience in software engineering roles. Or someone who has 5+ years' experience in CTO or VP Eng roles.

I have a theory that LinkedIn the product is only a platform for LinkedIn the company to tell us how great it is to work for them.

Add to that the "thought leaders", the quotes and other FB-ish artefacts, without forgetting a constant stream of brown-nosing and if you have 20% of the content that is interesting, you can consider yourself lucky.

In contrast, I hang in an industry specific slack with folks all over the world and it's so much better. I don't have a solution to offer with regards to an open platform, but I think curated communities are the way to go.

I think GitHub has a shot at becoming this if they added the right social features, I’m not sure if they should though.

Since both linkedin and github are owned by microsoft it seems kind of pointless for them to compete.

Well, there's that org chart...

https://www.extremetech.com/wp-content/uploads/2013/07/micro...

And maybe that next-gen professionals' social network can be the first open and free one?

Volunteers have to have somewhat cushy lives to be able to give their time and expertise for free. Stallman was willing to live by his ideals. He was also intermittently homeless while doing so.

These ideas of "We can rebuild it. We can make it better. We can also do it for free!" tend to be unrealistic fantasies.

Stallman also had the advantage of living within the university system for a long time while he built his reputation, so once he was on his own, he could use his reputation to solicit services. The average person has to do more to keep a roof over their head.

Yeah, he was often sleeping in some hacker space when he had no place of his own. It was like "homeless light," from what I gather.

Right? I think it could be done. I’ve been talking to friends about this for a couple weeks now.

I’m actually kind of interested in the people that have done “3000 applications etc etc.” The 100-300 applications are fairy common, but 3k?

Also, LinkedIn? more like LockedIn. There was a time where I enjoyed the service; I liked connecting with people doing interesting things from cool backgrounds.

But, I wonder if it hurts more than it helps... I feel like a lot of jobs are filled before they’re flown, every other job is meant for people who already have a job.

I like reading the job postings though; a lot of them are like non sequitur slam poetry.

This might be my naïveté (or the ipa), but I feel like it’d be cool to have a reversed scenario where I can get info on the team, the project, etc. before even reaching out (i.e. companies apply to individuals, but I think I may have reinvented recruiting? I know this may be unrealistic...)

Idk sometimes it feels like LinkedIn offers a non-service, talent will always be employed and/or sought; if everyone else is just a company’s 1000th pick aren’t they just wasting their time?

Author here. I scraped 8 years of data on Who is Hiring? threads. The data is available through the article and on GitHub. Spoiler alert: people want to hire full time software engineers in San Francisco. Happy to answer any questions.

If not intended as a learning exercise, why not use the existent API?

The author is still an undergraduate, so yeah, I'd file under "learning exercise".

I hope to engage in learning exercises long after I graduate!

Hearing this was pretty refreshing and encouraging. Thanks for posting that, and I mean it.

The article that I was hired to write was a web scraping tutorial, I did all of this to make it interesting.

I've always found it really "interesting" how much resistance the industry shows to hiring part time and/or remote, the two changes in hiring which would really ease their supposed labor issue. I refuse to believe the dogma about labor shortages until I see companies actually going out and hiring all the available qualified employees. All the nitpicking over what constitutes "qualified" which isn't about being able to do the work is just an excuse to exert butts-in-seats dominion over workers.

The number of electrical engineers employed in the US has been going down for years. One wonders what will happen when an entire generation is raised to think that “technology”=web pages and games. Who will build new computers?

The data on "Freelancer? Seeking freelancer?" is interesting and I think it can be misleading.

The disparity between freelancers seeking work and people seeking freelancers on HN is huge - way more people looking for work, than people looking to hire. That makes it feel like there is an oversupply of freelancers, which would drive down the pay rates.

However, I think it's more of a reflection of the HN audience than the broader market. I.E. HN readers don't hire as many contract/freelance employees. This aligns with what we've seen in practice as well. If software is the company's core competency, they will be less likely to hire contract or outsource.

On the other hand, there is a huge need for non-software companies to hire senior devs and they hire a lot of contract/freelance. Demand is high and rates are good ($120-200/hr)...at the right companies. So, if you are a contractor looking for work, I would target companies where software isn't their primary business.

Source: I run a company that helps senior software engineers find contract work. (Shameless plug: www.facetdev.com)

> I run a company that helps FAANG-rite-of-passage-waving software engineers find contract work.

FTFY.

Please don't do this here.

I get the comment is a bit snarky, but if you dig into this guys site it's required that you have worked 3+ years at a "top tier company" with a rotating ticker of FAANG and other giant SV behemoths on it's front page ad, which IMO is fairly pretentious and warrants a bit of a call out.

Neat. I'd be interested in seeing how many job posts are reposted each month, and which companies post the most. I have noticed many jobs/companies that are present almost every month.

The rainbow coloring of the graphs is distracting.

I had “plans” a while back to put together a cynically-named “Who’s Not Hiring” script that did just that. Look at which companies post the same or extremely similar jobs month after month. Life got in the way and I abandoned the project early, but it would be interesting to see the results of such an effort.

I'm on a team at my current job with ~10 other engineers, and at my previous job I was on a team of ~12. With growth, retirements, and turnover at both jobs we are always hiring. My rough approximation from both jobs (spanning about 10 years) is that we're able to hire one person every 6 months. I'm not a hot tech geographic region, and the field I work in is somewhat niche, but there are many non-cynical reasons a job might be posted indefinitely.

Indefinitely posting while hiring only once every six months is actually the poster child of the problem. If it really takes six months to find someone to fit the role, you might want to take a look at your hiring process because as you must be rejecting (or ignoring the applications) a lot of great, qualified people.

IMHO if someone did take the time to publish the metrics of companies who perpetually post the same positions, it would be a fair counter balance.

I'd argue six months is about average for the jobs I've seen and had over the years. Not every role needs an off-the-shelf front end dev.

A lot of things really do require significant domain and industry specific knowledge. For those roles, it takes a very long time to turn up even one minimally qualified candidate. It's easy to say "find someone bright who can learn on the job", but that's not practical in the cases I've seen.

Many things need either specialized education or experience in a similar role to be effective within a 2-3 year timeframe. When on-the-job training would take multiple years, it's worth spending six months to a year to find someone who can fill the role. At least in the industries I've worked in, the roles that require extensive and rare training/experience outnumber the roles that don't.

If you take six weeks to hire someone (let alone six MONTHS!) then you’re probably missing out on candidates who have other good, quicker options. I remember doing a phone screen with a major telco whose name starts with A, ends with T and has a T in the middle. It seemed to go well, then silence (as often happens with tech interviews). Well I did a few interviews at other companies, picked one, signed everything, moved my family across the country, and started work there. About 9 weeks after that phone screen, they finally got back to me with: “We thought the initial interview went great! How about coming onsite for the next step?” LOL

Our hiring process doesn't take 6 months -- we're only able to find a candidate, interview them, negotiate, and have them accept an offer every 6 months. Our biggest problem is just finding people to apply. In the last ~8 years I count only about 100 total applicants to our jobs (I'm not counting internship job listings where we got a lot of student applicants). We tried posting jobs on HN Who's Hiring for about 6 months and got zero serious candidates, so we stopped.

Out of curiosity, do you put a competitive salary range on your posts? I suspect that if you put a range that is above the median for your locale you might have more bites. Only other reason I can think is if your technology stack is outdated or you are located in an area where no one wants to live.

What's your definition of an "off the shelf" front end dev? And if you want to pigeon hole just front end developers (I meant industry wide), what is it you think can possibly be so special about the UIs of over half of everything that someone with many years of experience (possibly from a variety of industries) can't come in and quickly be effective? Isn't there value of someone who can handle new challenges over someone siloed into a single area?

I would say your comment come across as elitist and the evidence I have is also my personal experience. I've done it many times and rarely has anyone not been happy with the results thus far. Obviously there are industry specific areas of expertise that can be deep, and I can see where there are times you might have to hold out for the right candidate. But I think the figures you give reflect stem much more from personal bias than reality. Sadly, a lot of HR agrees with you which is why it's it's become needlessly hard to hire/find a job these days.

My apologies, I should have left "front end" out entirely. Front end has nothing to do with it, I was just using it as an example of a role that is frequently somewhat standardized, interchangeable, and often does not require knowledge of an additional domain.

My point was that many other roles do require expertise in multiple domains as well as industry specific experience. These jobs are relatively hard to fill. They're also quite common outside of consumer tech companies, in my experience.

I apologize if this seems elitist, but some things simply cannot be picked up in a few weeks on the job. Those jobs tend to have a lower supply of qualified candidates and take longer to fill.

People who are saying that the author should have used an API are missing the point; sometimes the only way to gather data is by resorting to scraping, and this article details how one could go about doing that, by using an interesting target: HN.

To the author: Good job, I'm sure many people will find your article useful! I wish I would've had this article at hand a couple of years ago when I needed to aggregate data from several websites that provided no API.

> sometimes the only way to gather data is by resorting to scraping

Nowadays, web scraping has a nonzero risk of hitting legal issues for sites when "the only way to gather data is by resulting to scraping", especially when an API already exists with a ToS/Guidelines on how the data should be obtained and used. And even moreso if the data itself has monetary value.

Thank you!!

HTML scraping should be the last option you consider to get data after all else fails.

Even though the Hacker News API (https://github.com/HackerNews/API) is somewhat old, it's a much more kosher way of getting data.

Even better is to use the public data dump in BigQuery (https://console.cloud.google.com/marketplace/details/y-combi...). Quick query to get all top-level comments in posts by whoishiring:

    #standardSQL
    WITH whoishiring_posts AS (
      SELECT id from `bigquery-public-data.hacker_news.full`
      WHERE `by`="whoishiring" AND type="story"
    )

    SELECT text
    FROM `bigquery-public-data.hacker_news.full`
    WHERE type="comment"
    AND parent IN (SELECT id from whoishiring_posts)

The API that https://hn.algolia.com provides can also be used! In my experience, it's more verbose and the queries are more customisable.

This is great! I've been wondering for a while how to efficiently load a comment tree without spamming the Hacker News API with hundreds of requests, and this seems like it would work nicely.

Example Node.js library that I wrote a few years ago: https://github.com/fraction/node-hacker-news-api

Agreed, unfortunately.

But the official Hacker News API is really painful to use. Specifically because each comment is its own API call (?! ).

I know the HN staff mentioned they're working on a V2 API, but until then the HN HTML tree is so clean, I think this is an exceptional case where scraping the HTML yields a cleaner result that's also less load on the HN servers.

EDIT: Like the other comment on this thread mentioned, using https://hn.algolia.com/api seems ideal here.

I second this. And it's true that the official HN API is painful to use. Algolia's is a lot friendlier.

I wrote a thin Python library on top: https://github.com/santiagobasulto/python-hacker-news

Here's an example:

    results = search_by_date(
        'Who is hiring?', ask_hn=True,
        author='whoishiring',
        hits_per_page=1000)

Full working example: https://gist.github.com/santiagobasulto/b3a3bb94a26c1798b965...

I am aware of the ease of use aspect when it comes to using BigQuery, however, I do believe that it is paid to use service.

You get 1TB/month free. The query above only takes 6GB.

While the article seem intended as a web scraping tutorial, it's good to remember that there is an official HN api [1] in case you really want to do data science on the site.

[1] https://github.com/HackerNews/API

Why didn't you just use the Algolia API? I used the Algolia API to build a 24 / 7 live stream of randomly selected highly voted HN vídeos https://www.crowdform.co.uk/hntv saved me a lot of time

The last section about when not to scrap warms my heart.

Same!! Like with most tools, web scraping can be used for nefarious reasons very easily. I'm glad he provided that caveat. Signs of someone truly trying to create good work and not just clickbait.

Thanks! It was important to me to include that in an article intended to teach the skill.

If you're still feeling froggy with your data, I'd love a follow up with some NLP term frequency used in the posts.

Did intern frequency drop or did the sheer increase in posts shrink their overall percentage? The concern I have there is the idea of becoming too "top heavy" where no one is interested in helping interns grow and only want to recruit experienced people.

You raise a great question. I also wonder if Hacker News as a platform tends towards more senior people, I know that I usually feel like one of the least experienced people in most comment sections.

Nice work! One comment - the plots show the fraction (not percentage) of postings for interns and remote workers.

Since we're in the middle of the Who is Hiring cycle, what's the best way to post a couple of jobs?

What happened on December 2011 and June 2013? Why is there no data?

A couple of months were missing for each data set, if you page through the "whoishiring" user submissions you'll notice them missing.

love this, beautiful.

Thanks you!

Reply on news.ycombinator.com