It has its hits and misses.
Hits:
1. We suss out a lot of people who've been a senior engineer on their resume for years but aren't up to date on practices we care about, like accessibility, semantic HTML, app architecture etc. That stuff is important, I don't want to hire folks who ship apps, but the apps aren't WCAG 2.0 compliant or poorly tested.
2. It can definitely help rescue a candidate that we're on the fence about.
Misses:
1. Some candidates blow it off or phone it in, and out of fairness to those that did well, we can't advance them. It's a shame too, because often the reason they do that is because they have other offers on the table, or other companies they're further along on, not because they have an attitude.
2. We say it's a four hour assignment, but in truth people who do well take around 6-8. It's not because the work takes 6-8 hours, but people go above and beyond, you can tell from the git commits.
Overall, I like it our coding challenge and won't hire without it, but I'm not saying it's perfect.