Hacker News Reader

Comment by jedberg | Hacker News Reader

I'm not sure if this is more of a pro in the ChatGPT column or an indictment of the USMLE. Although I don't think the USMLE is known as an easy exam, so I guess ChatGPT really is that good now?

View on HN

> In 2020, 98% of DO and MD students passed the exam on their first try. But if you don’t, you are given four attempts to pass each different level of the USMLE exam. If you fail, you receive a score report that offers feedback on the areas in which you struggled.

from https://www.ama-assn.org/medical-students/usmle-step-1-2/usm...

It would seem USMLE is not hard to pass.

Isn't this yet another case of "you have to memorize and regurgitate a ton of information" which people are bad at but something like ChatGPT is exceptionally good at?

I think this is one point that I don't see being made often enough. ChatGPT is exceptional at retrieval of information, but if it starts passing tests like the MCAT and LSAT and so on, I feel this is more that our testing is just crap, rather than ChatGPT being "smarter" than humans.

Being a doctor does require a lot of just plain information retrieval. Human bodies have a kajillion parts interacting in a kajillion-squared ways. And we usually don't get to look directly at most of the parts; we have to infer what's going on from very indirect evidence.

This isn't one of those "memorize nothing you can look up" situations. You just have to have it at your disposal before you can do actual medicine.

There may be a day when a computer can do most of the memorization scutwork and you can get back to the real work, but we're not there yet. It's not the test's fault that it's testing medicine as it's practiced today.

Reply on news.ycombinator.com