Hacker News Reader

Comment by omarfarooq | Hacker News Reader

Excellent, thank you for creating this. Just learning that this data is available. Only California? Are the other states not making the data available in the same fashion as CA?

Also, was there a personal trigger for you to create this?

View on HN

Thank you. This data should be available in all states, but each state has a proprietary site I have to write a new scraper for and write a new data pipeline for. I'm thinking about open-sourcing the project and gathering community support for the rest now that I have a good pattern set.

Yes, I was harmed by a doctor and the medical board dismissed my complaint. This made me curious about what the board actually does take action for, but this wasn't searchable because all that was available were PDF scans that weren't digitized/OCR'd. I also noticed some documents that were referenced in doctors' profiles had disappeared. As part of this, I archived every PDF available on the wayback machine to protect against that going forward.

I also want to give credit to a similar site that I took inspiration from called 4patientsafety.org, which did some of the same stuff I have here, but completely manually. I chatted with the creator of that and he agreed making an automated version would help because they fell behind updating it due to how intensive the manual process was.

Is there a link to the community where this is being developed? I was unable to find one on the site itself. I'd be interested in writing a scraper for my state and the digitization pipeline.

I'd be interested in helping out as well, looking forward to it

Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

Thank you for offering to help. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

You may want to look at courtlistener.org as inspiration--similar set of problems, but in a different space obviously.

Thanks but the site seems to be down. Do you have another link?

It's www.courtlistener.com

Thanks! I'm looking at the Contribute page and seeing some cool ideas.

> I archived every PDF available on the wayback machine

How did you tell archive.org to store the PDFs? It doesn't seem to have worked in all cases, since when I went to check out the single "order vacating automatic cancellation of certificate", there was no snapshot yet.

https://web.archive.org/web/20231009140931/https://www2.mbc....

I have to admit I missed a few hundred out of about 18k PDFs. I didn't keep track of which ones I archived on my initial pass, so I need to rerun the whole script which takes about a week with Web Archive's rate limit, so I've just been putting it off. I was also thinking people who visit them can click the "archive" button in cases it's not yet archived to crowdsource the effort, like you seem to have done.

I’m interested in helping out.

I have done a lot of research in medicine due to my poor experiences with even top specialists. Ignoring symptoms, denying side effects of meds I took, even though those were present on the label.

Do you have a discord?

Any plans on allowing people to comment on physicians?

Any plans of adding how much doctors have received from corporate interests?(as this data is available)

Sorry to hear you've also had poor experiences. I too had to delve into Pubmed to figure out what was going on with me after the doctor harmed me.

Thank you for offering to help out. I do want to refine the site and data a little more over the next few days and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo.

I've now made a discord here https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

Comments on doctors should already be enabled via Disqus if you scroll to the bottom of a profile page.

Yes, I definitely want to add that data!

I'm also interested in writing a scraper for my site (Michigan) if time allows.

Thank you. I'm going to refine it a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

Please open source this. I would love to help.

Open sourcing this would be a great idea and a lot of people would happily contribute, it already looks great. The biggest secret is that most doctors do more harm than good when a patient has anything worse than a cold. The psychopaths who couldn't make it as CEOs or serial killers end up becoming doctors, in charge of people's health.

Thank you. I'm going to refine the site a bit more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

In Washington, this information can be gleaned on a small scale from the Department of Health's Provider Credential Search - https://doh.wa.gov/licenses-permits-and-certificates/provide...

And the higher volume Open Data Portal dataset: https://data.wa.gov/Health/Health-Care-Provider-Credential-D...

Reply on news.ycombinator.com