Hacker News Reader

Comment by ankrgyl | Hacker News Reader

Can you tell me a bit more about your use case? A few things that come to mind:

- There are some ML/transformer-based methods for extracting a known schema (e.g. NER) or an unknown schema (e.g. relation extraction). - We're going to add a feature to DocQuery called "templates" soon for some popular document types (e.g. invoices) + a document classifier which will automatically apply the template based on the doc type. - Our commercial product (http://impira.com/) supports all of this + is a hosted solution (many of our customers use us to automate accounts payable, process insurance documents, etc.)

View on HN

Your commercial product looks very cool, but my use case is in creating an offline-first local document storage system (data never reaches a cloud). I'd like to be enable users to search through all documents for relevant pieces of information.

The templates sound very cool - are they essentially just using a preset list of (natural language) queries tied to a particular document class? It seems like you're using a version of donut for your document classification?

> but my use case is in creating an offline-first local document storage system (data never reaches a cloud).

Makes sense -- this is why we OSS'd DocQuery :)

> The templates sound very cool - are they essentially just using a preset list of (natural language) queries tied to a particular document class? It seems like you're using a version of donut for your document classification?

Yes that's the plan. We've done extensive testing with other approaches (e.g. NER) and realized that the benefits of using use-case specific queries (customizability, accuracy, flexibility for many use cases) outweigh the tradeoffs (NER only needs one execution for all fields).

Currently, we support pre-trained Donut models for both querying and classification. You can play with it by adding the --classify flag to `docquery scan`. We're releasing some new stuff soon that should be faster and more accurate.

Sweet! I'll keep an eye on the repo. Thank you for open sourcing DocQuery. I agree with your reasoning: my current attempts to find an NER model that covers all my use cases have come up short

Since you mention insurance documents, could you speak to how well this would extract data from a policy document like https://ahca.myflorida.com/medicaid/Prescribed_Drug/drug_cri... ?

The unstoppable administrative engine that is the American Healthcare system produces hundreds of thousands of continuously updated documents like this with no standardized format/structure.

Manually extracting/normalizing this data into a querable format is an industry all its own.

It's very easy to try! Just plug that URL here: https://huggingface.co/spaces/impira/docquery.

I tried a few questions:

  What is the development date? -> June 20, 2017
  What is the medicine? -> SPINRAZA® (nusinersen)
  How many doses -> 5 doses
  Did the patient meet the review criteria? -> Patient met initial review criteria.
  Is the patient treated with Evrysdi? -> not

Reply on news.ycombinator.com