Comment by swsieber | Hacker News Reader

Comment by swsieber | Hacker News Reader

So this is only tangentially related in that it's ML related. I'm working on a system where a user uploads a single document that corresponds to several documents we already have I'm the system (and we know which documents those are) and we are supposed to map pages from the single document to pages in the other know documents.

I'm having a hard time finding literature on problems like this. I think the closest thing it's like is the stable matching problem, but I'm not sure. My current algorithm doesn't allow for missing or reordered pages.

Does anyone have magic words I could google to find pertinent research/algorithms? I think if worst comes to worse I'll have to try to model the incoming page stream as something like a bayseian ... graph? It's been so long since I've done something like that (5 years), and that was in school.

It sounds like you're just trying to do similarity search? In which case, I imagine the key terms you want to be looking for are Information Retrieval and Recommendation Algorithms

The most basic thing I'd imagine (assuming text docs) is something like index each document by TF-IDF, and then the new document by TF-IDF, and then rank results by similarity

You're basically doing a google search on documents, but with a really big query (another document)

If I were you I would measure reconstruction errors from transformer predictions using input from the source document and then difference vs the target documents.

The challenge would be to get a good measure of the difference, perhaps us some cosine difference again from the transformer?

If you have specialist vocab you can train the transformers - so maybe you are doing legal matching?

I'd generate document embeddings for each page then put the embedding vectors in a similarity search database like Facebook Research's Faiss.

If the pages are ordered it sounds like a text alignment problem. Lots of literature on that especially in machine translation

Reply on news.ycombinator.com