Easy-to-Use Apple Vision wrapper for text extraction and clustering | Hacker News Reader

Easy-to-Use Apple Vision wrapper for text extraction and clustering

pypi.org

93 pointsbrulenaudet35 comments

The most interesting piece of code here IMO is the recognize() method, which demonstrates how to call Vision.VNImageRequestHandler and Vision.VNRecognizeTextRequest from Python code using pyobjc.

https://github.com/louisbrulenaudet/apple-ocr/blob/70c25b24b...

Note that Vision.VNRecognizeTextRequest is old tech that is less accurate than the newer ImageAnalyzer API from Apple.

These tools use Vision.VNRecognizeTextRequest because of legacy tech reasons - new APIs from Apple tend to be Swift-exclusive, while these tools are only built for ObjC

Choosing Vision.VNRecognizeTextRequest is choosing less accuracy than is currently available on the same platform

Sorry for my ignorance, but how does it work?

I understand that Vision is an Apple framework, but does it mean that if you are on MacOS, you can just call `import Vision` and it just works? No need to install any package/library as a wrapper? Didn't see anything special in setup.cfg or readme about the setup.

iOS too

https://developer.apple.com/documentation/vision/vnrecognize...

PyObjC is shipped by Apple, yes. (It even used to be installed by default.)

> PyObjC

Ah, thanks, I totally overlooked this. This is what brings all the Apple's Objective C API to Python.

1 - install_requires in setup.y adds dependency to pyobjc

2 - which brings the Vision package along, https://pyobjc.readthedocs.io/en/latest/apinotes/Vision.html

Apple Vision SDK: https://developer.apple.com/documentation/vision

There was a nice blog post a year ago from someone that used this framework and a rack of iPhones to OCR images in order to build a searchable meme database: https://findthatmeme.com/blog/2023/01/08/image-stacks-and-ip...

That's really cool. I found the HN discussion for it if anyone else is interested: https://news.ycombinator.com/item?id=34315782

Nice! We maintain one for C# too: https://www.nuget.org/packages/LiveTextSharp

Does using this over other OCR depend on your use case?

I just tried this to extract the text from the back of a nutrition bar label:

https://i.imgur.com/rRtuBCu.jpeg

The output missed almost all of the actual ingredients under "Ingredients:" (see: https://i.imgur.com/latOiqk.png) whereas Amazon's Textract had no issues at all (see: https://i.imgur.com/hAVzbUR.png)

(For Amazon Textract, I used the "Layout" feature and the output is a result of combining the lines under each "LAYOUT_TEXT" block.)

I also have no idea what I'm doing, so maybe the Apple one is better/can do the same and I just don't know how.

For one thing, apples OCR runs entirely on device. When scanning documents, if any are sensitive I don’t want them going to a cloud service.

To be fair, that's a pretty pathological image to try to OCR. :)

It's wrinkled and not flat so lines aren't straight, it's got big dark shadows going across text, and very low resolution for the letterforms. Even I have an awfully hard time reading parts of it.

OCR is generally designed for flat documents in relatively even lighting with a decent resolution. Different OCR software may do better or worse the further you get away from those ideal conditions, but using an image like yours to compare OCR software is like trying to compare a Porsche and a Ferrari by how well they drive through a muddy jungle.

Hahaha. Yeah, I figured I'd try it with a "harder" image for a personal project I'm working on — something you'd expect someone to take a photo of with their phone that isn't perfect. So I've been playing around with all sorts of OCR stuff. I was really surprised at how well Textract was able to read the text from that dark line with the fold in the image.

My iPhone picked up:

Nutrition Facts Serving Size 1 Bar (68g) Calories 240 Calories from Fat 45 Percent Daily Values (UV) are based on a 2.000 calorie diet Amount/Serving Total Fat 59 Sat. Fat 1.5g % DV 8% 8% Amount/Serving Amount/Serving % DV* % DV Sodium 150mg Potassium 230mg 7% Insoluble Fiber 3g Sugars 23g Trans Fat 0g Total Carb. 44g 15% Other Carb. 17g Cholest. 0mg 0% Dietary Fiber 4g 16% Protein 10g 20% Vit. A 30% • Vit. C 90% • Calcium 25% • Iron 10% • Vit. D 10% • Vit. E 100% • Vit. K 25% • Thiamin (81) 10% Riboflavin (82) 15% • Niacin (B3) 15% • Vit. B6 20% • Folate 20% • Vit. B12 15% • Biotin 10% • Pantothenic Acid 20% Phosphorus 10% * lodine 10% * Magnesium 15% * Zinc 10% * Selenium 10% * Copper 8% • Manganese 10% • Chromium 6% Oxide, DL-Alpha Tocophery Acetate (Vit. E), ferric Ortophosphate (Iron), Beto Carotene (Vit. A), Zinc Citate, Biotin, Niacinamide (Vit. B3), D-Calcium Pantothenate (Vit: 85), Phytonodione Vit. Kl, Manganese Gluconate, Copper Gluconate, Riboflavin (Vit. B2), Folic Acid (Vit. B9), Sodium Selenite, Thiomin (Vi. B1), Pyridoxion Hvdrochloride (Vit. 86), Ergocolterol (Vit. U24, «vanocoboiomin (Vit. 812), Chromium Chioride, Potassium lodide, ALLERGEN STATEMENT: CONTAINS SOY. MAY CONTAIN CERTIFIED TRACES OF MILK PEANUTS, WHEAÏ, AND TREE NUIS. WE SOURCE INGREDIENTS THAT ARE NOT GENETICALLY ENGINEERED. *Rainforest Allionce Certified" COCOR เมยESSUSL 1800C1948179-55T • 2013 CUF BAR & COMPANY • TRADENARKS AND REGISTERED TRADEMARES ARE ONNED BY CUF BARE COMPANY OR USED WITH REUMSSION CERTIFIED ORGANIC DY DA! HIGH IN PROTEIN - 0g TRANS FAT - 70% ORGANIC INGREDIENTS ADVENTURE SAFELY: ALL RISK AND LIABILITY IS YOURS. BEST BY 222 CLIF BAR FAMILY FOUNDATION.ORG clifbar.com 30J AN16G1 09: 09 H

I mean, that's about how well I did when I looked at that image, too...

GPT gives me:

Serving Size: 1 Bar (68g) • Calories: 240 • Total Fat: 5g • Saturated Fat: 1.5g • Trans Fat: 0g • Cholesterol: 0mg • Sodium: 150mg • Total Carbohydrate: 44g • Dietary Fiber: 4g • Sugars: 23g • Other Carbohydrates: 17g • Protein: 10g • Vitamin A: 30% • Vitamin C: 90% • Calcium: 25% • Iron: 10% • Vitamin D: 10% • Vitamin E: 100% • Thiamin: 10% • Riboflavin: 15% • Niacin: 15% • Vitamin B6: 20% • Folate: 20% • Vitamin B12: 15% • Biotin: 10% • Pantothenic Acid: 20% • Phosphorus: 10% • Iodine: 10% • Magnesium: 5% • Zinc: 10% • Selenium: 10% • Copper: 8% • Manganese: 10% • Chromium: 6% • 70% Organic Ingredients

The ingredients listed on the label are as follows:

Organic Brown Rice Syrup, ClifPro® (Soy Rice Crisps [Soy Protein Isolate, Rice Flour, Barley Malt Extract], Organic Roasted Soybeans, Organic Soy Flour), Organic Rolled Oats, Chocolate Chips (Dried Cane Syrup, Unsweetened Chocolate, Cocoa Butter, Lecithin, Vanilla Extract), Organic Cane Syrup, ClifCrunch® (Organic Oat Fiber, Apple Fiber, Inulin [Chicory Extract], Organic Milled Flaxseed, Organic Oat Bran), Organic Date Paste, Organic Sunflower Oil, Molasses Powder, Sea Salt, Natural Flavors, Cinnamon. Vitamins & Minerals: Dicalcium Phosphate, Magnesium Oxide, Ascorbic Acid (Vit. C), DL-Alpha Tocopheryl Acetate (Vit. E), Beta Carotene (Vit. A), Niacinamide (Vit. B3), Ergocalciferol (Vit. D2), Thiamine Mononitrate (Vit. B1), Pyridoxine Hydrochloride (Vit. B6), Riboflavin (Vit. B2), Cyanocobalamin (Vit. B12).

ALLERGEN STATEMENT: Contains Soy. May Contain Traces of Milk, Peanuts, Wheat, and Other Tree Nuts.

Does anyone know how Apple Vision performs on Japanese vertical, right-to-left text extraction? This is my biggest need around some machine translation stuff, but I’ve found Google OCR is the only thing that can do this reliably well, but also that it’s too expensive to run for hobby projects.

Surprisingly, it works! Vertical text is recognized as if it were normal text rotated 90° clockwise.

How well does GPT Vision read it?

I always used GPT-V for reading thorny documents (e.g. medieval French handwritten letters) and it would do a spectacular job, but they hobbled it at some point to tone down its enthusiasm for translation, so a lot of the time you have to tell it that it's important to your dead grandma's career or other tricks to get the output.

I haven’t tried via the API because it’s too expensive for what I’m doing (hundreds of pages of high-res scans of old Japanese books), but I should test out a few pages in ChatGPT.

Yeah, please test some in GPT Pro and let me know what answer you get. I have to do some mixed content later this year (Japanese magazines that fluidly meld English, LTR Japanese, and pop-outs in RTL vertical Jap). I've not tried them in Adobe Acrobat yet though, which I usually use for text recognition.

"Apple Vision" sounds like it has something to do with the Apple Vision Pro headset, but it doesn't. This is related to Apple's Vision framework, which provides computer vision algorithms.

Is it going to be the Microsoft Surface all over again?

Probably not because this product will be culturally relevant still 5 years after launch

optimistic take

Apple has a very strong track record of building high-end hardware that is expensive at first targeting pro user early adopters, then iterating and reducing cost over 3-5 years, eventually becoming the dominant player in a hardware category.

iPhone for mobile phones, iPad for tablets, Apple Watch for wearables, AirPods for fully wireless headphones. They have few failed product lines so this seems like a pretty safe bet.

I’ve been using textra for a wrapper on the Apple vision sdk

https://github.com/freedmand/textra

But this project calls torch and a bunch of other ML libs. So it’s not using Apple vision?

It's only using those other libs to do some simplistic statistical analysis of Apple vision's outputs. Further down you can see it just calls VNImageRequestHandler which is basically a call from Python to Apple Vision via ObjC in order to get any actual recognitions.

It uses python so I’m a bit concerned about speed. Are there any benchmarks for how fast this is if I want to use it for a real time application?

This is a python wrapper around apple’s vision framework. If you’re concerned about speed and/or don’t want python, then interacting directly with the vision framework is an option.

Don’t use this outdated API. There’s a new and far better one called ImageAnalyzer from Apple. This tool doesn’t use it because of legacy tech debt decisions they made

Anyone else's GPT spidey senses going off at how the documentation is written?

Apple-ocr is a utility for Optical Character Recognition (OCR) that facilitates the extraction of text from images. This Python-based tool is designed to help developers, researchers, and enthusiasts in the field of text extraction and clustering. It leverages a combination of various technologies to achieve this, including the Vision framework provided by Apple.

Reply on news.ycombinator.com