Show HN: Screenshot Hero - iPhone app to make all photos text searchable

In my limited comparison. Apple’s API performed way better than Tessaract. You can check your images using this app or their sample xcode project for vision, I think same API works on macOS too.

I would also vouch for google’s ocr API but I opted for Apple’s for privacy reasons and as they were easier to implement.

I think you have to fiddle with the internal settings of the OCR package you are using to get good results. For tesseract and pytesseract, there is a whole article on improving the quality: https://github.com/tesseract-ocr/tesseract/wiki/ImproveQuali...

My guess is Apple trained on their own massive dataset and better architectures that make their systems better than off the shelf ocr.

I am working in this area and one way I see good improvement is training an object detector to first detect words in an image, then you can pass that through tesseract/OCR software. Besides that, finetuning tesseract on data you want strong performance would be the next best alternative.

Here is a cool article from dropbox how they engineering their OCR system: https://blogs.dropbox.com/tech/2017/04/creating-a-modern-ocr...

This link below is a really cool project that showcases what building your own OCR engine would look like!

https://github.com/awslabs/handwritten-text-recognition-for-...

If it makes you feel any better, I had a similarly terrible experience with Tesseract. I spent days studying the docs and I failed to find a setup where it would produce anything useful even with the most perfect input.

I was also left with a feeling that I must have been missing some trick, because people on the web seem to be using it with good results.

You might like Screenotate by Omar Rizwan - https://screenotate.com/

It might be worth trying this for desktop: https://github.com/the-paperless-project/paperless

It handles OCR, indexing and search

I use Tesseract to extract text from my scanned documents in order to be able to grep through them. What surprised me, was that the lower quality (resolution) scans had actually much better OCR results. So it seems Tesseract works best with some specific font-size-DPI combination.

This is the script I wrote for that purpose (maybe it helps): https://nopaste.xyz/?ea1e6242a659ce5c#vAMyPpajo0OIeCGQe9Wkdq...

The results aren't perfect, but good enough for my purpose.

I had a similar experience working with OCR for a similar project as OP ( https://www.askgoose.com ). What we realized was tesseract's training data was significantly different to what we were trying to transcribe (screenshots of conversations). The sparseness of text leads to bad transcriptions. One workaround is to do some image processing before hand so you can break the image down into data that looks similar to the train data used for tesseract.

I've had success with first using imagemagick to cleanup the image, do some thresholding and then using the latest version (4.1) of tesseract. With tesseract you can also try to experiment with different --psm values for better results.

You get what you pay for with Tesseract, don't judge OCR performance based on it. Would highly recommend checking out FineReader if you're on a supported platform. You can feed it crumpled beer receipts and the results are still decent

Tesseract needs text on a white background to work well out of the box. For "text on images" try for example https://ocr.space/ocrapi

I don't know if this is how it works or not, but it is worth noting that searching for text in images is a much easier problem than transcribing images, and if you do the latter as "step 1" you destroy your ability to do the former: if you search for some piece of text you want to find all images that maybe if you squint a bit could be that piece of text, which allows you to deal with cases where your recognition is forced to make a bad guess that does the moral equivalent of rebracketing symbols that is difficult to work around in a later text-only search pass. If you can at all avoid it, always avoid transforming your input information until you know your query, lest you destroy information (this is what always made Microsoft Journal so epic for me, though I should be clear that I haven't used it in 15 years so for all I know they forgot this).

Is there a way to port Apple's Vision API to javascript for a webapp, or use their models without an Apple device like Apple Maps on Duckduckgo?

The data collected is, as mentioned, used to analyze crashes. In case of an error, some information is collected, that can be explained as being justified: device unique ID, to know if an error happened on one device multiple times, IP address is collected in web server access logs, configuration of the app could mean a lot of things, i.e. resolution of the viewport, etc.

If the Privacy Policy contains no mention of sending images to the server and storing the images and/or text contents, then ... it's not happening.

edit: This comment was posted before you edited yours.

> But as with all apps I’m particularly sensitive to giving bulk access to my photos (and other data).

I don't understand why iOS doesn't allow blocking internet access for apps. You can only disallow mobile data access, but not WiFi access.

The frustrating part is that in China they have the functionality [0], so it's there, but we are not allowed to use it for our privacy.

[0] https://old.reddit.com/r/apple/comments/69k1j8/the_chinese_i...

Try Google Photos.

I'm in the same boat. I wish Apple gave us a way to block internet access for specific apps. Currently, there is way to block cellular data for a given app but wifi is always unrestricted.

This looks very promising, and the price is reasonable.

But the web site is one page, without even an "About" section. All it has is an animation, some text, and an e-mail address. No information about the company at all. Even the whois for the domain is hidden.

For the sort of person for whom "on device" processing is important, information about the company has value.

Thank you! Glad it worked fine for you :)

Honestly, it hasn’t changed much. But Swift made it easier to make a native module (as compared to Obj-C which was defacto method a few years back)

Did you try the iOS Calendar tutorial[1] ? Besides this, I find useful looking into other third party modules, in concrete some module that is simliar of what you are building, if you are having UI or not, if you want to use Cocoa pods or not ...

Also a bit of tanget, but I'll take the oportunity to say that I wish third party react-native modules had some sort of standard documentation like flutter website, specially versioned documentation.

[1] https://facebook.github.io/react-native/docs/native-modules-...

I have learned it in the last few months. Its been really intuitive as a Native dev swithcing to work with RN more. A really simple example I made is this native module for showing the iOS 13 Context Menu when RN views are long pressed, if you are looking for superrrr simple examples https://github.com/mpiannucci/react-native-uimenu

Hi - I work on Expo.

One of the big focuses for us this year is to make that experience much better.

We've already taken some steps. (We now just give you a regular React Native project if you need one), but we plan to keep working on this stuff until you can get all the power you need to build something like Screenshot Hero with the ease of use of just making a website.

Object recognition using machine learning. Google scans all photos in Photos by default, but it can be turned off.

Try searching "credit card, [city], september 12th 2019", or "yellow cat" or "screenshot".

It can't transcribe text though.

Agreed: Cool app and certainly heroic and worth implementing.

As an alternative, Google Photos from the App Store also support OCR for still-images on its web version. I don't have an idevice on me to know if it's also OCR in-app. Give GP a run for its money, Screenshot Hero!

I wouldn’t. Swift is far more performant than using these React-type pseudo-native systems and the developer mentioned that he “fell back” to React because of an apparent lack of SwiftUI documentation — so why not use this project as a chance to actually learn it and create some documentation? He wrote up about how he built the app, adding yet another article to the boring canon of “I just want to use JavaScript and React and call myself an iOS developer.”

And he said that he used React just for the views — SwiftUI isn’t that hard at all and there are exact tutorials about listing things. Ray Wenderlich for example, the WWDC videos — there is high quality information out there but it seems like Swift and SwiftUI wasn’t actually a real consideration.

To be clear, I am not criticizing the app, what I am criticizing is the parent comment hoping more developers follow that approach. I happen to think that approach is lazy and aspires to the lowest common device denominator rather than building apps that are platform specific and optimized for each.

I have 6k photos on my phone and this works just fine. The search speed is mostly the same.

I have the same setup but less photos (around 6k, many on icloud). Does it not detect all 35k photos?

I will look into this.

By default, my app was only scanning screenshots. In the menu the 4th option should switch it to scan all photos, I think.

How would you define sensitive content? The real challenge here is to train a classifier.

With photos on iOS you can delete sensitive content on creation i.e. take photo, delete photo. The trash empties after 30 days, so you can access it there if you need to.

Take my word for it? :) I wish Apple had per-app-internet blocking feature but you can turn off internet and try the app, it will work just fine.

I don't plan to open source this particular project as I already have enough of open source action going for me.

Unfortunately not, I had to write most of it in Swift so cannot cross-compile.

I think this shouldn’t be hard. I will give this a shot.

Understandable, I will look into this.

It does index all the memes just fine. I should add inApp-purchase on the extra dank ones, jk.

Evernote had it even earlier. The founder, Stepan Pachikov, has been working on OCR as far back as the '80s.

Unless you're in a (big) library, isn't it faster just to.. look?

Not yet. But I assume that is going to be a useful feature.

It’s unlikely there’s any way to do that on iOS, but in theory you could do that on a desktop OS. Pretty sure there would be such a massive amount of text throughout the day, you couldn’t just do a simple filter search, you would need to somehow sort for relevance as well.

looks like a screen recording on device, standard iOS feature.

I recorded it in simulator using the following shell command (while simulator was running):

  xcrun simctl io booted recordVideo <filename>.<extension>

That's odd. It should be available on all stores. Try searching directly?

Thank you!

I think it will take some time, just like Swift did. It's has potential!

ikr! I did test XKCD comics when making this. I am primarily using this to index my memes :D

Does this link not work either? https://apps.apple.com/us/app/screenshot-hero/id1493170794?l...