Show HN: Page.REST – An API to fetch details from a web page as JSON | Hacker News Reader

Show HN: Page.REST – An API to fetch details from a web page as JSON

page.rest

114 pointslaktek32 comments

For absolutely 0-cost you can use http://proc.link which will return oembed info for ANY url.

General example: http://api.proc.link/oembed?url=http%3A%2F%2Fpage.rest

Youtube: http://api.proc.link/oembed?url=https%3A%2F%2Fwww.youtube.co...

Facebook: http://api.proc.link/oembed?url=https%3A%2F%2Fwww.facebook.c...

Very neat. But the question is, how they pay the cost of the server? I mean, do you think a web service without any API keys, subscriptions, etc. is reliable?

It's my own server and I'm the developer of the service ;) I pay $3/mo for hosting and $10/y for domain so it doesnt cost me much.

Regarding reliability: all the code is opensourced and if you want to use the service in production - you can run this on your own server.

Cool and just being curious, why did you run the server yourself? Just to have a playground or a demo for the project?

Idea was to make a profitable service out of this. But I'm bad interpreneur :)

I wanted to make an analog of http://embed.ly/code but lost interest in the middle.

I can code difficult logic and all the highload server stuff but when it comes to front end, billing, images etc.. I'm getting bored too quickly.

If someone wants to help me make a business out of this: mdxytop at gmail.com

What levels of traffic does your server see on a normal day? has it ever gone down from a spike?

Usual traffic is 0-1 req/sec. Even now it is about 2-3 req/sec.

It is running for more than 2 years now, never failed :)

Golang is pretty stable.

Also the server is behind Cloudflare which caches responses and protects from DDoS, this can be another reason for low RPS to the server.

I'm shocked to hear more and more people running an HTTP api from a $5 DO instance. It's enticing enough for me to learn, but I want to know, what makes golang so scalable?

I love the idea of just serving a production HTTP API from a single go file.

It's not DO. It's scaleway C1 arm server, even cheaper.

Golang produces native code with very good runtime with green threads which makes it very easy to do async programming and write efficient network code. But language won't save you from inefficient algorithms and dumb code of course.

> I love the idea of just serving a production HTTP API from a single go file.

You may have mistaken by single Go file in my service but it is actually a lot of code included spread across several libraries. In order to write this service I had to write my own open graph library, oembed library, html parsing library etc..

If I have to know the elements' selectors, why should I prefer this service over using a HTML parser?

It depends on the use case. Say you want to get all images on a page you'd just use `img`. If you are going to find JSON-LD blocks defined in a set of pages, you can use - `script[type='application/ld+json']`.

For MVP, I wanted to give the option of free-form selectors and then offer more targeted queries based on what people use.

Consider offering responses other formats, possibly protobuf. Then you can sell it as a solution for limited IoT devices where a full HTML parser is too much. If your service avoids having to host a custom server, it could be very useful.

Yeah, It kind of defeats the point/value prop of an API.

I would make the 5$ price and token validity much larger, like "4rem" or something, I was looking at the CC input field and thinking "seriously? how much will you charge?"

I bolded it for now. Need to rethink the presentation.

Not unlike YQL - https://developer.yahoo.com/yql/

Uhm this is interesting. Reminds me of: https://wrapapi.com

Looks interesting. I wonder what kind of market this app might serve. For larger apps, I would worry about support. 5 dollars per year tells me that the developer is doing this as a hobby. For small side projects, I can see tinkerers building this themselves.

It's actually a utility I built for my own use (I bootstrap Pragma full-time). I thought there might be an interest among other developers for a similar utility. That's why I thought to put it out (wanted to charge $5 so I can reduce abuse).

If you are interested in using it for a larger app and concerned about support, please drop me an email (see profile). I can set you up an instance with better rate limits and service gurantees.

Good to know! Thanks!

I did something similar a loooong time ago. Granted not as sexy.

https://github.com/keyle/json-anything

I wrote a follow up blog post about what I learnt from shipping Page.REST http://www.laktek.com/what-i-learned-from-building-pagerest/

Added support for OpenGraph extraction https://www.page.rest/#open-graph

Is there a await for JS Frontends to initialise before the scrape via a selector (i.e. Angular/Ember etc)?

It runs page scripts and wait after DOMContentLoaded. But may not be accurate in some cases. Do you have any examples to try?

Someone could really abuse this service, I don't see any mention of API limits.

The idea of charging $5 was a stop gap measure to prevent abuse.

There are limits set for access token too. I actually rushed to ship before putting out a TOS which mentions limits.

Zapier it.

If you are interested the same is possible with https://link.fish/hn-early-access/ which has a Zapier integration. It allows to send an URL and it then returns its data. However, you would have to send me an email to info@link.fish because the integration is still in beta mode and so only available after invitation.

How do you handle sites that have scraper prevention? Such as captcha, IP throttling, etc.

I'd pay $100-$500 per-month for a service that could reliably scrape some particularly difficult sites. With that being said, I'd need the service to be able to handle ~100 req/s in bursts and 2-4 req/s on average.

I can maybe help you with that. If interested you can contact me via info@link.fish .

Check to see if those sites have mobile apps. If they do, monitor outbound requests from the app and you'll find an API you can probably use.

Reply on news.ycombinator.com