wrapapi

We built a similar proxy at https://wrapapi.com/proxy to allow users to record and replay network requests when turning webpages into an API. I think it's possible to have an end-to-end secure and never-written-to-disk pipeline, provided you trust the provider is doing what they're saying.

The secure pipeline we ended up using involves:

1. Having a HTTPS endpoint for the proxy

2. Forwarding the captures to you immediately via listening WebSocket instances

Note that because the server upon receiving a request can immediately push it to WebSockets instead of relying on polling, no storage is needed

Whoops, our bad here! Certain accounts had the feature flag turned off randomly -- we should have fixed this, and you can use it now

Yes, definitely for HTTPS in general -- we have detailed instructions for each of 6 different platforms/browsers (e.g., Android, Chrome, Firefox, Windows/Mac desktop). However, the certificate pinning is actually really tough to get around for apps in Android and iOS these days, so those your mileage may vary there

Thanks for linking to this, chipperyman573! The parent comment used to be valid, but we added a dollar amount for the "Business" plan based on this and a few other comments; we are still willing to discuss discounts for non-profits, smaller companies, etc.

Thanks! We used this awesome library called React-Joyride (https://github.com/gilbarbara/react-joyride) which made setting up the product tour a breeze. Since our product tour is on a single-page app, it works quite well.

The most helpful part is that you can pass a callback which will trigger before/during/after each step, which can let you ensure that the state of the page matches what you're expecting. In our case, we use it to make sure that you're switched to the right tab, etc. Take a look! I highly recommend it.

@OhSoHumble: this actually came to mind for us too in terms of building better experiences on top of common government services. I'd love to chat and see how I can help, so shoot me an email at peter@wrapapi.com

Thanks webninja! The POSTing part is one of the biggest things we were trying to get right while not making it any harder to use than Kimono. Is there anything that was confusing when trying to learn it for the first time? If so, we're still trying to make it easier =)

This is designed for at least semi-technical people, but it's really not that hard to give it a try for simple sites. Try watching the video, and shoot me an email at peter@wrapapi.com if you run into any issues!

We're quite inspired by Kimono, and aim to be just as easy to use while handling use cases beyond scraping (e.g., fully automated form-filling, POST requests, etc.). One of the big feature requests we've been getting has been RSS feeds though, so we're definitely trying to get to full feature parity!

Let us know what page you're trying at peter@wrapapi.com, and I'll take a look and fix it ASAP.

For sites that load data using AJAX, we recommend you take a look at our Chrome extension (https://wrapapi.com/#/chromePlugin). Our philosophy isn't to run a full headless browser (similar to Phantom), but rather make it really easy to find the AJAX requests that actually load the data you need.

WrapAPI is meant to not only do scraping (reading information), but also to (1) perform actions with side effects and (2) allow for complex chaining

Let's say you have a web-based inventory management system or CRM that requires a login, but you want to take data a customer has sent you in a spreadsheet and automatically batch enter it into the CRM, which doesn't have that functionality. You could then:

1. Create an API endpoint that allows you to log into that system and return a state token

2. Create a second API endpoint that's parametrized the inputs of the form to create a new inventory entry

3. Chain those 2 API endpoints together so that the 2 actions are actually combined into one API call

Our focus is not only on getting data, but automating the many things that you or your company does with websites to save time

You can actually write an API endpoint that'll retrieve a state token (which includes the cookies). For one example, you can use view our Hacker News login endpoint at https://wrapapi.com/#/view/phsource/hackernews/login/latest

That endpoint will then emit a state token, which includes the session cookies. You can feed that state token into your next request and it'll authenticate you

Hi everyone! We just released the second version of WrapAPI

We have a new WrapAPI API Builder looks like a browser, and is as easy to use as one too. You can define your API's inputs with a quick tap on the address bar, and point and click at the data you want to extract.

We also have a Chrome extension is smarter and better-integrated than ever. It records your requests and It'll automatically create parameter inputs for the values that change between requests to the same endpoints. The contents of your captures are immediately ready for you to start defining outputs and data to extract too.

Let me know if you have any questions or feedback!