Hacker News Reader

Comment by jerrygenser | Hacker News Reader

Will this work for SSR applications? e.g. think old school net or jsp apps which make network requests then receive HTML which then needs to be parsed in order to understand the key pieces of information and then additional network requests?

I've found it relatively straight forward to reverse engineer SPA requests however with server side rendered apps, yow would your service handle that?

View on HN

Same question from me. I've got this exact use case I've been struggling with the past few days.

I work at a milk delivery company in the UK (The Modern Milkman). There's this website called findmeamilkman.com and I wanted to scrape all the milk delivery services that serve every UK postcodes to create polygons I can overlay on a map to identify competitors in each region.

I keep getting rate limited by the servers, and there doesn't seem to be any fetch/XHR requests on the network. Instead a SSR request that returns the full HTML.

If you're product could help me solve this by reverse engineering an API that would be amazing

Good question. Finding the request that's responsible for the action you want will be a bit trickier for SSR, but it's still possible for most cases. It auto-generates regex (for now) to parse out needed info out of the html template.

Another thing I've seen is that some of these old school apps are sending certain requests that don't modify the page but set server side context which subsequent requests are dependent on.

For example, set context to a particular group and then subsequent navigation depends is filtered on that group even though the filter is not explicit on the client side but due to state stored in the session remotely.

This can also have implications on concurrency for a given session where you need to either create separate sessions or make sure there is some lock on particular parts of server side state.

Would this type of this eventually be possible? Or at least hooks in able for us to add custom code such as session locks

Very interesting to hear about your experience here! We haven't come across a website that has this design and don't offer support for this just yet. We can certainly implement if more people face a similar situation.

Would be cool to use a proxy to MITM to twiddle the bits (with its own API) if the use case isn't supported by a browser or robotic process automation driving the app's client side UX.

I was talking about web apps. But yeah, for old school desktop apps or windows native proxy MITM works

Reply on news.ycombinator.com