HNHacker News
TopNewBestAskShowJobs

robk

3,187 karma · joined April 3, 2009

ex-Googler in SF, now VC in London at Hoxton Ventures. I love email. pop me a note especially if you're a European founder - robk at robk dotcom
submissionscomments
robk··on Ryanair – when every page is a dark pattern
Yet I can't ever think of a Ryanair flight I've seen go out with a single spare seat. There are plenty enough people who don't mind abuse of the save a few cents. Good luck to them. Keeps them off my flights.
robk··on Shoplifters Love Lego
By volume sure but by dollar organized crime is surely far higher
robk··on Most of Europe is glowing pink under the aurora
Amazing thanks for sharing
robk··on WindCon2024: A Joke Too Far for Many
These jokes are so tiresome. It was cute two decades ago but please let this stupid tradition die. It's become devoid of all humor and makes April 1 a hassle honestly.
robk··on Why are so many people being hit with £5 fines for 'counterfeit' stamps?
Well after the first time you just ignore the post cards wouldn't you? Easy way to handle returns too for sellers who can't cancel an order.
robk··on Why are so many people being hit with £5 fines for 'counterfeit' stamps?
I'd bet money in a forensic scenario they could look up the batch and then estimate the sequence and thus date of sale and pull the post office cctv footage.
robk··on Baltimore's Key Bridge struck by cargo ship, collapses
LLM for Poetic writing?
robk··on Suspicious discontinuities (2020)
Can you expound more? How did this work across different fields of sport? I assume a basketball scout can recognize talent early in their domain but how is it crossing sports?
robk··on Uber to pay $272M to Australian taxi operators
How in the world is that $500?!
robk··on SimplyNUC Introduces Onyx Pro: Featuring 10GbE SFP+ Fiber and Dual 2.5GbE Ports
The minisforum seems to be a better value than this one though.
robk··on Ask HN: Best web scraping toolset for 2024?
Three real paths I'd evaluate

0) Tools: Puppeteer and Playwright are the cleanest way I've found to get proper JS page rendering and behavior control. Node is well suited for this but some prefer python.. Since you sound concerned with blocks I'm guessing plain HTML scrapers like BeautifulSoup or Cheerio would be insufficient but they're more robust in terms of sheer volume and overhead of course.

1) if you have money the fingerprint avoiding systems like GoLogin are exceptionally good at avoiding detection. But they're not cheap so you would need to have a reasonable budget to use them well. I've had extremely high success with GoLogin myself and if budget wasn't a concern I'd just default to that.

2) less expensively you can use headless Chrome (Browserless.io has an excellent docker image for this) and then proxy using 4g/5g proxies. You usually pay by kb so you'd want to be savvy about blocking images etc to manage costs but with a good proxy and decent tuning this also takes you pretty far except some of the more onerous services like Cloudflare and Datadome. There are also v good captcha solving services that seamlessly integrate. I've had very good results with simple proxying this way and well crafted settings in Browserless like making sure stealth plug-ins used and user agent is properly done etc.

3) most inexpensive you can simply use Playwright on Browserless (Chrome Headless) and a captcha service with stealth plug-ins and running from a decent quality IP. I'm careful to check things like viewport size, user agent, ghost cursor etc.

Data center IPs usually raise flags and even the server should be selected to not have vpn ports or http ports open as those are also antibot signals among others.

--

I do #3 at a fairly decent volume (<1m page views per day) across a few dozen machines (each with a half dozen IP addresses and separate VLANs for each container) to scrape from some private sites and so long as I stay under a sane rate limit I've had years of success on many sites that are fairly strict about blocking browsers.

In all cases I parse the key parts of the page and dump it into mongo for async processing of the data and to allow fixes when sites change. You need to keep an eye on your ETL pipeline and alert when something breaks - I expect once a quarter I have to fix a selector change or something trivial as sites change.

This is also a good substack evaluating the various paid options for the toughest sites. https://substack.thewebscraping.club

robk··on Wi-Fi jamming to knock out cameras suspected in nine Minnesota burglaries
What's the character of Alaskan criminals? How different from the lower 48?
robk··on The failure of self-checkout technology
These seem to work fine in high trust societies. Today's America and the UK fail miserably at this owing to lack of customer honesty and organized crime exploiting the loopholes that are designed for individuals not organized groups that exploit them.
robk··on New PFAS discovered with novel testing method
Still makes me uncomfortable even using the shower when I'm down in Shallotte for the same reason!
robk··on Show HN: Beeper Mini – iMessage client for Android
it's very hard to socialize with non-techies in the US without it. you miss so much.
robk··on Why doctors in America earn so much
They're certainly not happy in Britain!
robk··on Cratering motor fuel sales in Norway show the death spiral that can end oil
Racist
robk··on Boox Palma review: A phone-shaped e-reader
It's just been rooted BTW which makes it easier. I absolutely love mine. https://xdaforums.com/t/hisense-a9-root-reward-offered-snapd...
robk··on US citizens with permanent disabilities get free lifetime pass to National Parks
Absolutely yes
robk··on Uber Eats will start accepting food stamps for grocery delivery in 2024
See for yourself on Amazon. You can get big bags of candy on EBT
robk··on Doing laundry on campus without a phone
The minimum sounds like a scam by the home owner to collect the breakage on their end.
robk··on Doing laundry on campus without a phone
I've found a decent number of exec assistants that have this issue booking on behalf of their bosses. Uber Black is preferred by the boss but they wind up using Addison Lee or similar for this desktop booking. So there's money left on the table from fairly price insensitive customers.
robk··on ChangeDetection, monitor any website change
Have you tried keepa?
robk··on North American English Dialects
Then drive twenty or forty miles south and you'll hear even more variants once you hit Charles and St Mary's counties.
robk··on North American English Dialects
Sounds like southern Wisconsin influenced. Wheaton is not far so perhaps he moved there or had family there (or the Wisconsin accent extends into wealthier Chicago suburbs)
robk··on How much better was DEC Alpha than contemporaneous x86? (2020)
I remember doing testing on Alpha for Windows NT 4.0 for my summer internship at DEC Littleton in like 1995 and was really surprised how fast it ran. Certainly faster than any other gui server I’d ever tried at the time.
robk··on America’s largest tool company couldn’t make a wrench in America
It was largely accelerated by the admission of China to the WTO under Clinton.
robk··on Show HN: Chrome Dev Tools Element Selector for Playwright and Scraping
Looks neat. How well does it make "smart" selectors like text contents contains"login" instead of a mess like //a/div[1]/span[3]?
robk··on Google Calendar and Assistant Reminders Will Migrate to Google Tasks Soon
It still drives me insane on Ios that there's no way to share to Google Tasks. For years I've asked for this simple option as my main work flow is to share from the browser to tasks for things to revisit later.
robk··on Ask HN: How would you write a ledger for micro payments without blockchain?
Formance https://www.formance.com
← PreviousPage 2 of 29Next →