Hacker News Reader

Comment by baby_souffle | Hacker News Reader

> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you.

The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money.

I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to.

One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.

View on HN

If this is the case, they probably outright do not want you as a customer.

Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there.

The actual on-sale item is potentially losing money for the business or at break-even.

------

A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.

> If this is the case, they probably outright do not want you as a customer.

Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit. I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount.

When I built this a few years ago, I had to write a lot of code to make all this work.

Now that's a simple "hey $agent, check the price of https://some-site.com/appliances/that-new-fridge.html and https://other-site.com/shop/appliances/that-new-fridge.aspx daily and notify me when it's on sale or has a promotion"

I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!).

Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale.

Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.

The tradeoff here is a classic false-positive vs false-negative issue. If the cost of the bad-bot false positive (which blocks your bot out) is lower than the cost of the bad-bot false negatives (which allow bad bots in), then it is still a good tradeoff, if a suboptimal situation.

The fundamental issue here is the way we use the internet is changing… and what we are seeing is some friction as that works its way through.

I have a business where I clearly don’t want any bots. A retailer, maybe they do.

How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices?

Not a rhetorical question, because most companies would welcome that interest from customers.

> How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices?

That's the thing; they can't, really.

Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of micro transactions system on the .md endpoint.

In basically every cat/mouse game defense is forever one step behind...

Rate limiting is ineffective against rotating residential proxies, though

Reply on news.ycombinator.com