That would help, but who defines which sites are required for the task? If it's the LLM you haven't solved prompt injection because the LLM can be persuaded to open other sites that the user didn't intend.
45,529 karma · joined February 16, 2019
> A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.
To me "recursive" is key—it transforms the traffic pattern from one that strongly resembles that of a human to one that touches every page on the site, breaks caching by visiting pages humans wouldn't typically, and produces not just a little bit more but orders of magnitude more traffic.
I was persuaded in another subthread that Nxtscape should respect robots.txt if a user issues a recursive request. I don't think it should if the request is "open these 5 subreddits and summarize the most popular links uploaded since yesterday", because the resulting traffic pattern is nearly identical to what I'd have done by hand (especially if the browser implements proper rate limiting, which I believe it should).
> A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.
There's nothing recursive about "summarize all the cooking recipes linked on this page". That's a single-level iterative loop.
I will grant that I should alter my original statement: if OP wanted to respect robots.txt when it receives a request that should be interpreted as an instruction to recursively fetch pages, then I'd think that's an appropriate use of robots.txt, because that's not materially different than implementing a web crawler by hand in code.
But that represents a tiny subset of the queries that will go through a tool like this and respecting robots.txt for non-recursive requests would lead to silly outcomes like the browser refusing to load reddit.com [0].
I'm not asking if you believe ad blocking is ethical, I got that you don't. I'm asking if it turns my browser into a scraper that should be treated as such, which is an orthogonal question to the ethics of the tool in the first place.
I strongly disagree that user agents of the sort shown in the demo should count as robots. Robots.txt is designed for bots that produce tons of traffic to discourage them from hitting expensive endpoints (or to politely ask them to not scrape at all). I've responded to incidents caused by scraper traffic and this tool will never produce traffic in the same order of magnitude as a problematic scraper.
If we count this as a robot for the purposes of robots.txt we're heading down a path that will end the user agent freedom we've hitherto enjoyed. I cannot endorse that path.
For me the line is simple, and it's the one defined by robotstxt.org [0]: "A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. ... Normal Web browsers are not robots, because they are operated by a human, and don't automatically retrieve referenced documents (other than inline images)."
If the user agent is acting on my instructions and accessing a specific and limited subset of the site that I asked it to, it's not a web scraper and should not be treated as such. The defining feature of a robot is amount of traffic produced, not what my user agent does with the information it pulls.
No, it's common practice to allow Googlebot and deny all other crawlers by default [0].
This is within their rights when it comes to true scrapers, but it's part of why I'm very uncomfortable with the idea of applying robots.txt to what are clearly user agents. It sets a precedent where it's not inconceivable that we have websites curating allowlists of user agents like they already do for scrapers, which would be very bad for the web.
[0] As just one example: https://www.404media.co/google-is-the-only-search-engine-tha...
> A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.
This is absolutely not what you are doing, which means what you have here is not a robot. What you have here is a user agent, so you don't need to pay attention to robots.txt.
If what you are doing here counted as robotic traffic, then so would:
* Speculative loading (algorithm guesses what you're going to load next and grabs it for you in advance for faster load times).
* Reader mode (algorithm transforms the website to strip out tons of content that you don't want and present you only with the minimum set of content you wanted to read).
* Terminal-based browsers (do not render images or JavaScript, thus bypassing advertising and according to some justifications leading them to be considered a robot because they bypass monetization).
The fact is that the web is designed to be navigated by a diverse array of different user agents that behave differently. I'd seriously consider imposing rate limits on how frequently your browser acts so you don't knock over a server—that's just good citizenship—but robots.txt is not designed for you and if we act like it is then a lot of dominoes will fall.
What about reader mode that is auto-configured to turn on immediately on landing on specific domains? Is that a robot for the purposes of robots.txt?
https://addons.mozilla.org/en-US/firefox/addon/automatic-rea...
And also, just to confirm, I'm to understand that if I'm navigating the internet with an ad blocker then you believe that I should respect robots.txt because my user agent is now a robot by virtue of using an ad blocker?
Is that also true if I browse with a terminal-based browser that simply doesn't render JavaScript or images?
Why? My user agent is configured to make things easier for me and allow me to access content that I wouldn't otherwise choose to access. Dark mode allows me to read late at night. Reader mode allows me to read content that would otherwise be unbearably cluttered. I can zoom in on small text to better see it.
Should my reader mode or dark mode or zoom feature have to respect robots.txt because otherwise they'd allow me to access content that I would otherwise have chosen to leave alone?
What coherent definition of robot excludes Chrome but includes this?
Website operators should not get a say in what kinds of user agents I used to access their sites. Terminal? Fine. Regular web browser? Okay. AI powered web browser? Who cares. The strength of the web lies in the fact that I can access it with many different kinds of tools depending on my use case, and we cannot sacrifice that strength on the altar of hatred of AI tools.
Down that road lies disaster, with the Play Integrity API being just the tip of the iceberg.
If the taxes earned from these transactions are then spent on things that also benefit the wealthy just as much as they do the poor, then the rich are double-dipping and poor people still end up net behind the wealthy. They lose access to something that previously was paid for out of property taxes in exchange for more revenue funding services that the wealthy are just as likely to use.
This model at least doesn't further exacerbate the regressiveness of the tax by only funding things used by the rich, but it doesn't restore balance.
That's why I say that the only way that you flip the tax to be progressive is if the proceeds benefit the poor disproportionately rather than benefiting everyone equally.
How does the math on this work? If a regressive tax affects poor people disproportionately over rich people and you then turn around and spend the proceeds on something that benefits everyone equally, you've just done wealth redistribution from poor to rich, no?
In order to counterbalance the effect of a regressive tax you would need to spend the proceeds on something that benefits poor people more than rich people so the disproportionate negative impact is balanced by a disproportionate positive impact.
Arguably paying for the MTA may count as doing that, given that poorer folks are more likely to be using it than rich folks (especially now post-congestion tax). The congestion tax becomes a tax on what is now a luxury (driving a car in Manhattan) that is used to pay for a staple (public transit).
I'm pretty sure this is exactly what op is saying, isn't it? The toll forces poorer people to change their behavior while letting rich people continue to do what they've always done with a barely noticeable dent in their wealth.
One big problem with Claude Code vs Cursor is that you have to pay for the cost of getting over the learning curve. With Cursor I could eat the subscription fee and then goof off for a long time trying to figure out how to prompt it well. With Claude Code a bad prompt can easily cost me $5 a pop, which (irrationally, but measurably) hurts more than the one-time monthly fee for Cursor.
What about the millions of software developers who have never even visited a pirate site, much less built one?
Are we including the Netflix developers working actively on DRM?
How about the software developers working on anti-circumvention code for Kindle?
I'm totally perplexed at how willing you are to lump a profession of more than 20 million people all into one bucket and deny all of them, collectively, the right to say anything about IP. Are doctors not allowed to talk about the society harms of elective plastic surgery because some of them are plastic surgeons? Is anyone with an MBA not allowed to warn people against scummy business practices because many-to-most of them are involved in dreaming those practices up?
This logic makes no sense, and I have to imagine that you see that given that you're avoiding replying to me.
Hacker News doesn't have opinions. Individuals on Hacker News have opinions. Different sets of individuals comment and vote on different threads. There's zero reason to suppose that it's the same people expressing both ideas or even that it's the same people voting on those ideas. To the contrary, I've spent enough time on this forum (way too much time) to know that there are whole sub-communities on HN that overlap very imperfectly. We self-select based on titles, topics, and even on time of day.
The only thing this kind of logic is good for is producing fallacious arguments dismissing someone's opinion because someone else holds a contradicting opinion.
And for that reason, I think your version exposes the flaw even more thoroughly: you can't reasonably merge a data set of stats on people's opinions on AI with a data set of stats on people's opinions on IP in the way that you're proposing.
To throw out random numbers as an example of the flaw: If 55% of people on HN believe that IP protection for media should not exist and 55% believe that GitHub stole code, it's entirely possible that TFA's condemnation only applies to 10% of the total HN population that holds the supposedly conflicting belief even though HN "statistically" believes both things.
And that's before we get into the question of whether there's actually a conflict (there's not) and the question of whether anyone is accurately measuring the sentiment of the median HN user by dropping into various threads populated by what are often totally disjoint sets of users.
All else being equal, yes. But NYC drivers are exceptionally skilled relative to the rest of the country. They're nuts in a lot of ways, but they're also far more respectful of pedestrians, far more aware of their surroundings, far more willing to drive slowly, and far more happy to stop for pedestrians even if they technically have the right of way.
NYC e-bike riders (especially the Citi bikes) are the opposite. They're far less likely to stop, far more likely to be going way faster than is safe, and far more likely to blow through stop signs entirely.
There's a well-documented phenomenon in safety that the "safer" choice in the abstract can actually end up being more dangerous because it feels safer, which leads to riskier behavior. I fear this is what is happening in NYC: incentivizing people to ride bikes doesn't get you 10x as many of the good, respectful, careful bikers you had 10 years ago. It gets you a whole bunch of reckless amateurs who buy the hype that bikes are safer and bike like maniacs. We may well find that that's worse for pedestrian safety than the well-known and well-regulated dangers of cars.
On narrow one-way roads with street parking, cars tend to go quite slowly and look out carefully for pedestrians coming out from between cars. At a stop sign you can check left and right, cross, and expect that cars will see you and stop even if they were for some reason planning on running the sign. It's even possible to safely jaywalk in most cases because the cars are going slowly enough that you can see them, judge their speeds, and cross before they reach you.
E-bikes as a rule do not behave that way on narrow roads. They will drive much faster than is safe (presumably because the road feels less narrow to them relatively) and end up outrunning their visibility and yours. I've never come close to being hit by a car in Brooklyn, but there have been numerous close calls with bikes, and they've all been on narrow roads with street parking where I was attempting to cross at a stop sign which the bike ran.
E-bikes are a totally different story. They'll shoot through narrow streets twice as fast as the cars do and blitz through stop signs without even slowing down. There have been multiple occasions where I was nearly hit by one of them.
They seem to think that because they're smaller they don't need to care as much, but at the relative speeds of cars and bikes on those narrow streets I'm pretty convinced they'd do more damage to me than the 5-10 mph car would.
Which argument? The one dismissing all arguments about IP on the grounds that some software engineers are pirates?
That argument is not only unpersuasive, it does a disservice to the rest of the post and weakens its contribution by making you as the author come off as willfully inflammatory and intentionally blind to nuance, which does the opposite of breaking the unproductive equilibrium. It feeds the sense that those in the skeptics camp have that AI adopters are intellectually unserious.
I know that you know that the law and ethics of IP are complicated, that the "profession" is diverse and can't be lumped into a cohesive unit for summary dismissal, and that there are entirely coherent ethical stances that would call for both piracy in some circumstances and condemnation of IP theft in others. I've seen enough of your work to know that dismissing all that nuance with a flippant call to "shove this concern up your ass" is beneath you.
This kind of guilt-by-association play might be the most common fallacy in internet discourse. None of us are allowed to express outrage at the bulk export of GitHub repos with zero regard for their copyleft status because some members of the software engineering community are large-scale pirates? How is that a reasonable argument to make?
The most obvious problem with this is it's a faulty generalization. Many of us aren't building large-scale piracy sites of any sort. Many of us aren't bulk downloading media of any kind. The author has no clue whether the individual humans making the IP argument against AI are engaged in piracy, so this is an extremely weak way to reject that line of argument.
The second huge problem with this argument is that it assumes that support for IP rights is a blanket yes/no question, which it's obviously not. I can believe fervently that SciHub is a public good and Elsevier is evil and at the same time believe that copyleft licenses placed by a collective of developers on their work should be respected and GitHub was evil to steal their code. Indeed, these two ideas will probably occur together more often than not because they're both founded in the idea that IP law should be used to protect individuals from corporations rather than the other way around.
The author has some valid points, but dismissing this entire class of arguments so flippantly is intellectually lazy.