Hacker News Reader

Comment by FriedPickles | Hacker News Reader

Claude code decided to just put my name and email in the User-Agent when scraping docs from the SEC. No clever prompting required.

It’s not a terrible idea really, but I wish it would’ve asked me first.

View on HN

That’s the fault of the SEC EDGAR tool. The Edgar MCP docs show setting an env var: ‘SEC_EDGAR_USER_AGENT="Your Name (name@domain.com)"’.

Claude just followed the instructions. Which is maybe even more risky?

Interesting theory. The Claude Code instance says it's using this user agent based on the guidelines in the EDGAR Fair Access Policy: https://www.sec.gov/search-filings/edgar-search-assistance/a...

...which indeed requests that bots supply `User-Agent: Sample Company Name AdminContact@<sample company domain>.com`.

So in this case, the site just asked nicely and Claude Code complied. Note that this wouldn't have worked for Claude Chat, which the author was testing, because the `web_fetch` tool cannot set the User-Agent or other headers.

How have you noticed that it did that?

I could see it right there in the tool call:

  Bash(curl -A "<my name> <my email>" "sec.gov...")

Why is it not a terrible idea?

If you’re making automated requests, I consider it a common courtesy to provide an accurate user agent.

Some services like Wikimedia will let you browse/download with rate limits IF your user agent is descriptive enough and not misleading.

Thanks, I wasn't aware of this. But to put your real name in the field instead of at least a pseudonymous id or more descriptive info but still have more bits of uncertainty user-agent for a public website, is that really a preferred practice?

As a website owner, if I saw someone scraping with a realistic looking name + email address I'd definitely give them more latitude than someone trying to hide the fact they're scraping. In my experience people who are hiding the fact are much more likely to be doing something nefarious.

Sound legit to me as long as it's prompted to use hardcoded "dario amodei".

Reply on news.ycombinator.com