Comment by yzmtf2008 | Hacker News Reader

Comment by yzmtf2008 | Hacker News Reader

TL;DR: use URI instead. URL is behaving this way due to backwards compatibility.

---

To quote a JDK bug ticket: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4434494

> We are well aware of the problem with URL.equals and URL.hashCode. The cause of the problem is due to the existing spec and implementation, where we will try to compare the two URLs by resolving the host IP addresses, instead of just doing a string comparison. Because hashCode has to maintain certain relationships with equals, namely if two objects are equal, they should have the same hashCode, the implementation of hashCode also tries to resolve the host in the URL into an IP address. As a result, we are facing problems with http virtual hosting, as described in the Description part, and performance hit due to DNS name resolutions.

> Unfortunately, changing the behavior now would break backward compatibility in a serious way, plus Java Security mechanism depends on it in some parts of the implementation. We can't change it now.

> However, to address URI parsing in general, we introduced a new class called URI in Merlin (jdk1.4). People are encouraged to use URI for parsing and URI comparison, and leave URL class for accessing the URI itself, getting at the protocol handler, interacting with the protocol etc. So, at present, we don't plan on changing the URL.equals/hashCode behavior and we will leave the bug open until Tiger, when we re-investigate our options.

When I was a Java developer ~8 years ago, "Use java.net.URI" was a whole sentence in my vernacular. Not just as a replacement for URL, but also for all of the dumb string concatenation people were doing to build urls and query parameters.

It was already ancient advice then, but I kept finding myself having to teach it to a new group of people, usually after some incident had happened (because they won't listen until something has already broken)

Yes, I’m constantly giving advice to Ruby and JS developers that “URLs are not strings” and to use URI libraries for manipulation. This is also true for phone numbers, emails, IPs, etc.

Hell, even Rails maintainers rejected a PR I opened that tried to get rails link helpers to work with URI objects.

I see why this was useful, though it still wasn't a good idea. This looks like it was added in ~1995, at which point (a) it was common for multiple hostnames to refer to the same server and (b) they were guaranteed to act identically because there was no HOST: header yet.

Your b) was actually only ever true for the HTTP protocol, but URLs are not strictly limited to that protocol. They are not limited to any finite set of protocols at all. You could easily make up your own protocol which differentiates between host names in a similar way as modern HTTP does. And you could have made that in 1995.

You could have made one in 1995, but looking over https://www.iana.org/assignments/protocol-numbers/protocol-n... I think HTTP/1.1 was the first protocol to send the host name like this. (Though I could be missing something, since that's a lot of protocols.)

Your 'a' case is still common.

Perversely, a well-timed DNS record update and cache expiry could make a URL not-equal to another URL made from the same string of characters.

That's the essence of a DNS rebinding attack, which can be used to bypass server-side request forgery vulnerability mitigations.

I might be wrong, it was a long time ago, but IIRC a different DNS rebinding attack was actually part of the reason this behavior was introduced to the URL class, to help protect against such attacks in Java Applets.

Hell, no need for a record update: just multiple A records for the same hostname, with a short enough TTL that two different URL instances could conceivably resolve differently.

You should also use a gradle or maven plugin called forbidden-apis to prevent people using deprecated stuff like this. I've been adding this to pretty much any Java/Kotlin project for years with some custom patterns to also prevent people using shadowed packages that some libraries expose. In this case, for forbidding java.net.URL.equals is probably a good idea. Might already be in the default patterns.

Also sounds like something a good static code analyzer should be able to detect.

I wish that was part of the Javadoc. I just looked up the Javadoc for jdk8 and URL, URL.equals and URL.hashCode and I would have no idea that it does DNS requests. I am a very seasoned Java dev (since 2001) and I am sure I read at one time about this but forgot. But I do use URL from time to time though and I am pretty sure to have used it with Map or Set somewhere.

A clear warning at top of URL, URL(String) would be good to make it recognizable in most IDEs when using it.

Out of curiosity do you get a compiler warning when you use URL.equals?

No and probably it will never happen in javac. Some IDEs may try to do static code analysis to identify possible uses of URL.equals (e.g. Map<? extends URL, ?>), but I doubt anyone would invest in this feature.

Looks like IntelliJ has it:

- equals() or hashCode() called on java.net.URL object

- Map or Set may contain java.net.URL objects

https://www.jetbrains.com/help/idea/list-of-java-inspections...

Doesn't look like it's turned on by default though, which seems strange. Presumably if you care enough to go into the settings to enable that specific warning you're unlikely to make the mistake in the first place. It's exactly the type of non-obvious gotcha that would be nice to have your IDE warn you about out of the box.

I agree and disagree. I mean, the thing is that outside of personal projects it's not uncommon to inherit a codebase from other people, sometimes quite a large codebase. Such a warning can be useful when eliminating boo-boos (acting as an lint I guess).

Wouldn't you want warnings when an inherited code base does weird things with possible incorrect behaviors as side effect?

Well I would consider DNS resolution on comparisons/hashing to be an incorrect behaviour as a side effect.

In this case Sonar will do a better job. IDE warnings are good for fixing them immediately - when they accumulate it’s very easy to stop paying any attention.

This is nonsense. The spec is wrong. There is no chance I would consider two URLs with equal local parts and query parameters, but different authorities resolving to the same addresses as equal -- that would be WRONG. The authority could easily provide different semantics for the same URL based on differing Host: values.

They know this is nonsense and that the spec is wrong.

It's there for backwards compatibility, which is not nonsense. It keeps the old code running.

Sometimes you need to break backwards compatibility for semantics. This is one of those cases.

I'm really curious as to why they don't make it, say, protected where the JVM wouldn't actually barf at runtime (or would it? I'm a bit too enmeshed in Scala bincompat at the moment), but where compiling from source would error out.

If I'm wrong: Why not invent a 'protection level' where this were a feasible thing... even if just to be able to remove old APIs and really force people to stop just adding @SuppressWarnings-deprecation all over the place. That's not a solution to anything.

Reply on news.ycombinator.com