Hacker News Reader

Comment by JaggedJax | Hacker News Reader

How is the sample `calculateETag()` function generating a weak ETag? It looks like it will generate a different hash due to any JSON formatting changes.

It seems like generating a weak ETag would take more effort since you'd need to either ensure consistent ordering and formatting of the JSON, or generate the Etag on the content before converting it to a JSON string.

View on HN

That's because it isn't really generating a weak ETag. From the article:

> You could make the `calculateETag` function format-agnostic, so the hash stays the same if the JSON format changes but the content does not. The current `calculateETag` implementation is susceptible to format changes, and I kept it that way to keep the code shorter.

They seem to agree, a true weak ETag implementation would probably be trickier and require more code :P I'd be fascinated to see how that might work in practice, though.

I could probably write one pretty quick, under the assumption that we are storing only JS-compatible JSON with no encoding hiccups (JSON sadly isn’t as standard as it appears at first glance…) just hash(JSON.stringify(JSON.parse(fileText))) and you’re done. This assumes the same parse and serialize methods are expected to be used at both ends, that they only normalize formatting and that you don’t have to worry about number representation doing weird things. I wouldn’t actually sort keys as sorting is technically a change in behavior and good browsers today do not re-order object keys for you, though your code can do that, of course. I considered skipping the second JSON serialize, but it makes a buffer out of an object so it’s easy enough to use. One could imagine a more efficient approach would modify the hashing to occur against buffer chunks of the JSON, but intentionally skip the whitespace. This avoids unintentional data serialization but obviously the parsing routine would have to match the recipient exactly to work correctly. And it still assumes you’re receiving oddly formatted but valid JSON, which doesn’t sound like a safe assumption to make. If your JSON varies in format I wouldn’t ever want to assume I’d be able to parse it correctly. I mean, what if a return character slips in by mistake amongst all the newlines?

I'm kind of thinking now, why bother with generating a weak ETag at all? Unless your backend is doing things that would commonly cause differences in JSON formatting for the same data, this is probably a rare case and not worth the extra effort or processing. Figure it out when you're at a scale that it actually matters, and stick with strong ETags for now.

It's good to know about this option for handling in the frontend if a system returns one though.

Yeah, I’ve never really heard of “weak etags” before in any sort of common usage of the term. Honestly, most people tend to skip etags by embedding hashes in filenames directly, this way you can avoid any bad proxies serving up stale content or dropping headers. It’s rare these days to be an issue given the use of TLS end-to-end encryption, but I’m sure it still occasionally happens. And yes, the more serious approach to possibly poorly formatted JSON is to “normalize it” into the expected format. It’s less about caching and more about ensuring what you serve to your front end is consistent, even if you are liberal in what inputs you can handle. E.g. if someone gives you XML, rather than write a front end that can handle both XML and JSON, pull the data out of both and make your own JSON later.

Since the author wrote in go, an approximate equivalent would sort the keys by default (assuming you decode into a map and Marshall that).

Not sure how I missed that, thanks.

Reply on news.ycombinator.com