Hacker News Reader

Comment by crazygringo | Hacker News Reader

No, it's a terrible API if retrieving the result '2000-10-26' requires downloading a 565 KB file.

I don't want to seem overly negative -- zipped CSV files are fantastic when you want to import lots of data that you then re-serve to users. I would vastly prefer it over e.g. protobufs that are currently used for mass transit system's live train times, but have terrible support in many languages.

But it's incredibly wasteful to treat it like an API to retrieve a single value. I hope nobody would ever write something like that into an app...

(So the article is cool, it's just the headline that's too much of a "hot take" for me.)

View on HN

It's historical data. There is zero reason to request it more than once a day, and the users of such data will be interested in wildly different filters or aggregates over that data.

If this were being used to get the current rates, then yes, it would be a terrible design choice. But there are other services for that. This one fits its typical use case.

> I hope nobody would ever write something like that into an app...

I have got some bad news for you...

Not directly API related, but I remember supporting some land management application, and a new version of their software came out. Before that point it was working fine on our slow satellite offices that may have been on something like ISDN at the time. New version didn't work at all. The company said to run it on an RDP server.

I thought their answer was bullshit and investigated what they were doing. One particular call, for no particular reason was doing a 'SELECT * FROM sometable' for no particular reason. There were many other calls that were using proper SQL select clauses in the same execution.

I brought this up with the vendor and at first they were confused as hell how we could even figure this out. Eventually they pushed out a new version with a fixed call that was usable over slow speed lines, but for hells sake, how could they not figure that out in their own testing and instead pushed customers to expensive solutions?

> how could they not figure that out in their own testing

This one is easy. Testing with little data on fast network (likely localhost).

Also, if there are any ORMs involved it could be that it’s not immediately obvious from their code itself that this would happen.

I’ve seen code that was using an ORM, where they needed to find some data that matched certain criteria. With plain SQL it would have resulted in just a few rows of data. Put instead with their use of the ORM they ended up selecting all rows in the table from the database, and then looping over the resulting rows in code.

The result of that was that the code was really slow to run, for something that would’ve been super fast if it wasn’t for the way they used the ORM in that code.

Seen this so many times by novice developers, whose work style is often "Trial and error, and as soon as it works, stop and change nothing." And it's so easy to slip in there because it works perfectly locally and often even in non-production envs, unless you've seeded them with production-like amounts of data, and with the ORM it looks fine. Like in Rails:

  Product.where(category_id: xx).filter {|p| p.name.include?("computer") }

  Product.where(category_id: xx).where("name like '%?%'", "computer")

PS: I know there's an Arel way of doing the above without putting any SQL into the app -- use your imagination and pretend that's what I did, because I still don't have that memorized :)

I just see recently an example of that, a REST call returning a few KB of data that would fetch a few million rows from the database and use 10+ GB of memory (unfortunately some people think that you should always use join fetches with JPA...).

Kinda feels like 20 years ago called and wants its argument back, like have you seen any javascript ever?

565 KB + the logic to get the big one is miniscule today by any reasonable factor.

True, though ironic that also OP is implicitly making the argument for GraphQL-like interfaces ("just specify the data that you want returned" rather than "get all the data, then process it"), which are themselves the New Hotness in some areas.

At risk of sounding pedantic, REST also allows for "just specify the data that you want returned" style APIs.

Something like:

    curl https://www.ecb.europa.eu/stats/eurofxref/api?fields=Date&order=USD&limit=1

A very fair point well-made!

I think a lot of people in this thread are glossing over a difference in definitions. Some people see "API" as "how to get data, even if that means all data with no filtering." Personally, I regard downloading an entire table as downloading a data model without any logic operating on the model, and an API as logic that returns only part of the model filtered in some way that interests me.

Why do you assume that 565KB even matters?

I've been building loads of financial software, both back and front ends. In frontend world it, sadly is quite common to send that amount of "data" over wires before even getting to actual data

And in backends it's really a design decision. There's nothing faster than a Cron job parsing echange rates nightly and writing them to a purpose designed todays-rates.json served as static file to your mobile, web or microservices apps.

Nothing implies your mobile app has to consume this zip-csv_over_http

They should ship parquet, supports predicate pushdown and is dramatically more compact, while you can't get row level data, it's great for analytical queries.

Totally agree! So much that I wrote a whole article about it a while back!

"Why parquet files are my preferred API for bulk open data" https://www.robinlinacre.com/parquet_api/

Parquet is a level better than csv but difficult to get customers to adopt and transmit in that format.

csvbase does ship parquet!

Just add ".parquet" - eg https://csvbase.com/meripaterson/stock-exchanges.parquet

Nice!

IMO this is the best simple option right now.

For sorted data you only need the relevant row groups which can be tunable to sensible sizes for your data and access pattern.

I'm an old fart, seeing the curl in every example is like nails on chalkboard.

565 KB, that's about 3 minutes download on a 28.8kbps modem I started getting online with...

When you got that modem were you concerned with how long it would have taken to type/write the information sent over it or were you just happy it didn't matter anymore?

I often wonder what I'll think of technology in say another 20 years but I can never tell if it's all just some general shift in perspective or, as you look farther back, if certain people were always just about a certain perspective (e.g. doing the most given the constraints) and technology changes enough that different perspectives (e.g. getting the most done easily for the same unit of human time) become the most common users for the newer stuff and that these people will also have a different perspective than the ones another 20 years down the line from them and so on.

For example, maybe you think it's crazy to ask for the exact piece of data you need, I think it's crazy to do all the work to not just grab the whole half MB and just extract what you need quickly on the client side as often as you want, and someone equidistant from me will think it's nuts to not just feed all the data into their AI tool and ask it "what is the data for ${thing}" instead of caring about how the data gets delivered at all. Or maybe that's just something I hope for because I don't want to end up a guy who says everything is the same just done slower on faster computers since that seems... depressing in comparison :).

How much data do I need to download before I can do a protobuf request ?

Mass transit files are often a single protobuf file that is like a megabyte or something.

I'm talking about zipped CSV files as easier to use than protobuf files. Neither is a request in this comparison.

Are you counting the client size? TLS handshake? Because otherwise the answer is none.

But what if I want the average over time? The query depends on every value, should everything be computed on the server?

Here's an API built on top of this data that allows for more fine tuned queries.

https://exchangeratesapi.io/

https://github.com/exchangeratesapi/exchangeratesapi

How much bandwidth does the average API documentation page use?

Does your client query the API documentation every time it's querying an API?

Well of course it does; how else would it know how to query the API? /s

> (So the article is cool, it's just the headline that's too much of a "hot take" for me.)

It's one of those headlines that we'd never allow if it were an editorialization by the submitter (even if the submitter were the author, in this case), but since it's the article's own subtitle, it's ok. A bit baity but more in a whimsical than a hardened-shameless way.

(I'm sure you probably noticed this but I thought the gloss might be interesting)

Reply on news.ycombinator.com