Hacker News Reader

Comment by stevoski | Hacker News Reader

Ah, I remember this specific file from my time at the ECB 15-ish years ago.

IIRC it was by far the most downloaded file on the ECB website. Tons of people, including many financial institutions, downloaded it daily, and used it to update their own systems.

IIRC #2 in the minutes immediately after the daily scheduled time for publishing this file, there was a massive traffic spike.

It was a conscious decision to make it a simple CSV file (once unzipped): it made it possible to serve the file reliably, fast, and with little resources needed.

The small team responsible at the time for the ECB’S public website was inordinately proud of the technical decisions made to serve this data in a single static file. And rightly so.

View on HN

This is something that powers a lot of data interchange but lots of people don't realize because they never work with older systems like that. It's not shiny. No frameworks.

I used to work for a large, old company whose products you have all bought and buy (and which shall remain nameless) about 15 years ago. I worked on data interchange between the systems used for record keeping of said products and various other downstream or parallel systems (same purpose, separate system because left over from merger/acquisition).

So basically bulk data import and export. The product was 15 years old at that point. Data interchange was via various fixed width or delimited (CSV but the C might be some other character or character sequence) files transferred to and from various SFTP servers.

It's been a while but there were probably like 20 or 30 different such data sources or exports going in and out. Worked like a charm. I bet they're still used without much change. The frontend was being rewritten at the time (old one was in Smalltalk).

I'm using this type of simple approach to build a SaaS right now. We need to spin up many VPS and provision them, and the fastest way to do that is with rsync and ssh.

But we didn't stop there: this SaaS for our open source browser product is entirely built like this: behind the scenes it's a collection of bash scripts that implement and execute the business operation of the SaaS.

So basically, it's a command-line interface to the SaaS. Think of it this way, say I didn't have a website, with login, and "click a button to open a browser", but instead people would write me letters, send me cheques, or call me on the phone. Then I can serve their requests manually, at the command line.

The reason I made it like this was:

- clear separation between thin web front-end and actual business logic

- nice command-line interface (options, usage, help, clear error messages) to business logic for maintenance and support to jump on and fix things

- inheritance of operating system permissions and user process isolation

- highly testable implementation

Maybe this is dumb, but I really like it. To me it's an architecture and approach that makes sense.

I'm sure this is not new, and I think a lot of good quality operations must be built via this way. I highly align with the author's stance of the composition of a few simple command line tools to get the job done.

Perhaps we can call this "unix driven development", or "unix-philosophy backend engineering"

browser product: https://github.com/dosyago/BrowserBoxPro (saas coming soonish)

I feel like you went way overboard with this one.. I think the first paragraph and an example of what you're doing that's similar to OP would've been plenty

I enjoyed their comment, thanks...

You're being a jerk, someone shared something they were proud of that's relevant to the discussion. What's the problem?

A lot of developers talk smack about CSV files (or tab delimited) but they really are the workhorse of our industry. They're simple, textual, compress really well, and as long as everyone on both sides has an agreement about the format, they're almost a standard.

sure, a lot of the criticisms of CSV are true, but given the above constraints it's really hard to beat them.

"everything is a file" still going strong

And they can be open in Excel for the best and the worst.

I know this file too, I was one of them. Of all the data sources we used, it was the best to work with.

Wow. I’m just thinking of the hoops I’d have to go through to host changing data as a static CSV/ZIP at my employer.

Architects would complain ZIP isn’t a format conform to their specs for this purpose. Compliance would complain there needs to be a check that no private informatie leaked. Risk that I should prevent bad actors from downloading the file. Web people that I need an approved change to add stuff to the site.

Yeah, but modern, approved practices don't really address any of that - they just hide it under a pile of extra complexity that's too big for most devs and managers' attention span, effectively disappearing it.

Do you know why they decided to host a zip file instead of just hosting the CSV and relying on HTTP compression?

HTTP compression is optional, so they either have to compress on the fly (wasting cpu) or provide multiple versions (complicating setup and deployment) or make some HTTP clients not work.

simgle zip file is really the easiest solution for cases when the file must absolutely be compressed

Many webservers allow you to serve a compressed file (stored on disk) and _decompress_ when a client specifically can't support the compressed encoding. Since most clients should support compression, this means you only use the CPU for the rarer case where the uncompressed data is required.

Eg, http://nginx.org/en/docs/http/ngx_http_gunzip_module.html

Or you can just zip it explicitly and remove complexity and future issues it they have to move to another web server platform.

But what if an analyst needs to access this data and run their regressions on a potato? Surely that use case is worth adding a few libraries to handle.

I would think running regression should be more demanding on the potato then decompressing ZIP file.

hmm good point, can you draft up an architecture plan using multiple microservices and redundancy via a kubernetes cluster and have it on my desk by Monday please.

What machine could anyone be running an analysis on where unzip is a limiting factor?

you heard them, a literal potato from the ground.

even a big old russet only has, what, like 32 bites?

I mean there are some potatoes that take more then four bytes to eat, but they are rare.

For some, if you slice them, you might even end up with only 16 bits.

Whoosh

Nginx or Apache would both cache these versions transparently. I think they just wanted to distribute as a zip

The business logic here is >15 years old. Http compression was only in early stages then, and you can guarantee that many client side scripts and libraries would jot have supported it. Zip was well known. Compress and place.

If I'm not missing the point here, this was, and still is, about offering the simplest, most reliable solution over a long period. This is a near perfect example of how to do exactly that. No changing formats, no moving requirements, no big swings in frameworks, apis, or even standards. And most importantly, no breaking your customers business workflows.

No edge case dependencies on the WWW server's configuration, and no sudden "why did we just saturate our external connection?"

No emergency change requests from the outage team that has to be impacted by other areas and fit into the infrastructure's teams maintenance windows and their capacity to address that.

No rebalancing of workloads because Jane had to implement (or schedule the task and monitor it) that change, Joe had to check and verify that the external availability tests passed, and Annick had to sign off on the change complete, and now everyone isn't available for another OT window for the week.

Or something.

> "The business logic here is >15 years old. Http compression was only in early stages then"

At least one of us is confused about history here; are you really saying that circa 2008, HTTP compression would have been considered immature or unstable?

Maybe in part because it encourages you to work with a local copy, rather than just hitting the hosted .csv repeatedly?

Just guessing. I have no idea :-)

It it exactly what most api optimization does. For example, batch queries merge several api calls into one to reduce api call counts. And it this case, it is perfectly optimized. You have literally one zip file, and there is no more.

I could see 2 reasons:

1 - save on cpu usage, compress once, serve many

2 - with zip you can have some rudimentary data integrity checks (unzip -t)

Another option would be save it as a gzip file and serve it raw with implied gzip compression.

Nginx does this: https://docs.nginx.com/nginx/admin-guide/web-server/compress...

Last entry: the directive is `gzip_static on;`

By distributing the data in a compressed file, you retain the benefits of the smaller file size at all stages (transfer from source, local storage, local transmission) until the data needs to be used in uncompressed format, all without any additional processing.

The CSV was available at https://www.ecb.europa.eu/stats/eurofxref/eurofxref-hist.csv but isn't up to date.

It's a bit scary to imagine the consequences if this file was somehow corrupted (wrong column headers for instance)!

Just wait until you inherit a service that, sans documentation, pulls in a web resource file that suddenly is no longer available :(

scale8 recently archived their GitHub repo without explanation and right after that the url for a GDPR opt-in list they were using changed. The product is now dead because the code doesn't handle the situation where that file might not be available.

(Scale8 is a web analytics and tag management software like google analytics)

I forgot to mention this error check even breaks the admin dashboard meaning you can't even see historical data anymore.

Just in case anybody else is in the same boat you can change the url in the code to the new one yourself.

> IIRC #2 in the minutes immediately after the daily scheduled time for publishing this file, there was a massive traffic spike.

Perfect usecase for bittorrent protocol imho. Funny thing is you can serve files with bittorrent protocol from Amazon S3 without any additional complexity or charges I think.

Do you remember why they decided to zip it instead of just relying on the compression in http?

Does HTTP compression perform well if you're streaming a file? Wouldn't it need to have access to the entire file to have the same result as zipping it?

Yes it performs well. You don't need access to the entire file.

Reply on news.ycombinator.com