calpaterson

Higher indeed. 40% no shows for a meetup I run (60-100 slots, meeting monthly)

Yeah the London ones were enormous back in the day. I seem to remember 100+ attendees. We only have a tenth of the population - so my expectations are appropriately "managed" :)

Haha, bold thread. I can understand the frustration as probably we all know this monkey business is too common - but I suspect that an open invitation to "name some names" is not probably not going to be a productive and useful as you might hope :)

Speaking of - I'm organising a coffee morning for HN readers here in Helsinki. Get in touch (email in my profile) if you want to come along

The financial system runs on a soft-whitelist system with extra checking and a shadow banning system.

This is why everyone is always asking everyone to demonstrate the sources of funds in finance.

Obviously it is not 100% effective, but it is somewhat effective. You cannot just jog in from Iran and enroll as a JP Morgan client

Yes - all of the code for everything is on github. The client you probably don't need - just use curl - but it is kind of cool to do `import pandas as pd; pd.read_csv("csvbase://meripaterson/stock-exchanges")`

I did try to say that it's all open source! Even in the first sentence! :)

There is also a podcast. It's more in-depth than the tv series and has been updated over the years.

https://www.bbc.co.uk/sounds/series/m000jf7j

It is not correct to ascribe the blame for this issue on a social divide between the public and the establishment/aristocrats/elites/etc.

Many of those culpable are pretty ordinary solicitors and technology contractors and not, eg, peers of the realm.

It is a really useful feature for ad hoc examination of datasets. Not every data-analysis is going into a massive, mission critical production build chain. Quite often you just want to look at the data in a REPL and IMO that sort of usage could bear to get a little easier

I even installed S-PLUS 8.0 in order to look for this - but Sdatasets doesn't seem to have been shipped in the base install. If anyone knows...

> There is no single answer to all of this, because it's too complex to boil down to a single number.

Having a single number that is a rough measure is still useful, though perhaps a (weighted?) average would be more useful.

Anyway, I just downloaded the Python tool (called "libyear" on PyPI) and ran it to quite quickly find three dependencies on my project that were over 2 years behind. That was helpful and I would use it again.

> The cause is that CSV is popular and is popular because it is incompletely defined. (See also: HTML, RSS.)

This is a surprisingly deep thought. I'm going to need a couple of days to reflect on this...

I'm sorry, quite right - you did. Using the computer locale.

The trouble is that people transmit data from one computer to another and so from one locale to another. And sadly, they do not always set the character encoding header correctly, if they even know what that is.

I mean take csvbase's case. It has to accept csv files from anyone. And christ preserve me, they aren't going to label them as "Win-1252" or "UTF-16" or whatever.

There is no alternative but statistical detection. And there is good evidence that this solution is fairly satisfactory, because millions of computers are using it right now! csvbase uploads run into more problems with trailing commas than with character detection getting it wrong at this point, that is your "Schelling Point" I'm afraid!

If csv files bring criminal liability then I am guilty.

Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Committee To Keep Csv Evil: https://discord.gg/uqu4BkNP5G

Jokes aside, talking about the future is grand but the problem is that data was written in the past and we need to read it in the present. That means that you do have to detect encoding and you can't just decide that the world runs on UTF-8. Even today, mainland China does not use UTF-8 and is not, as far as I know, in the process of switching.

I understand UTF-8 is mostly fine even for east asian languages though - and bytes are cheap

It's mojibake for "probably a bad idea" [in Chinese]

Yes you're right, UTF-8 technically does as well. I've never seen them in real life either.

UTF-16 BOMs do have a useful function as I recall: they really help Excel detect your character encoding (Excel is awful at detecting character encoding).

榥\ue0af侬펭懃䒥亷

csvbase is written in Python instead of Perl and FCGI instead of "trad" CGI but otherwise is a lot like a site from 1994...

One of the things I really need to sort on csvbase is permissions. Currently it has something very basic: you own your table and it's either public (meaning everyone has read) or private (meaning only you have read).

I really don't want to implement, eg RBAC, by myself but equally the state-of-the-art seems to be integration against pretty complicated external stuff (/cloud services) which is also undesirable when it's your side-project.

Advice appreciated. I suspect LDAP is not a fit but I will investigate it...

I'm sorry you're not a fan :)

I do take your point - to an extent.

I think the subject you're touching on is basically that of configuration. Configuration is what allows you to change the behaviour of code without making a code change. Configuration is of course very really powerful for good and evil. Someone below mentions ODBC which, yes, 100% is a great example. But the one that sticks in my mind is resolv.conf.

I would contend (:)) that csvbase-client is not doing "fiddly magic at package installation time" but supplying configuration for a url scheme. In much the same way that installing boto3 does for using the `s3://` url scheme.

"Magic" in my book would be monkeypatching the methods of other libs. I would hate to write `csvbase_patch` and have it execute on the import of a specific module. Like you, I prefer to live in a low-magic fantasy world.

Yes, though as I try to explain in https://news.ycombinator.com/item?id=39993176 the bit I was trying to get at is that for `pd.read_csv('csvbase://calpaterson/opcodes-6502')` there is no API surface at all. And you can do writes with `pd.DataFrame.to_csv`. I personally think that is fun.

PS: Thanks for all of your work over the years (decades?) on sqlalchemy. I'm sure you're not too surprised to learn that the lions share of code in csvbase is calls to SQLA core.

http is already registered as a protocol and you can indeed do that already. It's printed on every table page. eg on https://csvbase.com/calpaterson/opcodes-6502:

    import pandas as pd; opcodes_6502 = pd.read_csv('https://csvbase.com/calpaterson/opcodes-6502', index_col=0)

But writing back is harder. It's not easy to make pandas do an HTTP PUT and then insert the HTTP basic auth and so on. Plus (not discussed in the article) there is a cache to avoid redownloading the data when it hasn't changed, which is my personal bete noire in "data science" such as it is.

> Total clickbait title, obviously being able to make an HTTP request to an endpoint, and get a CSV back, is an API. Respecting the “accept” header to return different data formats, based on what the client requests, is totally fine/reasonable, but it doesn’t magically make an interface that’s designed for applications to program against not an API.

I'm sorry about the title. As I said below; that was really meant in the spirit of fun.

On the subject of csvbase's content negotiation - yes that is an API. That was covered here some time ago when I wrote about it before: https://news.ycombinator.com/item?id=37526047

The "no API" bit I'm talking about in this article is basically the "trick" (or whatever word you want to use) of avoiding having any user-facing interface and just hooking into stuff that is already there. There is no "API surface" here for the user to learn beyond a url scheme. I think that's nice. And it's mainly what I'm talking about.

> Also, while CSV is a nice/convenient data format for a data analytics use case (like this), it’s certainly not a format I’d choose for an API where clients are likely to be more standard CRUD-ish apps. JSON is great for those, CSVs (with their trickier parsing, “everything is a string” data types, and enforced flatness) are a pain in the ass.

Without wanting to sound too much like a sales pitch: csvbase does offer JSON. Try https://csvbase.com/calpaterson/opcodes-6502.jsonl for JSON lines (or https://csvbase.com/calpaterson/opcodes-6502.json (no 'l') for a paged plain-JSON interface).

I personally think there is no ideal format for this at the moment. JSON is very very large and slow to parse. CSV has well known problems though has massive compatibility and often works well in practice. Parquet is probably closest to the ideal and excellent in many respects but is quite complicated to parse (moreso than CSV? perhaps) and anyway is effectively unstreamable - actually quite annoying for something like csvbase where you really _don't_ want to materialise the dataset while serving it.

> I did think it was interesting to learn about fsspec, didn’t know about that!

Yes it is cool isn't it. Millions of downloads, terabytes of bandwidth of PyPI and no one has heard of it.

> Exaggerative title aside

I apologise for the title. Not intended to mislead, just to be a bit of fun :)

> t's really cool to learn about how fsspec [...]

I'm glad you took something from it. At the last place I worked I spent a lot of time building a library to persist dataframes (intended audience: data analysts) and in retrospect I wish we'd thought of the approach of just having a `csvbase://` url scheme via fsspec.

Currently it doesn't support Parquet which they would have needed though. Still some work to do for csvbase to accept Parquet uploads.

I love your tool. If you want a slightly more structured version of Gists that speaks parquet try my own tool, csvbase :). I pasted this data in as https://csvbase.com/calpaterson/opcodes-6502 and you can get parquet by adding .parquet to that (https://csvbase.com/calpaterson/opcodes-6502.parquet)

Here's datasette lite running off that file:

https://lite.datasette.io/?parquet=https%3A%2F%2Fcsvbase.com...

Really nice when webby stuff works together

Thanks for your advice. I agree and will try to improve on this for the future

There is an nginx server running on debian stable somewhere in the dark heart of Germany. But I do have numerous tricks (too many, probably) to keep things quick.

But there are still ways to be quicker. For example, the header photo is smaller than the vector diagrams on the page, by about tenfold.

That is very kind - glad to hear that you enjoyed it.

> Is there a reason to build a database on top of a filesystem abstraction rather than a block abstraction?

Oracle DB for a long time supported running on raw partitions which I think suggests that the answer is "not really". Snowflake (and I hear Clickhouse) can run on S3 which I think is more evidence against running on a filesystem. Not to mention the torrid time Postgres has had with fsync on linux.

I was tempted to define and cite the term more carefully before I used it but that leadened the article a lot right in the middle and so I cut it and just hoped.

It is a great concept and also a great book. I really enjoyed it but I've never found a convincing way to persuade people to read it. I read it on personal recommendation but that only works if it comes from someone you respect (as in my case).