calpaterson

The trouble is that images are a large fraction of the page size even for well optimised websites. Using smaller images via changing format often makes a significant difference in the page load time.

It's also relatively easy for new formats to be adopted - the html picture tag makes it really easy to serve the same image in multiple formats to allow for backward compatibility.

And of course image decoders are easily distributed - JXL's encoder/decoder are open source.

The combination of those two factors probably means that this will never be a settled area now. It's fairly easy for websites to adopt a new format and there are strong reasons to. Means, motive and opportunity.

I reckon Chrome will cave in eventually. The pressure on browser performance is intense. Unless they want to see Safari faster in benchmarks of image heavy pages they will have to adopt it.

The CMA is a quango:

> In the UK, the term quango covers different "arm's-length" government bodies, including [...] non-ministerial government departments

https://en.wikipedia.org/wiki/Quango

> The Competition and Markets Authority [...] is a non-ministerial government department

https://en.wikipedia.org/wiki/Competition_and_Markets_Author...

But this specifics of what is and is not a quango don't matter. The original claim implied that the decision was made by a politician for political reasons and this isn't the case.

It was blocked by a quango, not by a politician.

https://en.wikipedia.org/wiki/Competition_and_Markets_Author...

I reserve the right to use my holiday snaps as I want on my own blog :)

Well, it was me who did that test and frankly I only included it to give some context for people who've never heard of Parquet. If you want evidenced benchmarks I'm sure you can google for those yourself or indeed just run your own - the benefit differs as the data does (financial data obviously is mainly numerical).

I'm not, in this post, actually talking about Parquet being faster than csv but instead about how it makes data interchange easier (mostly that is about having an embedded schema).

You make a great point and I'm sorry about that. To answer some of your questions and try to add more detail:

Strings are UTF-8. Internally Parquet has the concept of "physical types" (ie how bytes are arranged) and "logical types". 64 bit ints are just physical with no logical type. DateTimes are 64 bit int physical types (millis since unix epoch - aka "Javascript time") with a logical type of "datetime" so you know to read it that way.

Bool columns are bits in a row.

The "data bearing" parts of a file can be compressed and usually are. Most are using snappy, a fastish, moderate compression codec. I believe some are also using zstd. One of the common questions people ask is: "how does it compare against .csv.gz" and the answer is favourably. Gzipped csv is even slower to read and write and usually about 30% bigger on disk.

Tanenbaum's book was the one that clicked for me, doing self-study some years ago.

Even though he has presumably retired now the book is still being updated:

https://www.amazon.co.uk/Computer-Networks-Global-Andrew-Tan...

I wonder if it is demoralising. Imagine any project you worked on for multiple years - and someone suggests it could be rewritten (and better) in a long weekend. How would you feel and respond? I might laugh a bit, but I probably wouldn't spend much time on it.

The part of FOSS I think would be demoralising for me would be dealing with all the people who assume that the maintainer of a piece of software is a customer service operative who exists solely to reply to them in a timely fashion.

The FDIC isn't funded from general taxation, it's funded by a specific levy on banks.

You can be failed for driving too slow in the UK as well. "Undue hesitance".

Having been to Finland I feel the reverse: it's much more hostile. It is not great that (as you implicitly described) they put zebras next to road turns and as you're crossing the road cars will occasionally turn right across you. Pedestrians are clearly also a lot more afraid of cars in Finland - as evidenced by their famous waiting at crossings for the green man even when there are no cars.

Finland is a very different country than the UK - much worse weather in winter but much much sparser - but looking at the stats it seems to do a little worse on road casualties than the UK on most measures. eg 3.8 road deaths per 100k pop in Finland vs 2.9 in the UK and 5.1 road deaths per billion km in Finland vs 3.8 in the UK.

I don't think Finland's road safety is as good as the UK though it is still one of the better countries.

> I would be curious though if the more expensive and involved driving license in many European countries results in better outcomes though.

It doesn't. UK road fatalities are very low compared to mainland Europe and it doesn't matter much whether your divisor for that stat is number of vehicles or number of miles driven. It's arguable how much of that is down to the driving test and how much down to other factors like rules, rule following behavior, etc etc - but I think at least some of it is.

> relatively calm (if somewhat pedestrian hostile) style of driving in the UK

The UK is pedestrian hostile? In comparison to where? It's always seemed to me one of the most pro-pedestrian countries, eg there is no such thing as "jaywalking": you can cross anywhere and cars are obliged to stop once you're "established in the road" - plus they are obliged to stop at all times on zebras.

> Now that https is ubiquitous, there aren't many caching proxies around to cause grief.

Well, there are still CDNs. csvbase is designed for a public cache for some pages. I haven't done much on this except for the blog pages, which use the CDN a lot.

I also have vague plans for client libraries that include a caching forward proxy as my experience is that most people export the same tables repeatedly. Likely that will be based on etags though so that the cache is always validated.

The designers of HTTP 1.1 clearly thought a lot about a lot of things, including caches.

Thanks for your thoughts. :) Keep in touch via email if you like (same goes for anyone else reading this): cal@calpaterson.com

You are most welcome. So glad it is useful.

This subject tracked here: https://github.com/calpaterson/csvbase/issues/29

Wow that is cool, really cool.

I will look into implementing Content-Range - what is it that you want that for? What's the usecase?

Thanks for that background. I'll try to update the post when I get home this evening

This is planned for csvbase at least.

> My reading of the HTTP spec suggests that csvbase.com is behaving incorrectly by not setting the Vary header properly

This is a good idea, I will certainly look at this. There are some planned features WRT caching coming up.

To address the comments you made in GP:

> How do I know that I can get csv files from that URL

That is a good question. The web UI could be better, of course. But programmatically, how do you advertise alternate representations? I'm not sure. Suggestions appreciated.

> Will the website always default to csv files or will my app break when they decide that XML is superior? (Well, obviously not, especially for a site called csvbase!)

As you say: csvbase won't change :)

But the other thing is that the HTTP client you use could decide to change it's default Accept header. If curl changed to "application/json,q=0.9;/" then suddenly you'd get json (I didn't mention in the blog post but that is also implemented)!

Oh dear. Perhaps a good idea to include the file extension or explicit Accept header when you're coding something that needs to last. But I do think it's nice to be able to copy and paste into pandas. That's my main usability case and I wanted that to be as smooth as possible.

I'm very sorry, you have found a bug. Dates and json lines currently not working happily together.

Try a table without a date column:

    curl https://csvbase.com/calpaterson/iris.jsonl

Is ".jsonl" the right file extension, do you think?

Well, one of my bugbears is that most open data is released in weird formats.

Take the boston housing dataset as an example:

http://lib.stat.cmu.edu/datasets/boston

This is one of the most popular beginner datasets around. It's often used in tutorials and when experimenting. Does everyone who wants to use that have to write some custom parser? Why? https://csvbase.com/calpaterson/boston.csv is just so much easier

Whoops, should be fixed

> a measure of the rate at which a development team consistently delivers business value

It isn't even usually this, because typically it's _developers_ - not the wider business - who assign point counts to tasks. At the end of two weeks you've typically done about two weeks of work so all that is actually being measured by velocity are the median numbering preferences of the team. eg People who like "fibonacci numbers" invariably achieve a velocity of about 6 * $DEVELOPER_COUNT per sprint.

In order to get a measure of business value you would have to involve someone who knew the relative value of the tasks. Usually that does not happen and agile has at least partially contributed to that by standardising the idea of a development team as being primarily made up of developers (with one token project manager who is there because they know how to manage people, not because they have special knowledge of business value).

Agile, at least as popularly understood, has moved the whole field backwards by entrenching the separation of developers from non-developers. If you try to work directly with the business on a modern agile team, you tend to get rebuked! You've cut your "project manager" (who typically doubles as your line manager) out of the loop and you're no longer making progress on the agreed team measure of story points. Frustratingly, trying to work on business value is de facto non-compliance on many "agile teams".

Yeah, tried Polars a couple of times: the API seems worse than Pandas to me too. eg the decision only to support autoincrementing integer indexes seems like it would make debugging "hmmm, that answer is wrong, what exactly did I select?" bugs much more annoying. Polars docs write "blazingly fast" all over them but I doubt that is a compelling point for people using single-node dataframe libraries. It isn't for me.

Modin (https://github.com/modin-project/modin) seems more promising at this point, particularly since a migration path for standing Pandas code is highly desirable.

> What they should do is give a popup saying "You have a slow cable, it could be faster, go rummage till this message goes away and mark the good one with a sharpie"

Android does this. "Charging slowly"

> But often the lower-hanging fruit is doing more joins in the DB, fetching fewer columns, and writing your query more thoughtfully.

Reluctant to add a "me-too" comment but I think you have it there. For whatever reason people are much keener to investigate different ways of scheduling IO than they are to investigate what the query plan looks like and why.

Unfortunately the majority of people who use asyncio in the real world do not have a need for concurrenct programming. They are drawn to it because they believe it is faster, which, generally, it is not. That isn't to say there aren't reasonable usecases - but most people using asyncio are writing webapps which spend a lot of time in the CPU generating or outputting JSON/GraphQL/HTML.

NAS bittorrent clients are often based on transmission. For example, Synology's is.

According to his benchmark run today on Python 3.9 on linux, dbm is still considerably faster for random access.

That said, what database dbm is using varies a lot by platform. I think on linux it's usually BerkleyDB.

I use dbm a lot to cache pickles of object that were slow to create. eg pandas dataframes

> I thought Stephenson gauge was chosen because there were a lot more rail miles and rolling stock using that gauge than Brunel gauge.

That was key reason. The other was that 7ft gauge limits how tight your corners can be versus standard gauge. There are other minor disadvantages, such as that because trains are bigger, tunnels need to be wider.

The 7ft gauge trains were probably a lot more smooth though. And that extra ~3ft would make them feel much more spacious. Luxury was a big prestige element on the Great Western.