DasIch

This 80% solution already exists in the form of trains and airplanes.

Self-driving cars that only go 80% of the way would compete against those forms of transportation which have established business models, networks and in the case of trains are subsidized in a way that self-driving cars likely won't ever be.

Additionally the key advantage of cars is flexibility, not having to figure out how to get to or from the train station or airport. In your scenario self-driving cars would lack this advantage.

Now granted this may still sound like an appealing proposition in countries with poor public transportation infrastructure but that substantially reduces the size of the overall market. Sure this might be attractive for the US but what about Europe? Asia?

Now of course there is an obvious answer to this problem: Keep the steering wheel and drive the rest of the way yourself. That just means you'll have a lot of drivers who won't gain experience at the current rate though. Not sure I would like to share the road with those people.

Let's assume for a moment there would be no opposition to nuclear in Germany. How does nuclear energy help with climate change?

Looking back to 2000, nuclear energy at peak was responsible for 32.1% of electricity production[1].

We'd need to at least double that. That would take a huge amount of money. More importantly it would take time. We can barely manage to build a single fucking airport in 1-2 decades. Building this many nuclear reactors would easily take this long, if not longer.

It seems to me that nuclear energy as a solution to climate change might look good on paper. Sci-fi also looks good on paper though and it's just as realistic.

[1]: https://de.statista.com/statistik/daten/studie/29295/umfrage...

It’s a significant deviation from what I would expect from a disk oriented database. So I would definitely expect it to be well documented, along with the reason for it, why the developers believe it is a reasonable (or even safe) choice, what assumptions went into that (such as availability of uninterruptible power supply) etc.

Additionally keep in mind that with EBS most people probably use network attached storage and fsync involves the network. Outage doesn’t just mean power outage, it could also be a network issue.

When you write to a file, you generally don't write to physical storage. Instead the writes get buffered in memory and written to physical storage in batches. This substantially improves performance but creates a risk: If there is some sort of outage before the data is flushed to disk, you might lose data.

In order to address that risk, you can explicitly force data to be written to disk by calling fsync. Databases generally do this to ensure durability and only signal success after fsync succeeded and the data is safely stored.

So ClickHouse not calling fsync implies that it might lose data in case of a power outage or a similar event.

I can't find anything about this in the docs except[1]. I also can't find any issues in their bug tracker related to clickhouse not using fsync[2].

I can however find code that actually calls fsync[3][4]. To be fair I haven't read enough to determine how this (doesn't) affect durability. Nevertheless I'm wondering do you have a source for this claim?

  [1]: https://clickhouse.yandex/docs/en/operations/settings/settings/#fsync-metadata
  [2]: https://github.com/ClickHouse/ClickHouse/search?q=fsync&type=Issues
  [3]: https://github.com/ClickHouse/ClickHouse/blob/355b1e5594119e036a2d62988bfa42bc8b1a1687/dbms/src/IO/WriteBufferFromFileDescriptor.cpp#L113
  [4]: https://github.com/ClickHouse/ClickHouse/blob/e765733a26cfc4cecc13c981686560338256a6b1/dbms/src/IO/WriteBufferAIO.cpp#L98

For me it's ad-hoc analysis on large CSV files. Large meaning well beyond what Excel would be capable of, often larger than fits into memory on my local machine (10s of GiB).

Sometimes I also use xsv to just do a step of the analysis and dive deeper on some subset using pandas.

In my experience both SQLite and Pandas aren't as fast as fast for large files. So they are not really good options.

Pandas is especially bad because it uses a column oriented data structure internally so reading from or writing to CSV is incredibly slow in Pandas. If you can use parquet that's not a problem but unfortunately parquet is not nearly is ubiquitous as csv :(

Some commands are benchmarked here: https://github.com/BurntSushi/xsv/blob/master/BENCHMARKS.md

`xsv` can do far more than just join and select, such as searching, filtering, partitioning, ...

For me it's the first choice when I find myself with a CSV file to work with. I've also encountered situations where just getting some raw data from a (slow) database and "querying" it with xsv ended up being the fastest option to get the results I wanted.

These are live streams for the chaos communication congress. The most important conference for hackers in Germany and certainly one of the most important conferences for hackers world wide.

You're looking for https://opentracing.io/

We don't just use it for traces across services but also as a means of profiling in production. Going from "latency is too high" to "this exact database query is taking too long" in a few clicks is amazing. I wouldn't want to work on a new project without it.

It doesn't affect a single AZ though. We see numerous problems with AWS services including but not limited to being unable to create new EC2 instances in any AZ.

This is creating a huge number of issues as you lose capacity with the AZ going down and can't recover by creating new instances.

That just means you should scale based on the work to be done rather than poor proxies such as CPU utilization. Also set a reasonable minimum and maximum based on observed load in production and review this as part of regular operational reviews.

It does reduce the competition of the "We provide Sentry as a Service" kind but it still allows people to create their own Sentry-like product of which they can also provide as a service.

I think for customers having different companies have their own approaches to solving this problem is vastly preferable to having different companies solve the same problem in exactly the same way.

You can also have a subquery in from as in

    select * from (select 1) as temp;

In what way is Vestager not qualified? Is there an example of an action she has taken that a more qualified person wouldn't have?

That's assuming you're making the equivalent of "beautiful pieces of furniture". There are quite a few people with open source contributions on their resume or their github account with repos that is not helping them, quite the opposite in fact.

Germany isn't the U.S. Germany has a 5th article[1].

Germany acknowledges that there are human rights (plural), that they occassionally conflict and that there are tradeoffs that must be made to achieve some balance and maintain all human rights in some reasonable form.

[1]: https://www.gesetze-im-internet.de/englisch_gg/englisch_gg.h...

Almost any application with an event loop doesn’t halt.

Spilo[1] is a Docker image that provides postgres bundled with Patroni[2].

The postgres-operator I linked earlier but also our setup on AWS (with one image per EC2 instance) uses that to actually run Postgres.

  [1]: https://github.com/zalando/spilo 
  [2]: https://github.com/zalando/patroni

I don’t have actual numbers but I did a quick search and most are a few GiB to tens of GiB, although there are a few hundreds of GiB large. In practice size is not the limiting factor, IOPS are because they all use gp2 EBS volumes. Databases that have huge IOPS requirements are still deployed outside of Kubernetes and run in i3 instances. In that case they still use spilo though, so basically the same system for backups and automatic failover as on Kubernetes.

That being said we also have an ElasticSearch operator that is used to deploy ElasticSearch on Kubernetes, there nodes running on i3 instances and the corresponding instance storage is used. Although used in production that’s still very new and sadly not open source.

EC2 instances do go down, EBS volumes fail, hardware fails. Maybe not as frequently as pods in Kubernetes get evicted but at sufficient scale it does occur frequently enough overall that you do need to find ways to handle this automatically without human intervention.

Once you’ve achieved that whether your database runs on a VM or in Kubernetes doesn’t make a difference really.

Granted, if your not at that scale, running a database in Kubernetes is probably not the best of ideas. That has nothing to do with Kubernetes though, that’s because running a stateful service with decent working backup, recovery and automated failover is difficult in any case. If that’s not your job, you’re probably better off using RDS or something equivalent.

https://github.com/zalando-incubator/postgres-operator supports point in time data recovery just fine and is used in production for 100s of databases at Zalando.

It's not democratic.

In Zalando, every PR made must be reviewed by at least one other engineer. Every application must be registered which involves giving it name, a description and associates it with the team owning the service. APIs that cross team boundaries must further be reviewed by an API guild, a group of people that are experts in API design, this also applies to events.

Events go through Nakadi, which enforces a schema and authorization. You can further tell which services are subscribing to such events and through the before mentioned application registry who is responsible for them.

Additionally compliance with various laws and shareholder expectations requires regular reviews, to identify and fix issues such as missing documentation, monitoring, SLOs, data storage restoration tests (in production), load tests etc.

The scenario you describe is not possible or allowed. As awareness and adherence to these rules is also part of performance reviews, it’s also not in any engineers in interest to do that.

You’re right of course that one needs to be aware of that.

What you need to keep in mind is that micro services have a huge fixed cost but the marginal cost of adding a new one is small.

In other words: Creating or migrating to a micro service architecture is very expensive but once it’s in place, adding a new micro service is trivial. In Zalando the goal is that you can do so within half an hour, that is take an idea, implement it and deploy it to production.

This leads to architectural decisions no sane person would make at a smaller company. For example let’s say you get data from an external company via SFTP and you make it internally available, so what do you do?

1. You create a micro service that polls the SFTP of the supplier for new files and sticks them into an S3 bucket 2. You create a micro service that takes the files from S3, parses, transforms into JSON and publishes events to Kafka. 3. You create a micro service that takes those events enriches them with some other data and republishes new events. ...

Especially when you use Kubernetes, you also start thinking about infrastructure completely differently. Why use multiple threads or processes for example, when you can just run multiple instances of your application very cheaply and temporarily run further instances to handle background tasks?

Before you abandon the idea of unions, you should keep in mind that the situation in regards to unions in the US is fucked up in a rather unique way.

So a much less radical approach of simply copying what other countries do, should work just fine much more reliably.

I don’t understand your argument. If you have lots of clusters, surely you’re upgrading using some kind of CD mechanism. So it shouldn’t make a difference in time to upgrade.

Zalando (disclaimer I work there, though not on K8S infrastructure. I just deploy to K8S) does upgrades this way: https://m.youtube.com/watch?v=1xHmCrd8Qn8

These motorcycle drivers are continuously exposed to wind pressure though, in her case she would be suddenly exposed to it which probably would be not unlike hitting a wall.

Agreed, however you have to take context into account. Python 2 didn't always have a bool type (granted, already a mistake), before it was introduced int was used instead. Making bool a subclass of int made it possible to use it even in combination with earlier code, which made adopting it trivial.

If bool wouldn't be a subclass of int, some projects probably would still be using int instead of bool.

issubclass(bool, int) == True in Python 2. They're not converted to ints silently they are ints.