mulander

In the article, mentioned Linux is mentioned as the underlying OS. Wonder what approach Ubicloud takes (if any) to have actual diversity in the software stack for the purpose of reliability and security. My assumption here being, that different OSes, while increasing the attack vector also make it more likely that the whole fleet is not susceptible to the same software problem or vulnerability at roughly the same time. Just something I started pondering about after seeing Hetzner, which is quite popular in the BSD land.

Because with schema-based sharding in Citus, the schemas are the sharding unit and the system can move them to new nodes being added to the cluster.

You can start with all your microservices sharing a single node, add nodes and have the storage layer distributed horizontally in a way transparent to the services themselves.

Hi, we open-sourced the project fully: https://www.citusdata.com/blog/2022/06/17/citus-11-goes-full...

There no longer is an enterprise version, what runs on Azure is the exact same Citus that you can run yourself. We even invested in Patroni, to make it easier for the community to self-host Citus with HA setups.

While we obviously want people using Citus on Azure, having Citus as a viable open-source choice is our path to achieving that. I wasn't part of the company when the acquisition happened so can't speak to that, but I can imagine how that could have made sales at the transition time unclear.

Personally I would also like to add, that the team is full of long term open-source contributors. We contribute both to PostgreSQL and projects around it (like pgbouncer). I understand and respect your reservation, but wanted to share my perspective on it.

Great observation! :)

We worked upstream to have `search_path` properly handled (tracked per client) by pgbouncer.

https://github.com/pgbouncer/pgbouncer/commit/8c18fc4d213ad4...

Check config.md in that commit for a verbose, humanized description.

They can be different, as long as your application can handle them being different (which makes sense for microservices as an example).

Schemas are groupings of tables and other entities that can be defined within a database. You can think of them like of a namespace in programming languages. You can have the same table definition, within the same database, defined multiple times (each in a different schema) and each holding different data.

By large and small we are referring to the amount of data each schema holds currently. They can grow over time and some of them may become very big while others will remain small (storage wise).

Hi Nathanba, I am a technical program manager for the Citus extension at Microsoft.

> Citus also exists for Postgres but their docs basically tell you that it's only recommended for analytics

Could you point me at the docs that made you think that? We find Citus very good at multi-tenant SaaS apps (OLTP), IoT (HTAP) workloads and analytics (OLAP).

> And it sounds like all it's doing is a basic master-slave postgres setup with quite a few manual things you have to do to even benefit (manually altering tables to make them sharded/partitioned)

This is true for now. We are looking into ways to make onboarding easier. That said, the time spent on defining a good sharding model for your data often leads to very good perf characteristics. Regarding the architecture, I personally find Citus closer in spirit/design to what Vitess is doing. Additionally, every node in the cluster is able to take both writes and reads, so I don't see the parallel to a basic primary/secondary setup.

> But with all that, as Joe Armstrong once jeered, no amount of type checking would catch the following bogus code:

It is worth to note that the stated problem, of accessing a file after it is closed was addressed in the Phd thesis by [1] Joe Armstrong in which he suggested a solution for exactly this problem by introducing a new testing & development methodology he named "protocol checkers" (9.1 Protocols, page 195). Relevant quote from the thesis:

> Given a protocol which is specified in a manner similar to the above it is possible to write a simple “protocol checking” program which can be placed between any pair of processes.

The protocol is a state machine, which would detect the attempt to write to a file after it was closed.

[1] - https://erlang.org/download/armstrong_thesis_2003.pdf

Developers are categorized as people with commit access to the project. So contributing patches itself is not enough to add entries to this specific file.

- Meetings are remote first. If at least one person is remote then the whole team video calls in. There is nothing worse than being a person on the virtual end trying to understand a mic grabbing voice from a room full of people.

- Schedule peer programming sessions, especially with less senior staff. Without the office environment they have less opportunities to grow.

- All meetings should have a note listing all decisions made.

- Promote async communication. Have people describe a problem instead of saying hi and waiting for a reply.

> just that it minimizes the attack surface

I stopped using multiple browser for exactly that purpose. It actually increases your attack surface. While compartmentalization of browsers helps somewhat against tracking you now have to keep all of them updated. RCE bug in one of them already compromises everything you have on your machine, so having multiple browsers just means the attacker can pick from a wider array of targets.

History is funny. KHTML (a KDE project) was repurposed to WebKit by Apple and integrated with Safari. Which in turn was taken by Google that built Chrome on it. They later diverged more and now other browser are being built on repurposed Chrome. It would be funny if Safari was really replaced by Chromium as that would mean the offspring has eaten the parent.

It used to have actual Windows specific crypto code[1] which has been removed in the linked commit.

I assume this has been ported from Windows and later never implemented the ripped out components. That said, I don't know the windows API so apart from confirming that they exist in Windows docs[2] I can't assess how valid their usage was.

[1] - https://github.com/microsoft/omi/commit/edbe231042173018c529...

[2] - https://docs.microsoft.com/en-us/windows/win32/api/dpapi/nf-...

Cool project. Have you known that PostgreSQL was initially written in Lisp?

> By a process of elimination, we decided to try writing POSTGRES in LISP. We expected that it would be especially easy to write the optimizer and inference engine in LISP, since both are mostly tree processing modules. Moreover, we were seduced by AI claims of high programmer productivity for applications written in LISP.

> We soon realized that parts of the system were more easily coded in C, for example the buffer manager which moves 8K pages back and forth to the disk and uses a modified LRU algorithm to control what pages are resident. Hence, we adopted the policy that we would use both C and LISP and code modules of POSTGRES in whichever language was most appropriate. By the time Version 1 was operational, it contained about 17000 lines of LISP and about 63000 lines of C.

src: https://dsf.berkeley.edu/papers/ERL-M90-34.pdf

It also had time travel (from the same paper):

> Lastly, POSTGRES supports the notion of time travel. This feature allows a user to run historical queries. For example to find the salary of Sam at time T one would query:

    retrieve (EMP.salary)
    using EMP [T]
    where EMP.name = "Sam"

> POSTGRES will automatically find the version of Sam’s record valid at the correct time and get the appropriate salary.

Regarding your project:

> Databases are implemented as directories containing one file per table.

PostgreSQL splits tables into 1 GB segments (1 segment is one file). This is done for filesystems that can't have large files. Isn't one file per DB too limiting in that regard?

Are you aiming to be ACID compliant? Can you expand on 'versioned'? PostgreSQL achieves that with copy-on-write, did you take the same approach? I quickly looked over the code, it's been a while since I read some lisp. The on-disk representation seems to be just key value records where each record can be a sexp(?). Where is the relational part and do you plan to implement a query language?

He assumed I was not working or doing work for a different manager (they implemented matrix management and he was at HQ 70 km away from the town I worked at). I made a judgement call as that migration was a $20 mln project. That corporation had time budgeting. I was supposed to account in a production tracking system every task I did in 15 minute granularity. I have a ton of crazy stories like that out of that place.

In essence, the guy was a micromanager with trust issues. They later replaced him, with someone that was actually worse.

> had fewer commits and SLOC compared to my teammates

Hope this makes you feel slightly better: https://www.folklore.org/StoryView.py?story=Negative_2000_Li...

It is impossible to measure someones productivity using metrics like commit counts and SLOC. There are times where I don't commit for weeks and provide actual business value by solving direct client issues without even a ticket being present.

At one time at my second IT job, I had a manager reprimand me (yelling over the phone) that I was away from my desk. I was remotely supporting a bank fixing consortium credit contracts (substantial amounts) after a migration. I was told to stop doing that. So, I told the support tech that my 1-up blocked this. He told the people at the bank; this went up the chain at the bank to the main accountants which resulted in a nasty call from the client to the skip manager of my manager. He called me back and asked me to resume what I did almost crying on the phone. Later that year I was selected employee of the year mostly because of that client constantly sending positive feedback on my help.

I would say that "source code is the blueprint of a computer program". If you look at the wikipedia entry:

> A blueprint is a reproduction of a technical drawing or engineering drawing [...] allowed rapid and accurate production of an unlimited number of copies.

and even if no-one reads the definition, the term is in common use. Spies stealing blueprints of weapons in movies / real-life being an immediate example.

Well, I would expect them to address this issue specifically as that is one of the selling points of fastmail. You can get banned on Google for nothing and not being able to reach a person with your data held hostage. Now, this is a comparable situation where data is held hostage unless the customers in Russia have a way to access and retrieve whatever is stored on their accounts.

On top of that there is a question of billing and refunds in the case of long-term plans being paid up front (eg. one year subscription).

The article touches on new signups being blocked, what happens to the current paying users in the region? Are they also blocked? How is fastmail helping them move/supporting them in this situation?

I am a fastmail customer in a different region (Poland). I am however looking closely at this as who knows, maybe I will be on the receiving end in the future?

> The problem the cartels will face is that there is no other business that makes as much money

This is not true. See this article[1] and HN thread[2]:

> “It’s good business,” El Polkas says with a shrug. “It makes a lot of money.” When I ask how gasoline compares to narcotics, in terms of overall revenue to Los Zetas, he rubs his index fingers together. “Fifty-fifty,” he says. “It’s approximately as profitable as drugs.”

[1] - https://www.rollingstone.com/culture/culture-features/drug-w...

[2] - https://news.ycombinator.com/item?id=18101141

Check the slides/talk from my comment here elsewhere in this thread: https://news.ycombinator.com/item?id=27163455

We did use DTrace to diagnose what was the root cause of a hanging insert (it was the same pending list cleanup as on this blog post). The whole talk we did was about DTracing PostgreSQL :)

We have been bitten by the same behavior. I gave a talk with a friend about this exact topic (diagnosing GIN pending list updates) at PGCon 2019 in Ottawa[1][2].

What you need to know is that the pending list will be merged with the main b-tree during several operations. Only one of them is so extremely critical for your insert performance - that is during actual insert. Both vacuum and autovacuum (including autovacuum analyze but not direct analyze) will merge the pending list. So frequent autovacuums are the first thing you should tune. Merging on insert happens when you exceed the gin_pending_list_limit. In all cases it is also interesting to know which memory parameter is used to rebuild the index as that inpacts how long it will take: work_mem (when triggered on insert), autovacuum_work_mem (when triggered during autovauum) and maintainance_work_mem (triggered by a call to gin_clean_pending_list()) define how much memory can be used for the rebuild.

What you can do is:

- tune the size of the pending list (like you did)

- make sure vacuum runs frequently

- if you have a bulk insert heavy workload (eg. nightly imports), drop the index and create it after inserting rows (not always makes sense business wise, depends on your app)

- disable fastupdate, you pay a higher cost per insert but remove the fluctuation when the merge needs to happen

The first thing was done in the article. However I believe the author still relies on the list being merged on insert. If vacuums were tuned aggressively along with the limit (vacuums can be tuned per table). Then the list would be merged out of bound of ongoing inserts.

I also had the pleasure of speaking with one main authors of GIN indexes (Oleg Bartunov) during the mentioned PGCon. He gave probably the best solution and informed me to "just use RUM indexes". RUM[3] indexes are like GIN indexes, without the pending list and with faster ranking, faster phrase searches and faster timestamp based ordering. It is however out of the main PostgreSQL release so it might be hard to get it running if you don't control the extensions that are loaded to your Postgres instance.

[1] - video https://www.youtube.com/watch?v=Brt41xnMZqo&t=1s

[2] - slides https://www.pgcon.org/2019/schedule/attachments/541_Let's%20...

[3] - https://github.com/postgrespro/rum

I remember Ada always having very precise and helpful error messages like this one:

literal_string.adb:5:33: warning: wrong length for array of subtype of "Standard.String" defined at line 5 literal_string.adb:5:33: warning: "Constraint_Error" will be raised at run time

then when you run the app it behaves as advertised

$ ./test

raised CONSTRAINT_ERROR : literal_string.adb:5 length check failed

There are many others and when I saw llvm improving error messages for C and C++ (which I saw happening before Rust was a thing) I always thought it was inspired by Ada and it's helpful messages like:

expected private type "<type name>" defined at ...; found type "<type name>" defined at ...

"<name>" is undefined (more references follow); possible misspelling of "<name>"

and the many others that were just there when I first tried Ada around 2008/2009.

I used it when it was only a way to build a trust network. I stopped using it when they started adding chatting and other functionality. Never even once used the chat feature or anything else they later bolted on.

I don't think there is an explicit filter installed. Quick search reveals a companion that can pronounce names and Jesus is not recorded as something the character can say while Mohammed is allowed and voiced. Though this might not be censorship but just something that wasn't recorded. Fallout also evolved into an online game. I haven't played it but that might lead to character name restrictions.

> Codsworth is known to say the Sole Survivor's chosen name if it is an option, although it may be shortened, extended, or have a word omitted. A list of spoken names can be found here.

> Ironically you can be Mohammed but you can't be Jesus. I know I missed a bunch of the good ones, feel free to add some below.

[1] - https://fallout.fandom.com/wiki/Codsworth

[2] - https://steamcommunity.com/app/377160/discussions/0/49688113...

In Fallout one and two, the main character had the option to kill NPC child characters. The game went so far as to mark you with a specific perk "Childkiller"[1] after performing that act.

From the linked fandom wiki, quote by Tim Cain about the subject:

> This led to the child killing controversy. We said look, we're going to have kids in the game; you shoot them, it's a huge penalty to karma, you're really disliked, there are places that won't sell to you, people will shoot you on sight, and we thought people can decide what they want to do. [...] This of course contributed to our M-rating, however, Europe said "no". They wouldn't even sell the game if there were children in the game. We didn't have time to rewrite all the quests, we just deleted kids off the disc.

[1] - https://fallout.fandom.com/wiki/Childkiller

Last time I tried matrix, and mind you that was over 3 years ago. Was with riot.im and trying to use the e2e encryption. The major surprise was not being able to suddenly decrypt older chat messages when OpenBSD changed the User-Agent string for Chromium which as I understand was used to establish the device identity. This is not something non-technical people can or want to diagnose.

I don't have experience with large SQLite databases. I had PostgreSQL take over 40 minutes for recovery on a 2 TB database set to checkpoint every 30 minutes. How long it takes will depend on the WAL size, the frequency of checkpoints and the operations that were performed before a crash/snapshot.

The only problem with atomic filesystem snapshots is the time required to perform recovery which can be avoided. The recovery process is of course reliable.

You are correct and what I had in mind was the required recovery. I do trust recovery but performing it can be very time consuming.