samaysharma

(Samay from Citus)

This blog post is quite different than Cloudflare's old pipeline. I can think of three fundamental differences.

(1) Cloudflare's pipeline used an old version of Citus. On that version, Citus was still a proprietary database that forked from Postgres. Since then, Citus became an extension of Postgres (not a fork) and went open source.

(2) Features highlighted in this blog post weren't available in the old version of Citus. Example features are hash distribution, distributed roll-ups, and sharding by site_id and partitioning by time.

These features help in two ways: (a) They simplify the overall pipeline quite a bit. You don't need to rely on a single Postgres node and Kafka queues to do roll-ups. (b) Your end to end price-performance improves by 4-5x. You can ingest larger volumes of data through sharding + time partitioning. You can also do parallel roll-ups inside the database. Neither of these were possible back in the day.

(3) Similarly, Cloudflare outlined issues around single point of failures around Postgres / Citus: https://blog.cloudflare.com/http-analytics-for-6m-requests-p...

Postgres streaming replication and Citus' HA solutions have come a long way in the past three years. These points shouldn't be an issue in newer versions.

Overall, the database space is an exciting market, where products continuously evolve. This blog post presents a reference architecture that uses new features in Citus - that have been added in the past three years and that our customers are using today.

Part 1 of the blog post here: https://www.cybertec-postgresql.com/en/are-triggers-really-t...

"It has been a crazy 24 hours for GitLab. More than 2,000 people tweeted about #movingtogitlab. We imported over 100,000 repositories, and we've seen a 7x increase in orders." That's quite an impact!

Link to original post published on Twitter's blog: https://blog.twitter.com/engineering/en_us/topics/infrastruc...

Citus Data (https://www.citusdata.com/) | Technical Solutions Engineer | Full-time | San Francisco

Citus is worry-free Postgres. Built to scale out, Citus is an extension to Postgres that is available as open source, as on-prem enterprise software, and as a fully-managed database as a service.

We are a fast-growing database startup in San Francisco. We founded Citus Data with a vision to combine the best aspects of relational databases with the elastic scale of NoSQL. And we’re building a team with some of the best minds in distributed systems, databases, and Postgres—along with engineers with extensive operational experience managing millions of Postgres databases in the cloud.

As a solutions engineer, you will be responsible for analyzing customer requirements, understanding their use-case, and building an end-to-end solution using Citus technologies. You will need to quickly prototype solutions, run proof-of-concepts to demonstrate value, and also help customers move to production. Every customer is different, and thus adaptability and being a quick learner is crucial. Your day to day role will involve talking to customers, writing code, performance and query tuning and debugging and solving problems.

You can learn more about the position and apply here: https://www.citusdata.com/jobs/solutionsengineer

Author here:

This post just notes some back of the envelope math I do in my head in terms of what to expect with a moderate box (r3.2xl or so). It is not the limit of how far a single node PG can go and I'm sure you can go much higher with a very well tuned PG on larger boxes :)

The replication and high availability related documentation is here:-

https://cloud.google.com/sql/docs/postgres/high-availability

https://cloud.google.com/sql/docs/postgres/replication/

Any timelines on high availability support? This would be pretty helpful

From https://status.aws.amazon.com/: "Update at 12:52 AM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour."

(I think the AM means PM)

Looks like the dashboard is fixed https://twitter.com/awscloud/status/836662601090134017