Citus Unforks from PostgreSQL, Goes Open Source
citusdata.com
citusdata.com
> With the release of newly open sourced Citus v5.0, pg_shard's codebase has been merged into Citus...
This is fantastic, sounds like the setup process is much simpler.
I wonder if they have introduced the Active/Active Master solution they were working on? I know before, there is 1 Master and multiple Worker nodes. The solution before was to have a passive backup of the Master.
If say, they released the Active/Active Master later on this year. That's huge. I can pretty much think of my DB solution as done at this point.
We're working on making Citus masterless. In all openness, we evaluated two different approaches to this in the past six months, and wrapped up the design for one. This design works well on the cloud, and we already demonstrated a working version: https://youtu.be/_nun2S6EdWo?t=411
For on-premise deployments, the primary challenge is set-up complexity. We're now prototyping one of those designs to know more: https://github.com/citusdata/citus/issues/389
We expect to share all the details and a concrete timeline in April.
also in Turkish: kolaylıklar dilerim :)
Or is Citus taking over the master/master replication? (or is it doing something different?)
It seems at least theoretically possible.
Since the Citus master executes distributed queries by sending regular SQL queries to the Citus workers, you could already use BDR servers as workers and replicate the data between pairs of workers in different data centers and copy over the metadata on the master manually. However, some distributed joins and data loading features wouldn't work.
For all features to work, and to replicate the master, you would have to compile Citus against BDR, which probably requires a few code changes.
Which of these are supported:
1. Full PostgreSQL SQL language
2. All isolation levels including Serializable (in the sense that they actually provide the same guarantees as normal PostgreSQL)
3. Never losing any committed data on sub-majority failures (i.e. synchronous replication)
4. Ability to automatically distribute the data (i.e. sharding)
5. Ability to replicate the data instead or in addition to sharding
6. Transactionally-correct read scalability
7. Transactionally-correct write scalability where possible (i.e. multi-master replication)
8. Automatic configuration only requiring to specify some sort of "cluster identifier" the node belongs to
Have a donut and look at our marketing spreadsheets.
I'm so tired of "seamless" "effortless" "simple" distributed database lies. There's mathematical theorems as to why there is no free lunch.
On PostgreSQL language support, we're updating our FAQ to have more information: https://www.citusdata.com/frequently-asked-questions Since the PostgreSQL manual (and its feature set) spans over 4K+ pages, we found that the best way to think about Citus' capabilities is from a use-case standpoint. If your workload needs distributed transactions that span across machines, or large ETL jobs, Citus currently isn't the best fit.
Citus supports sharding and replication out of the box (#4, #5). On #6, reads go through a master node (metadata server) and you see what you write.
We don't have #7. The way in which we implement this also has implications on your other questions. Multi-master (no single metadata server) is by far the biggest feature request that we receive: https://news.ycombinator.com/item?id=11353866
If we go with the approach in https://github.com/citusdata/citus/issues/389, you will be able to configure #3, #6, #7 through PostgreSQL's streaming replication settings. We still won't support distributed transactions that span across multiple machines.
On #8, could you elaborate a bit more? Do you mean a logical identifier for the node?
Also, it's hard to write a concise reply on a topic that requires so much context. I'd love to grab coffee with anyone who's interested in diving deep into distributed databases. Feel free to shoot me an email at ozgun@citusdata.com
Do you know when you're planning to release Citrus 5.0 deb/rpm packages?
As soon as they're built in PGDG! Our Docker image just builds on the PostgreSQL 9.5.1 image, then installs a .deb we built.
I've been wrapping up all our packaging work during the past week, but not having a OSS release yet was the final blocker for getting into well-known repos. We'll probably have a post about this in the near future.
[0] https://en.wikipedia.org/wiki/Affero_General_Public_License
Hint: How did you (the developer) originally acquire a _copy_ of the AGPL'd sourcecode?
If you have a copyright license that adds a condition to the grant of permission, its hard sell to argue that the exception trumps the condition.
If you do not have a licensed permission from the author, you now have to prove that you are the legal owner of the copy. It could be said that they intended to address people who pirated a copy from a torrent site (or a BBS, since this is before the time of torrents), but its a major claim that someone can own a copyrighted work without, 1) purchasing it, 2) having a copyright license to it.
When it comes to GPL in particular, the question is also if the recipient is a owner or a licensee. The license text specifically call "you" to be equivalent with "licensee", which makes the claim of ownership even more dubious.
If not I have misread and quite a few other people have misunderstood as well I guess.
See for example MongoDB and others that distribute the core under AGPL and the drivers under MIT or something so that you can actually use it for something without having to make your product AGPL licensed as well.
Edit: did some quick googling and found this in another HN thread about some other nice but AGPL licensed software I've never heard about again:
"""The Opa web site says that I cannot: "Firstly, `‘if I'm using AGPL Opa to develop an app does it need to be AGPL, too?’'. Long story short — yes, it does." """ [0]
IANAL but being a nerd I have read a lot of licenses and it seems pretty clear to me that thats what companies has in mind when they license something under AGPL.
Unless someone can prove me wrong I hope we can stop spreading the misconception that AGPL somehow isn't viral.
https://www.gnu.org/licenses/why-affero-gpl clearly explains what the license intends as does the second paragraph in https://en.wikipedia.org/wiki/Affero_General_Public_License.
Based on what GNU says in the first link it seems kind of reasonable although I will not dare to use it without checking carefully with lawyers as this obviously this isn't what a lot of people think.
The only difference between the AGPL and GPL, other than the name, is that the AGPL requires you to offer to provide the source of your modified version of the software to all of its users. In short: network interaction equals distribution.
It says nothing about what licenses may apply to any software that interacts with the AGPLed software over a network. Anyone that suggests otherwise is misleading you or are mislead themselves.
Edit:
Note that MongoDB's drivers are MIT-licensed because they are made "part of" the client software. (A)GPL virality would come into play for drivers incorporated into clients.
Note that MongoDB's drivers are MIT-licensed
because they are made "part of" the client
software. (A)GPL virality would come into play
for drivers incorporated into clients.
That said it seems more than one company thinks it is viral over the net (Odoo, OPA comes to mind) and I would really be interested if someone who really understands licenses would explain.
If the network part of AGPL only affects the original AGPL software then it even kind of makes sense although I still wouldn't use it myself except as an extortion scheme :-P
Should you redistribute a modified copy, and not license your modification accordingly, the recipients do not get an implicit license to you modifications.
The only way your code will get licensed to others is when you explicitly say so.
If I write a program that connect to a database, my program will happy work fine without the database software. I could replace the database with a different one, and the program would work identically. Outside some database specific quirks, programs that connect to a database do so as independent works.
An other aspect is that if you change the standard library of a programming language, the programs that uses it will utterly change too. If you change the database software, most well written software won't be effected and will run identically as if it connected to a older version of the database.
And last, software that connects to a database won't share memory with the database software. Programs that uses a standard library will share memory with the standard library. The connection between a program and it's standard library is significant closer than the connection between a program and a database.
Edit: the PostGIS extension is GPL, and that license choice has been very successful. Hopefully the AGPL works out at least as well for Citus, I'm just not familiar enough to know what the implications will be in this context.
But then it doesn't really belong or aim to be in PostgreSQL proper, at least IMO, so I think that's fine.
The functionality (i.e. distributed processing and horizontal scaling) that Citus has done is something that I predict will eventually be part of standard PostgreSQL, but it will not be Citus's code (unless they change the license, of course).
I think the model could work in this case, both for the business and for PostgreSQL in general.
Companies of any appreciable size will be happy to pay for support if they choose to make Citus a part of their critical infrastructure. And the industry reached an inflection point where there are enough companies want as much of their infrastructure to be open source as possible, that you can run a company where most of your stuff is open source, while still making a ton of money (like RedHat, CoreOS, Docker, etc)
But a sign of getting investors is also that they see that there's still potential of making a lot of money, in spite of being open source.
That's true here too. Citus mentioned they spoke to their board (and presumably their investors too) about this change.
AGPL means the only people using it have to licensed the same.
AGPL only requires open sourcing any modifications you make to the software when you give users direct access to a server running the software, which seems like something you would never want to do in case of a database.
You can use database servers running AGPL software in a closed source SaaS: http://www.gnu.org/licenses/why-affero-gpl.en.html
Beyond that, we have a few other things in the works for the future that will cover other revenue models.
Several things have changed over the last two years that allowed us to make this happen: Most importantly, we've continued building out the product for a more broad user base, grown with more customers and users, received further funding as validation, and expanded both our team and product to offer additional revenue generating services. All put together, open sourcing Citus is something we've always wanted to do, and we are excited to continue building on it for many years to come, with the help of the community and our enterprise customers.
Can you publish competitive positioning of Citus vs Actian Matrix (nee ParAccel) and Vertica? I'd love to compare them side by side - even if it's just from your point of view :-)
First, Citus is not a traditional data warehouse. We position Citus as the real-time, scalable database that serves your application under a mix of high- concurrency short requests and ad-hoc SQL analytics (i.e. think both random and sequential scans for a customer-facing analytics app). The default storage engine for Citus is the PostgreSQL storage engine, which is row-based. This is in contrast to many data warehouses, which often use a column store and/or batch data loads, and are focused purely on analytics. The trade-offs you get are: - Citus vs. DWH performance: DWH and Citus both have a similar parallelization for analytics queries (multi-core, multi-machine), but most data warehouses typically use a columnar storage engine instead of a row-based one. Columnar storage is designed for faster analytics queries, so that makes columnar DWH generally faster on longer running analytics queries. However, this comes at the expense of (1) concurrency and (2) short-request performance (think simple lookups, updates, real-time data ingest) vs. Citus' row-based storage. If you've tried having 10s of concurrent connections to Redshift for short lookups, or performing 100s/1000s of inserts/updates to power your application, these limitations will be familiar. This is to be expected, as Redshift is not designed as a real-time operational database, but an offline data warehouse.
In essence, the two classes of products are more complimentary than substitutes, even while they have some overlaps in their analytic capabilities. Something like Redshift will give you fast offline analytics, after you move your data in batch (via S3); Citus will directly power your analytic apps in real-time; without ETL'ing your event/user data back and forth between separate OLTP and OLAP databases. Both can be extremely fast: Redshift can run complex data warehousing queries that take an hour in a few minutes, Citus can scan and aggregate 100 million records in a few seconds, while simultaneously ingesting your events in real-time.
I hope that provides some clarification on the workloads. There is a lot more, including columnar storage and product approach (re: implications of extending Postgres 9.5 vs. forking Postgres 8.x), and I’ll dive into those in separate comments as well.
Currently, my favorite is Vertica, but I do have concerns about its future under the stewardship of HPE.
I'm quite interested in what Citus brings to the market and will be following its progress closely. Once you have a more rounded story for the traditional Data Warehousing purposes, I can recommend it to my clients for evaluation purposes.
In terms of a sweet spot for you, here's a free tip for your sales: target customers of Unica (well, IBM Unica now). That's one application that would definitely benefit from your Operational Analytics positioning - lots of data ingested throughout the day, lots of queries to run for the analytics.
Precisely because cstore_fdw is built for PostgreSQL, and Citus is PostgreSQL (see Part 3), however, you can still choose to use cstore_fdw as the storage engine for your Citus cluster. Citus will still parallelize the queries as you'd expect it to, but instead of hitting row- based tables, they will hit columnar ones. cstore_fdw has certain limitations, importantly it is not updatable; so we don't consider it as an alternative to a data warehouse. Rather, it is useful if you are archiving your quickly growing timeseries / event data on PostgreSQL or Citus.
> cstore_fdw has certain limitations, importantly it is not updatable
Does it support insert, or are tables completely immutable once created?
To load or append data into a cstore table, you have two options:
You can use the COPY command to load or append data from a file, a program, or STDIN.
You can use the INSERT INTO cstore_table SELECT ... syntax to load or append data from another table.
Note: We currently don't support updating table using DELETE, and UPDATE commands. We also don't support single row inserts.
While this might appear as an implementation piece at first, it has important product implications, and might even impact how you might want to think about your database stack. By not forking the core database, you are choosing to always stay with the core PostgreSQL product. For starters, you get the uber-cool (and uber-fast) JSONB type that came with 9.4, or the recently checked in UPSERTs, or the popular PostGIS extension for geospatial capabilities. More philosophically, the moment you use forks of database, you know you'll be diverging over time. And when you introduce new databases and/or piece together many different ones to build one application, your development cycles will only get costlier and more complex over time.
This was a long answer to a short question, but hopefully useful. Let me know if you have questions, or any feedback using Citus – would love to hear your thoughts!
If the data is big and I need to run analytic queries then I think I have to use a columnar storage format because row-oriented formats cause too much overhead for aggregation queries that usually need to process single column efficiently. If I use CitusDB as an analytical database, then it's comparable with Redshift, Hive etc. As you said, they're suitable for offline data but Can I use cstore_fdw in CitusDB and able to take advantage of real-time nature of Postgresql? Maybe I can push hot data to a table that use row-oriented format and move the data periodically to another table that uses cstore_fdw and execute queries that fetches data from both cold storage and hot storage tables? If CitusDB makes it easy for me, then I think this is huge.
I guess another use case is using CitusDB as distributed data store and executing filter queries such as "SELECT * FROM table WHERE partition_key = x and predicate1 = y ...". Instead of using multiple Postgresql instances and routing the queries in application level, I can just use CitusDB that takes care of replication && query routing && sharding etc. I think it can also be comparable to databases such as Cassandra, Mongo (using jsonb) since they also have similar use-cases.
Or should I think CitusDB as distributed Postgresql?
> If I use CitusDB as an analytical database, then it's comparable with Redshift, Hive etc.
A particular difference is in response times and concurrency. Data warehouses and Hive are great for reporting use-cases, but not for use-cases that require fast responses and have many users like analytical dashboards. This is a use-case for which Citus is particularly well-suited (see for example the CloudFlare dashboard).
> Can I use cstore_fdw in CitusDB and able to take advantage of real-time nature of Postgresql?
Yes, since cstore_fdw and Citus are both developed by Citus Data we made sure they're fully integrated. We've even seen some deployments that use a mixture of columnar- and row-based storage in a single distributed table.
We find that row-based storage generally has better ingestion performance and more indexing possibilities. Citus can do very fast execution of analytical queries by parallelizing over row-based shards and using the indexes on each of them. However, if you only need a small number of columns and have analytical queries that are not very selective, you can use columnar storage just as easily and even mix and match (might require some support).
> I guess another use case is using CitusDB as distributed data store
Yep, Citus can definitely be used for that by using hash-partitioned tables.
This would be complimentary to what Citus does, which is distributing the load across multiple shard instances (each with their own cores, benefiting from the parallel work in 9.6).
Some Postgres committers work on Citus as well (e.g. Andres Freud), so I'm sure this has been thought through before.
It's a good product, and it was even fairly easy to do a major version upgrade / cluster relocation. At least as easy as such a thing can be. :-)
Your queries tend to break down into ones where you're hitting a small number of shards (such as when we serve data for the analytics page), or else ones where you tend to aggregate up into temporary, non-distributed tables for further analysis (such as you'd do for infrequent business reporting).
All told, it's been one of the easier tools for us to operationalize, largely thanks to the fact that it's "just PostgreSQL." The one thing I wish we had was better documentation of what kind of consistency guarantees to expect, although for an append-only store like our current use case, that's less of a concern. And to be fair, Ozgun and the other guys at Citus have always been really happy to answer any questions we have.
With this code now going open source, it should be pretty easy to look into these sorts of internals.
It looks good, will have to wait and see what actually makes it into 9.6 and how well it supports all the possible queries. This is something that many commercial databases have solved so Postgres is pretty behind in vertical scaling.
One thing I'm having trouble with is finding information about transactional semantics. If I make several updates (to differently sharded keys) in a single transaction, will the transaction boundaries be preserved (committed "locally" first, then replicated atomically to shards)? Or will they fan out to different shards with separate begin/commit statements? Or without transactional boundaries at all?
In fact, I can't really find any information on how CitusDB achieves its transparent sharding for queries and writes. Does it add triggers to distributed tables to rewrite inserts, updates and deletes? Or are tables renamed and replaced with foreign tables? I wish the documentation was a bit more extensive.
"for customers with large production deployments, we also offer an enterprise edition that comes with additional functionality"
Since I heard last year at PgConfSV that you will be releasing CitusDB 5.0 as open source, I've been waiting for this moment to come.
It makes 9.5's awesome capabilities to be augmented with sharding and distributed queries. While this targets real-time analytics and OLAP scenarios, being an open source extension to 9.5 means that a whole lot of users will benefit from this, even under more OLTP-like scenarios.
Now that Citus is open source, ToroDB will add a new CitusDB backend soon, to scale-out the Citus way, rather than in a Mongo way :)
Keep up with the good work!
To use open source code, the more permissive the license the better. But to actually open your own code, BSDL is a very tough sell.
That's also why they use the AGPL. With database systems, even if they were under the GPL, some competitors could just modify the system and run it on their own server with improvements, and offer just the service to their clients. Again, the improvements go one way only: since the competitor would not distribute the modified system, as it's running on their servers, they would not need to distribute source changes. With the AGPL, that loophole is closed.
They are of course free to release their code under any license they wish. I just think releasing code under the *GPL when you profited from a liberal BSDL is a douche nozzle thing to do. But knock yourself out! This tells me all I need to know about the company.
Would CitusDB have been created at all were it not because they could sell a propietary fork as they have been doing in the past?
I'm a GPL zealot, to the point that I've used the GPL as a weapon and as a shield against others in multiple capacities ("You own all my code when you employ me? Sure, as long as I get to dictate the license"). However, I would never take someone's 2 or 3-clause BSD-licensed product, and relicense. Those of us that value sofware freedom value the rights of other licenses that believe the same.
We all see why it was done in this case, however; In order to ensure software freedom in the cloud (someone else's computer), there isn't another license to use. the BSD license completely breaks down in this use scenario, and the best we have is AGPL.
I'd say cloud usage is to 3-clause BSD what Tivo was to GPLv2.
If this were true then Cloudera, Horton and a whole bunch of other companies would be out of business, yet in reality they are doing really well. All that AGPL is doing for Citus is:
1. Turning away people (customers) who are religious about licenses.
2. Eliminating any possibility of this code ever being integrated into PostgreSQL
My $employer paid Hortonworks for a support contract and I have no qualms declaring publicly that it was a total and utter scam. (We are a Java shop and know our shit)
If I go into too much detail I'll write countless pages like my internal report on why we needed to switch, but the bottom line is that Cloudera's products are well and honestly documented, while, as of last year, Hortonworks' products are simply one land-mine after another.
Their management platform (Cloudera manager) being closed-source is barely a mark on the comparison analysis when you compare it to Ambari in practice. Ambari is a bad joke and I'd rather do without it after spending significant time using it and trying to extend it.
And I could go into excruciating detail as to how Hortonworks abuses the Apache License to try and force lock-in. It's disgusting and pathetic.
Cloudera also makes (Apache Licensed) Impala, which IMHO is a pretty cool product.
Another company worth mentioning is Databricks, which leads Apache Spark development.
https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...
I work for a BI consultancy and we don't even bat an eye until we hit billions of records in a primary fact table.
Certainly the DB server does need to scale vertically to some extent as you pass through the orders of magnitude > 10M. A good columnstore engine is also worthwhile to consider.
https://www.citusdata.com/blog/15-marco-slot/402-interactive...
"For customers with large production deployments, we also offer an enterprise edition that comes with additional functionality"
Software isn't free to produce, and the need to make money off software isn't something companies should be ashamed of. In fact, nowadays I'm leaning towards trusting OSS with clear financial sustainability over software whose long term existence seems shaky.
I use a major open source system with enterprise features and support but don't pay for any of those options. I've used it for 3 years and it's been invaluable. No pressure to start paying for anything. Some of its premium features have actually become free over that period. But I wouldn't decide that all open source systems with premium features are safe based on that experience.
Yes, that's what you're seeing right now, but in the past Citus (used to be "CitusDB") was a superset of the entire PostgreSQL codebase. During the lead-up to the open source release, we removed the use of any static methods or internal machinery and rewrote the installation process to use the PostgreSQL CREATE EXTENSION command. Additionally, we moved all of pg_shard's DML functionality into Citus to unify the product line.
So ultimately CitusDB was a fork but is now entirely an extension.