The Apache Cassandra battle highlights major problem with open source projects | Hacker News Reader

The Apache Cassandra battle highlights major problem with open source projects

techrepublic.com

79 pointsCrankyBear45 comments

This is a curiously vague article. Apparently there were issues with Cassandra's project leadership not addressing trademark issues, causing the Apache Software Foundation to remove Datastax from their leadership position. Um, ok? The author here seems to be sowing FUD about the future of the project, and by relation, all ASF projects. Anyone with knowledge of this situation care to chime in?

This is not quite what happened.

Datastax employees were the controlling influence on the PMC, and the PMC failed to police copyright issues (including with Datastax marketing materials), mostly because they did not know this was their duty (and partially, in at least the case of some individuals, because they did not have any desire to participate in this duty on a volunteer basis).

They were also seen to have poorly nurtured other involvement from the wider community.

To some greater or lesser extent these charges were valid, and the community (not just the board, at least on the non-trademark issues) felt improvements would be welcome. But certain members of the board behaved in what appeared to be a very hostile and childish manner, by threatening Datastax with removal of every employee from the project. (I've been in heated arguments with these individuals for their behaviour, and I'll note they disavow any intended hostility, though do not disavow their actions)

The ASF did not remove Datastax from anything, although they were I think involved in Jonathan Ellis (Datastax CTO) being replaced as chair.

Datastax recently posted that they were planning to put much of their new feature development into their commercial version. They claim this all had nothing to do with stepping back from the project, but in all likelihood this is a "rising above it" PR position. That said, it probably is a good business decision, and may have happened sooner or later anyway.

Reading this apparently biased article in favor of DataStax, it seems like no one is coming out of this situation looking good. I personally believe that DataStax may have overplayed their hand here since several of the high-powered Silicon Valley companies use Cassandra. If DS' commercial offerings aren't up to snuff, one of them will undoubtedly step in to help steer development of the source in a direction that benefits them.

Cassandra is one of the better big datastore offerings out there, and the doom and gloom narrative played by the article seems a bit disingenuous to say the least.

There's a difference between use and pay the bills to develop; fundamentally, the companies paying the engineers and testers and hardware bills thereof will control the direction of the project regardless of what the pmc wants. Or the company footing the bill for all the documentation and code samples [1]. It is also a little weird that java driver development seems to be run by DataStax and not the apache community. Hopefully this won't cause datastax to fork from apache cassandra.

[1] http://www.datastax.com/2016/10/take-a-bow-planet-cassandra

I honestly don't see how companies, that aren't monetising Cassandra, providing resources to the project is a major risk to Datastax. This only improves Cassandra's mindshare in (and share of) the market, while leaving them to corner the large corporate client base as well as target their engineering efforts on things that can directly yield revenue.

Although I think you overestimate the inclination and ability of these corporations to invest - at least to date, only Apple has demonstrated any capacity or willingness to do so, and they are fairly slow about it. Their lawyers and antiquated deployments lead to very few (but quite big) improvements, and so far all of them have needed help from Datastax to be incorporated. This is despite a largeish team of people with direct experience of participating in the project; no other SV company has any such employee at present, and bootstrapping such a team would be non-trivial. Instaclustr claims they intend to do this, but it remains to be seen how successful they will be.

I've committed a few Instaclustr patches in the past month - they weren't huge, but they exist (and I appreciate that).

Also some patches coming out of FB/Instagram (with a new committer as well).

> the doom and gloom narrative played by the article seems a bit disingenuous to say the least

Would you have read it if it said "a bunch of nerds argued about nerdy things on the internet, and in the end, everything will be OK" ?

Nothing is stopping Datastax from creating a non-ASF open source fork if they want, right? The apache license is quite permissive.

Datastax sells Datastax enterprise, which is a closed source fork.

Yes, they could have a non-ASF open source fork, but what's the purpose? What do they gain? That just fractures the community - the same community they need in order to sell DSE to users.

They would gain control over the management of the open source project, which is apparently what they wanted? If they don't want an open source version they have control over the management of, then that's fine too of course.

But yes, forks of open source projects are never something one wants, they are a last resort, but one you always have with open source.

Apparently what they wanted? There's no evidence they ever tried to use their overwhelming majority in the PMC to do anything that even remotely resembled control. What they did was fail to enforce rules, primarily around trademarks - they never did anything at the technical/engineering level that was even remotely controlling (and in many cases, they openly worked with engineers from their competitors to make it easier for those competitors to avoid forking).

- non-Datastax Member of the PMC

True, I over-stated it, sorry. I didn't mean to suggest they were controlling. All I meant was that apparently they don't want anyone else telling them what to do with the project, including with rules around trademarks?

I'm curious what impact this will have on other ASF projects, specifically Mesos. A number of the features required for a smaller or medium sized organization to actually use Mesos in production are only available in Mesosphere's DC/OS. And whenever those shortcomings are brought up, the answer is "oh well if you really need authentication beyond htpasswd, you can pay for DCOS." Like the article hinted at, I hope the ASF tries to do more to work with vendors to help OSS side of projects flourish rather than just be lip service.

If you contributed a reasonable patch to Apache Mesos, implementing a feature like enhanced authentication, and then that patch was reject with such an excuse, that would be a problem.

But Mesosphere choosing not to contribute specific features back is their choice. The long term belief in a healthy community is that enough people will contribute patches that make the project better.

There are many, many ASF projects that are healthy.

Take HBase... A thriving community, committers and PMC members from dozens of organizations, no open source vs commercial interest collisions.

Projects backed by a single commercial entity are suspect, but can work fine as well (Kafka for example).

Disclaimer: Apache HBase committer here.

Cool, do you see HBase being used in prod? (Outside of Google that is). Who's using it?

It's used pretty extensively at Adobe.

Google doesn't use HBase. Facebook does use it for Facebook Messages I believe.

I heard Flurry has something like 2k+ nodes. Though post Yahoo acquisition, who knows what's going on.

My company does quite unfortunately; we don't really have enough data to make it necessary and it was justified w/ vague notions of scaling blah blah blah.

Hbase is approaching a decade of age. It is well understood, rock solid, and pretty much ever error is googleable. There are numerous companies that run it in production. Hbase may not get the press Cassandra does because of it is not as easy to setup due to some external dependencies (HDFS and Zookeeper). But its very much production ready for some epic data sets.

Facebook uses HBase pretty massively.

https://code.facebook.com/posts/321111638043166/hydrabase-th...

There is an interesting article about HBase vs Cassandra et al here: http://www.infoworld.com/article/3066358/analytics/hbase-the...

I cannot validate or refute any of it. Hoping some here can.

I've seen hbase used in production at a number of startups, and worked at one where it was used. It's very powerful and flexible, but it's also not like running any other database. Setting up Cassandra is like "java cassandra.jar". Setting up HBase is...someone's job for a while is going to be "running HBase".

Anyway, I'm talking about small-ish startups that aren't super researchy technological trailblazers. There are definitely people outside of microsoft (I'm assuming that's what you meant instead of Google as Google already had BigTable and HBase started out as a Powerset project and so ended up at Microsoft) running HBase in production.

Facebook, Salesforce, Google (just the HBase client APIs, though), Flurry (now Yahoo), Yahoo, Apple (Siri), Pinterest, Twitter, Rocketfuel, Huawai, SiftScience, just to name some.

I work at one of the above :) where we run many, many thousand HBase servers.

> I'm curious what impact this will have on other ASF projects

It might cause those committing the code, building the tests, and writing the doco for other ASF projects to question whether their project is being justly managed. I only watch one ASF project, Apache Groovy, and it has a similar imbalance between fluff managers and grunt workers.

Although all 9 members of its PMC are committers, only 4 of them have ever committed code since Groovy joined the ASF via its incubator 18 months ago. The other 5 (i.e. chairperson Guillaume Laforge, Jim Jagielski, Roman Shaposhnik, Konstantin Boudnik, and Andrew Bayer) are all committers but have never committed any code in that time. I can't find any other technical contributions from them either. All 5 are also ASF members, whereas only 2 of the other 4 PMC members who do grunt work are ASF members.

There's 10 other committers in Groovy, most of whom are more active contributors than those 5 I named, and in my view have far more merit to be on the PMC than them. OCI, the company doing Grails consulting, a few weeks ago enlisted an active PMC member (Paul King) as an OCI consultant "to coordinate contributions to Groovy". If Groovy's governance at ASF doesn't radically change fast to give PMC voices to the people doing the actual work instead of those ASF politicians, OCI might tap into the discontent and "take over" in similar style to DataStax, or even fork it like LibreOffice.

Sometimes members of the PMC are on the PMC to help committers do things in the Apache way - Jim (in particular) is a member of the ASF board and likely mentored the project early on (I'd go check to be certain but I'm mobile at the moment).

Because it's 18 months since Groovy joined the incubator and 12 months since it became a top-level project, it could be time Jim Jagielski and all ASF members not actually contributing to Groovy leave its PMC so that the proportion of those actually committing code and similar goes up from its present 44%. OCI is ready to pounce if they don't.

The PMC can (should) invite active members of the community - there's no fixed size, the PMC can add members at any time.

There's no need to worry about ratios of people committing code - the PMC is about guidance, not about writing code.

I believe Apache Groovy implementations should be led by their technical people, so its PMC should only have people in it who can and do code, test, and write docs. Groovy's past problems have been due to managerial sorts running things.

Do you have a personal vendetta against Groovy?

That comment is from a newly-created anonymous user responding with the word "personal" to a roll of facts and figures by someone who's real name and photo can be found by following the user link. If anyone (even someone anonymous in HN tradition) wants to challenge the facts I presented, present other relevant facts (not speculations on motives), and show how a different conclusion can be drawn, then I'll happily reply to them. I believe the Grails consultants at OCI will try and take over Apache Groovy, DataStax-style, irrespective of my own personal feelings on Groovy.

I'm a lurker and created an account. I have worked with Groovy/Grails for the past five years at multiple large corporations as well as one state entity (both employment and contract work). I was actually surprised on how many companies use it. Also, I have never heard anyone with such a negative response to it. You can take off your tinfoil hat now.

I've never worked with Grails, but used Groovy a lot from about beta-10 to v1.8, though have since moved on to other languages, even trying to bring Groovy-like ideas to those ecosystems, most latterly Go.

I'm positive about Groovy's widespread use for builds and scripts on the JVM, similar to the way Bash is used on Linux. It doesn't get used much for building actual systems, even though some of Groovy's project managers tried to repurpose it for that. I've long had concerns about Groovy's governance, and believe that's the reason for its problems, e.g. Pivotal, OCI.

I have a feeling Apache leadership probably did the right thing here. It is after all, their brand. Recently there have been some strange things happening with the Cassandra project. Hopefully not another rethinkdb.

For example, a longstanding Cassandra bug makes it hard to build the database from source if you have a modern version of maven installed. Though Cassandra builds itself using ant (why?), simply having modern maven installed is a big enough incompatibility to break the build. That's weird.

Another recent weirdness is that Cassandra was shipping a non working command interpreter (cqlsh) for at least 3+ months as recently as a few months ago. That's now fixed in v3.9. I have to admit, the first thing I thought upon seeing that brokenness shipping in the binary distribution was - Is this because Datastax needs more sales?

Can you link the jira for the mvn/ant bug? Or, if it doesn't exist, can you create it?

Maven has a notorious reputation of behaving differently (especially with dependency resolutions) in newer versions of its release which has lead to build failures in some of the projects that I have been involved in. So it may not really be Cassandra's fault here.

Sharing is hard. Nobody making money can work. Many people making money can work. Just one party making money, that's a mofo.

Full disclosure: I work for Couchbase, and our management tends to describe DataStax as a key competitor.

I've been a Cassandra user for 5+ years and have been watching the drama unfold on the mailing lists the past few months. I have no business relationship with Datastax and have never paid them money for any of their services (except for attending the Datastax sponsored Cassandra Summit a number of times). I'm not a committer but I have been an active member of the community opening bugs, participating in the user mailing list, Stack Overflow, maintaining a client library, etc.

From my perspective it seemed like the ASF board members had already prejudged the situation before engaging in any public discussions on the Cassandra mailing lists. The first interaction was about why Cassandra doesn't ship with its own client drivers[1]. The board member believed this to be a sign of Datastax "controlling" Cassandra because they have their own high quality open source client libraries. As a user and a former maintainer of a client library, this theory came off as completely ridiculous. The project has never shipped a production ready client driver in tree. It is simply impractical to be able to do that for all the languages for which there are drivers. I've got code in three different languages in production right now and none of them use Datastax drivers. If you read that mailing list thread, that is basically what everyone says on it (including the new Casssandra PMC chair), and yet the conclusion that the ASF board member takes away is that Datastax is controlling the project by having out of tree open source client libraries.

That ASF board member also got upset when a question on the mailing list was answered with a link to Datastax hosted documentation[2]. Again this seemed like an overreaction from the board member over a fixable problem as justification for a Cassandra shakeup.

Then recently there was a new wave of threads about Apache and Datastax[3][4]. Both sides come off looking pretty bad in these threads. Its clear that at this point a lot of people in the Cassandra community have negative feelings toward certain members of the ASF board. It also seems like those board members have become very defensive about what has transpired. Fortunately people have started to calm down and there has been a few reasonable emails from both sides.

From my perspective the ASF came off looking like they care more about the Apache trademarks than they do about the health of the community. I do believe there were some actual issues that were identified as places for improvement for the Cassandra community, but I don't think it was necessary to come in wielding a sledge hammer. It seems that the Cassandra board was not given an opportunity to fix the issues (mostly by increasing the number of project committers) before the board forced the changes. This has resulted in a lot of hurt feelings and a general sense of distrust between the Cassandra community and the ASF.

All that being said, I'm not particularly worried about Cassandra's future. I was sad to see Jonathan Ellis leave as the chair person, but hearing that Nate McCall was stepping into that position put a lot of my concerns at ease.

I hope that the Datastax folks know that the community appreciates all that they have contributed to the open source Cassandra project, even if the ASF doesn't.

[1]: https://lists.apache.org/thread.html/72a884fa8f35cbed23135c8...

[2]: http://www.mail-archive.com/dev@cassandra.apache.org/msg0914...

[3]: http://www.mail-archive.com/dev@cassandra.apache.org/msg1003...

[4]: http://www.mail-archive.com/dev@cassandra.apache.org/msg1002...

I feel like I've sent more than enough emails on this subject already, so rather than nit pick tiny things in your comment with which I disagree, I'm just going to say that I agree with your closing sentence.

Thank you for your general civility on the lists. I'm glad that you are on the PMC.

This isn't a "major problem with open source projects," it's a problem with the governence model of one specific project, and a conflict between the ASF and the company holding a controlling majority on the board.

Other than "governance is hard and politics sucks," I'm not sure what the "major problem" actually is

> a problem with the governence model of one specific project,

I think multiple (not many, nor all) OS projects have a leading vendor supporting the codebase in various ways. For poorly adopted or niche OS projects, hardware, computer languages, etc, this can be a deathstroke to the project.

Yeah, but they were saying it was a major flaw with all of OS. It isn't. In fact, if your project isn't incredibly popular, this likely won't effect you.

I haven't followed this specific Datastax vs ASF discussion, but have seen similar battles played out in recent years between some popular projects and ASF and even ESF (Eclipse foundation vs Vert.x project for example)

Does anyone here know what value do these foundations bring to projects, in this day of cloud hosting/computing that make these foundations worth for the projects?

Yes Open-Source projects can have political issues too. I don't see how this is any different than the politics of closed source software. Why the author wants to disparage Open-Source is beyond me.

Reply on news.ycombinator.com