The Drupal Crisis | Hacker News Reader

The Drupal Crisis

unleashedmind.com

266 pointsddw102 comments

The reality is that this sort of bloat happens in every mature software project. Some of it will get fixed, some of it will persist. You take the good with the bad, but at least there IS a community behind the project.

However, what Drupal has going for it right now is critical mass. It has finally gained widespread acceptance with government agencies and large companies... it's an exciting time to be part of the community, and to a large degree this can be attributed to having Acquia really champion the project with enterprise clients. No other company was previously capable of doing this. Because of this singular fact, Drupal will remain relevant for many years to come.

The only reason this sort of discussion is even able to happen is because the Drupal project is open source... can you imagine how much shit is buried deep within every single product OpenText, for example, maintains? Have you ever tried to do anything truly custom with Microsoft Sharepoint?

Let's try to fix the problems, but let's also recognize that Drupal has a lot going for it at the moment.

One thing the Drupal community has no shortage of is cheerleaders.

Dries keeps going on about something Koolaid related... not sure what he's talking about, but it sounds tasty!

;-)

> However, what Drupal has going for it right now is critical mass.

http://www.google.com/trends?q=drupal%2C+joomla%2C+wordpress

I'm not sure how much we can infer from the above, but Drupal has been about the same since 2009.

While i'm loathe to take the "Drupal Cheerleading" side, Google search volume and platform adoption are definitely two different measurements. The number of globally installed Drupal sites has grown dramatically over the 2-year period you mentioned. http://drupal.org/project/usage/drupal lists the number of sites that are actively pinging back to the Drupal.org servers over time, and the project lead maintains a crawler that attempts to count the number of sites that actually run Drupal, regardless of their "phone home" status.

Those numbers have been growing consistently, in some cases jumping by as much as 50% per year. That doesn't mean that Drupal is great software, or that it will continue to grow at that pace, but it's absolutely not flat. In fact the author of the original blog post is pointing out one of the significant problems that comes with growth: the number of people using and building sites with Drupal has grown much, MUCH faster than the number of people who actively build and maintain the software itself.

I was referring primarily to enterprise adoption - there have been quite a number of high-profile wins recently in terms of Drupal finding its way into companies/gov agencies that previously would have used proprietary software like Sitecore, Sharepoint, etc.

But yes, the total number of installed sites has gone up dramatically as well. I guess the side benefit to demand outstripping supply of talent is that it's a great for companies who provide Drupal-related products and services. Every one of Acquia's Enterprise partners currently have more work than they can handle.

This isn't limited to Acquia's enterprise partners. Every freelancer I know with Drupal in their portfolio is turning down gigs left and right because they're out of capacity.

Most shops I've worked with have given up on finding out of the box talent and have started hiring developers and training them on Drupal.

All of the recruiters I've talked to in the last eight months that where trying to fill a Drupal-related position where tearing their hair out trying to find loose talent.

In short, there is HUGE demand for Drupal talent right now.

I'm actually seeing a shift in the marketplace to more Wordpress projects than Drupal.

What segment of the marketplace are you referring to? Corporate clients, large nonprofits, government and .edu doesn't seem to be using wordpress for much.

This is probably true on the small end of things, and I've had discussions around Drupal vs. Wordpress in enterprise-ish environments, but Wordpress is still not a serious contender in the enterprise space.

I'd be interested in talking to the folks at Automattic to see where they're trying to take things though. Seems like everybody wants to play in everyone else's space. I have a sneaking suspicion that by the time all of the major open source CMS projects are on an even keel the target will have moved and frameworks will be "the thing".

>>Have you ever tried to do anything truly custom with Microsoft Sharepoint?

Yes. Sharepoint is buggy as hell.

The whole cms product category is in crisis. Developers would rather use frameworks, and testable deployable code. The inexperienced user wants something very simple, but secure, so they will chooses hosted solutions, wordpress.com hosted wordpress is now 50% of installs. Other cloud solutions will start to grow. The commercial cms market is a mess too, too many products, mostly very old architecturally, and in terms of code. And then everyone has their own in house solutions. Even the rise of statically hosted git based blog platforms shows signs of the problems.

don't forget that Drupal is both a CMS and a framework

So are lots of cms systems, like typo3 and silverstripe. But almost no one uses the frameworks outside the cms. The product category is not really well defined architecturally any more.

I started with Drupal around the time version 4.7 came out, and I always thought that the project would have done better for itself by focusing more on its potential as a framework rather than a CMS.

Drupal has always seemed to have something of an identity crises. Some people think of it as a fully-featured CMS, some think of it as a framework or platform, and then you have a ton of people just trying to use it as a more flexible blog.

I think that when the community exploded in size, it had only really begun to wrestle with the identity question. The lack of some definite focus as Drupal's popularity expanded probably contributed to the current situation.

Frankly, I think that the majority of the current problems would be solved by trimming all of the fat from the core. That would allow the core developers to focus on making the core fast, stable, and flexible. Let the wider community maintain and provide any extra functionality.

>> Drupal has always seemed to have something of an identity crises. Some people think of it as a fully-featured CMS, some think of it as a framework or platform, and then you have a ton of people just trying to use it as a more flexible blog.

This has been my take on it, too. (I've been developing Drupal sites for clients almost exclusively for the past 4 years.) The confusion persists because Dries Buytaert believes it's possible to be all things at once. In his keynote at DrupalCon SF last year, he brought up the question of whether to focus on becoming an easy-to-use CMS for people who don't want to code (competing with WordPress) or being an awesome platform for developers, and answered it by saying "both". (My apologies for paraphrasing.)

Without any clear direction on this, the core definitely becomes vulnerable to bloat as there's no clear model with which to approve and reject features.

That said, is Drupal truly in a crisis, in the sense that the project will die soon if the core maintainers don't act immediately? I'm not so sure. I don't feel like the slow uptake of Drupal 7 has as much to do with bloat and half-baked features (of which there are a few, like Dashboard, but not an overwhelming amount.) Rather, the problem is with a few important contributed modules still not being ready for prime time. In particular, modules for nodereferences and breadcrumb management are still in flux months after Drupal 7's release.

In time, those will catch up too and Drupal 7 will likely have a few if not several years of good life in it. That's a long time in the life of a Web site, and hopefully long enough for Dries to realize that he needs to step up and make some tough decisions about which direction he's going to steer the community toward.

I agree and I believe if they simply picked a direction, be it a CMS or Framework, it would indeed trim out a lot of the fat from the core like you mentioned.

The fact they're trying to be all things to all people is what will likely cause the project to fail. Which is unfortunate, since I do like using it.

>> Frankly, I think that the majority of the current problems would be solved by trimming all of the fat from the core.

The challenge, of course, is that everyone has a different definition of "Fat" versus "Muscle." The developers who actively maintain the codebase are a tiny, tiny percentage of the overall community that uses it. Many of the features that are frustrating and annoying to maintain are popular among the non-developers who use Drupal for site building.

Balancing the usage needs of those people with the maintenance woes od the developers is one of the big challenges.

The other is dealing with the identity crisis you describe; the lack of willingness to focus on a manageably small set of use cases or target audiences became a real problem once Drupal's growth curve started swinging up around 2007 or 2008. Now, simply declaring that things are "more focused" will orphan large numbers of users who came on board and started doing their own thing with Drupal while it was less focused.

It'sa conundrum.

"Frankly, I think that the majority of the current problems would be solved by trimming all of the fat from the core."

There are 100's of projects who proclaim to do so, and all of them remain obscure. By having a core that does, well, nothing (no blog, forum, commenting, news workflow, permissions, ... systems), as a developer you sill have to go hunt for 'modules' of which you usually have no idea how well they work and if they'll be maintained next year.

In theory the 'plugin' system is nice, in practice it turns into the situation where the 'core' rolls on like a freight train, leaving 'plugins' by the wayside, and users scrambling to patch together the known working versions and plugging the holes left by the incompatible versions. In the end, you still have to do a bunch of programming - but oh, before you can, you need to spend weeks to learn all the ins and outs of how to do so.

The answer is in slowing down development speed. Yeah it sucks because you can't work on cool new stuff, but maintaining backward compatibility and having a large base of usable modules, that are actually vetted by a larger group than just the one or two people who work in that module and only use the bleeding edge version themselves anyway is the only way to build a platform that people can rely on being there new year, and the year after.

Having supervised or been involved in the build of about 300 Drupal sites I can say:

Small Core!

Seriously, since I didn't have to use Drupal I haven't in new projects. I found that the conflating of configuration in the database (at least in D6) made true test-driven development and agile maintenance a chore. It made good development processes more difficult than necessary.

Drush went a long way toward fixing this, as did Features and a few others, but it always felt like it was stuff added to Core to remove stuff from Core. That always felt clumsy to me.

I owe Drupal a huge amount in terms of how its use helped what I did and the artists we did it with, but the project, as most do, has grown too big at this point. It needs a reset and a focus on deployable quality, scalability and additive functionality rather than subtractive.

Is "small core" a movement to decouple the framework from the product?

Indeed. If you've ever worked with Drupal in large scale production environments, the need for this is abundantly clear.

As for scope, we were launching at times 5 sites a week.

Ah. Yeah, I've worked with Drupal for 5 years or so and have been baffled by the orthodoxy that framework and product must be the same thing. My guess is part of it might be political: it would be harder to centrally control two projects rather than one.

Probably more like String Theory. An idea that there must be a singular product that can satisfy everything at once. (We could even call if the Master Control Program! ;)

Note that the original "Drupal Crisis" blog post is already a few weeks old, and a lot has happened since then.

For example, the author wrote this shortly afterwards: http://www.unleashedmind.com/en/blog/sun/crisis-conclusions

You'll want to read that (and especially the various drupal.org issues linked to within it) for a more nuanced understanding of the situation.

Otherwise you are not getting anything remotely resembling an accurate view of the larger Drupal community's thoughts on this topic.

Number of open bugs is a poor measurement of software quality. Anyone who has seen the bug-tracker of a large project will testify to that fact. If anything, a large number of open bugs is an indicator of a large, popular and mature product.

Mozilla's Bugzilla currently has 46951 open bugs within Firefox and Core.

(Disclaimer: I know little about Drupal. I'm only responding to the argument itself, which is a fairly common fallacy)

It might be a poor measurement of quality in general, but in Drupal's case it speaks to the growing pains Drupal core (and Core development in particular) is going through right now. Sun's post is specifically about new and untested/unproven UI modules that were forced into Core by Dries, that have added to the burdon for Core developers, as well as the increasingly complex (and very often poorly documented) Core code base that is making it very hard (or impossible) for new Core developers to get going.

One key thing to keep in mind here with regards to Drupal is the release cycle for Core. The current release (7.x) took three years to come out, so when things get shoved into Core it takes a looooong time to take them out, or even to improve their underlying APIs. This is as opposed to Contributed Modules (Contrib), which is where much of the power of Drupal is, and which evolves at a much faster rate than core (for example- Views went from 1.x-2.x-3.x in the time it took to develop Drupal 7).

So yet another rant bemoaning Drupal's internals wanting a more elegantly decoupled internal architecture. Decoupling Drupal's internals in the ways suggested would be no easy feat and a fool's errand to even try. What's the objective? "So we can have more reasonable unit tests." That's sort of hilarious. Drupal is not exactly a test-driven culture. "So we can reuse these components in other frameworks." Seriously? You want to reuse Drupal's theme registry some place else, like wanting your bike, your car, and your golf cart to have the same size wheels because that'd be more convenient for you?

Drupal project leads need to avoid getting distracted by this noise. Drupal is not and should not try to be an architectural masterpiece because the typical Drupal adopter has no idea what a unit test is anyway. Addressing these complaints will add no value for the typical Drupal user. Drupal is utilitarian. Drupal fulfills the need to slap together a web site quick and make it pretty, and you don't want to know what's going on underneath. If you want architecture and unit tests then there are plenty of other frameworks out there to consider instead.

>Drupal is not exactly a test-driven culture.

Yeah, the bug count for D7 is consistent with that.

>the typical Drupal adopter has no idea what a unit test is anyway

Developer accessibility has long been an excuse used to justify bad architectural decisions.

Did you read the same article I did? He wants to revamp the architecture because it's horribly unmaintainable and is burning out core developers.

The answer to that is obvious. We just need to replace them with core developers that have more durable filaments!

Of course. Why didn't I think of that?

No, what's burning out core developers is the endless architecture debate.

Perhaps you should listen to the core developers who started this debate, when they explain what's burning them out? For better or worse, architectural decisions in Drupal's past have resulted in very large workload for a relatively small number of developers, and great pressure from others in the community to "Keep the system running" in the face of numerous feature requests and so on.

There's a difference between "Let's turn Drupal into an architectural masterpiece" and "Let's refactor some of this stuff to make it more manageable." Drupal has long had a culture that devalues planning and deliberation while glorifying the "firehose of code." Attempts to iron out the best path for a complex change can only go on for a few days before they're shut down with cries of "Talk is silver, code is gold!"

Obviously, the other extreme -- endless unresolved attempts to come up with the "perfect master plan" -- is just as unproductive. But there's a healthy middle ground that needs to be maintained for a project of this size to avoid crumpling under its own weight.

There is not a single mention of either unit tests or "reusing components in other frameworks" in the original article. Did you read it?

To get a better context read the discussion threads linked from this article, which inevitably lead back to the "small core" discussions. The Drupal project has been increasingly distracted by debating fantastic hopes and dreams, and there's a definite sense of immediate issues not being addressed because everyone is waiting for the great architecture debate to be settled first, which it never will.

You're saying, because the typical end user doesn't know what tests are, it shouldn't be a high priority to have the core platform testable?

> Drupal fulfills the need to slap together a web site quick and make it pretty...

You had me until that. Usually you can tell a site is Drupal by how ugly it is and how awful the code is. Also, putting together a site in drupal is far from quick, unless you've been working with drupal for years.

Reminds me of Jeff Eaton's talk at Drupalcon this year (http://lanyrd.com/2011/drupalcon-london/sgkbz/).

Personally, I think Drupal needs to be forked already and stripped of its non-essentials. As much as I like the Drupal community, it's always felt extremely rigid and even more so when Drupal 7 was finally released.

In one of the Drupal Camps for the release of D7, more often than not I would overhear PMs and developers saying things to the extend of, "I will not touch D7 or its documentation until 6 months from now." It's been around 8 months now since then, and not once have I had to deal with a Drupal 7 site (as all of them were D6).

I don't follow the politics behind the project, but to me the biggest indicator of something awry was D7 still not being acceptable for a production site. It's much more fragile than D6 and the post clearly illustrates why, with the issue counts.

I recently had to help a rather technical client simply get D7 -running- with common modules. They'd have been better off simply installing D6, which they'd done before without any voodoo needed.

While this is mainly about technical issues, it does mention Acquia/Dries, so I'll give my 2 cents to that part:

There is a lot of FUD these days about Dries and Acquia's role in Drupal. Dries wrote some blogposts that try to answer the most common questions people have.

The criticism about the bug count explosion in Drupal has already been addresses with an "Issue queue thresholds for Drupal core":

http://buytaert.net/issue-queue-thresholds-for-drupal-core

The role of Acquia in all of this: "Does Acquia suck up all the Drupal talent?":

http://buytaert.net/does-acquia-suck-up-all-the-drupal-talen...

"Why Acquia acquired Cyrve and GVS":

http://buytaert.net/why-acquia-acquired-cyrve-and-gvs

Full disclosure: I work for Acquia. Not directly on the Drupal codebase, but I have a tech position. They really are a good people with the best intensions for the community and Drupal. They try to give back by e.g. free trainings, sponsoring camps and hackathons, having "gardening days" where people work on anything (old modules, new modules, core bugs, ...) that they think is needs some work, free hosting for any Drupal community project/blog/website and many many things more.

We really need a new "Godwin's Law" to deal with usage of the term FUD, because at this point it seems to be used any time a corporation or project feels they are being unfairly criticized. Do you have some actual examples to prove that the critics of Dries and Acquia are spreading FUD and not expressing real concerns about the project?

I tell you what- every time I hear an Acquia employee use the word FUD against a critic it just makes the critics look right.

Also, note how Dries addressed approximately 1 of the 10 or so issues outlined in the "Does Acquia have inappropriate influence over core" post.

Props to you, eaton and greggles for showing up on this thread, and not in disguise.

Maintaining disguises on the internet is freakin' EXHAUSTING. We're too busy trying to get a roadmap for the next round of refactorings ironed out. ;-)

http://drupal.org/node/1224666 http://drupal.org/node/1273344

Dude. I thank you.

Yeah, big ups for your clear vision, eloquent writing, and just being persistently awesome here.

Vote for eaton!

Sorry, I'm not a native english speaker. I picked that term up during the whole discussion :)

When I said "FUD" I wanted to pick a word that expresses the fact that even outside of Acquia, there are people on both sides of the issue. As far as I read the threads, it's not like the vast majority of people are strictly against the direction Drupal is moving in. I'm simply not embedded enough inside the community to give you any specific examples, sorry :(

I just wanted to say that it's important that these concerns are raised and discussed. From what I hear in the meetings and read in the emails, Acquia is really dedicated to advancing Drupal both on the enterprise side (which helps site builders get contracts and gets Drupal publicity) and the developer/user side by contributing to/maintaining modules and fixing bugs (core and contrib).

p.s. I don't speak for the company, I just wanted to relay the impression I get from the inside :)

Alex this argument is just ridiculous. After all you've been through with the Drupal community lately you should know that for all intents and purposes I haven't seen any conclusive evidence that Acquia is hurting Drupal. They aren't using the community to do work that only benefits them, because they aren't doing any work at all that only benefits them, everything they do is available for anyone that wants to use it. I for one have no found any of your arguments sound at all, you use the Overlay as an example, assuming that since you don't use it, no one else does and it doesn't belong in core. This is patently false, lousy logic, and until you can demonstrate where they are abusing the Drupal community for features or code that would only be usable/benefit Acquia, your argument and concerns are uncertain, and you are trying to cast doubt as to their motives.

If that isn't FUD, I don't know what is.

What argument are you referring to, that the usage of FUD to shut down criticism is currently endemic to software development communities (the comment you responded to)?

Here's the thing Mikey, I never said Acquia is doing work that "only benefits them", and I also never questioned their motives (unless assuming that profit as a motive for a venture-backed startup is controversial).

Overlay, however, is a great example of the (subtle) shift in power within the community. Now that Dries has a paid staff he is able to have things developed by paying for them directly rather than by building consensus or waiting for a volunteer to take it on. That's a real shift, it's easily seen in how the overlay was built and found it's way into core, and even Acquia employees can see it was a problem: http://groups.drupal.org/node/170999#comment-575529. To quote from the "other" Alex: "The Overlay example, however, is different, in my opinion. This, I do believe was a clear case of Dries knowingly making an unpopular decision, Acquia being the only company willing to fund the development, and the development not have happened without such funding."

So, please rest the FUD talk, you obviously don't have any real examples of me spreading "FUD" here (obvious to me, because your accusation is false), and it's really nothing more than a lame attempt at shutting down a discussion that really needs to take place.

I just started to do new projects with D7 (was playing with D7 since its release, but only now got new commercial project at hands).

While everybody complains about bloat, you can turn off most of it. But new developer features just awesome!

For example, I had hard time incorporating CDN into D6 heavy loaded project. Now it is easy since D7 support different backends for storing files (via wrapping PHP streams)! You no longer need to rewrite URLs to point to correct CDN location. Same goes to private/public files. Now managing what to allow user to download and what should go through access control much easier.

New DB abstraction functions. While they are nowhere close to full blown ORMs, but they provide enough abstraction and most important - your modules can plug into process of executing SQL queries! (it was possible in D6, but in D6 you had to work with raw text, which was very annoying and prone to errors).

As for bloat... They moved a lot of often used modules in core. Fields in core - idk. I can imaging projects which do not need CCK, but from my practice - every single commercial project required CCK. So for me personally Fields in core make perfect sense.

So. Even if there are a lot of bugs still open - it is fine. D7 is really good product which helps with building scalable and hackable (in good sense) projects.

"As for bloat... They moved a lot of often used modules in core. Fields in core - idk. I can imaging projects which do not need CCK, but from my practice - every single commercial project required CCK. So for me personally Fields in core make perfect sense."

Yeah, that's what we all thought at first. Unfortunately (and I've found this out the hard way) this is naive.

Moving CCK's functionality into core means no feature releases for (at least) the next three years. Meanwhile, Fields in core stomped the single largest game-changing feature to hit CCK since it's creation: multi-field support. So now, if you're in a situation where you need to delta-sync multiple fields, you have to bail on the Fields API entirely and implement hook_node() or a custom form.

In other words, the grim reality is we've traded 30 extra seconds of work (drush dl CCK) per install for a frozen feature set that is incomplete when compared to CCK 3.0's functionality in D6.

The DB API is also seriously problematic. From a maintainer's perspective this means 100% of the code in any 6.x modules have to be at minimum tweaked, typically completely rewritten. This also means that developers have to memorize two data manipulation models, one of which has fucked up proprietary syntax requirements entirely unlike the SQL it's meant to replace. Not to mention the new DB API leads to rampant code bloat. What was a simple one-line select now takes anywhere from 5-20 lines of OO fuckery if you want to adhere to D7's nebulous style requirements. You even take a performance hit since all of that crap has to be run through additional layers of parsing before it gets cooked down to SQL. This isn't abstraction, it's added complexity with no win attached.

1. It's not clear what you're referring to by "multi-field support", but if it's something that's only available in the 3.x branch of CCK for Drupal 6, note that that branch is not stable anyway (only has an alpha release and should not be used on production sites). The 2.x branch is the one that's stable.

Perhaps you're talking about http://drupal.org/node/695636, in which case you'll see that the new Field API allows that feature to be implemented in a much cleaner way in Drupal 7, and there's already a contributed module for Drupal 7 being worked on to do it (which only has a beta release itself, but appears on the surface to be further along than the equivalent Drupal 6 CCK code).

2. Your comment about the database API is incorrect. Simple one-line select queries do not need to use the new OO syntax (and in fact, should not). See http://drupal.org/node/310072.

As for queries that do need it, it seems like you've missed the point; it's not "added complexity with no win attached", but rather the purpose is to support things like cross-database compatibility and access control. Sure, it was simpler in Drupal 6, as long as you didn't mind that your code might not work at all on certain websites :) If you're writing custom (site-specific) code for Drupal 7 and still don't care about that, you can likely use the simpler syntax for more things than it is intended for (although it's obviously still not recommended).

CCK 3's stability isn't the issue here. Given the module maintainer's track record nobody in the community doubts that 3.x would have reached maturity. That development effort was halted when it was announced that Fields API was going in core. In the mean time you still can't sync multiple instances of multiple fields using pure Fields API, not to mention the huge number of field types entirely unrepresented in the core implementation.

So again, we have traded 30 extra sections of time spent on an initial site installation for a feature locked partial implementation that cannot change for at least the next 3 years. Even if the implementation was complete, given how quickly the web changes I don't think any of us can confidently say that it will be well positioned to meet the needs of the day 3 years from now and now that it's in core, it cannot grow between major number releases, which is really what this is all about.

My statement about the database is not incorrect, but I recognize that it's an unpopular position. I am well aware of the stated design goal of the new API. The problem remains that core devs have troweled a ridiculous amount of additional complexity into core to satisfy a small contingent of potential enterprise users that are hell-bent on running MSSQL or Oracle.

I recognize the tradeoff here. On the one hand you have ease of use. On the other you have huge stacks of enterprise client cash. I don't think anyone's confused about which way Acquia went on this one.

I'm an Acquia Project Manager that started working with Drupal in a company that I owned and was one of the PM's for the drupal.org redesign.

2 quick things...

First, I'm struggling with relating Acquia to the MSSQL and Oracle work that was done with the new API. Didn't that come out of AF83 / CG getting a significant amount of investment from Microsoft? Hasn't the decision to put it into core led to a lot of really good conversations and partnerships with other technology providers including everything from additional funding for contributed modules, core, drupal.org features and other things? To be sure, it's easier to develop for just one DB, but it creates a choke-point for adoption that has the potential to stifle a lot of really good things that come out of it. While Acquia has supported the new API, I think it's a stretch to say that Acquia went for the cash as I think we've only done one or two projects of any significance that were not based in MySQL. I've heard of several other projects, and even some really big ones, but they're all at other shops. Singling out Acquia is a clear case of misperception here, and I think if you're going to bash, you should do so fairly.

Second, what's wrong with developing in a direction because you're being paid for it? SOMEONE needs to make money off this, and contributions like the ones your pointing out are both significant shifts in what a group of invested organizations, developers, peers, and business-types decided was the right way to go. The sell-out argument is weak - nobody is running a charity because there are too many of us that have to have jobs so we can pay for things. I agree that the endless pursuit of revenue is not the only thing that matters, but I struggle to see where our (both Drupal's and Acquia's) purpose has become unmoored from our profit motive. The new DB system, fields in core, overlay, and the myriad of other D7 changes are big. There are sacrifices. However, they will continue to improve as more development work gets done and we'll see significant incremental improvements in what many smart, dedicated, and yes... even non-Acquia people agree is the right direction.

tl;dr: Don't piss in my pocket and tell me it's raining.

I understand what you're getting at but I've been with the project since 4.7, and I feel like the time for happy-fun-time talk has passed. Like most community developers, I supported the overwhelming majority of the changes that went into D7 when they where being discussed among the community. Now faced with the reality of D7, I see that some (maybe most) of these additions where a mistake.

Assuming Dries' keynote wasn't based on random numbers, Drupal had reached the "1% of total websites" threshold well before D7 was released. There was no choke point. Look at the following web server statistics http://greatstatistics.com/ MSSQL support doesn't service the majority of potential consumers of a CMS, just a small minority of potentially well-heeled clients that run Microsoft-based environments.

It's counter-intuitive but I've come to realize that sucking module functionality into core is stupid. Example: your choice of forum, aggregator, or blog. All crusty, useless modules that any serious implementation will either ignore, patch heavily with additional contributed modules, or just go with a fully contrib alternative. Why do these modules all suck? They've been stuck in core and thus cannot change. When a major release is planned everyone's busy focusing on the shiny new stuff to cram into core and modules like this are typically neglected. Why should the fields API be any different, given the track record of the last five years?

Also, you may not want to lose sight of the fact that it makes Drupal shops (you know, the little guys who are busily engaged with growing and maintaining Drupal's adoption base) look bad in front of clients when they've been on the receiving end of Acquia-sponsored D7 marketing blasts, meanwhile we're waving them off from migrating because core isn't stable and critical contrib modules are either unstable (RC$n or beta) or entirely non-existent in D7.

I know several contrib module developers that have thrown their hands up in disgust and gone on to work with other communities while new developers are struggling with the unprecedented complexity in D7.

I agree, the changes are big. Some of them are awesome (new menu structure, new AJAX api). Others range from obnoxious and/or useless (overlay) to actively harmful (see also core bug queue explosion). There was a large and VERY vocal contingent among the community that was calling for smallcore years before D7 shipped and a blind eye was turned by a vanishingly small yet disproportionately powerful minority, and now it's biting everyone who isn't vying to become a Microsoft Enterprise Partner in the ass.

And that is my take on the Drupal community as it stands currently. You don't have to agree with me but it would be a mistake on your part to assume I'm an outlier.

Er, your accusations about Acquia here are both incorrect and offensive.

For the record, Acquia was only marginally involved in getting the new database API into Drupal 7 in the first place (probably 95% or more of the work and technical direction for this particular feature was contributed by non-Acquians). Also, given that we (Acquia) own a MySQL-based Drupal website hosting business (and don't really have MSSQL or Oracle expertise on staff so it's certainly not our preference to have clients who want to run Drupal on those systems), the implication that this was some kind of Acquia-directed conspiracy really makes no sense.

Also, the primary motivation of the new database API was certainly not to support these proprietary database systems anyway. Other databases whose code is free (especially SQLite, which is now supported directly in Drupal 7 core) are much more important. And cross-database support is definitely not the only useful feature of the new database API, just one of them. I would suggest reading e.g. http://www.garfieldtech.com/blog/database-tng-lands for more information.

Regarding CCK, it's hard to understand why you are so concerned about features being "locked" in Drupal core for the next few years? For one thing, they're not locked (certain features might still be added to Drupal 7 if they don't change things in a way that will break existing code), but more important, the entire way Drupal works is that core functionality can easily be extended by contrib modules to meet additional needs. As described in the link I provided earlier, the functionality you're looking for is apparently being implemented by the Field Collection module in Drupal 7 (and due to the Field API, it's being done in a way that knowledgeable people seem to agree is much better than any implementation that would be possible with CCK in Drupal 6). So it's not really clear why are you are apparently comfortable typing "drush dl cck" to install CCK for Drupal 6, but not comfortable typing "drush dl field_collection" to install this (or other) modules you might need for Drupal 7.

Sounds like what they need is a wycats type figure to ride into the breach and clean house for them. Rails could easily have ended up in this situation if it weren't for the efforts of yehuda, and obviously others.

Sun has very good points, but fortunately movement has really begin to shift towards moving cruft out of core, even just recently with initiatives to get rid of the PHP filter among other things. Along with it, Dashboard should go, especially considering that Homebox was a perfectly adequate contrib module which users could be pointed to.

I do disagree with calling out Acquia though, as all the infusion of cash from large institutions adopting Drupal that has come from having a big name to point to is overall helpful for long-term development.

I don't think he's ranting so much as proposing the way forward, and fortunately, people have started listening. I think sun was at the tipping point and now things will finally begin moving steadily towards small core.

I've worked with Drupal since 4.7 on many projects. I was excited for the release of Drupal 7, but was amazed at how bloated it was. I've completely given up on Drupal now. You WILL spend more time working against it's api then you would just building something custom with a traditional framework.

Drupal should really look at codeigniter/expression engine for an example of a core/platform separation that works really well.

Good example of what happens over time when a project says yes a lot more often than no. Sounds like a almost impossible task to maintain and probably not an easy beast to use and get good performance out of.

Might be similar to something like zen art which is stuck with a lot if bad php4 practices and horrible organization. Add to that a vaporware rewrite.

I have to work with Drupal 6 on occasion for parts of a number of projects with my employer.

Before I was hired I'd had zero knowledge of Drupal or anything to do with it. "How hard could this be?" I thought.

Let it be said beforehand that I tend to use OOP practices when coding.

With that in mind, I was given some Drupal related task in my first week and my mind was utterly fucking blown. My OOP stuff was useless when dealing with the procedural nightmare of a Drupal module. I had to learn how to deal with Drupal's pointless reclassification of HTML when using its APIs to create forms, and had to get used to the idea that the people behind it are dangerously obsessed with multidimensional arrays, to the point they're used for everything (check out the module list function, which returns an array whose keys and values are identical).

As a developer, the tools included within Drupal that are intended to make life easier are not abstractions, but obstructions. The documentation is neither help nor hindrance, because there is barely anything that qualifies as documentation.

When it came to performance, I was just as blown away as I was by the development process. It's hard to even use performance and Drupal in the same sentence. Everything requires a horrific number of database calls. My colleague and I ditched the horrific concept of nodes and developed our own solution that, because Drupal is neither a good framework nor a good CMS, had to hack a lot of things to fit in. So we cut down on database calls but it was still slow. I logged calls to module_init() and noticed modules were being initialised not one, but up to 4 or 5 times on a single request. There were no doubt a good 20 or 30 db calls in that.

Then there are the practises it encourages amongst poor developers. Why not roll your own code when you can download a module that is barely any better than the Drupal core itself. Abstract existing functions within PHP to the point of them being abstruse and then spend the rest of your day trying to make sense of the non-existent documentation. And you end up with a project with a modules list that could stretch to the moon and back and an extra few hundred MB of bloat.

Considering I came to Drupal with no prior knowledge, bias, or experience, it couldn't have made a worse impression if it tried. This isn't entirely on topic of course but I find it hard to disagree with anyone who says it needs sorting out.

Yeah I had a similar feeling with Zen Cart recently, my mind was blown by just how many files were included in each page and the complex paths through which it happened, also the very high complexity search queries which perform terribly.

I think the issue is this software is more designed with allowing non programmers to get most of the way to a functioning website in mind rather than programmers. To someone with limited knowledge of what's under the hood they probably love the choice they get with modules and configurations abstracted away from the code.

Here's some follow-up from a somewhat different cohort.

Does Acquia exert inappropriate influence on Drupal core?

http://groups.drupal.org/node/170999

Reminds me of this: http://www.jwz.org/doc/cadt.html I wonder why this seems to happen in some open-source projects and not in others?

May i introduce you to the danish Content-Management Framework "TYPO3" which is very successful and widespread all around Europe. Currently, it is being rebuilt from scratch using the home-made PHP Framework "FLOW3" (this one is huge!). Curious people may have a look at the TYPO3 main page [1] as well as the FLOW3 page [2]. Give it a try, it would really deserve it!

[1] http://typo3.org [2] http://flow3.typo3.org

When you recommend a framework, you better tell us why it's better than the rest.

Even though Typo3 originated in Denmark it's huge in Germany. I think Typo3 is more popular than Drupal in Germany.

Typo3 is actually called Contao now and unless you can read German, the documentation is really really weak in English. Lots of documentation, but it's mostly in German. I've been looking at it and one of the main advantages is the ability to maintain multiple sites in one CMS, as opposed to having a CMS for each site you have. Also has the "live update" feature, so you can always have the most updated version with a single mouse click. Although this is $9 extra per month, it's one of their key features.

That's not true. TYPO3 is an open-source CMS/CMF licensed under the GPL. Contao seems to be a spin-off of it.

TYPO3 has its strengths in being very modular (even the core is nothing more than an accumulation of (system-)extensions), powerful (*nix-alike) user handling capabilities, its own domain language for website-integrators (confusingly called 'TypoScript' [it's not really a scripting language, more of a description language]), a very very sophisticated handling of workspaces (they may "shine" through each other) as well as a very flexible templating mechanism (called TemplaVoila). Its multi-domain and multi-language concepts are imho superior to what i've seen somewhere else (XLIFF coming in the next few months). Of course, there's generally no "this one fits all" solution and due to its complexity, TYPO3 has a rather steep learning curve. However, held in capable hands, TYPO3 rocks in use-cases described above and mostly sucks if you just need another simple content management system, since it's rather huge in its nature. A last comment on the documentation: yes, it is very strong in Germany and because of that most tutorials and books are in german, but there is a drive towards i18n. However, the main and most important doc stuff is in english. [1]

Give it a try, i think it's really worth it!

[1] http://typo3.org/documentation/document-library/

Contao is just rebranded TYPOlight, a lightweight CMS inspired by Typo3.

Those who use it, how does it compare to joomla? Joomla is the closest I have come to having to maintain a full blown php cms and I found that to be quiet bloated as well.

Drupal has a lot of problems. It's still much better than Joomla, IMO--Joomla has a lot of really deep architectural WTFs and I find it much harder to work with, both as a software developer and as an end user.

CCK and Views allow you to do fairly complex stuff within Drupal without writing code. There's definitely an upper bound on performance with such a method, but you can get stuff out the door pretty quickly. Joomla doesn't really have an equivalent, and adding new content types in Joomla isn't novice-friendly. (Also, Drupal's Taxonomy is way, way nicer than Joomla's arbitrary Categories/Sections stuff.)

I haven't used Joomla but have worked on Drupal sites. It is possible to build a reasonable and scalable sites in Drupal, but often times you'll find instead is novice web devs have chosen Drupal because they know very little about relational databases and don't want to know anything about database design. Drupal gives you custom object storage and custom object views without having to write any code, so you can imagine the skill set of the people this appeals to and the kinds of architecture their sites have.

On the flip side, just because you know how to write code doesn't mean your designs are going to be good.

The "everything is a node with attributes" concept that Drupal requires makes you think first about data, then about presentation. For example, if you want an organized list of data, you make custom nodes (the model), then a view, which is a decent design pattern for most sites. Views are incredibly powerful and can do nearly everything most sites would want.

Compare this to other CMS's that make it difficult or impossible to tack on functionality without writing a lot of raw database code (not to pick on them, but Wordpress developers seem to have this problem in spades).

Also, Drupal treats caching as a first class integrated system, which makes performance much better than other options.

yep, everything-is-a-node-with-attributes makes me think about my data; realise it doesn't fit drupal's rigid content heirarchy; remember that, no, views are not nearly powerful enough; and go back to django.

Joomla has very extensive community issues. I've written a bit about it over here[1] and you can check out Johan Janssens (Joomla co-founder in exile) blog[2] for more in-depth analysis. The bottom line is - take a look at Nooku[3] - especially if you're looking for a framework that makes sense (a corresponding CMS is underway).

[1]http://valanx.org/index.php?option=com_content&view=arti... [2]http://johan.janssens.me/ [3]http://nooku.org/

We need to stop painting lipstick on a giant pig

Perfect analogy regarding the state of this project.

It's a lot more fun to write new code than it is to fix and refactor old code.

Bug bounties? But then the question is "Where will the cash come from?"

The community, does Drupal have an official organization behind it?

http://acquia.com/

That's the unofficial organization. And, yeah, they definitely have some cash (recently got a $15M investment round). ;)

The official org is the Drupal Association (https://association.drupal.org). Given that they made $400K off the Chicago Drupalcon this year (after expenses) in addition to their regular association membership fees (and perhaps money from the European conf, Drupalcon London, although I'm not sure about that), there's should be money for bug bounties.

The Drupal Association was, if I understand correctly, a non-profit until recently. It has merged with DrupalCon Inc., the corporation that put on DrupalCons.

https://association.drupal.org/node/1584

The "Drupal Association" prior to this year was an unofficial name used primarily to describe "Drupal Vzw" a Belgian non profit.

Drupalcon Inc. is a non-profit in the USA. They are both non-profits, just in different countries.

Neither has been merged/folded/acquired: much of the business undertaken by the Drupal Association moved from Europe to USA making Drupalcon Inc the main body doing business. There was a governance update to officially make Drupalcon Inc the main focus of governance/efforts to match what was coming to be in practice.

The link you included, https://association.drupal.org/node/1584, does indeed have more information but it doesn't say the things you've said in your comment.. ;)

>Drupalcon Inc. is a non-profit in the USA.

Ah, that's the root of my confusion. I didn't know there was such a thing as a non-profit corporation.

The perception that Acquia = Drupal is a HUGE problem for both Acquia and the drupal ecosystem. If this perception persists then you can guarantee that more Drupal firms will go the way of Development Seed, who not only stopped working with Drupal for their clients in favor of mapping and data visualization, but who also just released their new site using Jekyll and GitHub pages: http://developmentseed.org/blog/2011/09/09/jekyll-github-pag....

And if all of our shops turn away from Drupal, or turn towards a forked version, Acquia is definitely going to be feeling forked themselves. This is a Mambo moment, and Dries + Acquia should have a healthy fear of a fork.

Disclosure: I own a 20 person open source consultancy that does mostly Drupal work, and we are Acquia "partners".

Behind the community, no.

There is no organization that directs the development of Drupal itself. The Drupal Association supports Drupal initiatives such as redesigning drupal.org and migrating the project to git. As <a href="https://association.drupal.org/about>they say themselves</a> "The Drupal Association has no say in either the planning or development of the Drupal open source project itself."

Acquia was co-founded by the Drupal "benevolent dictator for life", Dries Buytaert. He maintains separation of church and state.

Seems like they have to do what Apple did when they changed their OS core. Backward compatibility will probably have to be an afterthought if the project is to survive.

Actually, backward compatibility has always been not just an afterthought, but completely thrown out the window with every major x.0 release.

I think it's important to note here that this isn't just about "is core bloated" it's also about the new elephant that has emerged at the center of Drupal's ecosystem: Acquia. I think Nedjo (a permanent member of the Drupal Association) puts it best here (http://groups.drupal.org/node/170999#comment-575044 ) and here ( http://groups.drupal.org/node/170999#comment-580704 )

Some quotes from those two comments: "What Acquia means for the overall Drupal project is a question that's key for the project's future.

Currently 3 of 8 Drupal Association (VZW) board members are on staff at Acquia. This is not due to Acquia stacking the board with its own people. All three have been active in the DA since its early days, before Acquia even existed. Rather, in my view, it is one of many reflections of the convergence of Dries' roles and choices in various domains of the Drupal project: in the Association, in Acquia, and in Drupal core. Sure, this convergence has benefits. But it also has major risks that we all need to be cognizant of and active in addressing.

The influence of Acquia (or another company) on Drupal core may sometimes be explicit, and it's worth looking at specific patches that may or may not reflect this issue. But I think much more to the point are the larger factors of scale, resources, access, and influence.

By analogy, there's the name Drupal itself. We all through our participation and contribution help build meaning for the name, but ultimately its use is controlled individually by its trademark owner, Dries. Any company or individual may apply for a license to use the trademark for commercial purposes, subject to the conditions, which include notice that the requirements for usage may change. Basing a major company, product, or service off an external trademark implies some risk. Does Acquia have differential access to the trademark based on Dries' ownership of both Acquia and the trademark? In practice, the most high profile private branding of Drupal products and services are mainly in Acquia: Drupal Gardens, Acquia Drupal.

"In approving or rejecting proposals and patches, [Dries] gives special weight to comments made by people he trusts and respects for their past contributions to Drupal." About Drupal: Core Developers.) In practice, Dries has gathered around him in Acquia an ever growing number of the core code contributors he most trusts and respects. Included in this pattern are core branch maintainers--for both Drupal 6 and Drupal 7, the individuals Dries chose as branch maintainers were subsequently brought onto staff at Acquia. In Angie (Webchick)'s case, her work at Acquia looks a lot like a continuation of her previous work as a core maintainer.

Even with a high level of technical skill and insight, getting even a minor change into Drupal core can require months of work (assuming you aren't in the elite ranks of IRC superstardom). Contributing larger changes - like significantly rewriting a major API system - is a huge investment. Before making such an investment, any company would have to have a high degree of confidence in its likeliness of success.

Acquia has on staff all current committers to Drupal core, and talent to burn, including a lot of the inner circle of Drupal core development, backed by venture capital and a presence in every main area of Drupal products and services. Does this in itself mean that all of Acquia's core initiatives will go in while those of other companies will languish? No. Does it mean that Acquia is uniquely positioned to invest in long term core development, to shape Drupal's future with a degree of confidence that others can only dream of? I think the answer's clear: of course it does."

Do you know who hurt the project most this last year? You, by your constant attacks at the DA which takes away whatever little community time Dries have. This needs to be noted.

chx- your personal attacks would be distressing, if they weren't so hilariously silly. Yes, my attempts at keeping the Drupal Association accountable, and specifically howling against the DA taking money out of my pocket and putting it directly into the pockets of my competitors hurts Drupal, causes global warming, and is almost completely to blame for world hunger.

For anyone who wants to know what this angry fellow is referring to (self dealing on the Drupal Association board), you can find it in my complaint on groups.drupal.org: http://groups.drupal.org/node/162604

"In practice, Dries has gathered around him in Acquia an ever growing number of the core code contributors he most trusts and respects."

What's the alternative? You are a business owner as well! Would you staff your company with people you trust, with people you worked with, with people who proved their expertise? Is that not the number one choice a business owner would go with?

I staff my company only with people whom I distrust, but whom I know I can defeat in hand-to-hand combat. At first it seems like a bad work environment, but the beatings continue until morale improves.

<blockquote> Does Acquia have differential access to the trademark based on Dries' ownership of both Acquia and the trademark? In practice, the most high profile private branding of Drupal products and services are mainly in Acquia: Drupal Gardens, Acquia Drupal. </blockquote>

Be careful asking a question and then citing marginally evidence as if it proves an answer to the question...

The question is if they got those trademarks fairly. The evidence you cite next is that their brands are high profile. These two things are not really connected.

To try to actually answer the question, I don't believe Acquia has extra access to the trademark process. Having gone through the trademark process I can say that an external team of lawyers manages and adjudicates the process which keeps Dries at arms length from all decisions.

But even without an audit of the process of granting trademarks it's pretty obvious that "Drupal Gardens" is a name which meets the guidelines for a commercial use of the trademark since it doesn't imply the nature of the service. "Acquia Drupal" is probably covered under fair use since it truly is the Acquia version of something.

Sorry if it wasn't clear, but that was a direct quote from nedjo, so I'm not actually citing anything other than him. To be honest, I'm not sure what the deal is with the trademark, but I do think it's problematic that it's in private hands.

May be its time is just over? I mean that the time of personal CMS or blogs or just small personal sites is definitely over.

For Drupal - they got lost momentum, lost a lot of active developers, lost fans, lost community. The game, when everyone wants his own site in the net, is over. Enjoy your FB page. ^_^

I think you can see the same situation of stagnation in Wordpress also, and, I hope, finally, in PHP world in general. ^_^

Not at all. The issues described in the linked post have arisen _because_ the community has grown so much and so fast over the past few years.

Community of whom? Active developers or passive users? Development and usage of a product are not the same thing. ^_^

Yes, it has grown, and now it is going flat. Try to visualize the trend - imagine a bell curve (the only type of curve hackers know) and point out where the current time stamp is - on a growing or falling slope? ^_^

OK, to look a little bit more professional: doing a major rewrite of already mature project is a tricky thing (look what a terrible mess Gnome3 is or a recent Ubuntu), because users are passive and tend to use what they call a stable versions.

We can see the same situation with Python3 (although it has real improvements and much less buggy) and with ruby19, where the Rails guys are pushing it as hard as they can, but inert users still want 1.8.

So, users are happy with drupal6 and variety of third-party modules, while independent developers are losing their interest.

Facts, not baseless opinions, please. For example: http://drupal.org/project/usage/drupal

Clearly shows both Drupal 7 usage (and consequently Drupal usage as a whole) continuing to increase at a steady, fast rate.

  cooper@ubuntu:~$ apt-cache search drupal
  dh-make-drupal - Create Debian packages from Drupal modules and themes
  drivel - Blogging client for the GNOME desktop
  drupal6 - fully-featured content management framework
  drupal6-mod-addtoany - addtoany module for Drupal 6
  drupal6-mod-cck - cck module for Drupal 6
  drupal6-mod-commentrss - commentrss modules for Drupal 6
  drupal6-mod-contemplate - contemplate module for Drupal 6
  drupal6-mod-filefield - filefield module for Drupal 6
  drupal6-mod-i18n - i18n module for Drupal 6
  drupal6-mod-imageapi - imageapi module for Drupal 6
  drupal6-mod-imagecache - imagecache module for Drupal 6
  drupal6-mod-imagecache-actions - imagecache_actions module for Drupal 6
  drupal6-mod-imagefield - imagefield module for Drupal 6
  drupal6-mod-imagefield-assist - imagefield_assist module for Drupal 6
  drupal6-mod-inline - inline module for Drupal 6
  drupal6-mod-ldap-integration - ldap_integration module for Drupal 6
  drupal6-mod-lightbox2 - lightbox2 module for Drupal 6
  drupal6-mod-masquerade - masquerade module for Drupal 6
  drupal6-mod-openid-provider - openid_provider modules for Drupal 6
  drupal6-mod-pingback - pingback modules for Drupal 6
  drupal6-mod-site-verify - site_verify module for Drupal 6
  drupal6-mod-tagadelic - tagadelic module for Drupal 6
  drupal6-mod-trackback - trackback module for Drupal 6
  drupal6-mod-views - views modules for Drupal 6
  drupal6-mod-views-charts - views_charts modules for Drupal 6
  drupal6-mod-views-groupby - views_groupby modules for Drupal 6
  drupal6-mod-xmlsitemap - xmlsitemap module for Drupal 6
  drupal6-mod-xrds-simple - xrds_simple modules for Drupal 6
  drupal6-thm-arthemia - arthemia theme for Drupal 6
  drupal6-trans-ru - Russian translation for Drupal 6
  drush - command line shell and Unix scripting interface for Drupal

This is from Ubuntu 11.10. In Fedora 16 they have a drupal7 package, but most of the pre-packaged modules are for drupal6.

Well considering that there are about 12,000 projects hosted on Drupal.org and that it has it's own packaging system this is a pretty pointless example. I've never met anyone using apt to install PHP applications. Given the ways that apt usually installs a PHP app, I'm not really surprised either.

That is not useful information. It only indicates that someone with package commit privileges decided to package Drupal 6 for Ubuntu 11.10. It doesn't mean anything about usage, developers, community, etc.

To put this into perspective, Debian once had a Webmin package in the distant past (about 10 years ago)...at a time when we had far less than a million new downloads per year (might have even been in the low hundreds of thousands). We're currently at about 3 million new downloads a year, and growing every year, and there is no Ubuntu or Fedora package for Webmin.

All you can assert based on a lack of packages in the standard OS repo is that there is a lack of packages in the standard OS repo.

Reply on news.ycombinator.com