“Drupalgeddon2” touches off arms race to mass-exploit powerful web servers
arstechnica.com
arstechnica.com
D5: https://www.drupal.org/files/issues/2018-03-28/sa-core-2018-...
D6: https://www.drupal.org/files/issues/2018-03-28/SA-CORE-2018-...
D7: https://cgit.drupalcode.org/drupal/rawdiff/?h=7.x&id=2266d2a...
D8: https://cgit.drupalcode.org/drupal/rawdiff/?h=8.5.x&id=5ac87...
That's the D6 version, but same concept would apply to other versions.
It takes a relatively large team to reliable shield software as complex as modern CMS from abuse and the mitigations have to be employed so timely that one cannot wait until this is high enough on someone's to-do list that has to do this for 10 different CMS.
Something like jekyll or something.
Most websites are mainly read only anyway. No need to throw in a programming language and store the data in a database.
The thing is, non-tech savvy people should be able to use(and deploy) and understand the static-web generators.
Funnily enough, having worked on all kinds of websites for the past 15 years, i'm still at a loss when asked what the all-round good solution is.
To further complicate things, things change. A website that was a perfect fit for a Wordpress template easily grows beyond that. And a site that seemed ideal for a custom Django/Rails backend might eventually settle into the kind of functionality groove where you begin to wonder if an off the peg solution might not have been a better idea.
Turning that CMS into a distributed publishing platform is tricky, but scaling static content is easy-mode from top to bottom.
However, I see a couple of problems standing in the way of more widespread adoption of static site generators:
1) I've never seen a static site generator that non-technical users felt comfortable with. Programmers like terminal interfaces. Your average user in 2018 has probably never opened the terminal.
2) Possibly even more significant, I don't think the people making the choice to use a CMS for a simple business-info or blog site are aware of the trade-offs that they're making, or of the benefits that a static site will provide down the line. For a lot of small business owners I've spoken with, Wordpress is basically synonymous with "web presence."
I wonder if there's a market for a web-based GUI like what Wordpress provides, but which runs jekyll under the hood and uploads the files to s3 or something like that. Is this what companies like Squarespace and Wix do?
(disclaimer: Netlify employee)
That said, it didn't help that you don't support Gitlab, which is where all my sites are, and which is the more important reason why I couldn't get it to work.
We (Graphia) took a similar approach for our document management system. Essentially it looks and acts like a regular CMS but it sits on top of a git repo instead of a database; and publishes via Hugo.
It's not intended to be a fully-fledged CMS but the API would support it without much work; the UI is definitely the time consuming portion.
http://www.graphia.co.uk for anyone who is interested, not quite ready for prime time just yet. Soon.
Feel free to ping me using the contact info in my profile or at @benaiah in our Gitter room (https://gitter.im/netlify/NetlifyCMS) if you're interested in discussing this elsewhere.
Imho if you are really wanting to write a blog and host it yourself, this should not be too hard to learn. Wordpress doesn't install itself automatically, either.
We also have a pretty neat solution for site generation/hosting (JekyllPro) which is rolling out next month.
Using these two, none of the contributors realize they're using Jekyll. It's not as user-friendly as Squarespace, but it's pretty darn good.
1: https://github.com/Wiredcraft/jekyllplus
2: https://wiredcraft.com/blog/the-new-starbucks-cn-website-bol...
I remember having to purge front page extensions off many a server. Cathartic feeling that one.
Forestry + Netlify + Bitbucket is a good combination.
I would work on this if I wasn't already running another startup.
I spent a lot of years working in tech consulting and did a lot of CMS integrations. We desperately wanted to push clients to Contentful and away from ornery beasts like AEM, but Adobe can sell AEM by saying it's part of the "marketing cloud" and that it's magic power is in user engagement and analytics and whatever other marketingspeak lies. Contentful was the developer favorite, but it was hard to convince businesses it was in their best interests even though it totally was.
That's not only the same gnarly problem as cache invalidation but with a metric ton of filesystem writes on top of it.
There's also Netlify CMS, but I haven't done much with it. I think the GitHub requirement puts people off, and work on the GitLab--which has free private repositories--support issue is slow.
What we need is something like Publii that's not packaged in a way that's obviously meant to make closing the source easy. That story is too old, and too common, to believe their intentions are good.
Is there significant overlap in their functionality?
You know it's all trivial garbage...except if you want to replace it you need to support it, train them to use these tools and at every point you're going to encounter this resistance about WHY you're doing it and you say well security and speed and they're like it's fast enough and we've never been hacked.
So until the site grinds to a halt and they get also get hacked they basically view you as a blocker.
That said, back on subject:
Look, this is the main reason you devs tend to hear disdain from us sysadmins when we hear you want to run some new webservice in whatever language fad of the day (node, python, perl, php, asp.net, ruby, scala, go, etc) instead of generating files with your app that can then be serviced to the user using the tried and true tools (apache, nginx, hiawatha, haproxy etc) that have had sometimes decades of public internet exposure to claw thier way through.
Ah, but what about $functionality. Most webdevs think they need a scripting language backend when they really don't. There are uses for it and it has it's place, but it's been way overdone.
Relevant discussion: https://groups.drupal.org/security/faq-2018-002
Sidenote: I think everyone should check out hiawatha webserver, over at (https://www.hiawatha-webserver.org/). Hugo wrote it with security in mind and it by default addresses many things that no other webserver does. It's also gplv2 (which is a big deal for me), and it's performance is second to none. I'm not a big php person, but if I were to be doing a webapp where a backend like that would be needed, I would probably try to go PHP-FPM with hiawatha(mbed TLS)/haproxy and a really nicely hardened boxen and edge firewall(you are using nftables now right?).
I'm on mobile but a quick ddg show: https://www.hiawatha-webserver.org/weblog/64
I used to work a LONG time ago for a company that built Yahoo Small Business ecommerce websites. They used a tool called RTML ("Robert T Morris Language"), which was essentially HAML, powered by Perl, and constructed using a GUI (no text editor interface). Select where you want a node > New node > select node type > Save...each action was a full page load. The process was excruciating, and I was in charge of developing tools to automate the upload of "template" files from the developer's machine to the client's Yahoo Small Business store.
At the end of the day, the customer would load up all their goods into the Yahoo database and hit "Publish", at which point a multi-hour job would kick off to run their RTML against the database. The result was a completely static ecommerce website. Only the cart/checkout pages were dynamic, and those were using shared code.
The development process was a nightmare. The client experience was really bad (want to fix a typo? You're going to wait a couple hours). But I'll be damned if it was possible for anyone to hack those things. There was nothing to hack.
Static site generators are simply a file-system caching layer. Drupal used to have one of those and it was a terrible idea as the number of page variants on a commercial site very quickly hits file-system limitations and performance is appalling. Better to use a proper CDN for a similar result but with massively reduced latency due to content being served from the edge.
No. The article is about a security holes from 2014. It does not need a team to keep your software updated. I have been running a bunch of CMS for many years now and keeping them updated takes me almost no time. If I accumulate all the time I spent on it, it's probably a day per year or so.
Think about all the schools that paid someone once for this. The Solution is easy but the structure of responsibility is not there.
it's a bad idea to host and maintain
a presence with a CMS yourself
Talking about schools. I would say it depends. If the kids run the CMS I would say the learning outweights the downsides of having some VM hacked that only holds public information. So for a CMS with public information I would say: Let the kids have fun with it!
There might also be use cases where that's not a good idea. Really depends on what the CMS is used for.
This... simply isn't true.
Sure, you can just set your system to update and reboot once a week, and then when an update is incompatible with some of your software, well, spend the time to clean up the mess.
This is going to break every few years, and that mess is going to take significant effort, especially if you haven't touched that software for a few years.
A more proactive approach is to watch the security mailing lists, updating when appropriate, but that's an hour or two a week of effort... remember, you have to keep your full stack updated; your cms, your webserver, your os, etc, etc... you need to watch for holes in each of these things, then when there is a hole, you need to figure out if applying the update will break your current configuration, and then apply said update
Either way, it's a significant amount of skilled labor.
Sysadmins don't get paid what SWEs get paid, but we ain't free.
To go deeper into what others will respond to this with, yes, distributions cover some of this. But consider that every few years your distro will go out of support.... and you will have to figure out how to upgrade that, which is often pretty painless, but it does require either a high tolerance for risk or a lot of reading.
Then consider that other bits under your stack; your webserver, your php, your cms, all those packages/versions are sometimes changed mid-release and often changed on those major releases.
Hosting a Drupal site on Pantheon or Acquia does NOT relieve the site owner of testing and applying patches, fixing the site when a patch breaks something, etc.
In terms of hosting, the LAMP stack itself is very stable; setting Linux, Apache, and MySQL to auto-apply stable updates is very unlikely to break Drupal. Even in PHP, going from 5.x to 7.x required some careful testing, but minor updates are generally fine.
The security advantage of open-source CMS software is that the community update process reminds site owners to do this stuff, and gives them the code to do it.
In contrast, a custom-developed CMS can sit there with security holes in it for months or years, and the owner will have no idea until their site is hacked.
This exploit was far more lenient in timing, there was an advance notice of a serious vulnerability, and the automated exploits were weeks later this time.
There are certainly many people running a CMS themselves that don't have the capabilities to keep them reasonably secure. But it's not that hard if you have a reasonably small attack surface, e.g. few plugins and no user-generated content/accounts. And of course you need to subscribe to the relevant security mailing list.
Currently they have to deal with at least 5 different CMS, 4 shopping systems, and some hand made sites with at least 4 different frameworks. 2 backend languages. And legacy code with self written frameworks in PHP 4 on at least 2 important sites.
If you try to bring it up there's always some inexperienced overseas contractor with 6 months of experience who will say "It' s a few clicks to install and a few plugins to protect the security 100% no problem, what's so difficult? why do you make a big deal out of it?".
What's my incentive to do things differently?
There's a spread between your life is ruined and nothing happens at all and it entirely depends on what you've done with the account/site.
This is the catch. Depending on what was done to your site, there's no "getting rid of the malware" for $100.
Those cheap hosting providers don't provide automated backups of your database. You probably have a backup of your site's files -- that's probably not perfectly current, but close enough maybe -- but I bet you haven't got regular backups of your database.
Drupal is one of those many CMSs that stores tons and tons of code in the database, including executable PHP. So how do you go about ensuring that all of that is clean, and changing all the passwords that may've been compromised, and making sure there are no other backdoors or shells left behind, for $100?
This is horrifying. Why was this ever done?
Thankfully they removed this option in Drupal 8, the latest version. You could also restrict users from accessing the functionality so it wasn't that terrible. In practice few sites actually use the option, but when they do it can make troubleshooting a giant pain in the ass.
My projects had a patch to remove that entire module from core on each build.
Now which is easier—make a new module to serve this page or filter the output of that page, or just enable PHP code for this page, and write it directly in PHP on this page only?
Instead they just believe that they are 100% secure then when they get hacked they act all surprised and with great hypocrisy say "security is our number one priority at shitshow.com. we take security extremely seriously.".
Otherwise I'm not suggesting that there's some great incentive. As we've seen with huge hacks like Equifax and many more companies we know right now they just get a slap on the wrist and so they continue to try to use "we are sorry" PR statements after the fact as their strategy.
I'd change that to "enterprise CMS". Experienced sole proprietor web devs have often moved to lightweight, non-enterprise, non-feature-packed CMSes, this being one reason.
It takes some expertise to do these correctly, but it's not rocket science. I taught myself and have self-hosted Wordpress and Drupal sites for 8+ years with very few security incidents.
That said, I'm moving my Wordpress and Drupal sites to platforms like WP Engine and Pantheon. But not because of security concerns--because the platforms will accelerate our development process by making it easier for developers to create staging instances and deploy to production.
What could go wrong.
All software has bugs, the easier and quicker to fix them and roll out those fixes the better?
If your auto-update system also means a single RCE vulnerability can then embed malicious content into the program itself, that is worse than non automatic updates.
If php has permission to extract an archive with new files over the current ones, it has permission to write a malicious file over a current one.
I’m not talking about a vulnerability on the wordpress central infra that delivers a malicious update via the legitimate update path.
I’m talking about a vulnerability in the millions of wordpress installs that have filesystem permissions to overwrite themselves, to facilitate “auto update”.
I don't do as much with WP these days, but have worked with it some over the years, and work with multiple folks in the geographic area who primarily do WP work (friends with folks who run WP meetings and do all-WP work, etc).
I've seen dozens of exploited systems in the last 2-3 years, and been asked to help 'clean them up'. Every single one of those was some variation on some bot exploiting some code (in core or in a plugin) that allowed for some obfuscated code to be written to a publicly accessible directory, with names like "includes.php" or "sys.php" (things like that) - common words that no one who is not a WP expert would even think to question in the first place (no one leaves BACKDOOR_EXPLOIT.php as a file name, for example).
Yes, you can install stuff like wordfence and similar tools to continually scan your disk to look for 'infections', but... why allow those to be written to the drive in the first place? I know the answer is "convenience", but it's been a huge community price to pay.
Since this is done extremely rarely, that leads me to believe that its more of a cultural problem (perhaps with the WordPress team, even) than a technical one.
Spent 5 years in shared PHP web hosting, can confirm that most of the people are muddling through with the bare minimum of understanding to get WordPress going.
That is inherently not the default for wordpress, and at that point you may as well be installing via deb/rpm.
I didn’t say scheduled updated are bad I said a self writable web app is a fucking security nightmare.
I've found that clicking a comment's timestamp to view a permalink to it will usually allow replying.
What signing keys? Wordpress's automatic updates aren't signed, so your trust is horrendously misplaced.
Someone already did the work for them to implement it[0], and rather than commit it, a Wordpress developer wrote a blogpost saying signing isn't really that important[1].
[0] https://core.trac.wordpress.org/ticket/39309
[1] https://medium.com/@photomatt/wordpress-and-update-signing-5...
On shared hosting I’d have thought a user-account cronjob calling the app via cli (not via curl) could do the updates - but wordpress has always veered for “offer a shit solution to everyone” over “offer a secure solution to some subset of everyone”.
my default for wp installs is to just take write permission off everything all the time, then re-enable it when I know I will be doing something needing to write (updates, etc). There are likely some actual WP users who are not technical yet actually use the system to do their own content updates, but almost everyone I know who's using wordpress is managing it on behalf of their clients, doing all the content updates for them. These people could/should probably get familiar enough with ssh and stronger security practices, but the 'one click update' for themes/plugins/core seems to give a very strong (but ...problematic) sense of security.
I did that more than once back in the good ol' days before the fancy schmancy automatic updates. After you do it a few times, though, you'll probably just give up and modify the file permissions so that WordPress can update itself at which point...
If the user that WordPress (or the web server stack, I suppose) is running as has write permissions on the directory where WordPress lives, it can update itself directly without having to FTP to localhost or whatever. Of course, typically we'd say it would be a bad idea for the application to be able to do that but blah blah risk assessment blah blah trade-offs blah blah ...
For a long while, this was a widely accepted/standard way of hosting multiple customers on a shared web server, using something like suexec+php-cgi or (even better, IMO) mpm-itk. Then, each customer's WordPress site would run under its own uid/gid in order to isolate them from one another. A single customer could break their own instance or allow it to get hacked over and over again, but at least it couldn't get to other users' sites. I'm not sure how the large shared web hosts are handling this nowadays or if anything has even changed.
(Edit: After reading your other comments, mgkimsal, it's obvious that you already know all of this but I'm not going to delete/edit my comment now.)
So many of the problems in WP stem from the extreme 'hackability' from the early days, and a resistance to push for more security by default (the lack of ability to even attempt to install updates via ssh is, imo, a good example).
Well, after several years not much went wrong. There was a single case where the updated broke the update process, thus forcing users to a single manual (still one-click) update. I.e. the worst that happened is that in one instance wordpress was as bad as any other major CMS by not having working automated updates.
What went right however is that probably thousands of wordpress infections have been prevented by automated updates that work 99% of the time.
Wordpress surely isn't perfect, but the fact that they have automated updates that work most of the time means they're more secure than any of their major competitors for now.
You're talking about whether something has gone wrong with the auto-update system, specifically.
However, the parent comment was referencing the risk inherent in granting a PHP application write permissions on its own files. Doing so means that almost any exploit in the application can result in persistence, because malware can be written into (hidden in) core Wordpress files. You would not necessarily know how often this happens, because it happens to individual sites.
The Wordpress community decided that the risk of not patching quickly outweighed the risk of allowing Wordpress to write its own files. I happen to agree with them; unpatched vulnerabilities are like candy for script kiddies and fall easily to automated attacks. And most people running Wordpress are not sophisticated.
That said, for teams that are on the ball about managing updates, running Wordpress with locked down filed/directory permissions is definitely more secure.
The last time switching off comments helped (as far as I can remember but note I only remember the more serious secholes) was 13 years ago and the only reason that wasn't called Drupalgeddon because barely anyone used it back then and naming them wasn't in fashion (and we had two more RCE bugs the first half of 2005 anyways before we kicked out the old XML-RPC library and replaced it with a better one) ... but DRUPAL-SA-2005-002 was very, very long ago. And while Adrian wrote form API around the same time for theming purposes, we ran with it to avoid that kind of bug happening again. We? Or... just me? should I take the blame? It was an ... emerging decision but a lot of it were on me. As they years has passed, form API became frighteningly complex and a lot of that is certainly on me, someone misused it and bam! Drupalgeddon2 . It makes me sad but I do not feel guilty. I felt guilty for Drupalgeddon because that was a silly one, more of a process fail than a technical one, this was not silly, this was just too complex. Frankly, catching the bug in D7 was very near impossible, whoever ported this code to D8 should've seen it because it clearly violated the most fundamental form API principles but when you are busy porting so much code, it slipped through the cracks. Just as Rachel said for Drupalgeddon: I shouldn't feel guilty, others could've caught it too. And, after all, I am out now so it has nothing to do with me any more. Or that's what I am telling myself these empty, sad days and nights.
This is the first I'd heard of you chx, you have quite a storied history surrounding your untimely departure from Drupal.
If nothing else you sound like a great developer, and I hope you've found new projects in the FOSS community which can benefit from your expertise.
So I'd like to configure a bunch of redirects to bounce people from the old Drupal urls to the new Medium pages.
Will I have to do it by hand for each URL I care about, or are there some tools to help doing that, like scraping google or my logs for a histogram of incoming links, to determine which ones matter the most?
I'd appreciate any links or suggestions about tools or techniques to migrate out of a CMS like Drupal and redirect old links to the new pages. How to play well with google and SEO, how to normalize the urls, what kind of redirects to use, what nuances are there about which urls to redirect, etc?
If you wanted to just do something quickly and then in slower time make the finer adjustments you could simply spider and download your current site with wget and then drop the resulting static html into Netlify.
It's free and I just did that with an old Wordpress site that I didn't want to take the time to upgrade and deal with but needed to make secure right away.
I kept a copy of the Wordpress install to work with locally and the live site now just returns static html hosted on Netlify that can't be attacked.
[0] - https://lukespear.co.uk/site-relaunch-2018-hugo/
EDIT: I have another Drupal site that actually needs to stay as such, and have other Hugo sites already, some of which were D7 sites. Just for context.