Today's Email Incident | Hacker News Reader

Today's Email Incident

github.com

256 pointsbuttscicles172 comments

Recognizing that it is not politically correct in these parts to pose this question, I sometimes wonder if the Github team (which seems like a highly-skilled, competent team) regrets building on Rails.

It seems they have been able to use their high skill and competence to laboriously steer a bulky and (dare I say it?) heavily-engineered framework to serve Github as the modestly fast and fairly reliable site that it is.

But would not they be happier had the efforts exhausted on scale, reliability, and repair been available to spend building features had a faster, less confusing platform and framework been selected?

I know, it's antithetical or heretical to imply that Rails is anything but a model of efficiency.

I've stopped building on Rails because of things like this. Point releases shouldn't break functionality. I got tired of regression testing my apps for framework changes.

When I read the blog, the first thing I thought was "boy am I glad I don't use rails". I realize all frameworks and toolkits have issues, but Rails seems to be particularly bad.

I think the big take away here is the importance of apparently redundant sanity checks....

What would be some potential alternatives?

Django?

Play framework, Grails, Node, Django, Revel...

ASP.NET MVC works pretty well these days. It's open-ish source as well, as in you don't need to pay to deploy it on Mono.

I've used it with NHibernate for the ORM and Postgres.

And don't forget the Web API http://www.asp.net/web-api

To rails? Thousands- so much depends on what exactly you need.

  Python? Django, Tornado, Twisted, etc
  Go? "net/http" + Gorilla (http://www.gorillatoolkit.org/)
  Java? JSF 2.0, GWT, Spring, Tapestry, etc etc
  C#? ASP.NET MVC

This list could be almost infinite... or at least could be discussed infinity :)

I've started a side-project using Go + Gorilla. It's actually very nice, but I don't think it's quite as productive for the early-stage as Rails is.

I can certainly recommend Django - I've been working with a large-ish Django codebase for 3 years and have yet to see a security vulnerability as bad as the ones Rails seems to have every month.

Any suggestion that implies Golang + Gorilla is comparable to Rails is poorly informed (and I really like Golang).

"You know what you could do instead of using Rails? Implement your own ad-hoc version of 60% of Rails!"

I don't think that is fair; I did say "so much depends on what exactly you need."

Maybe you don't need to whole kitchen sink. Also its not as though there are not other Go libraries to get other functionality from such as DB/ORM functionality.

What's a good Golang ORM that you could use in a production app?

If people want something like Rails in Golang, it probably makes more sense to point them at Revel. Gorilla is more like Sinatra, for better or for worse.

Yeah, I don't think Gorilla or Revel are quite mature or stable enough at this point to consider as a Rails replacement. And I'm speaking as a (extremely minor) Revel contributor - it's still very young, early adopters only!

Web years are a little bit like dog years. I feel like Revel will grow up fast.

For the record, I have written some Go, but not worked on Revel specifically.

While fully realising that the language itself is pretty terrible, Java + Spring + JavaMail is a pretty good framework and as a bonus involves minimal XML these days.

I'm sure one of the alternative JVM languages would make this setup tolerable to work with, even pleasurable, but I don't know them.

OCaml [1]

[1] https://news.ycombinator.com/item?id=2235204

google app engine (based on python)

Symfony2

Sinatra and Padrino are some of the more common Ruby alternatives, especially for smaller sites.

Mojolicious - http://mojolicio.us

They all have issues - sure - but I've never encountered one so likely to break so spectacularly. Then again, it feels like ActiveRecord is a more complex beast than most others I've used.

I've mostly used Hibernate and NHibernate, and SQLAlchemy a few years ago.

I got tired of regression testing my apps for framework changes.

I get tired of paying taxes and brushing my teeth, too, but I still do it, because life will get worse in a hurry if I don't.

I think he's probably talking about dealing with the fallout from failed tests rather than the tests themselves.

Trying to understand where you're coming from here... I should write tests for every portion of the .NET Framework my app uses because it might break on a security update?

If Microsoft had a history of breaking things with .NET framework security patches, then yes, absolutely. That's what people are saying happens with Rails, isn't it?

What are you, crazy? No, what you do is that you jump ship.

To be honest, I've had a lot more problems with gems external to rails than problems with rails itself; and this particular case seems like an issue that should have gotten caught via testing.

"regrets building on Rails."

I'm surprised they haven't tried to build their own framework / programming language (like Facebook with HipHop). It wasn't a bad idea to start with rails, but now they subject themselves to the issues of Ruby and Rails without an alternative

I sometimes wonder if a good alternative to django/rails is to use the most minimal framework that's reasonable to start with, and then gradually customize it away.

This actually sounds like a promising idea, though only if you've got insane dev chops on the founding team. In the Ruby world, Sinatra would be the obvious place to start. On the other hand, you might just end up reinventing Rails.

Tried it. You spend way too much time solving problems that are already easily solved in the existing larger ecosystems.

Yes indeed, and also leaves everyone in the muck when the lead dev leaves. Your left with a load of code you had wish you had documented as well as the previous frameworks. The value of existing frameworks is the documentation and stack overflow answers.

I love Sinatra, but there's a long-running joke that if you look at the source for any Sinatra application with even a little complexity, you'll find a half-baked re-implementation of Rails inside. I've built apps like that and it's definitely not an exaggeration.

Ah, so following Greenspun's tenth rule [1], perhaps we have this:

  Any sufficiently complicated web application contains 
  an ad hoc, informally specified, bug-ridden, slow 
  implementation of half of Ruby on Rails.

[1] http://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

That was my road to Ruby: Developing in PHP "mvc" frameworks. Even had a PHP active record implementation. Create function was used heavily ...

http://php.net/manual/en/function.create-function.php

That's probably almost the exact quote I saw :)

Yep, especially as there are sinatra stacks that already do that for you in a sane fashion, e.g. Padrino.

Also: yes, building from minimal makes you appreciate the work that big frameworks have already done.

Why would bringing a framework in house be better than using open source?

I'm confused by all these people that thing 'Rails is crap, clearly people should roll their own everything since that will be more reliable and better engineered'

If you think Rails is crap, the only answer is to look to 'more mature' platforms, not roll your own. Spring/Hibernate might be a reasonable answer. Django is not, being at a very similar level of maturity. Nothing in PHP even begins to approach the level of maturity.

Software engineering is hard. Really hard.

No software is a panacea. Decisions are made which favor some use cases over others.

When you are as large as google or facebook or github, those decisions which don't naturally mesh with your workflow become more painful over time. And when you have enough in-house resources, the cost to working with the existing solution far exceeds that of extending it on your own or building your own.

Hiphop is a good example where Facebook saw certain things they wanted to optimize in PHP, and it made far more sense for them to work on their own thing (many of those decisions underlying hiphop make sense for facebook but not for most companies deploying PHP)

When Github started, they didn't just use Rails: they used Rails and shelled out to command line programs to integrate with git. If you have misgivings about engineering at Github now (I don't; I have the opposite of misgivings about they way they handle software), imagine a site built in the equivalent of shell scripts.

I think the short answer is probably, nobody at Github has a lot of macro-level regrets at this point.

I think the real issue is that the Rails community should be supremely grateful that they have a team like Github's building on the framework they've chosen. They all get to draft off the deployment/security/development/reliability work Github sinks into the platform.

And, painful though it will surely be for HN to hear this, snags like the one today are how frameworks eventually get to be solid. Spent much time looking at Java Spring's track record?

I think people forget how transformative Rails was. When it came out it blew people away. We all make fun of the 15 min blog today, but at the time it was really impressive. Nowadays it's not so amazing by comparison because everyone's caught up.

Honestly, I think it's great that this all gets aired out in the open. The amount of transparency behind all of this is a breath of fresh air.

Rails has had its share of growing pains; it really sucks that a security patch had a regression and I think this gave people a lot of flash backs to the pre Rails 3 days. By the end of this, however, we'll end up having a solid security infrastructure.

I'm willing to wager that 90% of the time these kinds of issues just get swept under the rug and silently patched.

"We all make fun of the 15 min blog today, but at the time"

I made fun of it at the time. "Great! Now audit and deploy it."

> I think people forget how transformative Rails was. When it came out it blew people away. We all make fun of the 15 min blog today, but at the time it was really impressive. Nowadays it's not so amazing by comparison because everyone's caught up.

This was true for young developers or people without the right experience developing web sites at the time.

Already in 1999 our company was using TCL framework built as Apache module that was quite similar to Rails. Following some designs similar to AOL Server.

We just happened to be in Portugal and never had the press coverage that Rails did.

By the time Rails appeared we had already moved into .NET/C++ to improve the scalability of our product.

The same scalability issues that Twitter, LinkedIn and other American companies found out with Rails while deciding to move to JVM based alternatives.

>>This was true for young developers or people without the right experience developing web sites at the time.

Very true. Transformative my ass. I was already pretty efficient at building websites without rails.

It was transformative, it just wasn't for you. It made me enjoy building websites significantly more than doing them with PHP and HTML did, and whilst there's always been other options none of them were as simple to get running as Rails.

I'm probably not the only one who found it transformative, you're probably not the only one who didn't, and that's fine. A piece of technology doesn't have to be useful to everyone to be transformative.

What company in Portugal do you work for?

That company was called Intervento, before they were acquired by EasyPhone and together became what is known nowadays as Altitude Software.

I live in Germany nowadays.

Thats why he says transformative. Most of the stuff in Rails was "nothing new", but they got it all together in a package, relased it and _advertised why you should do stuff like that_.

The technology behind Rails is was nothing new, it was just very well applied in a fresh language.

I wasn't blown away. Nobody on my team was blown away. If you had pretty much no real experience then maybe you were blown away....

What blows me away is just how many arrogant programmers there actually are out there..

I agree, it's massively arrogant to think your platform of choice was a real game changer because of your ignorance as to what other people were already doing.

I'd be interested on which MVC framework you feel in the web community changed the whole conversation on how to develop web applications before Rails became popular. Maybe you could school us on this alternate history?

Personally, I have a different recollection. I seem to remember the PHP MVC frameworks and ASP.Net MVC and and whatever the hell Java people use all came out because Rails and Django became popular. I do remember Django helped but as an outsider to the two frameworks I seemed to get the feeling Rails was clearly the thought leader.

The Java ecosystem had MVC a long time before .NET and a long time before rails mvc was even conceived. You seem to be missing a chunk of history.

Rails made MVC popular among lesser developers (bottom feeders, small projects, not much money - this may have changed now). A lot of organisations were using MVC as an enterprise design pattern a long time ago.

I think smalltalk devs were using the MVC pattern in the 80's and it's been in various Java enterprise stacks for a very very long time. It seems that everyone else caught up when being a low paid front end 'developer' became cool in the USA.

"Controller Model View Controller (MVC) is one of the most quoted (and most misquoted) patterns around. It started as a framework developed by Trygve Reenskaug for the Smalltalk platform in the late 1970s. Since then it has played an influential role in most UI frameworks and in the thinking about UI design."

- http://www.martinfowler.com/eaaCatalog/modelViewController.h...

I remember building pseudo-blog-like thing with an ISAPI web application in Delphi with WebSnap back in... 2001? Simply by dragging and dropping components and filling in a few things like table names and columns.

Understand, that doesn't take away from Rails' achievements, but it does underline that people in tech, especially web stuff, have a very short memory.

Github is not a website in the normal sense. Github is a layer over top of Git. One of the best wrappers for libgit/what ever there is (regardless of Language) is the Ruby rugged library which Github pretty much built. Before that was grit which was also Ruby based. And what Github was built on.

Everything else is just extra they added on. Their core product is a Web based Git.

But I sometimes wonder this as well, I love Ruby a ton. But I wonder how Github would have done had they used another language PHP, Python, Lua what ever.

But then again the same thing can be said about Facebook, would they need less servers if they switched to another language?

It's become fashionable to hate on rails, and honestly I understand the motivation--it isn't cool and cutting edge anymore. It's slow. It doesn't do concurrency well. Ruby is nice for what it is, but it lacks the power and stability of functional languages, and lacks the performance of go or javascript on v8.

But rails is just doing what it's always done: trying to optimize developer productivity for front end web dev and routine web tasks. In this it's far ahead of the pack, and for many businesses, both small and large, this is exactly the right thing to optimize. Say what you want about github, but they push a lot of features and push them fast.

Rails is not for crafting algorithms, but when you need to crank out 6 new pages with various complex crud views and forms by Friday, and by the way can it be exportable to cvs and oh can you also do paypal and oauth integration and user avatars with thumbnails, and keep it all reasonably readable and organized... the sad fact is that for all of rails' heft and lack of cs cred, trying the above with any other framework after you know rails will drive you insane, because despite how great the language is or how many connections your server can handle, you are again and again spending days on things that take an hour in rails.

Ultimately, I don't think the way forward from rails is to search for a partial reimplementation of rails in some other language. Rails is the best we've got for rapid front end and shallow back end work. So use it for what it does well. For what it doesn't do well, use something else. Personally, I think it's silly to have an ORM written in ruby. ActiveRecord is for the most part a lovely api, but its guts should be a separate service written in go or something. Building a complex recommendation engine or game AI? Write clojure or haskell services. But when the data comes back and you need to dice it up and lay it out as html, with a hundred little ohs and ifs and buts, it's not a fun time without rails.

Intereting. Your arguments sound a lot like the usual arguments in favour of PHP.

>>It's become fashionable to hate on rails, and honestly I understand the motivation

Well, they were pretty vocal about calling everybody, in no uncertain terms, for using frameworks other than Rub/Rails, an idiot. There is plenty of bad will that they've been accumulating because of that and this is simply the backlash.

If you really think you are the best there is no need to insult developers who do not use your frameworks. Unfortunately that is exactly what they did. People have long memories.

And in my opinion they deserve every bit of it.

The comment below by user mhartl demonstrates the attitude of pretending everybody else is shit.

>>Ah, so following Greenspun's tenth rule [1], perhaps we

>>have this:

>> Any sufficiently complicated web application contains

>> an ad hoc, informally specified, bug-ridden, slow

>> implementation of half of Ruby on Rails.

>>[1] http://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

The nerve to call other frameworks bug-ridden as if their shit didn't stink. Specially in light of all the security problems with Ruby/Rails that have come to light recently.

Who is "they"?

they=Ruby/Rails community

I don't care about cool or cutting edge or even performance. I hate rails because it is built by people with no understanding of security.

But most of all I hate it because it uses so much magic.

What makes you think RoR was built by people with no understanding of security? Rails was one of the first frameworks to do things right (CSRF protection comes to mind).

I was looking for the date when RoR changed their templating so it always output sanitized content and I found this:

http://stackoverflow.com/questions/698700/escaping-html-in-r...

Correct me if I am wrong but it seems to me that RoR still isn't shipped with a safe default?

Edit: RoR added auto-escaping around February 2010, Django had it since November 2007

http://yehudakatz.com/2010/02/01/safebuffers-and-rails-3-0/ https://code.djangoproject.com/wiki/AutoEscaping?x=52&y=...

You're wrong. It has auto-escaped since Rails 3.

http://yehudakatz.com/2010/02/01/safebuffers-and-rails-3-0/

Rails changed the default to safe output about three years ago. 3.0 escaped output out of the gate, and 2.3 switched the default in the 2.3.6 neighborhood.

Prior before that, the standard way to sanitize html output was to write <%= h foo %> instead of <%= foo %> in your templates.

Their (abuse) of the Yaml parser. It is one thing to use a potentially dangerous parser on a file that you aren't going to let anybody touch either (your database config); it is downright stupid to use a dangerous parser on untrused user input (json) a format it shouldn't even parse to begin with.

And as of late tons of similar bugs have been found. CSRF and database parameterization was well known issues when Rails was written, so yeah props for that. It is just not enough anymore.

Well, the magic is what enables the big productivity boost. It's a higher risk, higher reward approach. In some domains that added risk is unacceptable, but for many applications it's a very worthwhile tradeoff.

I think often when people say "I hate rails", they are really saying, "I hate building the types of apps that rails excels at". Which is fine, and makes sense, because rails excels at making apps that can be very cool and innovative on the product side, but are usually pretty boring on the engineering side--it doesn't really get interesting for the engineers until it's time to start replacing the lower level tiers of rails with something else. And while secure best practices are certainly stressed (and included as defaults), conservatism isn't seen as a valid reason to hold up progress. So you get both the speed and the occasionally bumpy ride that strategy entails.

That has nothing to do with it whatsoever. They specified why they hate about rails, the magic. Personally, I simply cannot stand magic in frameworks I use. That's because I don't become truly productive when using a framework until I fully understand how everything works.

It is not enough that it does work, I have to understand how it does. Otherwise, I'm constantly bogged down when implementing every single feature thinking about possible security and performance issues down the line. The performance thing might be premature optimisation in some cases, but the security definitely is not.

That's why I hate when frameworks or IDEs have so much magic -- I'm looking at you Xcode, I can't count the times I was wondering if a particular statement would cause a memory leak or not before I finally understood most of the related magic Xcode and the toolchain for iOS dev do.

For me at least, magic saps my productivity, it doesn't boost it. It also offers a steeper learning curve to proper mastery to get to that point where I'm actually productive. So yes, I haven't tried ruby on rails but I can definitely see myself hating it if it has too much magic.

I can somewhat relate to what you say, and that is perhaps why I prefer Django, as it has just the right amount of magic for me.

Have you taken a look at the play framework? Yes it is inspired by rails, nearly as productive and very little magic.

I don't get why you think I must secretly hate what rails is used for -- I don't, at all. And yes when rails was written the productivity was so much better than its competitors, rails was the choice.

Today, not so much. Don't get me wrong if it is an internal wiki or other system then rails makes sense, since if you get hacked you can always find the guilty person.

As for conservatism? Dude I write HTML5 games for a living and I am not arguing for JSPs here, just Play or Django or Sinatra (Ruby is great, don't get me wrong) but Magic is going to bite you at some point.

I haven't personally looked at the play framework, though a former colleague who is steeped in both rails and scala did a pretty thorough comparison a few months back. He is definitely not a rails partisan, and much prefers scala to ruby as a language, but his conclusion was that while play has a solid core and does many things right (including some things better than rails), it comes up short on convenience and utility functions, templating, and ecosystem.

I think the ecosystem in particular is an area where rails magic makes it possible for the community to create very powerful gems that just work, and is probably where the most time is saved vs. other frameworks that have more of an explicit bent. I understand the skepticism towards the implicit approach--it has certainly bitten me before, and I try to minimize magic in my own rails code wherever possible--but again, if you're optimizing for productivity, it can be a worthwhile tradeoff.

Take devise as an example. Despite using all kinds of implicit and convoluted magic underneath, it can save huge amounts of time, especially early in a project, and is well tested and proven in the community. I'm not aware of a lib in any other framework that can handle so many boring authentication issues (sessions, roles, encryption, email confirmation, lockouts, remember me, forgot password flow, session expiration, redirect flows, etc. etc.) with such minimal code and configuration.

Sorry if I put words in your mouth. I do think a framework like play has more conservatism in its blood, which is great. It's good that there are other options for when security, stability, and scalability are at more of a premium than developer productivity .

Favoring Ruby over Scala... Insanity! ;-)

But seriously, maybe your friend just wasn't all that well versed in Scala. I'm no pro (yet), but on the topic of Utility functions, I haven't run across any big holes. The Collections API is very Fluent (in the Martin Fowler terminology describing Ruby's Array), so it's hard to fault that. In fact, it goes quite a bit farther.

You have Option, Try, Either to use FP in a way Ruby-only programmers probably won't ever grasp. I certainly didn't. I've never seen any real Ruby code that comes close to a foldMap (foldLeft(Tuple2[Accumulator, Map])), or a Try pattern match, or (you could go on and on...).

Templating I don't get unless you're wed to HAML maybe. There are similar libraries for Scala, but it's hard for me to imagine someone being able to compare Twirl (Play's templating) to ERb and declaring ERb superior on any metric. Twirl is arguably simpler, definitely far faster, has no identifiable magic, and Scala+Twirl makes it silly easy to avoid a whole class of common problems with branching logic (hello id=4 my old friend!).

Play has First Class Forms. Rails doesn't. I'm sure Rails will some day. Declarative data binding is simply better. Safer, simpler, provable correctness.

I'd give Play a second look. :-)

To be fair, the one gap I've come across is in String manipulation. A few minutes of googling and a little tweaking and I've got a quick slugging function though: https://gist.github.com/sam/5213151

And then there's stuff like writing:

  10 seconds

Which makes me giggle a little inside every time I write it. :-D

On the other end of the spectrum, there's all the things Play gives you Rails doesn't: Futures. Actors. Async. Chunked Responses (which maybe Rails 5 will have?). WebSockets. Awesome XML (it's nice to use a lib that does not suck if you have to deal with it). Better JSON support. Functional Programming. Pattern Matching. Performance. Built in (Memory) Cache Concurrency. Functional Testing with Selenium. A built-in production ready web-server. Continuous Testing. Deployment Packaging. Bootstrap. LESS compilation. SBT: A single tool that provides file system watchers, Tasks, compilation, packaging, dependencies (including Git). All the JARs a Jar Jar could Jar if a Jar Jar could Jar Jars.

The only significant downsides I've run into are:

1. Sharing your code as a package is harder. Much harder. Nexus isn't nearly as low friction as rubygems.org. 2. Binary Compatibility concerns means sometimes your best option is a source dependency.

Git dependencies have been a good solution to #1 for me so far, but I know the day will come when I'll have to figure out hosting on Maven Central.

All that stuff by Friday... I can do that in Java / Spring. It will also be high performance, extendable and maintainable.

Rails brought very little to the table in that respect. Not to people who actually know what they're doing anyway.

Are you using the same Java/Spring I'm using?

Would be nice to see some side by side code of Rails and Java/Spring doing roughly the same things, both written by people experienced in those technology stacks. These debates aren't very useful without code to look at.

I'm sure Rails would be less code and quicker to production. That's not to say that it can't be done quickly in Spring - Java.

The whole 15 min blog thing is a good example of fast Rails, it's also a good example of non production ready i.e. maintainable and extendable, rails.

Rails isn't the problem. If it wasn't this bug that caused the emails to be exposed, it would have been some other bug. The problem was that there wasn't enough test coverage, and there weren't enough runtime assertions to shut off failing software. (if to_emails > 10: logging.fatal("Sending message to too many emails %s", to_emails))

So if I'm driving my car and the accelerator pedal gets stuck... it's my fault for using the pedal without proper extensive testing? Even if the pedal got stuck after fixing my door locks because the manufacturer stuck the pedals as a side effect with no previous notice?

There is no excuse: change of behavior shouldn't happen when installing security patches, specially if it's not mentioned anywhere in the patch release notes (which were reviewed by the GitHub team).

Okay, I get your point, they should have tested it, you're right, but saying that Rails is not the problem is kind of being in denial.

It's more like taking a homebuilt experimental aircraft out for its first test flight. When the engine blows up 30 seconds into your flight and you crash into a house, you'll probably wish you turned the thing on on the ground before actually flying it. Sure, the engine was faulty, but any of the parts could have caused the failure. The root cause was that you didn't even try to see if the thing worked in safe conditions.

Security updates are not experimental! They're security updates.

If it works fine and it breaks with a security update then it's the framework maintainer's fault.

Like I said: yes, the engine failed, and the engine shouldn't fail. But it's unreasonable to expect that if only the engine had continued working, the test flight would have gone fine.

This is not a product that is marketed as something still in testing. Rails is supposedly mature by now.

A closer analogy would be if you were an airline and a technician from Boeing made some Boeing-recommended safety changes on a plane before it took off and then the engine blew up.

Sure, you should have had proper oversight of what said technician was doing but the bigger part of the blame is on Boeing for a faulty safety change.

I'm not talking about Rails, I'm talking about Github's mail-sending program. Rails is the engine. The email sending script is the experimental aircraft.

Blame Rails, I'm fine with that. But it's not going to prevent a serious problem with the email sending script in the future. To do that, you need to write some tests.

So with Rails, applying security patches is "like taking a homebuilt experimental aircraft out for its first test flight".

Hmm, I don't think I want to start my next commercial airline business on Rails.

FWIW, popular advice for TDD is "don't test the framework, test your own code".

I previously assumed the cause was some marketing intern pasting email addresses into the wrong form. Better training (and a slap on the wrist) would prevent it from happening again.

But it's somewhat scarier knowing this was due to a bug triggered by a security patch. Who knows what yet-undiscovered bugs are still in there.

That said, there's no such thing as bug-free software and I still find GitHub to be trustworthy.

I don't think anyone doubts github's trustworthiness, but as an enterprise customer, I do wonder if they have tight enough controls. Marketing emails are an area that companies should be spending extra time testing as those screwups tend to be irreversible.

Very concerning - they accidentally distributed their entire customer list and now I have to handle the internal PR fall out re: my email address and that those of non-technical co-workers (board, accounts, finance, engineering managers) have been distributed globally and then subsequently pastebined. Times like this I'm glad that our code is hosted internally and access to it is controlled internally.

https://dl.dropbox.com/u/109905580/Screenshots/qg~abn4fl601....

https://dl.dropbox.com/u/109905580/Screenshots/73dw6%7Eq_h4h...

I wouldn't normally consider my email address a secret. I assume the concern is over spear phishing?

Email address isn't secret, the use of certain tools for some however is. Enterprise is a different game than your usual early stage web/product startup. Politics are real, exist and credibility is on the line when you stick your ass out to vouch for a company/product outside of the company standard. Base cost for this software is $5k/yr for one seat pack (20 devs), whereas the competition - Atlasssian is incredibly price competitive (10 users for $10, 25 users for $1,800) - keep in mind competition also exists in the form of TFS (ergh) which is free for MSDN subscribers and included in most enterprise licensing agreements.

Sorry, still not sure I follow. So the issue is that, for example, the corporate office could discover that a certain department is using what--in their opinion--is an overpriced solution?

An overpriced insecure solution.

Exactly. Perception management for those who will never read or care about the technical explanation but will know about it tomorrow when the industry publications they read publish it in their early-morning newsletter.

Now imagine some poor developer stuck with TFS or Visual Source Safe right at this moment trying to convince his line manager to change to the "overpriced insecure [non-standard] solution."

For enterprise, decisions on what tools are used (or shortlisted) are almost always made by those who don't use them thus why it is important to manage perceptions. TL;DR - enterprise is broken.

Confirm. Perception management is one of the key things that starts coming into play when you have people far up enough the chain that they don't use the tools.

Perception is actually everything, it turns out. :-(

If a competitor is looking to market their version control solution to enterprise this email list is going to be a very good place to start.

On the flip slide, if a developer cares about their tools and is looking to jump ship then they now have a shortlist of companies and contact details as to whom to approach within the company. ;-)

Atlassian isn't going to spam Stash or BitBucket to the exposed users. They'd get crucified.

There might be a few users who were already frustrated and switch on their own, but I think the big problems are an ongoing spearphishing risk and a bruised ego.

Come again? Atlassian doesn't email you when you need to repay or when you had paid?

This wasn't "spam" it was an accident email send. Things pretty much every single giant website has accidently done at least once.

I was responding to my parent comment's concern that GitHub's competitors (e.g. Atlassian) would use the leaked/pastebinned GitHub emails as a mailing list.

I've already received an "invite to check out Dropbox" to an email address that only GitHub knew.

As did I... I replied tongue-in-cheek and the kid sent me what appeared to be a pretty heart-felt apology. Probably just saw a quick chance to get some upload credit, and didn't pause to think he was spamming a list of professionals in high-tech :)

It looks like you printed that email and then splashed some water on top. Why would you link to that in the first place?

Proof that I was one of the enterprise customers who had their information disclosed and thus is directly affected by this unlike 99% of the other posters in that run-away lol ruby thread which is unfortunately at the top of HN.

This is bad - real bad. Goes to show you if the 'golden boys' can make mistakes such as these, what about the other 96% of companies handling your info?

I don't see what's bad about it, well except for the bug itself.

Computer software is bound to have bugs, and if you ask me. a company that discloses those bugs, thoroughly explains the cause, addresses how to avoid them the future and apologises to their customers is on the right track.

Don't get me wrong, you software should be very stable, but when the inevitable happens, well this is the way you address it.

A modern mail server should have features to prevent too many users from being present in the To: field. This will also prevent a lot of the Reply All problems we see regularly. This can be implemented as part of a Quota management system.

Wow, I had to scroll a long long long way to finally get a reasonable suggestion on how to fix problems like this. A problem that potentially EVERY software has, regardless of framework or language. Kudos to you.

Exim does this since forever (http://www.exim.org/exim-html-current/doc/html/spec_html/ch-... - look for max_recipients). I would assume other MTAs do so too.

Problem is, options like these feel nice until they annoy you because you do in fact need more recipients. At that point you build a workaround into your software to send a new mail for every max_recipient users which brings you right back to square one with the additional technical dept caused by potential bugs in your batch sending code (off-by-one errors for example)

Be careful what you wish for :)

Just like one can have safe_parse() and unsafe_parse(), there could be an elevated API that could be safely used when vetted. For instance, when the SMTP server rejects a request it could put it in a special queue which requires manual confirmation.

This type of problems happen so regularly. In fact, a Japanese trader accidentally punched one too many zeros and lost a lot of money for the company. This type of problems can get very expensive in an instant. As an industry, we tend to think of input validation in passing because we don't pay enough attention to the problem domain.

This is potentially a pretty big security hole for apps, right? For example:

    posts = current_user ? current_user.posts : Post.public
    posts = posts.where(:user_id => params[:user_id]) if params[:user_id]

In the old Rails, that code is guaranteed not to show any private posts. But with the new Rails it'll show you everything written by a user.

Granted, it's pretty awkward code, but I can imagine situations where people might be inclined to write something like that.

I think that sort of code is extremely common. It's a bit less awkward if `current_user` is never nil, but sometimes a non-saved Guest record, letting you write code like this:

    @posts = current_user.posts
    @posts = @posts.where(user_id: params[:user_id]) if params[:user_id]

In case anyone's interested, I think this is a Github Issue on the same bug:

https://github.com/rails/rails/issues/9813

Thank you for posting that (one of the only informative contributions on this whole thread). That issue description is worrisome because the problem actually isn't limited to `default_scope`, which many people avoid, but also affects just `scope`.

I'm impressed. I thought it much more likely that somebody carelessly pasted a bunch of addresses into a "To" box instead of "Bcc".

Would it make sense to just always put the recipients in the BCC field to ensure you don't accidentally leak email addresses if/when something like this were to happen again?

"putting them in the To: field to enhance deliverability"

Yeah. Is there a strong argument against sending separate emails though? Selfishly curious because that's one of the things we've been doing to prevent this sort of mistake.

When they start taxing emails, those separate emails are going to cost a pretty penny :P

A lot of enterprise spam filters are triggered by a burst of incoming mail from a single sender, sending one email with a bunch of to: is less likely to trigger them.

I've had to deal with too many stupid spam filters in my life.

I suspect there are spam-filter considerations. Also, some email clients distinguish between "email addressed directly to me" and "email I happened to get through some mailing list or bcc".

Not really. They thought they were emailing administrators at the one organisation. From the recipients point of view, it would be much easier to deal with if you saw who it was going to. A bunch of BCC recipients within the same organisation is going to lead to confusion.

Wait, so what exactly is the situation that's at issue here? We upgraded yesterday, too, and have both scopes and class methods that we append onto has_many/belongs_to relations, but haven't experienced anything out of the ordinary. Now I'm terrified we have some unexpected behavior lurking somewhere that's not tested or something.

> In this case, when using the scope method to define an Arel scope on Organization, the where clause of the scope is overriding the condition imposed by the Organization#teams association. The part of the WHERE clause meant to restrict the query to Team records related to the Acme organization was dropped.

This doesn't happen in all cases -- all the cases where I've done this appears to be fine. And a scope just blowing away an association in all cases would be too obvious to not be caught.

Is it because both the scope and the relation select by the same attribute? Does that attribute have to be `id` for this bug to occur?

Or does it have to do with the fact that they're nesting a query within the where clause to determine the IDs? (I don't see why this would matter).

I guess I'm off to a REPL unless anyone here knows more concretely what the issue is.

EDIT TO ADD:

I can't reproduce this behavior. I just spun up a new Rails 3.2.13 app, and created those two models. Going through what they do in the console, my results differ on the potentially dangerous part:

     1.9.3-p0 :015 > teams = acme.teams.using_octocats_scope
     Team Load (0.3ms)  SELECT "teams".* FROM "teams" 
     WHERE "teams"."organization_id" = 2 
     AND "teams"."organization_id" IN 
       (SELECT id FROM "organizations" 
        WHERE "organizations"."has_octocats" = 't')
     => []

I do get an empty array, as is supposed to be the case. I notice that my nested select remains a SQL statement, rather than being evaluated to '(1)' as in their example.

So... I don't know if I'm doing something wrong, or if GitHub has other code / configuration settings interacting strangely here.

Can anyone else reproduce?

EDIT AGAIN:

Here are the four files you need to reproduce yourself. https://gist.github.com/losvedir/5202121

ONE MORE EDIT:

https://github.com/losvedir/test_github_thing

There's a repo with my history in the console. Dunno why I'm seeing the behavior I am.

I can reproduce it. I'm very worried also because I love using scopes. Here is a repo that shows the problem:

    https://github.com/pjungwir/scope-error

I think it is actually a mite simpler than the gist in your "EDIT AGAIN."

Yeah, very strange. I saw your comment so I decided to put mine on github, too.

    https://github.com/losvedir/test_github_thing

I'm going to bed now, so I can't really compare mine to GitHub's to yours right now, but it seems to me that sometimes it's a problem and sometimes not. I dunno...

I can reproduce it reliably by first following a has_many relationship, then adding a scope to that. Adding a where clause (directly or inside of a class method like in Github's example: it's the same thing) is okay. Also the scope must filter on the same column as the foreign key. So this wouldn't cause a problem: `scope :recent, lambda { where("created_at > ?", Time.now.utc - 1.week) }`.

SELECT "projects". FROM "projects" WHERE "projects"."user_id" = 1 AND "projects"."user_id"=5

This expected SQL doesn't really make sense to me. It would be better if your example didn't include selecting twice on the same column (not a join) with an equals clause, as I'd expect that to be removed (it's redundant and would always produce zero results). So the resulting sql makes more sense (replace where clause when new where clause uses same col):

SELECT "projects". FROM "projects" WHERE "projects"."user_id" = 5*

Looking at the github example, they used xx=1 AND xx IN(1,2,3), which makes more sense as it could possibly produce results, and I see why they were doing that.

I realize those two conditions ensure the query will never return results, but the point is to let you combine business conditions at a higher level of abstraction, which may indeed result in zero results. There are many cases where this sort of thing is useful, and it's important that the conditions be cumulative.

Bad incident, great transparency and handling. Just don't do it again please... :)

Sending emails always requires more attention than expected. Everyone, even Apple in 2007, made some severe mistakes in delivering emails. By using a professional SMTP relay (disclaimer: we provide SMTP+) this wouldn't happen since, for example, every A:/CC:/CCN: recipient is splitted from the original message by default.

http://pastebin.com/JX1PdiMB

You'd think the .edu addresses would go to bitbucket and get free unlimited hosting.

How would one unit-test an app like this, where you have to talk to an external database and over SMTP at a specific time of day? Seems like in a case like this thorough unit testing needs to be very extensive to catch a bug like this, yet it would have been worthwhile.

You would only need to unit test the programmer written ORM code against a seeded database to be sure you are getting just the data you expected.

You probably wont catch new issues with the ORM (which you should not be testing) but you will ensure you wont put this bug back into production.

And this is why you shouldn't use ORMs. Magic blackboxes always bite you in the ass eventually.

Yeah, why use SQL or language bindings, just write into the B-trees themselves.

Because there's no middle ground between ORMs and packing your own varints into registers.

I'm not the one that implied all blackboxes are bad.

https://yourlogicalfallacyis.com/slippery-slope

I'm so excited to see this posted everywhere. It will be just like reddit where everyone makes a loose connection to what they think is a logical fallacy and then just shouts that instead of taking time to actually put thought into a post. And I can just imagine the smugness that goes along with posting that.

He implied black boxes are bad. ORMs are black boxes, so are dozens of things that programmers use every day. I understand some of the arguments about why ORMs are bad, but it hardly stems purely from them being a black box.

BTW: My post is not an example of a slippery slope fallacy.

Yes I would definitely trust an application much more if each and every one of its SQL queries was lovingly hand crafted by a human being.

Because if there's one thing computers are truly bad at, it's transforming high-level things into lower-level things.

As an expert on security, what makes you think that in 2013 every webapp out there should still be backed by SQL and running SQL queries?

I mean is SQL something inherently so secure that all webapp devs should use?

If you're not a DB expert, what would be your rationale for using SQL instead of other alternatives?

It's not like if every website out there was using SQL. Heck, not even all sites are using OO languages.

So while I agree that an ORM is nice if you're stuck that particular Java/C# + SQL hell I fail to see how exactly OP did imply that he recommended objects and SQL...

He said ORM sucked. Not that he was using "objects" and SQL...

SQL is not intrinsically insecure.

Come on, the parent comment more or less implied that ORMs are bad, not that he was using flat files or a non-relational DB (many of which are often wrapped in ORMs anyway).

If there is one thing computers are in fact bad at, it is understanding intentions, an ORM in many cases is a translation of an intention from one level to another.

In my experience lovingly hand-crafted SQL for your application does not take as long as you think it will and does not subject you to the interpretations of an intermediary. ORMs are useful but there is something to be said for writing the SQL you want to do what you want with a database.

Until someone gets the bright idea to push all your complex queries down into stored procedures. Then it becomes a mess, the dba quits, you go back to an ORM, have some weird issue like this Rails bug happen, then someone says, "Hey, what about using raw SQL...we'll just make the gnarly bits into stored procedures..."

Circle of life, man.

Is this sarcasm?

The excuses are on an on. I admire Github guys. They give a super cool impression to the other outside world and they're doing a great job. But the recent outages and the problems are giving high attention to Github. It's giving negative sign for the people who are interested go for Github way of hosting code. Either their own premise or on github. giving detailed analysis is a good tech thing... it fires developers heart and we are still going admire what they did to resolve this issue but definitely this isn't a good sign for github.

I whipped up a quick example Rails project to demonstrate the issue. It's a lot like the Github post, but a little simpler: for instance, there's no real reason to talk about class methods:

https://github.com/pjungwir/scope-error

I'm pretty concerned about this bug, because scopes are one of my favorite Rails features.

For quite some time, I have been using Sequel (http://sequel.rubyforge.org) as an ORM with Postgres instead of ActiveRecord, and have never been happier. Overall, Rails has been a great web-framework, but ActiveRecord has been it's sore spot, at least for me.

Also, using the sequel_pg (https://github.com/jeremyevans/sequel_pg) on Postgres ensures that even raw SQL queries are type-casted properly, plus it supports streaming and cursors, while also being very very quick.

If you want to stick with Ruby/Rails and relational databases, but think ActiveRecord might not be your bag, give Sequel a shot. It also fits nicely into lighter frameworks such as Sinatra.

Non-technical companies which do this tend not to apologise - it draws attention to the mistake which most of their (non-technical) customers never noticed in the first place.

I guess github doesn't qualify for the non-technical crowd :)

I'm trying to figure out how likely my apps are to be effected by this same bug/behavior change as Github's, and if I should delay installing the security update without more investigation.

Not a great situation to be in with a security update.

Seing as how this list of customers are about to run out of their trial for Github Enterprise, it'd be funny if atlassian or any other hosted Git providers were to make use these emails for their own gain.

a little more on the incident http://chadscira.com/post/5148b1f0f93e934b550112be/Github-le...

atlassian? :)

i bet github grinned when they saw that email popup

Atlassian builds a JIRA connector for Bitbucket and Github:Enterprise. To test the connector, an installation of Github:Enterprise is required.

Internally we use Stash of course ;)

Frameworks are innocent.

Rails security point release breaks apps by changing the ORM.

News at 11.

I'm not sure why you found it necessary to post this comment. I found the original article to be a well-written warning about a rather serious bug that might affect a lot of people. But your comment seems to be sarcastically suggesting "move along, nothing to see here."

I'm not sure why you found it necessary to post that comment, or why I found it necessary to post this one.

And I can't figure out why either of you two posted. Or why I'm posting.

I guess it is what it is.

I don't think the GP is criticizing the article, he's using the article to criticize Rails. Not so much "nothing to see here" as "well, what did you expect from Rails?".

(I'm explaining what I think the GP meant, not my position; I'm an old Perl guy, still undecided on the overall trade-offs offered by Rails)

Yes, I was pointing out that Rails made backwards-incompatible changes to core functionality in a point release primarily targeting security. I don't know how much more succinctly I could have originally put it, and I thought that it was obvious that this is a Bad Thing. You can't reach everybody.

The "backwards-incompatible change" was part of a security fix, not an unrelated non-security change.

Github's code doesn't rely on the security flaw to function. Their code should keep working after the security fix.

Anyone who has ever consumed an API to build something they don't want to break understands this.

Yes, it is obvious that this is a Bad Thing. That's why, no matter how succinct, I don't think your comment is particularly useful. I didn't find that it added anything to the discussion.

Not every comment on Hacker News is useful. I'd say I was going for poignant if anything.

I took it more like "oh look, another Rails security flaw, big surprise". Not "move along, nothing to see", but rather "take pause and reconsider using Rails".

The point isn't that they had a security flaw, it's that the patch release fixing it changed the god damn default ORM's semantics!

I wonder if this is Karma for all the hyperbole and bad nature comments that the Rails community has thrown at Java. This is some really good schadenfreude. Please keep them coming Rails/Ruby.

Reply on news.ycombinator.com