Twitter is built on open source software | Hacker News Reader

Twitter is built on open source software

twitter.github.io

129 pointsAshuu53 comments

Twitter is also a leading silicon valley tech company with hundreds of extremely intelligent employees. You could easily argue that they may have one of the best engineering staffs in the world. Given that, of course they would use open source versus buying enterprise products.

In terms of engineering staff, they are the exception, not the norm. A company like home depot or kelly moore paints would have a tough time relying on open source for all their operational software.

I might put SpaceX, Google and even companies like Boeing very far ahead of Twitter when it comes to "one of the best engineering staffs in the world"

EDIT: I added Google because of robots, AI and driverless cars, I'm not really sure Twitter is on that level.

That's just nitpicking. Twitter is considered a top tech company to work for in SV, IMO. Their hiring standards are no different than Google's or Facebook's.

What you're doing is stupid. All you're doing is placing software engineering behind all of the classes of engineering. Why not mention companies like BP, Pfizer or Toyota ?

In terms of software engineering Twitter is absolutely one of the best around.

Citations or evidence required for your claims that Twatter is absolutely one of the best around.

This is one of the more tiresome class of comment on HN. The parent was self-evidently just an opinion.

> All you're doing is placing software engineering behind all of the classes of engineering.

Is doing so wrong?

Calling what Twitter does -- SMS with ads and RoundRects -- leading software engineering is an insult to software engineers. If twitter built cars and planes, I would walk.

It's no insult. Given enough scale, simplest things like posting 140 character messages become engineering challenges.

A company like home depot or kelly moore paints would have a tough time relying on open source for all their operational software.

For vertical software systems, probably. For applications developed in-house, I'm guessing that even at Home Depot or Kelly Moore it's the other way around -- more and more, you have to make a good case for using a proprietary library. The connotations of "open source" have changed somewhat, and open source libraries are becoming the norm, even at large old-school software businesses, where the lawyers hate open source.

Shouldn't it be "Twitter is building open source software"? Pointing out the obvious since most of the software companies are built on open source software but only a few build one.

True dat.

I think close to 100% of the internet is built on/with open source software.

But it's a major headache to get your in house libraries/software released as open source. There's reservations because by default everything developed is considered proprietary and a trade secret. It's difficult to convince management to give stuff back and/or see a benefit, the larger the org the harder it gets. Then comes legal with the legal derp.

And finally you can't just put stuff on Github and expect it to thrive, you need resources to maintain your open source projects and again convince management beforehand that it's not just a liability.

Can you list a few benefits of releasing custom coded solutions to the public?

The only one I can see is building culture/name for yourself (which I don't consider a trivial thing), but I can't think of anything else.

I built a number of those open source libraries (Summingbird, Storehaus, Algebird, etc) - often more than 50% of our contributions would come from outside Twitter. Scalding has far more outside activity than inside, where a few talented folks are holding down the fort.

Sam, you don't hold down a fort. You just hold it. Forts don't fly away. <3.

A scala type hierarchy DOES feel a bit like a pillow fort on a blustery day.

> <3

You probably mean this [1], but all I see is this [2]. What's wrong with me?

[1] http://www.urbandictionary.com/define.php?term=%3C3&defid=11...

[2] http://www.urbandictionary.com/define.php?term=%3C3&defid=13...

Sure. Your grateful users might add a feature you didn't have time for. Or they might find a security issue and send you a patch as a part of responsible disclosure. Or they might start hacking on your codebase and actually do something cool. Then you can hire them/buy them out, turning your codebase into a nice recruiting tool.

Think about this: if Twitter open sources 100% of their code, what would change? Would Twitter clones pop up all over the place? No, because Twitter is the software + infrastructure + name. Most companies think that their code contains some type of secret sauce that makes it special. In reality that's not true. Sure, Google may hide their exact PageRank algorithm, but they don't need to hide their web server code. Or their indexing algorithm. Companies like Twitter are even better for this: they do absolutely nothing that's really proprietary.

For an example of this on a much smaller scale look at TheTVDB (http://thetvdb.com/), their entire site source is OSS, yet there are no clones.

For some companies where the barrier to entry is very high or requires a lot of market share to compete, this may be true. But for others where the investment in actually building the thing was the only barrier, giving the code out to the world can be a terrifying prospect.

You are right, it's not always appropriate. However, in most cases, it's just fine. Here's a list of companies that I think could open up at least 95% of their code: Netflix, Hulu, Amazon, Tesla/Ford/GMC/Toyota/BMW/any car manufacturer, Apple, Home Depot/Walmart/Safeway/any retailer, Mint, Twitter, Facebook, Google, and many more.

http://tom.preston-werner.com/2011/11/22/open-source-everyth...

I don't think it's anywhere near "close to 100%", like you're claiming.

IIS is still quite widely used for many web sites. Netcraft's latest survey results suggest just slightly under 30% of all web sites are served by it. So that's some closed-source software powering a significant portion of the web, at least.

And it's quite safe to say that those IIS installations are running on some variant of Windows, which is yet another generally closed-source software system. Then there are other non-web services (DNS, FTP, and so on) handled by such Windows systems.

There are still a surprising number of proprietary UNIX systems out there in production, on the public Internet. We're talking HP-UX, AIX, UnixWare, and even BSD/OS in some cases. There are many behind the scenes, indirectly supporting web sites and other publically-accessible services.

There is a lot of other networking gear that runs proprietary software, too.

Is open source software important to the Internet? Absolutely. Is it the "100%" you're claiming? Absolutely not.

Yes, you're right if you take it that way. I didn't mean the internet is built exclusively with open source software, but the majority of developed applications contain at least a bit of open source software, like a library, curl, Apache, PHP etc. That's much closer to 100% I think. Something where you can say "we took something for free, we give something back". Not that that's required in any way.

There's some high profile exceptions like the infrastructure parts you mentioned, but noone expects you to open source your router and unix config, there's nothing worth contributing and collaborating on here.

The one exception I've seen though is Microsoft stack applications. For whatever reason open source libraries aren't as widespread there. Sure, there are some and with things like NuGet it's getting easier to use them, but a lot of developers don't seem to bother going outside of what Microsoft offers out of the box.

I think the value here is that they can help sway other large companies to support the open source movement.

I actually built this page a few years ago when I was working at Twitter during a hack day (bonus points if you can find the easter egg!). It's fun to see it on HN again.

Anyone who is interested in Twitter's current open source contributions should definitely check out the open source section of their engineering site as well:

https://engineering.twitter.com/opensource

Isn't it that clicking on the logo starts showing Twitter birds in the background?

Amazing work! :)

Thanks! :D

There's some undue harshness in this thread. Twitter have been _huge_ open source contributors to the Scala and big data communities with original works like Scalding, Summingbird, Scala School, twitter/util and tons of other code. The work they have shared has been a big part of us at Snowplow being able to build a scalable open source analytics platform.

Keep up the great work guys.

It's worth considering the value of IP to a business like twitter. Fundamentally the mechanics of twitter are exceedingly simple - especially when you consider the complexity of apps like Facebook and Google+.

They key difference between Twitter and a would-be competitor is scale (and that's why the actual mechanics are not simple at all), but you don't get to that scale on the strength of technology alone. More importantly, once you're at that scale you become exceedingly hard to dethrone, for primarily non-technical reasons.

That is why Twitter can afford to be extremely generous when it comes to releasing IP. Of course they don't have either, so thanks Twitter!

While Facebook and Google are constantly one-upping each other with their open data center designs and open server architectures, Twitter contributes project after project that are applicable to a far larger group of people. Very much appreciated.

I don't mean to put down Twitter's contribution to open source – which is substantial – but I think you're underestimating Google and Facebook's contributions when you suggest that they are only "one-upping each other with ... open data center designs".

I'm more familiar with Google's contributions, but I would imagine both high-user-visibility projects like Android and Chromium and more developer-oriented projects like protobuf, the Closure Compiler, the Dart and Go programming languages, googletest, Guava, Guice, WebM/WebP, V8, Breakpad qualify. I could consider all of these "projects that are applicable to a large group of people". This, on top of large contributions to other key projects like the Linux kernel and LLVM.

I know less about Facebook's contributions, but I do know that their open source projects includes HipHop/HHVM, and they contribute significantly to Hadoop, LLVM and Mercurial (among other high-profile projects).

I think the point here is not on the amount of contributions, it is about: can you build a Twitter clone using exactly the same opensource technologies as Twitter, or how easy is it? Twitter mainly use open source technologies and they will contribute back if they created something useful - even the projects are the core components of Twitter.

For Google, it would be much harder, as they use so many proprietary (backend) technologies, GFS, BigTable, Chubby etc.

Gumbo as well:

https://github.com/google/gumbo-parser

(And LevelDB, Snappy, Angular, GoogleTest, tcmalloc and google-perftools, lmctfy, etc.)

Interesting to see that Twitter Bootstrap isn't on that list.

That list is only for twitter's github repos, bootstrap has been moved to its own github organization[0].

[0]: https://github.com/twbs

That's because it's no longer Twitter Bootstrap. @mdo works at Github, @fat at Medium.

We worked closely with the bootstrap community to move the project to an independent home as part of the 3.0 release, you can read about it here: https://blog.twitter.com/2013/bootstrap-30

https://about.twitter.com/press/brand-assets

The first rule of the Twitter bird is: You do not change the form of the Twitter bird.

The second rule of the Twitter bird is: You DO NOT change the form of the Twitter bird.

The third rule: If someone Tweets without referring back to Twitter, the game is over.

The fourth rule: No speech bubbles, no inadequate portions of space around the bird

The fifth rule: If this is your first experience with the Twitter bird, you have to Tweet with Twitter.

A walled garden built on open source software. Thanks Twitter!

It's one thing to take advantage of open source software, but it's another to contribute back to open source. The point of this page is to show the latter…

Yes, that may be so, but perhaps (in some cases) it is because they feel legally obliged to:

"This repository is published in order to share code and information and is not intended to be used directly outside of Twitter. We provide no guarantees of bug fixes, ongoing maintenance, compatibility, or suitability for any user outside of Twitter"

https://github.com/twitter/mysql

the world is built on open source software...

This is cool and they should probably do something so that things like https://github.com/twitter/killdeer are no longer listed on the page.

It was a strange experience to scroll down, see something that sounded interesting and then end up at a repo like that.

I don't get why they won't just open source their main app code. It's already been mentioned, it's not like the concept is difficult or couldn't be cloned...it's simply not practical for many reasons. But being able to have the code if really cool from an educational aspect.

What is the significance of the color in the top right hand corner of each repo-box?

I'm guessing programming language of the project.

It shows the language that the majority of the code is written in.

For example, go to https://github.com/twitter/twitter-text-js and click on the red bar.

More info at https://github.com/blog/1037-highlighting-repository-languag...

Twitter uses open source because they do not have the resources to build and maintain custom infrastructure, nor do they think infrastructure is what distinguishes their business. It's that simple.

Only half that statement is true. Twitter has tons and tons of custom infrastructure. Most of those projects are largely built and maintained by Twitter. But their business is built on network effects not trade secret infrastructure, so releasing it as open source helps more than it hurts.

We owe Twatter a big thanks for showing Rails on Fails doesn't scale (via so many fail whales), and so guided many a new startup to explore Scala, Erlang, Python, C++11 - basically things with a good chance to help not hinder your startup.

Thanks Twatter!

And how much of IPO money is going back to open source ?

Reply on news.ycombinator.com