Changing the Culture of Python at Facebook [video] | Hacker News Reader

Changing the Culture of Python at Facebook [video]

youtube.com

81 pointssandGorgon31 comments

Open articleView on HN

The punchline:

- Python is popular at Facebook

- Python 3 is the default for new projects

- Changing to Python 3 needed an internal advocate to update build and dev tools to push the change

- Facebook is a big user of the new ``asyncio`` module

Perhaps the most subtle point of the talk is that Facebook managed to change from 2 to 3 from the efforts of just one person. Of course there were more folks changing code than just the speaker, but it appears he was able to heavily influence the culture change. Enabling 3 for the build tools probably made a big difference.

Changing Python versions is hardly a "culture change". Changing to Go or Rust might be.

The main problem is library support, not the language changes. A large company like Facebook can afford to put someone on library compatibility to make this happen.

Apparently that wasn't even necessary. The big change for library compatibility was simply changing the settings of the build system. According to the speaker.

In my own experience, most libraries are 3-compatible.

I'm not sure if you watched the entire video, but I seem to have gotten the impression that all through 2014.. he fixed hundreds of libraries that did not work with Python 3.

I watched it while working on other stuff. Must not have noticed that part.

But note that a small company will have correspondingly fewer dependencies. So I'm not sure a small company would need a dedicated upgrade engineer.

The size of the company doesn't dictate the number of libraries used. The scope of the work does.

Size of engineering team and scope of work are correlated, in my experience.

I used to have issues with machine learning libraries and python 3. Were they fixed recently?

For the most part new projects have been doing a good job of either working with Python 3 out of the gate or making it a main priority (Issue #1 on Tensorflow was Python 3). In general, you'll be fine with Python 3 now. Some older, academic-type packages are still only Python 2.7 (Though might be easy to port).

For those still stuck on 2, the most likely culprit is not paying attention to the difference between text and bytes. The required change will be using decode when you want text and encode when you want bytes. Not too much trouble for most projects. But it is a tedious change that cannot always be find-replace'd away.

Plus, teaching new hires to use py3 as default, putting in linter warnings to get rid of old bad habits, adding in 2to3 compat layer libraries by default and switching to 3 as default after that seems to have helped.

All stuff that can be done by 1 person at your company. Go forth and live in the __future__!

The cost of 1 person varies heavily though :P

> - Python 3 is the default for new projects

What teams are using Python at Facebook now? So far job requirements seem to be mainly C++ and PHP.

Do they use Cython to speed up? What about PyPy?

Instagram uses django

The Production Engineering org at Facebook uses a lot of Python. Check out https://code.facebook.com/posts/1040181199381023/python-in-p...

not many, just some op-y type jobs

one of the interesting things in this video is that Facebook pretty much invented an equivalent of Flatpak/Zeroinstall/Click-packages to deploy their python apps.

It's actually not super hard, Unix has pretty good support for this, and a lot of companies have something like this internally. The gulf between "good enough for a company" and "good enough for the world" is pretty huge.

Plus, the extent to which people can't do this because they have no/have a terrible build system is generally underestimated.

I've had great success using pex to bundle up tools that use a few dependencies. It's a lot simpler than tools that try to bundle a full python distribution.

I wonder why someone hasn't built a simple loader for Windows that wraps pex in an exe with pythonxy.dll. You'd pretty much solve distribution of Python scripts on Windows.

there are lots of projects that do this - we had a QT app with embedded cherrypy that we bundled using cx_freeze.

Worked in production at a hundred different merchants

https://sourceforge.net/p/cx-freeze/mailman/message/34695906...

And PyInstaller is quite complicated.

The speaker says, "dependencies are associated with the application (as they should be) rather than with the system"

That's certainly a practical strategy, but it reduces flexibility/modularity. Go-lang shares the same strategy for deployment. Perhaps it was inspired by Java-users.

There's good and bad there, if you need to patch a security vulnerability you may need to redeploy tons of packages on each host (each package that contains a copy of the bad lib code) versus patching it once. Depending on how you deploy that might be no big deal or it might be.

Then again, patching it 'once' usually involves restarting all processes that are dynamically linked anyway (and really hoping that it loads the new lib). If deployment is fully automated and scales well, the only painful thing is the restart to begin with :)

We've changed the link from https://youtu.be/nRtp9NgtXiA. Please submit the full URL, since HN kills links from shorteners as they break the referential integrity of the post.

interestingly, this is because of mobile apps. I used the share function in the youtube app, coupled with the "copy link" intent on android that pops up. This is the link that was given to me and i pasted it here.

On a mobile device using a native app, it is quite hard/impossible to get a different url. Could you whitelist the youtu.be url ? because I'm guessing this will start to happen more frequently.

Or, perhaps hn could just auto-dereference the links to a whitelist of the most popular shorteners & then everyone can be happy?

Thanks, that's a good idea that has come up before. We've got it on our list to look at.

Part of the reason Youtube provides the youtu.be link is that their systems disambiguate between mobile and desktop websites.

Mobile versions have the 'm.' subdomain, and provide a less optimized experience for desktop.

I would also argue that the youtu.be link is a canonical form, since the identifiers are identical to the source. It is just an optimized canonical proxy.

Reply on news.ycombinator.com