Hacker News Reader

Comment by Twirrim | Hacker News Reader

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

View on HN

Ask yourself how much Python code you think Google has. Then multiply it by some large single-digit number, minimum. Then compare the effort of spending 30 seconds per line on that code vs. writing a new Python interpreter/compiler/runtime, especially when you can trivially get people who are capable of doing that.

There's a reason why the large companies often end up working on new runtimes/interpreters/compilers like HipHop for PHP, Hack, and so on, rather than working on the code bases written in those languages. It is very easy for it to not just be easier to leverage in at that level, but an order of magnitude or two easier. Or three.

It's interesting to think of the dynamics of developing an application (presumably) many times more complex than its host language. At a certain point, changing the language implementation becomes cheaper than making sweeping changes to the application.

I don't know PHP, but in the case of Python you will spend the next ten years ironing out subtle incompatibilities between CPython and this new thing.

Added up, it is easily more than 30s per line.

"Added up, it is easily more than 30s per line."

30s per line of current Python code, not compiler code.

And I'm just providing a number to put some numbers out there. 30s/line to convert a Python program to something fundamentally different like Go is probably an underestimate, yes, but then, if it's an underestimate that means the budget for the Python reimplementation is that much larger.

It also really helps the "subtle issues" when you control both the implementation and the code running on it; it means for every subtle issue discovered you have the option of either fixing the implementation or fixing the original code not to tickle the corner case. It's much harder when you only control one side or the other. It's not Google that will be having massive problems with corner cases.

I wouldn't think that a tool for transpiling Python to Go would be harder than creating a whole new runtime. Most of the work is already done; you can get the compiled ASTs directly out of Python itself.

Of course, if it started out as a "this seems like a cool project", that skews the "a is more efficient than b" ratio significantly.

Sorry, I think my post may have been excessively specific; I've edited it to say "writing a new Python interpreter/compiler/runtime" instead of just "writing a new Python interpreter". I was trying to make a generic point, not one specifically about a Python interpreter. (At least, if I'm interpreting your reply correctly.)

I mean, you can get something out of transpilers--not typically maintainable code.

I take it you've never worked on a large code base with a large team? Even ignoring the actual task of rewriting the logic in a different language with different idioms, the change is going to be a massive hit to productivity for many months as people learn the new architecture of the rewrite, the tooling, and idiomatic Go.

For a project the size of YouTube, that will be millions of dollars of engineering hours and weeks/months of lost productivity for an unknown gain and almost guaranteed bugs. It's a terrible value proposition so it's better to squeeze every last drop of performance out of the code base you have, which at the scale if this project includes paying engineer(s) to work on a completely new runtime.

Because Python running in Grumpy can import Go packages, I'd assume (I work for Google, but not on YouTube), that Python libraries could then be migrated pretty seamlessly and incrementally to Go.

It's been said a couple times in this thread already but basically, the YouTube Python codebase is very large and the cost of a rewrite is prohibitive.

Statistics would be very interesting to the "useless internal trivia" types.

Creating a new Python runtime in Go is obviously non-trivial, but so is rewriting the heart of YouTube in Go. Heck, YouTube may be a bigger codebase than Python for all I know.

And there's a lot more side benefit in a better multithreading Python runtime for Google for other Python code Google has (or hosts), whereas the benefits of a YouTube rewrite are more narrowly limited to YouTube.

Just because the site is popular doesn't mean the codebase is big. It's a simple site. I could probably write YouTube in Go in an afternoon and I don't even know Go.

You could perhaps write a feature-bare clone. Here's some basic features:

    - search (the various search pages)
    - trending
    - channels
    - subscription
    - video uploads
    - history
    - comments
    - likes
    - upload
    - video editing (most of this is in browser, but still)
    - livestream
    - video analytics
    - payment and ad management
    - video comment review and moderation
    - translation
    - captioning
    - video replication, multiscaling, caching, and resolution management based on network speed
    - video recommendation
    - cards, overlays, annotations, etc.
    - music and sound effect search
    - antispam
    - dmca and copyright tooling

And that was 5 minutes of poking around.

> resolution management based on network speed

Not so much a feature of YouTube as it is a feature of HLS/Dash... but yes, it means you've gotta transcode the source video into multiple different bitrates.

Even outside of that, I can override the bitrate in some instances. But yeah, I was mainly talking about the necessary replication, cache, and transcoding (and I actually forgot about the multiple codecs, which they also provide, in addition to multiple resolutions/bitrates)

This comment made me wonder what you consider to be a complex site.

Not a video host with comments. That's practically CRUD. A very solved problem.

Well, go on then, why spend just an afternoon when you could spend 5 whole days. Make a YouTube that's 10x better, host it from your closet and watch the megabucks roll in as you take down Google.

But, really, what site cannot be boiled down to practically CRUD? That does not help us understand what you believe is a complex site.

Is this supposed to be some kind of no true Scotsman?

See "Code: It's Trivial" https://blog.codinghorror.com/code-its-trivial/

That's a question I have as well. I'm guessing they have a very massive cpython codebase, and the trade-off was worth it.

I'd assume that's true, but they also didn't know how well this would work. A 'little' project to make Grumpy is probably much easier to sell to management/throw away if to slow than "Let's rewrite all our code in language X to see if it goes faster".

The thing is, you can not use Python in production at Google. YouTube was acquired a long time ago and a rewrite of the inherited Pyton code base has not happened for very good reasons.

Maybe there is too much to rewrite.

It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.

I think he's talking about moving their Python codebase to Go, so they don't have to run Python anymore.

You're probably right. He seemed to be questioning the translator so I thought he was suggesting replacing CPython piece by piece with a Go implementation.

It could be part of the auto migration effort.

Reply on news.ycombinator.com