Go + Services = One Goliath Project
engineering.khanacademy.org
engineering.khanacademy.org
I would love hear some thoughts from others that made the move, especially anyone that decided to move back to a monolith repo.
For a more concrete example, I recently built a service ("scraper") that scraped data and upserted a large tree of structured data to postgres in a transaction. Writes were only allowed from scraper, but "api" could SELECT data for reporting to the frontend "web" as much as it wanted. In the future, "api" might make be refactored to make internal HTTP request to "scraper," so they could have totally separate databases.
This approach is a nice balance (IMHO), since you start off with monolith but it is broken down into cleanly separated modules. Each module can potentially become its own micro-service if or when the time comes.
In terms of implementation, this can be easily done, for example we do it JVM/Kotlin where each microservice is its own project that produces a binary/jar. All the projects are part of a multi-project build. Lastly we have a common project for shared code, utils, types, enums, interfaces, etc and an application project that loads/sets up all the microservices from each project. Works great so far. And when you do have break up 1 project into its own service, the effort is fairly manageable.
It is much easier (imo) to pop into a repo of one of our services and look through the code in it's enitrety and see when things were last touched and where the issues are. I would make the argument that the "inevitable corners that fall behind and have questionable security" is something that is inevitable in any codebase that grows to a certain complexity, and microservices (or SOA in general) make it much easier to see those things as they are decomposed.
The more you break down your code (the smaller the size of the services), the more maintenance need is created from the division, and the easier it is to fall behind on it.
You need CD or this is an accident waiting to happen.
The overhead introduced with having such a setup in a corporate environment that has not-so-well-though-out requirements and design is ridiculous. Keeping track of dependencies, arcane knowledge of inter-service dependency quirks being siloed and hidden, keeping individual services up to date, dealing with older services and their interaction with newer services till they "migrate" to newer tooling/common code, etc.
Everyone then skirts around the fact that the problem could potentially be Microservices or a micro-repo setup. Instead, they throw process, sign-off, complicated promotion pipelines and just plain warm-bodies at the problem in an attempt to mitigate it. But the damage is done, velocity has slowed to a crawl and everyone is miserable, especially when having to explain the whole thing to newcomers.
When I worked at a pretty large software co, a team there were always adopting the latest techniques and tools but their deployment pipeline was an over architected disaster that nobody could reliably deploy. Reason? Their CI/CD system consisted of a person manually clicking around to build Jenkins Jobs. There will be other such silly nonsense (hopefully less extreme) in other corporate environments too.
Note that they are not all going to start/stop at exactly the same time and during development they may even crash so each microservice should survive such transitions. You may have two different versions of service code or even completely different implementations or two different version of the API in use at the same time. This can happen as different services evolve at a different rate. And you may want to transition to a new version in a piecemeal manner so as to not bring the whole service down. All these considerations complicate things. So ideally you have factored out these common tasks in shared library/packages. And ideally you write your code such that if necessary more than one service can be compiled into the same binary for performance reasons.
In a monolith some things become easier since everything dies at once! But somethings become more complicated - such as supporting evolving code. And monolitha require more discipline to keep things modular. Over time this gets harder and harder. Lack of modularity means you have to understand a lot more code and when you evolve things, more code will have to change and there may be unforeseen side-effects. And scaling can become harder.
We're still building out our deployment system to better support multiple services, but we're planning to redeploy all of the services when library code changes (which is something we're trying to minimize).
All of this ensures that we don't have trouble with services lagging behind on critical updates.
1. In the 10-5000 employee range: tech giants are a different beast when it comes to team accountability.
In the microservices organizations I've seen, this isn't true. The other service teams provide an API and, like any other SaaS you use, you do not need to be able to read the implementation. You would only work on that code if you switch service teams.
Like I said, though, we do want to minimize the library footprint. The vast majority of deploys in this new world will be single service deploys, with the benefits that come with that. We already deploy our monolith several times a day. These services will speed that further.
We ideally want one language. But Apache Beam (which is behind Google Dataflow) doesn't yet have production support for Go. More importantly, though, we have no time pressure on switching the Kotlin code over, so that's a long way out.
[1]: https://engineering.khanacademy.org/posts/kotlin-adoption.ht...
I've been in companies that moved from monolith to microservices and I saw it work well except when teams had such tight cross-service dependencies that they had to get other teams' ok before making changes to internal details of their service. Then developer velocity was slower than before because it took time to make the cross-team discussion happen and political capital to make other team care when they have other priorities.
Ultimately, though, if we find that this plan reduces velocity, it won't be that hard to change later.
At my employer, I’m spearheading a wholesale reimplementation of outdated process automation software, turning everything into Django web apps.
I’ve been working with a monorepo and monolithic deployments to maintain development velocity but recently started transitioning the CI/CD pipeline to deploy each application/service in the monorepo independently. The pipeline packages common assets (including, e.g., manage.py, common HTML templates, and the dependency spec...all housed in the same monorepo) into each app directory before the deploy stage.
Meanwhile, local developers clone the entire monorepo, and when they launch localhost, all of the services come online simultaneously. (That’s the goal, at least!)
I was already excited to see my work come to fruition, and now I’ll be keeping an eye on Khan Academy, too!
Our current plan for local development is to continue cloning the monorepo and firing up all of the services. Go services don't take a whole lot of resources, so we think this plan will work fine for quite a while.
Forget everything you have heard about micro services. Most of it is bullshit from people who don’t actually think for themselves.
Developer productivity is very hard to maintain in a monolithic app as the number of developers increases and the legacy code piles up. Breaking up services and giving each dev team control over their own codebases enables them to develop their own products at their own pace.
If you only have one dev team, microservices are a lot less attractive. However, there are still some benefits, such as being able to refactor parts of your codebase in isolation (including perhaps rewriting them in different languages), and the ability to individually adjust the runtime scale of different parts of your codebase.
No one wants to touch them so they sit around unmaintained until an unrelated change or unpatched security issue comes around. Suddenly you've got a big problem with a mystery codebase.
That's why most companies are promoting devs to do ops .
So then you have devops
Containerization, autoscaling, service discovery, tracing, metrics and monitoring et al - lot of it is required to do larger scale, distributed systems. Even if you do not call them microservices.
IMO the only reason to use micro-services is the one they mentioned - you can have different parts of your system running on different machines so they can be spun up independently. But I think most people aren't "web scale" enough to need that anyway.
No need to over-engineering modularity with distributed systems algorithms into the mix.
For those who are curious, here is a classic article on why rewriting code from scratch is a bad idea: https://www.joelonsoftware.com/2000/04/06/things-you-should-....
For a more in-depth analysis on the unforeseen challenges of microservices in particular, I would encourage a lot of careful research into how other companies have tried and failed at this. In particular, I might look at Uber's ongoing difficulties.
All I have to say to the Khan Academy engineers is to buckle up because frankly, moving from Python 2->3 is not that hard and you have no idea what you are getting yourself into.
Microservices have an immense cost, and you have to make sure they're worth it. Many teams years ago found it a nice pattern and implemented it because why not, and now we're at the "oops this isn't actually amazing" part of the cycle.
What other people say about scaling teams is true, but I have a few other points as well:
- personally, I spent 6 months writing tooling for service lifecycle management and setting strict conventions. This was before the microservices decision and I was recruited to do “devops” which gave me a lot of head room. :p
- when talks about microservices surfaced I had to fight for several months to get the lead devs on-board with tooling and conventions. From the infra/cm/systems management side we’re used to manage many thousands of “configuration items” - devs are usually not, and many underestimate the value of proper automated lifecycle management.
- once everyone was on-board we all used a common language. Big win.
- I could form a “devops” team to help develop the tooling further and the infra platform as well.
- almost all teams worked in mobs - amongst other things it really helped with the ownership part, something that’s absolutely crucial. Well defined domains and accountable mob teams, just awesome!
- quality rose by a mile. Small commits and somewhat robust tooling under the eyes of a mob.
- graph the pains. If outdated versions are a problem - put it on a graph, green, yellow, red. Show services in relation to other services - the application is the sum of all connected services.
I could go on, but the post is getting long! :)
The more i write and talk about it, the more I realize that it is about ”externalities”, so to speak.
A microservice is just code - one small piece that does one, tightly defined thing. We’ve been doing code always and smaller pieces of is easier to deal with and reason about.
The structure around keeping 100s or 1000s of moving pieces in concert is where a lot of the work is shifted. It takes teamwork as well as a common vision and language. The above sentence tangents “culture”.
A service have in a way two interfaces - one business (the work it’s doing), and one technical (how it does it; a port publishing an endpoint or whatever).
No business process to support, no service to manage.
The lifecycle of the technical service will consist of a bunch of actions that will be taken, perpetually, until the sunsetting/decommissioning of the business process. Actions: init, code pushed, deploy, monitor, trace, update, decommissioning etc.
You take these actions and put then in a lifecycle circle and you have a nice powerpoint!
As much as possible, preferably everything, in this cycle have to be governed by conventions and automations.
- Automatic follow-up if a service have no upstream or downstream services for example. Why do we have a dangling service?!
- Or, you have a key service tied to an SLA, but upstreams are not matching this?
- you have services that have not been touched in a timely maner.
- etc... just drop the relevant team an automated slack message with the option to initiate whatever is required to keep the lifecycle churning.
With many thousands of assets/ci (configuration items) almost everything have to be automated or you will grind to a stop eventually.
If you can couple business process to automated technical service management - big wins!
Take the best/reasonable parts from ITIL (concepts/principles), mix it with principles from the agile manifesto and the 12-factor app. Automate the lot of it.
Doing this in practice gives you dev & ops.
It’s quite a journey that is more difficult the bigger you are. It scales though, so start small, prove the concepts, and grow organically.
We had a big monolith where I work.
We’ve been slowly, but surely isolating parts of the monolith as separate deliverables, extracted into their own repos. But only when appropriate, and not as a forced exercise.
The remaining “monolith” is still pretty big, but it does (mostly) represent one logical deliverable, so effort to split it up has some what stalled.
There’s been small points of friction, but nothing near as painful as we used to have it. No way we’re going back.
So everything in moderation. Micro services architecture is a tool. Use it when it’s the right one.
It's very limited when you started to do complex thing. Example, let's say you are building websocket. You will have a hard time to write type safe websocket handler to process the payload from client for all the events...
I started to do Rust/Crystal and both of them are better than Go(performance, type system).
Yet, whenever I build something for work, I come back to Go :-(. I told myself to use Rust or Crystal.
Then I realized that Go is a practical language. It compiled fast so it makes testing easier. The cross compiler just make it so easy to build binary run on everything thing. And the limitation of Go makes it very consistent on how you do thing. This makes working with Go become faster event by the fact that it slows you down on other parts.
So I think Go is a language that people easier to fall into because it has the speed of interpreter language like Ruby/Python(or even faster) during development and have a better performance/type safe story.
C# and Java are slower to compile and offer way more options to do the same thing.
Here's Uncle Bob take on testing with Go: https://m.youtube.com/watch?v=2dKZ-dWaCiU&t=36m40s
Not for any meaningful work in my experience. As a matter of fact, I found the change/compile/run loop in golang to be slower on projects I've been working on due to the fact that it doesn't support incremental compilation, so any change I make ends up recompiling the entire program and writing out a 100+MB binary anyway. Compared to a Scala project I worked on before (and Scala is notorious for slow compiles), after the first compilation, all modifications happen very quickly as only the respective classes are re-compiled.
> Here's Uncle Bob take on testing with Go
Again, this doesn't apply for any non-trivial/large project. On a project I'm working on, it literally takes 7-8 minutes to do a clean build + run all unit tests in golang.
Same for tests which are cached by default so during typical development only a subset of tests are executed and compilation time can be a big part. Leave full tests for CI.
An anecdote I found from 5 years ago:
On my 1.7GHz processor it takes 10 seconds to build the whole standard library from scratch (300k lines of code).
It's fast AF.
2020 is going to be in an interesting year at Khan Academy ;-)
Of course the obvious thing missing in the article is how they expect to deliver new business features while recoding everything in a new language.
It's fun to read this and the Etsy thread currently on the frontpage as well.
For new features, the new parts of the GraphQL schema for those features will be written in Go as part of the new services. Our frontend is already in large part a single page app in React which requests data via GraphQL, so the frontend for the features will look just the same as it would on our monolith.
One thing that might not have come across clearly in the blog post: we're already well on the path to using React everywhere (we started using React a week after it became public… 6 years ago?). We made the decision to move to GraphQL in 2017, so we've already got a lot in our GraphQL schema. Finishing those switchovers will make our move to Go happen more quickly.
My guess is that the framework for them thinking about this is that they have already been thinking about migrating languages to get better performance and the Python3 migration seems like completely wasted effort if you then throw it all away to go to another language shortly after.
I see people say things like this a lot but my experience is that while other languages are 10x or more faster than python in some benchmarks it's very rare that computation time dominates server latency or that servers are running at 60%+ cpu across all cores.
If 90% of your service latency is not directly on the cpu and/or you haven't profiled to see that the performance bottleneck is evenly distributed across all tasks, then it's super dangerous to migrate to a new language thinking that will fix it.
I hope people inside Khan Academy know this and it's just a clickbait blog. If they really think "go is 10x faster than python so we'll only need 1 server for every 10 when we migrate" then I think they'll be disappointed.
* they moved from a monolith to a microservices architecture; concern that any of the services in the request path could add latency just because of the overall runtime speed is slow is a legitimate one.
* their primary deployment method is Google App Engine where you are billed by CPU used. Any change that consumed less CPUs has a tangible effect on their costs
Add in extremely poor error messages, lack of generics, having to generate loads of code for various things, the ability to crash whole services if your program does something incorrect, excruciating error handling will all slow you down.
Also, while I agree Go’s error handling isn’t very elegant, it does force you to explicitly consider every potential error, which in my experience makes uncaught errors far less likely than a language with bubbling exceptions.
> Go is faster to iterate
holds true considering the total lifespan of the project. Golang is more explicit thus requiring more time to define every type but I've never refactored so fast and safe a codebase. In Python the fact that is dynamic makes it more difficult to safely iterate over it (is more statically-typed Vs dynamically typed). About error handling, it's not perfect, but the code is readable, easy to follow and easy to reason about.
> Iteration in go is simply harder as you have to be more explicit rather than sketching something out.
I will completely agree that prototyping in Python is way faster. Python is my preferred language for throwaway/prototype code.
> Where do you get this idea from?
It's solely based in my experience. I hope it contributes to the conversation.
Things are also easier to change in go because of the type system and interfaces. The former catches most the obvious incompatibilities, the latter ensures that abstractions don’t leak across different system boundaries; whereas in Python there is a tendency to pass a do everything objects across the system.
Error checking has improved substantially with error wrapping in go 1.13. Not only can you locate precisely where your system failed; you have to be explicit about handling errors. I do concede that pre error wrapping the error handling was garbage.
Most applications spend most of their time waiting for the database or network. I suspect the fastest programming languages are those that have the lowest thread/process overhead. If most apps spend their time waiting, then a language with 10X lower process overhead can handle 10X more processes.
Yes, NodeJS solves one problem by having low process overhead, but it also fails to take advantage of parallelism in modern processors. Ideally, I'd like to see a system with both.
Java, or Kotlin, using one of the reactive frameworks.
I'm now converting one server to Go (although not the heavy one), and it really runs fast and uses much, much less memory. It also starts in in less than 1s, whereas the django application takes 5 minutes, because of some stupid problem in static file collection.
Python is fine as a teaching tool, to prototype in, and to use in notebooks as a wrapper around numpy, scipy, etc., but not to run in production.
Indeed, there are many benefits these more performant languages have over Python aside from raw single-core performance. For starters, more efficient concurrency and parallelism can help reduce average latency when combined with a quality async webserver. Then there's gains due to shared memory across threads.
So in many cases-- absolutely, you can only need 1 server vs. 10 when you migrate. It's thus not fair to say that these gains are "very rare".
So, no, things are not as simple if you're not dealing with toy projects. And no, you can't assume that it's the same for everyone if you're not in their shoes.
Your comment is pretty much the equivalent of "I don't see a bug. Works for me."
But even if you nail the types to the board (e.g. assert isinstance(...)), use (the somewhat weak) Mypy wherever you can, and have good test coverage, you still have to grep your code base for usage of, e.g., .keys(), eyeball hundreds of modules for subtle Unicode madness or hunt for the odd division, replace every sort() that doesn't use key= yet, etc. The todos add up and someone has to go into the code and change those lines.
The str/unicode misery is one of the biggest gripes I have with Python. I'm glad this unpleasant knot has been mostly untied in Python 3. I came to the conclusion that the transition would have been much easier if Python 3 just concentrated on the separation of bytes and (unicode) strings. The other features could have been in Python 4.
It's a bit like IPv6. If it would just solve the address space problem, most would have moved to it already. Instead it comes with a lot more baggage. And each additional feature has it's own uphill battle for acceptance. So nearly everyone is dragging their feet, citing their pet peeve with the technology.
How do you automatically infer the intention of somedict.keys()? Is it going to be used as a list or as an iterator?
Those are just off the top of my head. I don't remember all the cases where 2to3 tripped over and produced garbage. But there were too many cases to put actual faith in automatic conversion.
It might work if your code is kind of new and homogenic. But looking at how much trouble Dropbox had, even with all the tooling and Guidos they could muster, I have the feeling that your positive experience with 2to3 might rather be the exception than the rule for old and big code bases.
Yes, there's a wrapper. You use it, add a comment "this is wrapped in the migration to 3" and move on.
> How do you automatically infer the intention of somedict.keys()? Is it going to be used as a list or as an iterator?
If it's being iterated on first thing, it's an iterator. If list methods are called on to it, it's a list. This just hasn't been a problem for us, sure, it took some looking at, but it wasn't more than 30 seconds per case.
> I have the feeling that your positive experience with 2to3 might rather be the exception than the rule for old and big code bases.
Maybe so, but the codebase was ten years old and hundreds of thousands of lines.
What you should do is use unique names. But that will give you ugly code like myFoobar.foobar_addBar(). The kind of code that makes the pythonic crowd cringe.
Another problem is making code too generic with regards to what types it consumes, instead of nailing it down to the few types you're ever going to use here. This makes it hard to reason about your code months and years down the line. How is this method used in the rest of the code base? Do all callers expect an int? What if my method now happens to return a float?
And there's also abuse of duck typing. Throw around a lot of objects, sprinkling methods and other members on to them as you go. Then when you consume the object, just look if it has the method you want to call. This makes any kind of type checking and static type inference useless.
And then there's a whole lot of Python 2 libraries where you get the feeling that the authors didn't give too much thought about whether they are dealing with str or unicode. The method might just call .encode(...) on one of its arguments without being too sure what it is.
And every one of the mistakes that result from the above practices might only pop up when your code has already been shipped to the customer site.
Khan Academy rationale might be false, as the transition path might be easy from Python 2 to Python 3.
But in the end they will have the same stack as before and that's what they clearly try to avoid. Given that it makes sense to transition from a dynamically typed language to a statically typed which offers more compiler feedback.
Nice: re-branding. I can't wait for the, maybe "consolidated computing" manifesto (aka turning micro services back into monoliths).
In the latter part, that must be a disingenuous dichotomy. You don't really believe that just because an avenue exists we should include it in an evaluation?
In this case, it leads to a higher concern about minimising the cost of the operational services than you might have in a for-profit organisation. In all the strategic planning I have been involved in with NFP, we always have the "what if worst case scenario arises" plan and in that plan the ability to scale down to bare minimum operational cost is key. It may not be conscious but I suspect that may be part of the reason the performance savings from moving to Go are so attractive in this case, where most profit-making companies just ask the question of whether they can afford to pay for the servers with their current margin or not and if they can they have more important things to worry about.
- The decision seems to be primarily a software architecture one, without much mention of all the other architects whose input will shape how the finished product is run and supported. In a modern software development environment, all the other parts of the org should be consulted on greenfield work to "Shift Left" anything that may need to change down the pike. Design in a silo leads to ineffective products.
- They're going from "hmm we need to upgrade from Python 2 to Python 3", to "we need to redesign everything in a new language with a radically different software architecture". This is definitely the second system effect. It's going to take years to make this thing reliable and sunset the old product.
- They're porting over the logic? Even if this is actually the right move, wouldn't a clean-room implementation potentially give better outcomes?
- Why are they continuing to use App Engine if the writing's on the wall for 2024?
To your first point, "The decision seems to be primarily a software architecture one...", this project has had involvement of the whole engineering team since the beginning. The whole org is on board with this change. It's definitely not happening in a silo.
> This is definitely the second system effect. It's going to take years to make this thing reliable and sunset the old product.
I hope not, but obviously we're not done yet, so I can't say how long it will end up taking to completely decommission the Python 2 app. What I can say is this: there are aspects to this project that are _simplifying_ our system and, for what's left moving from Python to Go, our intention is to port the business logic as close to a straight up port as we can get.
> - They're porting over the logic? Even if this is actually the right move, wouldn't a clean-room implementation potentially give better outcomes?
_That's_ second system effect, to me. We can't change everything and fix every problem now, so we're focusing on the changes that will help us move from Python to Go faster.
> - Why are they continuing to use App Engine if the writing's on the wall for 2024?
I don't think Google Cloud is disappearing in 2024, for one. Beyond that, again, we're not changing everything about our architecture. The way our data is stored is staying the same.
Given the seemingly strong chorus of voices responding with cautionary tales about why you might want to rethink this plan, and the number of engineers in your organization, it seems more likely that you have some dissenting voices who have either been too scared to speak up or have already been shot down.
Even in this thread, there's a chorus of voices sounding caution based on their limited information of what our situation looks like, but there are others who see why we're doing this, based on the same limited information.
We absolutely do know the risks of this project, which is why we're doing this as incrementally as possible.
[1]: http://thinkrelevance.com/blog/2011/11/15/documenting-archit...
I’ve been down this road. Deep down this road. Let me just give you a heads up on something I didn’t consider at the time: Most template languages do not parse every single node, one by one. In a sense they are just doing string concatenation. Not so with server side rendering and React. I’m not saying it can’t be done but just realize it is going to take a lot more compute power. Caching is great of course but won’t help you if you plan to customize user content during the server side rendering as well. My recommendation is that you don’t do any user authenticated stuff during SSR.
Also consider how you are going to handle cookies if you do plan to make authenticated requests to server side rendering. Also solvable but for some reason people had the hardest time understanding why we had to forward cookies to the domains we controlled in an API request and definitely not to any other servers.
I’m not sure I would pick React for an SEO driven website. It is hard to get a competitive “time to first byte”. Unless of course you can pre warm a cache of every one of your pages.
Lastly, you’re going to need Node for the SSR. I’m sure you know this but that might take you out of app engine and into cloud compute. Not a big deal but thought I’d mention.
Good luck! It is doable. If you ever want to chat about how we solved some of these problems I’d love to save you some time if I can. Hit me up in my profile email.
We've been doing SSR for quite a while now and are improving our CDN use as we go along. We already took steps to ensure that there's no user-specific information showing up in our server-side react rendering which would damage cacheability.
Our frontend infrastructure team essentially owns the React render server. I'll let them know you offered to chat.
I disagree with this. It's a Python project's dependencies that make it hard to move from 2 to 3, and most libraries have been updated.
Of course, you could argue that it isn't easy to migrate a codebase from one major version of a language (or framework, or database) to another, but when you eliminate easy from your vocabulary it becomes harder to describe different levels of difficulty.
source: no python services at my company are going to be migrated to python 3; it’s all moving to a JVM
I'm going to call BS on that one.
If you're having issues with Python 2, then it might make more sense to switch to another language instead of upgrade to Python 3. But going from Python 2 to 3 is much easier than switching languages completely.
Python is not a perfect language. There is no perfect language. It sounds like your company just had a reason for switching to a JVM language and the Python 2 EOL was a justification to start.
Source: All python services at my current workplace are in the process of being migrated to 3.*, and I'm doing one of the main ones at the moment and it's a breeze, including compiled c-extensions.
For curiosity, a good list of actual python3 syntax changes: https://docs.python.org/release/3.0.1/whatsnew/3.0.html#over...
Is Django a big enough project for you?
Did you know that Django was not only successfully migrated from Python 2 to Python 3, it was ported in such a way that for many years it used the same codebase in both languages ...
Perhaps that's the biggest advantage of porting from 2 to 3. A lot of the code could run in both languages.
Is Dropbox open source? Is Dropbox even a typical Python application representative of the challenges of porting from 2 to 3?
My hunch is that the challenges of porting Dropbox to any other language have to do less with Python more with the need to deal with a filesystem at a lower and more granular level than what typical programming languages offer. Thus everything needs to be rewritten in bazillions of ways to handle the bazillion corner cases.
Go has consistently been 10-20x performant (allowing for dramatically reduced hardware needs), easier to maintain, and more productive to produce code in than our previous Python (Twisted) and Perl (AnyEvent).
Hopefully KhanAcademy has solid telemetry data in both legacy and new code so they can quantify benefits. They will also have a learning curve for managing multiple micro services vs monoliths. Accessing shared data will be a problem they will likely have to solve. We've opted for each service controlling its own data - no reaching into another service's data behind its back. Everything through APIs. This gives the microservice the ability to alter its datastore as it needs to and not be blocked by other teams' need to update how they access the data.
Debugging a distributed solution is much harder than a single service. Distributed tracing, consistent structured logging with log aggregators that let you do fancy searches (like Splunk), and application telemetry and metrics will be even more important than before.
We have established the rule that each service owns its own data.
We've already got Stackdriver set up to give us distributed tracing and have set up standards around logging.
1. Write in Go an exact reimplementation of the current Python codebase. Use the same database schema, front-end HTML/JS, test suite, and so on. To whatever extent possible, use the same names for classes and functions. Check the reimplementation correctness by using a comparison tool that calls both the Python and Go version of a page/function/search and making sure that they produce the same results.
2. Change the production code over to the Go version, perhaps using a ramping strategy where X% of servers are running the Go code, and you gradually increase X, while monitoring vital statistics like server load and response time.
3. Now that the production site is running Go, incrementally split off components into their own services.
This approach leads you to the same destination, but with a lot less risk. It is very unhealthy to have a situation where the production site is running one codebase but all the developers are working on another codebase. Note that you will realize the benefits of Go (performance, type safety) after step 2, which is much sooner than OP's plan.
Joel Spolsky's classic essay about how you should never do full codebase rewrites is worth reviewing:
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
I’m thinking that defining parts that can be moved to separate services, and start consuming these could be a way to organically transition to a new architecture.
The key to the success of the project was that the API was an exact match, and we could compare both implementations for exact requests. The deploy strategy of the new version:
- Reply the real traffic to the new Go service comparing the results with the old one - Then implement a toggle feature than enabled different traffic sources to use one backend or the other - Keep changing backends to the new system and ensure that metrics were unaffected
Having e2e and integration tests for the Golang project was of a huge help, since we could fix all differences using TDD.
Although we changed some of the implementations to take advantage of Go constructs, just a 1-to-1 replacement would have had a huge performance impact.
(Speaking as someone working at a firm that did choose (before my time) to start a ground-up rewrite than ran about 3 years over estimate).
After the tests are in place, you break off small portions into microservises and ensure tests pass.
You pass a small percentage of traffic through the new arch, fixing bugs and leveraging telemetry.
You eventually slide all traffic to the new architecture.
You want to get something receiving traffic asap as to start getting feedback. Often this means taking something smaller or simpler out of the lagacy codebase first.
Doing a complete, side by side replica is a recipe for disaster. Think MVP. We've even introduced traffic routing based on feature sets so you can route users who don't use edge case features the new arch which has yet to incorporate the those features while keeping others on the old arch.
Since we're using GraphQL federation, we have a gateway which serves up our complete GraphQL schema, pulled together from a collection of services behind the gateway. We can move individual properties over from our Python monolith to new Go services and the clients will never know. Plus, we can do side-by-side testing in the gateway by making a request to the monolith and to the new service and comparing the results (something we don't have yet, but plan to).
This is definitely not a big bang rewrite. It's about as incremental as it can be, because we can move individual GraphQL properties over and the gateway stitches the result together.
I'm not entirely sure how well the line of code: "import six" would fix any compatibility issues nowadays.
I don't think there is any question here: Python 3 is a complete disaster. Years and years of engineering effort wasted on changing string libraries. Sadly, the Python leadership refuses to acknowledge the failing, perhaps because such an acknowledgement would challenge their omnipotence ... it would, and it should.
The Python 2/3 split is by far the most annoying thing about Python. I don't develop software in Python but about half the time I've had to use a Python library or program the problem of 2/3 incompatibility has cropped up. Some projects don't make it clear whether one or the other is required, leading to further confusion.
If anything the Python 2 EOL could make a bad situation worse. Like Khan Academy, each Python 2 package maintainer will be forced to make a decision: move to Python 3 or abandon and maybe move to an entirely new language. It think many will choose to abandon, leaving these packages to rot.
Second on the list are the multiple package managers (or things looking like package managers).
Third on the list of annoyances are native extensions, driven by the poor performance of Python itself. These extensions make it difficult to use certain libraries across operating systems.
So as a non-Python developer I don't look forward to the occasions when I must use a Python-based piece of software.
Multiple package managers: There has only been 2 big ones from my long-term general usage of python. Easy-install and pip, the former of which is falling in favor but still semi-supported. Pretty much everything runs off of pip. What may be confusing you, and does confuse me at times as well, is their naming and the installation instructions as provided by library authors. E.g. some say python setup.py -install others just tell you to "pip install" it. Some would say use "setuptools", etc. Other would tell you to use things such as "conda" or "anaconda", pipx, and to create virtualenvs. All secondary, but things that should not ideally distract you from just plain using pip.
3. This has also been getting a whole lot better in the last 5 or so years. Microsoft has been funding dev-time to make the ecosystem for python (including extension compilation) much more pleasant in the Windows space. Also, the package managers and library authors are doing a whole lot better in that binary distributions are much more prominent so the compilation of the extensions never has to happen on your machine.
I take it that you've never tried the obscure and un-loved pwntools package :)
It's _literally_ the reason why I went with Ruby instead of Python all those years ago.
I've become enamored with package management in Go — not perfect yet but efficient, simple, to the point. The backwards compatibility enforced at the version level is also great — you often find Go code years old that keeps running just fine. I like things that you set up once and may just forget, that's where real productivity is found imho — it doesn't matter that I can do X in 1h if I have to do it every other day, I'd rather spend a full week or even ten, and solve it forever.
I think Js is comparably simple but I've heard so many horror stories about dependency management that I just don't know — I've yet to use Js in prod myself at work and I don't look forward to this day.
Here's the thing: it does not matter how great a language may be while I'm writing it, because that's 10-25% of my time; what matters is that everything around, from setting up dev environments to shipping passing by devops, especially as a one-man/small team, can be done "simply enough". And that, IMHO, is where Go is miles ahead of most other languages from a philosophy standpoint.
I tend to feel very positively about Rust for it looks to be an extraordinary intelligently lead project, with comparably 'real' benefits that extend beyond the code page (but I've yet to use it myself to confirm first-hand).
We see the importance of these topics so clearly with Py2/3: none of the problems between these two have anything to do with what's in the code, with programming; all of it has to do with the ecosystem, with the real and much larger task of maintaining codebases, managing teams and deploying 'stuff' in ways that work with the environment (whether tech, people, knowledge, politics, what have you).
The move to 3 by Python has been a failure in that regard, and IMHO it rests on the shoulders of an entire community who chose to stick to 2 now regardless of what happened then. Well then is now and the result is chaotic.
I'm not worried about Python itself — the language is incredibly popular, especially in academia, and we need to double the programmers population earthly each year so that's a sustainable amount of new projects written in Python 3 every day. Old 2 projects will be but a drop 10 years from now simply because of this number effects of growing tech at such an insane rate (it's been about true since the late 1940's, uncle Bob has a great take on it in his latest appearance in The Changelog podcast).
Anyway. I can't wait for py2 to die and py3 to become the only Python.
Very productive, until an unpatched security issue in a dependecy from 6 years ago bites you in the ass.
So looks like there has been at least 120m in donations!
I still find it interesting that for a relatively obvious feature set of fast compiles, fast startup and fast runtime there really isn’t anything mainstream out there to compete with Go.
I really hope something like Kotlin, Swift, ReasonML or even AOT JVM/.NET brings something to the table soon. Or perhaps I’ll just have to wait for WASM to really take off server side.
All I can think of is D, but it’s not quite mainstream. Are there any other less popular languages that meet all 3 conditions?
(depends what "fast" vs "slow" means - are we talking about milliseconds vs a second or two, or startup times so horrendous they cripple your devs' ability to iterate and tests?)
We _could_ manage our own Kubernetes clusters and such, but Cloud Run is pretty similar to that and takes away all of the management headache. There is essentially zero code difference, should we decide to change our deployment strategy later.
We're using Google Cloud Datastore for persistence, and that automatically scales in both servers and storage, so it has worked out nicely for us as well.
The nice thing about Go is it performs well in each of these use cases by ensuring nothing in its tool chain is slow, or produces slow code.
Slow is in the eye of the beholder I suppose, but I guess I’m using it here to mean within an order of magnitude of its peers.
> fast compiles, fast startup and fast runtime ... mainstream
C is this but it's harder to write secure/correct C, the standard library is smaller, and there's no canonical toolchain in the same way as Go.
They also said a faster language will improve their server.
In a typical codebase, it's probably 98% very specific and known data types linked to the variables in your code. With the remaining 2% being things that are just "easier" to solve with dynamic typing rather than coming up with complicated interface/inheritance hierarchies that you typically find in compiled languages.
And with type-hinting now being there in python, you have a very good way of "codifying" that dynamic or duck-typing. Such that you can expect almost 100% knowledge of all the data types coming in/out of your classes/functions. At this point, I'd argue it's got one of the most robust "type" systems out there, if one can call it that at all. Just don't use text editor + MyPy, or VSCode for your python development, and you'll be in good hands. I.e. Use PyCharm.
The other metric would be what a person coming from established, full-fledged IDEs such as VS would expect. With PyCharm, you get intellisense almost on-par with what you get from VS for C#/VB, assuming you use type-hints that is.
Not trying to be difficult, but I'm being honest about them providing a really good python experience that is for the most part free. There is no need to putz-around with VScode, json configs, plugins, mypy, etc and still end up getting a relatively inferior experience. Doubly so for new-developers.
Go has static types, and that distinguishes it from Python, Ruby, and Perl. But the trend now seems to be for languages to move towards static typing anyways; 2005-era Python was wrong about that.
It also has interface{} and that's a non-trivial commonality with Python, Ruby and Perl.
Given Python's incredible rise since 2005, and even in the past few years, I'm not sure we can say they were "wrong." It's serves a purpose.
In Python you have a lot of ways to make sure someone does a thing. Exceptions are good for making sure an error is handled. Context managers make sure a resource is cleaned up properly.
I might be wrong but I feel like writing something like jquery (with its fluent API) would be really tough in Go.
Some example languages that come to mind here: Python, Ruby, JavaScript, Clojure, Groovy.
Go actually comes close here, even though it doesn't have a REPL and isn't very dynamic, because of its focus on minimizing cognitive overhead and getting the job done with minimal fuss.
Other languages carry a lot of community baggage. Java is one of the worst IMO...if you try to hire Java programmers, it's going to take a lot of effort and risk to find and reject applicants who've read too many design pattern books, are architecture astronauts, or come from an enterprise-y background. The signal-to-noise ratio is just really poor.
Now look at Ruby, its stack is gigantic and cumbersome.
I guess Go will try to resist that, but over time Go will become burdened with a huge stack of outmoded software.
Stateless web servers are one of those things which are ridiculously and easily parallelizable, you can scale a web server horizontally easily. The ROI of rewriting everything in another language as opposed to just adding more web servers would probably take years unless you’re running at a ridiculously large scale. For the cost of one developer’s fully allocated salary, you can throw a lot of hardware at performance issues with web apps.
I prefer statically typed languages, but performance isn’t one of them. Besides, how much processing is a typical web app doing?
I think it's more that given their specific codebase it's similarly difficult to switch to python 3 as it is to switch to a number of other, entirely different programming languages. Once you recognize that rough equivalency, then it's worth considering the stability, compile times, necessary production resources, etc of those other programming languages.
If your project's specific dependencies are so intrinsically stuck on a python 2.x implementation you might be caught between having to redesign that dependency in-house or switching to a language where you wouldn't need to do that in-house work.
But that's almost certainly not the case. Even if their existing codebase relies very heavily on the small subset of Python2 features that require manual porting, switching to Python3 will be much less work than rewriting everything in a new language.
I will agree that porting to Python 3 would be less work than porting to Go, but we believe that the difference is less than people would suspect.
If you announce a plan, you're making a bet (sometimes with your reputation) that's it's going to be successful, and telling the world about it.
If however, you didn't announce it, you could still get the benefit of success, or make it looks like learning the lessons of a failed experiment.
As mentioned in the post, a small piece of our GraphQL schema is already in Go running in production. This blog post isn't just "we're thinking of doing this thing". It's "we've already done a bunch of research, thinking, _and_ built some of it."
Unless the existing codebase is mired in technical debt and completely unsalvageable or cannot scale further, this seems like a very radical move.
I find that this is the outsider's view of a great many products. Things always seem a lot simpler on the outside.
In Khan Academy's case, I think a lot of folks just think of our site as being a collection of more-or-less static pages with videos on them. There's a lot more going on than that, though. We've got a CMS that supports articles with math and interactive elements, in addition to the videos... and many, many exercises with hints. All translated into dozens of languages.
We have to remember every exercise people have done so that we know which ones to present to them next, and we need to display that progress when they look at topic pages. Oh yeah, and if they're in a classroom, we need to present that progress to teachers (or coaches/parents, outside of the classroom). Teachers can also assign content.
Now, we're also offering features for school districts: https://www.khanacademy.org/district
Plus, there's the official SAT prep, which connects to the College Board directly to provide personalized guidance about what to work on... and that's only one of the test preparation areas of our site.
And, as you can imagine, there are a bunch of other features and aspects of the features above that I'm not mentioning. It adds up.
Over the past year, we've started leveraging our CDN (Fastly, who have been great) a lot more. That said, for us a lot of logged out traffic still carries the weight of logged in traffic. A logged out user can start doing math exercises and we'll keep track of what they've done. If they then create an account or log in, that activity is associated with their account.
Khan Academy may look like a content site, but in many ways it's more like a "learning app".
I've come to some fairly comfortable conclusions:
1. If your team is small; don't do it. The mental overhead of that architecture will bring your team's ability to deliver down to it's knees.
2. If you've got a long-lived app with lots of legacy code; don't do it. You will have to maintain and add new features to the old codebase because it's easier, which means you have more things to rewrite.
3. If you're small-ish/mid scale. Don't do it. Kubernetes and similar are tools for people handling Facebook/Google/Netflix kind of loads.
4. If your organisational structure doesn't fit (i.e. you don't have enough people to split into smaller teams); don't do it.
Scaling is considered a good problem to have, right? That means that your current, ugly monolith is actually successful, and somehow the first thought is to replace it all with a complete—but trendy—unknown?
If the engineering team is so unhappy about the architecture then Go is the wrong choice. Maybe they could consider service oriented architecture and some refactoring/tech debt time so they feel happier about that codebase. Pull some of the code out into modules and then start figuring out where bits of the codebase would actually belong, while still being one deployable.
And then after that, if you really want to, distribute it over the network, and then start thinking about porting it.
Otherwise, throw away your first successful prototype at the first instance and go all in on distributed architecture and microservices. At least then you have the luxury of figuring it out from scratch.
It's almost religious. The reaction some people have when you suggest monoliths is completely baffling. It's apparently just "known" that it is the correct approach to all problems, so, y'know, it's embarrassing for you to have even suggested otherwise.
If you release a bug to prod in your monolith, it is exactly the same as a dependent microservice releasing the same and bringing the cluster down. You get a crash either way, and at least with a monolith you're not spreading your call-stack over the network; it's all in memory.
This seems like a response to the blog post as opposed to the parent comment.
We absolutely recognize how added network boundaries changes the app in big ways.
One thing I wanted to mention: we're _not_ going the Kubernetes and service mesh sort of route because our experiences thus far show us that there's still a lot of rough edges. We're sticking with App Engine because it generally just works. Scales down essentially to zero and scales up well with the traffic. So our services are all going to individually be running on App Engine.
Plus we're not going "micro" with our services. They're each fairly decent size, own specific parts of our data, and are owned by specific teams.
This has one big backdraw however. Not only do you need to write in two different languages, the rewriting in C requires a lot of care, as the language protects you much less than the high level language you implement for.
One big attraction of Go is, that it is high level and productive enough, to be the main implementation language, and for time critical stuff very efficient. So you don't have to cross language bareers to implement speed critical code and you get the full type and memory safety in the whole stack.
Interestingly, one can write Python extensions in Go, so in most cases that would be my choice these days for speeding up critical code paths in Python.
Hardware is cheap. Hardware is on the "accessible to a 3rd world middle class person" level of cheap.
But with enough scale, it adds up, while the costs of a rewrite don't. And the difference between a language like Go and one like Python is on the hundreds of times.
i wish them all the best, but would've been much more impressed if they'd done it, not simply announced to do it.
No question Python 3 is better than 2, but it's not better enough to justify the move. People will only move when they absolutely have to. That isn't progress, it's inefficiency.
But that doesn't mean it's a great fit for all projects. Personally, I've come to find that code in statically typed languages is easier to maintain over time, especially from a big team. I guess a lot of Python folks agree, which is why Python 3 allows static typing as well.
At a certain point, server costs _do_ add up to real money and some applications are not purely database-bound. Go's tooling makes it almost as fast to work with as a scripting language, but with much better performance. The language itself is certainly not as succinct as Python, but I think it has made reasonable tradeoffs.
Also: there's already a lot of JVM on the web.
Finally, I'll just note that _not all_ Python 3 migrations are that hard. It depends on a lot on the libraries used.
Python not being statically typed also means all of the breaking type changes they made with string now being byte or similar means crashes are not revealed until possibly production if your unit tests didn't get that oneeeee edge case right.
If I'm going to do a new project in the future, I will demand it will be a statically typed language. I'm sick of dynamic languages.
1. the Go team is working very hard to ensure that there are no such compatibility issues. Code written for Go 1.0 should still compile with Go 1.14 beta today.
2. It's possible there's more we could have done along the way, and I tend to think that statically typed languages make it easier to safely refactor more ruthlessly. But I do think we've actually done quite a bit of change incrementally along the way. Our move to React on the frontend and GraphQL on the backend have been good examples of that. Plus, we did a huge refactoring a couple of years ago to draw better boundaries in our monolith, and that has made a move to services possible.
Whenever I use python I run into problems with versions and dependencies. And the whole community just tells me to use pyenv or virtualenv and it will "fix all my issues". Only it doesn't.
YMMV on the exact number, but that's been my experience several times now.
I know it can be done; I've seen it done, I've done it myself. But refactoring without even the rudimentary static type system Go has just becomes an increasing nightmare at scale.
And I use unit testing in Python, etc.
But, flipside, yes, Go isn't a great language for just bashing a script together in. Maybe not the worst, with a bit of library work, but not a great language.
Regarding the dependencies, you have tools on top of virtualenv, such as pipenv/poentry, which handle dependencies, and are easy to use. Biggest issue that I've encountered would probably be when two or more dependencies require the same package, with no intersect between supported versions. I don't think Go handle this any better, thought.
Type hints (and mypy for static type checking) are a must, and coupled with a good IDE, they really improve the productivity. I'd say that mypy's type system is more advanced than Go's, but it strongly lacks in type safety (due to the fact that majority of the libraries are not taking advantage of it yet, and that Python is still a dynamic language by its nature, and there is no runtime type checking).
I don't know much about Go. How does it avoid dependency conflicts?
It’s ridiculously easy to build things in go. The default tooling works just great. It’s a nice fit with docker for building tiny containers.
Perhaps the nicest thing though is how easy it is to write fast http servers. The default server is pretty good, but there are also so many choices for faster http server frameworks. Middle wares are easy to write and share. I can’t say that I truly understood how http servers worked until I started using go.
Is Go really significantly faster to compile for similarly sized projects?
Although compiling overhead of large JavaScript codebases, that make up most of modern websites, kills any kind of benefit in improving build performance of some backend service.
Not only going from a monolith to microservices but also changing the language? This is a mistake that rookies make. This will be one of those post-Mortems where they will sheepishly admit they bit more than they could chew, and it wasted years of productivity.
There’s no reason to move to Go. Stick with Python for now. Migrate safely to python 3. Once everything is stable, start breaking things up into thrift or protobuf services. They don’t even need to be microservices but you need the contract. Once that is stable migrate to whatever language you want. But at this point you will have the well-defined api and test cases. Trying to do too much all at once is a no-brained disaster.
Apparently not mapping developer hours to real money keeps not being a thing outside consulting shops.
Seems pretty risky to me.
Besides, the reality is that most business software out there gets rewritten every 3-15 years (really depends on use-case and conditions, but on average 4-5-6 years is a good bet). After some time it's just not worth it to keep refactoring, you'd rather start anew with hopefully better tech and certainly with better knowledge of your problem — they say you should write everything 3 times to make sure you really nailed it.
In many businesses, these rewrites would constitute a new major version, more comparable to the feeling we always got in the waterfall era — new version = big changes, new UI, new stuff. That's when it's possibly lethal, if you really break the thing, and that thing is your product, not a means to it.
[0] IT disasters now part of modern life - https://www.afr.com/technology/it-disasters-now-part-of-mode...
[1] Number of IT failures at banks and other firms is unacceptable, say MPs - https://www.theguardian.com/business/2019/oct/28/number-of-i...
[2] The Biggest IT Failures of 2018 - https://spectrum.ieee.org/riskfactor/computing/it/it-failure...
we can argue whether these are "rewrites", but big changes can be rewarding but are inherently risky. balancing this is hard, and rewrites uncover and introduce the unknown unknowns.
There are obviously cases of failure. I don't know the stats. I'm speaking mostly of small/medium businesses, where size and complexity are different.
i think one thing we can agree on is that the only thing users hate more than change is breakage.
rewrites are a valid tool in a long-term strategy, just as debt is for finance. but for most people, incremental change has a bigger, smoother RoI. it sounds like this is kind of what KA is doing. although the timeline seems aggressive and the whole thing absolutist, being stuck on Python 2 is a risk now, too.
Data on small businesses is hard indeed. I'm only speaking anecdotally from the MSP / software shops perspective, the "tech guys" of most businesses who don't in-house IT. Also from a European perspective, so that might make a big difference — we're, ahem, let's say not as involved, interested, or capable in all things "technology" as a general population (I didn't say luddites but that's how it feels sometimes, compared to the vibe I get in NYC or rich Asian cities).
(Ironically, that post is about Netscape and the rewrite, Mozilla, ended up growing as a nonprofit far beyond Netscape's original scale.)
The so-risky-it's-almost-certain-to-fail approach is a big bang rewrite. Stop the world until the rewrite is done.
That's not what we're doing. A tiny bit of our GraphQL schema is _already_ running in production atop new services written in Go. We're going to do this step-by-step, while keeping the site running, _and_ adding committed features next year.
It's still a huge investment and comes with risks, but it's an incremental process and we'll be able to track the progress at every step.
And if you have the interfaces in hand, the language itself becomes considerably less important, since more of the code is subsequently dependent on your own system and not really the outer ecosystem. It's the projects that have coded "to the metal" on their existing platform that have the biggest issues with keeping up their flexibility.
Our system isn't perfect, but a couple of years ago we spent a good deal of effort detangling our monolith. This plan wouldn't have been an option had we not done that.
This is just dogma. Often it's the wrong choice, but sometimes engineers have very good reasons to re-write a system. As with all decisions, tailor the solution to the situation - not the other way around.
My projects (e.g. https://tmuxp.git-pull.com, https://unihan-etl.git-pull.com) are both python 2/3 compatible.
I learned by reading through other projects like Sphinx, Flask, Werkzeug, and SQLAlchemy:
- Flask: https://github.com/pallets/flask
- Werkzeug: https://github.com/pallets/werkzeug
- Sphinx https://github.com/sphinx-doc/sphinx/tree/v1.8.5 (see compat: https://github.com/sphinx-doc/sphinx/blob/v1.8.5/sphinx/util...)
- https://github.com/sqlalchemy/sqlalchemy
Helpful blog posts: http://lucumr.pocoo.org/2011/1/22/forwards-compatible-python..., http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/
As for automation tools, for 2/3 compatibility, I used futurize and huge codebase to great success: https://python-future.org/futurize.html. I started with this:
futurize --write --stage1 --unicode-literals --nobackups <files>
2to3-2.7 -w -n <files>
Also, wire Travis / your CI system to have your tests run on python 2 and 3.
If you want to test your python 3 codebase live on a subdomain / same SQL/NoSQL DB, be careful about jobs/tasks! Pickle version mismatches and stuff. Use a separate redis/whatever DB for the deployments.
After you're finally on python 3, you can use https://github.com/asottile/pyupgrade to modern your code.
[1] For the other 2%, use conditionals in your requirements file:
enum34==1.0.4;python_version<"3"
ushlex==0.99.1;python_version<"3"If we would have been able to have a 2&3 compatible codebase live in a week or two, we absolutely would have done that. Incompatible changes in some libraries we use, App Engine first gen to second gen changes (which are for the better, but still a big deal), the choice of storing some pickles permanently, plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase), and other factors that aren't coming to mind at the moment mean that this is not a couple week thing for us.
Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect.
Also, I can't speak for the technical specifics of your project, but I would like to speak a bit from my prior experiences:
> Incompatible changes in some libraries we use
What are those libraries? There may be python2/3 forks available on pypi. For instance, on Peergrade we went from pyPdf -> pyPdf2, boto -> boto3 (that one is a lot of work). We were still able to stick on the python 2 codebase, but gain python 3 forward compatibility.
In the case of very specific packages, we had to do forks. We had to move some patches from a python 2 only library and port them into a python 3 only library, then do one of those version constraint things.
> App Engine first gen to second gen changes (which are for the better, but still a big deal)
I'm not familiar with app engine so can't speak to it.
> the choice of storing some pickles permanently
Can you clarify this?
I can't speak for it without seeing it, but would it be possible to serialize the data to json, use python import strings (https://devel.tech/tips/n/djms3tTe/how-django-uses-deferred-...), then write a migration to port the old pickles over, and have something more portable?
It's possible, depending on what's being stored, it'd also help you migrate to golang if the data stored in it could be consumed by a go service.
> plus a need to really verify unicode handling all over the place (especially in a 10 year old codebase)
Yes I had a lot of pitfalls with this one. Areas to look out for are hashing functions. They are very strict in whether they're dealing with bytes or strings.
Are you already using unicode_literals? Those can be implemented gradually in a current codebase.
How is your test suite? Sometimes having test coverage, even naively, can also act as a smoke test to catch unicode issues.
Selenium tests can get a lot of coverage for very cheap to verify behavior at a high level.
> Moving to Go is more work than moving to Python 3, but in our particular case it's not as much more work as people might expect.
I think the services + golang part is smart, and can't speak for the specifics of your codebase.
I will say in hindsight, I feel the effort / hours I put into upgrading a Python 2 codebase -> 2/3 and eventually 3-only made it much easier to breath. Cleaner syntax, no unicode headaches, and no need to have the lingering prospect of a language exodus looming over the head.
One negative aspect we haven't talked about those with "gradual" python 2/3 migrating is breakages. There are refactors that end up being done that when pushed out risk breaking - and when they're purely internal code changes. There is a business case to eliminate the tech debt because the value equation: A python 2 codebase, even with warts to it, is meeting an EOL in the next few days (https://pythonclock.org/).
Even assuming golang + microservices is the final destination. Amortized, on the projects I've been on, moving to python 2/3 (or 3-only) paid back. If there's opportunities where the python codebase could be split into apps / separate wsgi entry points, then have golang services replace them later - that could be an option. Even without golang, the (probably huge?) refactors involved in moving to python 3 and being "(micro?)service ready" has benefits.
Regarding the pickles: we are essentially doing what you're suggesting. We're going to migrate them to JSON. This is one of those things that is going to be a pain to do and we'd have to do it whether we're switching to Python 3 or Go.
Also, I'll note that the Python 2 EOL is only half real. There's so much Python 2 out there that, while the PSF is no longer supporting it, there will be people supporting it as needed.
I do agree with you about Python 3 being much cleaner. But we have to do significant rewriting of our devserver, a lot of our data access code, and some other libraries that I don't remember offhand. Again, it's quite specific to our codebase. People with a "normal" Django app are unlikely to have such issues.
Go doesn't make things easy. It asks you to repeat yourself. I don't like the lack of basic functions like a generic map / filter function.. I know Rob Pike just says "use for loops", but it feels so unnecessarily unexpressive. When I see map, I know what's going on almost immediately. For loops take more reading to understand. Nil pointers shouldn't be, yet still are, a thing (developers aren't perfect - why can't the type system help?). It feels like a straight downgrade from Python from a code clarity perspective. And it's typed, sure, but the type system doesn't let me express constraints that other languages allow me to do that would prevent entire classes of bugs. It doesn't feel worth it compared to Python. Yes, the core language ends up being comparatively "simple", but simple building blocks doesn't guarantee a simple overall system. And my company is very diligent from an architectural perspective.
But then when I look at Python, I'd rather just use Javascript with Lodash, esp. when it comes to the treatment of functions as first class objects[0]. Throw Typescript in there, and you get, in my opinion, a better type system than Go, so unless language performance is a major constraint (which it hasn't been for my company, our DB usage patterns it the biggest thing instead), why would I want to use either of these rather than Typescript?
[0] Edit: Dumb and wrong, I meant its treatment of anonymous functions. I don’t like lambdas.
If I was a business owner, I'd love Go. But what I can't really figure out is why so many devs love it. It's far from the worst thing in the world, I don't hate it, but I just can't get excited over it either.
As someone who writes — I mean prose, whether fiction or not (mostly not: essays, technical, etc) — I had to come to terms quite early with the dichotomy you explain here.
There is a place for the "beautiful language", the parts of it and the ways of using it that make it a pleasure, as a writer and as a reader. This, unsurprisingly, usually demands a whole lot of additional work on top of the 'direct meaning'.
Then there's a place, in casual communication, in business, in marketing, in essays as well, in technical docs, in speeches, in a lot of places, for the 'direct meaning', or close to that. The efficient use of language, when all form recedes in favor of meaning, of concepts, of getting that 'other' to get what you mean.
It's just that, in human language, you do it all with the same tools, we use formal or topical subsets of a vastly larger ensemble. In programming, we're more likely to use different languages [themselves subsets of human language if you think about it, but let's forget that for the sake of simplicity].
And there are programmers among us who love to dabble in the form, like some writers would spend 10%, twice, ten times the effort crafting just "better form" over an already well-defined idea/story. While other programmers, or at other times, just focus on getting things done. Cue the spectrum in-between.
So it all depends what we put in our code, as programmers, as human beings I guess. Is that thing a personal statement? Or is it just garbage code to temporarily expedite some roadblock? How do we approach complexity, bottom-up from the simplest elements/code, or top-down with the most expressive almost-meta entities? Maybe some side-line out-of-the-box angle? See how we'd word all of these, in human languages, as in code. We just wouldn't say the same, nor code (select languages) the same.
I don't know if I explained it well. But looking at it from a human language writer, it all seems clear now. The whole rat race of languages, the churches, the sheer effort put into form when meaning has already been solved 10 times by others, the strong NIH syndromes... it's all so common in traditional writers circles. We're all just writers, really!
The lack of quality batteries included also leads to "there is more than one way to do it" through the choice of unofficial libraries. My experience is that the extensive standard library of Go brings more normalisation.
I think, in order to understand why so many people love Go, you first need to understand why so many people love C.
Go is C with most of the warts removed (for application development). GC, easy strings, maps, easier first-class function syntax, code formatting, modern standard library, easy concurrency, etc.
Yes, there are plenty of places that C can go that Go cannot - operating systems and embedded devices are high up on that list. But for back end application development, it's great as long as your application is of a moderate size and you don't need specialized data structures.
That covers quite a bit of ground.
So what part of C is not in the list of warts averted in Go?
A simple language with only a few things to learn. Everything built up from those few things.
1: unless designed by a committee with no such single-minded aim
That is the reason, I like Go so much. It is still a very simple language, but improves in the key parts of memory safety, having a GC, better type checking, and a few high level constructs, most of all having first class functions and closures. These enable many of the features of "higher" programming languages. You can do mapping functions over lists in Go quite fine.
For those Go would beat anything that lacks convenient green threads (Type Script included, also Java, C#, C++), anything that is dynamically typed, and anything that is interpreted.
Not sure why business owners should love it either. Not enough features in the language coding productivity suffer in a long run. Also if the developer needs Go because other languages are "too complex" maybe said developer can't produce proper code anyways.
Secondly, C/C++ is like the third or fourth most commonly listed programming language in job listings. If you think all but 2 or 3 languages are niche, that is not what the word means.
Most software problems are not about solving them faster, it’s about combining existing components in new ways and figuring out to orchestrate it all.
I don’t care about copy elision, heap fragmentation, perfect forwarding, when my performance is being lost in the communication between services. What I need is a better architecture and more scaling, not concerning myself with if this loop is being vectorized, or that object is being moved instead of copied, and other minutia which inevitably ends up wasting your time when writing C++.
Niche does not mean "stuff I don't personally use at my job", but that is the only definition under which Typecript is not niche and c++ is. C++ in 2019 had the 4th most job listings according to Indeed. Calling that a niche is absurd especially in comparison to Typescript.
There are also many places which just ask for Java/C++ experience for no apparent reason. Amazon is like this, all of their job listings mention C++ but only a very small percentage of the code base is in C++. There is at least 10 times as much Ruby code and it is part of systems that most engineers will have to work with, but no job application mentions that.
It's bit meant to be exciting, it's meant to be effective, efficient, and reliable.
In typescript, if I have a bunch of "User" objects where each user has an "name" and "age" field, and I want to print a comma separated list of all those names formatted as "name -- age", it's simple. I write:
let userNamesList = let userNames = users.map(u => `${u.name} -- ${u.age}`).join(", ")
console.log(userNamesList).
In go, to get the same level of expressiveness, I'd have to write the following stuff around it:
func mapUsersToStrings(f func(u User) string, users []User) []string {
result := make([]string, len(users))
for i := 0; i < len(users); i++ {
result = append(result, f(users[i]))
}
}
userNamesList := strings.Join(mapUsersToStrings(func(u User) string { return fmt.Sprintf("%s -- %d", u.name, u.age) }), ", ")
It's harder to point to many other cases because, simply enough, people don't write such cases. The fact of the matter is just that go libraries make very sparing use of first-class functions because the lack of generics prevents it from working well, and so we end up with an entire language ecosystem where code is harder to read and with fewer simple abstractions reused across it.
You can still write any code without good generics or first-class functions, you'll just have more programmers writing more code with less clean abstractions.
def a(n):
def b():
print(n)
return b def a(n):
return lambda: print(n)It is annoying that you have to give the function a name, though.
def b(x):
return x + 1
x => x + 1This is a code smell for sure.
You are basically saying that one should never need a multi-line lambda. Obviously, I disagree.
I will mention a core aspect of my argument, to add to your point.
When you make a named function and then use it separately somewhere else, there is a loss of locality. The logic is now further from its point of use. This is a cost, and sometimes it is an unreasonable one.
It's a very intentional language limitation, much as semantic whitespace is an intentional limitation.
And there is actually a language difference. A for loop is a statement. A lambda is an expression. Python doesn't let you put statements into expressions.
I've sometimes wondered whether code would be universally clearer if all you could do in a loop is have a one-liner or call another function/method (mainly when looking at my own code within loops and thinking WTF!!!).
Would probably cause issues when teaching programming though. It would be interesting to have a compiler switch that enforced this...
foreach ( $item in $list )
{
if ( 1 == some_function( $item ) )
{
continue;
}
else
if ( 2 == some_function( $item ) )
{
break;
}
else
{
another_function( $item );
}
}
Streams and repeated application of short anonymous functions are a much better solution to the method chaining issue.
If js supported better streaming tools, they'd be less common in js too.
I've written snippets or data structures one way or another as I'm sure many have, and in my experience the results have been mixed. This and the fact that reduce has no bearing on your app architecture, I do see this as arguing over a small and vague gain or loss.
I’m having a difficult time understanding what you mean here. Can you offer an example?
I have never felt compelled to write a multi line lambda.
fooBaz(x => {
// multiple lines of logic
})
Of course, sometimes it is better to create a named function away from the point of usage, but I don't think that is always the case.
You end up seeing functions declared within functions so that people can work around this limitation and it is grotesque.
There are a lot of cases like GUI programming where you need a lot of handlers which are suitable for lambda because they don't have a good name, and should never be called manually, and often they have multi-lines of codes.
> “Typescript is a hack on top of [a] [supposedly] server side language that is [awful]. ( Node doesn't even support int64 )”
"supposedly" in this context meaning Js is not worth being called a "server side language", in the commenter's opinion (citing lack of int64 support for example). This also implies in subtext that "server side languages" would be of a higher category/value than languages for whatever other use-cases.
I personally don't agree with any of that, just helping communication here.
A short comment would help here.
But oh man it's the same discussion every time when Go comes up here
Edit: Also, for loops are just a way of implementing a mapping, where you have data A that you want transformed into data A*. The map is the fundamental concept here, not the for loop.
In fact in many languages much of filter() can be implemented in the for construct rather than its body.
The map/filter is more fundamental in a mathematical sense. If you turn a list of A turned into a list of A*, the 'mapping' is the fundamental concept, the 'for' loop is just an implementation detail. Maybe the computer did it in parallel, or in random order, or I asked you to turn a pile of towels into a folded stack of towels. The same way if I say 'want to watch a movie' - the movie the fundamental concept, whether it's digital, film, etc. is irrelevant.
Same with map, they are a constrained for each that is guaranteed to be a correct implementation.
And maybe that is part of the problem with a generic 'map' your not really sure the underlying implementation, is it parallel, clustered, serial, etc? So you end up with map(), parallel_map(), mpi_map(), etc, and how to you control the parallelism. Do you just let it default or do you have levers to control the batch & interleave. Pretty soon, its not such a simple construct anymore.
It's pretty trivial to write your own custom map/filter/reduce though.
Ways to make your Python experience nicer:
- almost never try to use map or filter. List/set/dictionary comprehensions are more pythonic and will be easier in the long run
- learn about standard library stuff, especially the methods on dictionaries, as well as the collections package. If you’re thinking about lodash, Python tends to have replacements that end up being better tbh
- if you really want a local function, just creating an inner def statement is fine. But KISS should usually make this rare.
Not a phrase you hear every day.
For some reason, JavaScript haters always seem baffled when confronted with a person who doesn't hate JavaScript.
In backend systems, I think that the overall architecture and management of data are much more interesting problems than the code. I feel like Go helps direct more of the thinking toward the overall rather than creating beautiful abstractions in the code.
We'll see if I still feel this way in a year.
The big picture is that JS is the English of software languages. It’s not beautiful, it steals from everyone else, and it’s never the best tool, but it’s becoming - more and more - rarely the wrong tool.
I expect this trend to continue. Some genius will get JS to compile to native bytecode (WASM is the intermediate step). And, it too will be neither the best language for the job, nor the wrong tool for the job.
The proof is in the pudding, nothing so wrong (JS) can be so right - and yet it survives, nay thrives.
The Saxon words are direct and burly. E.g. oak, brash, death, iron, etc. The Latin words are multisyllabic (e.g. multisyllabic :) ). The Norse words are just plain fun (e.g. Yule, law, heathen, oaf).
It's nice to be able to choose depending on context. Similarly, I find versatility to be nice in JS, as the community goes from embracing OOP to a more functional style.
What if the end goal is not to do something elegantly, or perfectly, or with superb efficiency?
What if the goal is just to solve a problem in a way that's a little better than what came before?
That's why the authors of YouTube built YouTube in Python, and the authors of Khan Academy (re)built Khan Academy in Go, instead of bickering about nitpicks on HN.
I could be mistaken, but this sounds like they went ahead with the default JVM settings, where it tends to use as much memory it is allowed to (which makes sense from a utilization and efficiency perspective). If memory usage is a concern, the JVM can be tuned for such.
You can use a calculator to get precise, proven settings for any supported JVM: https://github.com/cloudfoundry/java-buildpack-memory-calcul...
Go is an efficient platform, however in these switches it usually goes something like "we took something hugely overbuilt, with layers and layers and layers and abstractions and abstractions, and rebuilt it with minimalist Go and now it's faster", which, of course it is.
Not really. Kotlin was just a better JVM language developed by the JetBrains folks before Google adopted it as a first class Android language, but I don't believe it was specifically developed for mobile initially.
The fact that people are doing this tedious optimization is strong evidence of the fact that Go needs a generational garbage collector with bump allocation in the nursery.
The JVM has a fast generational GC, and as a result you don't have to do this kind of optimization to get good allocation performance.
The JVM has been fine-tuned for 3 decades (JIT and GC) whereas Go is pretty primitive in a lot of its backend.
Even so, the page linked starts with "Back in April 2010, Russ Cox charitably suggested that only fannkuch-redux, fasta, k-nucleotide, mandlebrot, nbody, reverse-complement and spectral-norm were close to fair comparisons".
Since Russ Cox is one of the Go co-developers, lets see the ones he accepts are fair. Of those 5 are present in the page, on 3 of which Java wins with decent margins (fannkuch-redux: 18%, k-nucleotide: 21%, reverse-complement: 17.5%) and on 2 of which Go barely comes ahead (fasta: 6.3%, spectral-norm: 6%).
https://www.techempower.com/benchmarks/ ( real world benchmark with networking / serialization ect ... )
Get over it yes Go can be faster than Java it's not a secret.
Highly debatable. Even then, you see that golang ranks in the 102nd place for the plaintext benchmark, almost 6x slower than Netty, which is very established in the JVM world. Same with the JSON benchmark, golang is in the 126th place, ~3.5x slower than Netty and Vertx.
> but Go usually use between 3 and 10x less memory.
The JVM by default will use whatever memory is assigned to it. This makes sense from an efficiency and utilization point of view, as it generally aims for maintaining good throughput (whereas golang is only tuned for latency). The JVM now ships with new low latency GCs (ZGC and Shenandoah) which are currently available in experimental phase.
My point still stands, this is a third party library that is not commonly used, and depends on another third party library, fasthttp, which itself comes with caveats from the its author.
https://blog.jetbrains.com/kotlin/2019/04/kotlin-1-3-30-rele...
They seem like they're committed to supporting new features, but you're probably right they will lag on some things.
Developers need to be a lot more disciplined about performance and efficiency. I'm glad Khan went to Go, but man all those years wasted.