Hacker News Reader

Comment by bjt | Hacker News Reader

Whether intended or not, there's an undercurrent of "you're all so dumb for using Python" (or Ruby, or PHP, or other similarly performant language) here. I want to surface that and question it a bit.

It's totally reasonable for a company to choose the Python/Gunicorn option if they already have a bunch of people who know Python and they don't need to serve tons of requests per second.

Even if they do need to serve tons of requests per second, it's totally reasonable for them to still choose Python/Gunicorn if the cost of the additional servers is less than the cost of having to support multiple languages. Or if they get a lot of value from libraries that are unique to the Python ecosystem. Or if they care more about quickly iterating on features than driving down server costs.

I agree that there's a point where it stops making sense, and there are plenty of engineers who don't recognize when they're past that point because they keep doubling down on sunk costs and things they're familiar with. But let's not be too quick to assume people are in that camp when we don't know all the tradeoffs they're facing.

View on HN

I don't really get what the point of this post is. Is it really a dig at python? Python can handle thousands of connections in a single thread no problem with basic enough stuff.

Is what the author did supposed to be impressive? Is it supposed to make python look bad?

I don't get it. Seems like run of the mill stuff. Python might struggle at the same level of concurrency (was it like 15k?) but you can still do 10k connections easy enough iirc.

I read it more like: "what if we just use threads instead of avoiding them at all cost and complicate things to get concurrency"

Which I read as "I have no experience whatsoever with modern python, and async is something that catches water".

Async is typically less prone to error and complexity than thread code, and also typically faster/lighter for io (in python). I can't see a reason for this not to be the case in other languages too.

Note: these opinions are mine

In terms of error-proneness, I would say the hierarchy is this:

Actors < CSP ~ Async << Threads

In terms of "getting started" difficulty, I would put the order like this:

Async < Actors ~ CSP < Threads

In terms of first-order maintainability in large projects I would put the order like this:

Actors >> CSP ~ Threads > Async

Async is on its face easy to grok, and saves you a ton of problems with locking and the like, and you can run it on a single threaded system with the right type of dispatcher. However, small amounts of complexity rapidly devolve into towers of async calls that cannot be untangled from a spaghetti pile, and there's not really satisfying ways of figuring out how to unwind error-handling in async, except for the basic case of "I truly don't care if this async fails".

I will say though, it's spectacularly easy to write messy and garbage code in all four of these concurrency models (I certainly have). My preference for actors comes from four opinions:

- some systems (if you want to be flippant, microservices architecture) use actors to encapsulate failure domains, which is really fantasic, and truly the #1 reason to use actors

- 99.8% of the time no need to write mutexes, and good actor systems basically won't deadlock unless you really try hard. (also true for async system btw)

- gives you an organizational framework to write well-designed and well-engineered systems.

- with a small amount of discipline "not having a spaghetti ball" scales with complexity (I find it takes a lot of discipline to not have a spaghetti ball with async in the more complex cases)

The async stack traces I've come across in Python are extremely simple to grok. I've never had something I couldn't figure out. (Using aiohttp, aoipg, asyncpg).

I can't even say the same for synchronous Django. Sometimes it's just the quality of the tool you're using, not the higher-level concept it implements.

You can trivially implement async on top of actors, (like exists in Elixir's Task module), and generally most async in elixir should go through that model. But if you're in python-land you really don't know what you're missing. In elixir, my tests autopartition the state so that each test exists in it's own "universe in the multiverse", so I can run concurrent, async integration tests with stubs, database transactions, even http requests that exit the vm (through chromedriver) and come back, finding their own correct partition of the test mockset and database state, and I have a few custom extensions to the multiverse system like process registries, and global pubsub channels. We're talking hundreds of highly concurrent integration tests that run through completion in seconds.

I made a post some time back about maintaining 64k connections in a Ruby process a while back: https://www.wjwh.eu/posts/2018-10-29-double-hijack.html . It only stopped at 64k because I could not be bothered to rig up multiple IPs so it eventually ran out of ports.

Just having a lot of connections that do trivial stuff is not very difficult. It becomes way more interesting when all of those connections need to access shared data structures and whatnot.

It seems like doubling down is the standard thing to do. Here's a video about how Instagram bugs engineers to do fewer string manipulations in Python instead of using a faster language https://youtu.be/hnpzNAPiC0E

You ... are not ... Instagram.

Okay. Get over it.

Python is fine. Threads are fine. Go is fine. Clojure is fine. Java is fine. PHP is ... okay I won't go that far. :)

You fix the problem when the cost required to fix the problem is finally less than the opportunity cost of fixing something else.

I would prefer not to deal with "scaling up" for workloads that could run in a single process on commodity hardware.

I've mainly worked at companies that would be bankrupt if they used Python the way Instagram does.

Django and Python somehow handled websites for entire newspapers on far wimpier hardware just fine--which makes me wonder if "cloud" (ie. non-deterministic memory and I/O accesses due to sharing with other tenants) isn't the problem rather than Python.

You need a market. You need paying customers. You need cashflow. You need features. You need a business plan.

You don't need scaling. Ever. To first, second and third order approximations.

Your company has a higher probability of bankruptcy than needing scaling.

Newspapers run everything behind a CDN, so Django isn’t a bottleneck. The things they do that can’t just be CDN buffered (like ad targeting and paywalls and comments) end up being supplied by outside vendors. I think Django fits well for a news org, but it’s good to understand why it fits well.

PHP is fine too. Check out https://laravel.com/ and tell me that doesn't look like a joy to work with.

How much of the old "A fractal of bad design" post would you say still applies? That one post showcased so many footguns right at the language level that it spooked me away from it forever.

A lot of those are still applies, but these days you can ignore those bad parts and only use the new good parts just like how you would use javascript. Still, I only use php occasionally but I wish it's not as eager in treating string as number in many cases.

You can get a good sense for what "modern PHP" looks like by browsing through PHP The Right Way (https://phptherightway.com/).

PHP has grown up a lot over the last ten years.

if you have an hour to spare, this (https://www.youtube.com/watch?v=wCZ5TJCBWMg) is a great talk by Rasmus Lerdorf, the creator of PHP, on the design of the language, the reason it ended up the way it did, and the ways that it is evolving for the better.

I took a look, doesn't seem like a joy to me.

It's too stringly-typed for my tastes, seems like a lot of errors you can make will only be caught at runtime.

I just got roped into a project with the promise of Django, but the project lead ended up deciding to go with Laravel, so I've been looking into it for a few days.

It's a resounding no from me. If you're stuck with PHP, sure, this beats the WordPress style of...well...I can't really find words to describe how bad it is..., but it's still miles behind anything else. It full of strings and other things that you just have to memorize with IDE integration even with specialised extensions still worse than other languages with standard IDEs.

"String-programming" is mostly optional. Modern Laravel is introducing type-safe alternatives. For instance, instead of doing validation like:

  'gender' => 'in:male,female,other'

You can now do:

  'gender' => Rule::in(['male', 'female', 'other'])

Which, coming from Go, I very much prefer.

Modern PHP is pushing for type-safety (via type-hinting) and Laravel is following this direction as well.

And it has JIT support out of the box, while PyPy still needs to fight for adoption.

> Even if they do need to serve tons of requests per second, it's totally reasonable for them to still choose Python/Gunicorn if the cost of the additional servers is less than the cost of having to support multiple languages.

How hard is it to get up to speed on any other tech stack? ASP.NET Core is extremely fast and the learning curve is close to none, for example.

If someone was able to wrap his head around backend development with Python I'm pretty sure they have the mental fortitude to onboard a tech stack that doesn't suffer from major performance problems.

This is where my head always goes as someone who does mostly Asp.Net Core. Why is it always between something like C++ or something like Python? Nowadays with middleware and endpoint routing, asp.net core can be almost as simple as flask (even if that’s not idiomatic or what the docs show you).

Or Azure Functions if you want to go the serverless route. Coupled with Visual Studio, you can build high quality, large scale apps at about the same effort of blinking a LED on Arduino.

Are we using the same Azure Functions? I spend more time messing with functions.json and determining binding types and waiting five minutes for error messages to come in than I ever did when I simply provisioned an app service and deployed code.

Feel the same way with lambda.

I’m most comfortable in .net core but I recently learned Django because I kept hearing how productive it was from HN and all the startups around me use it, but Im starting to feel like I chose to move the wrong way. I will say I like how Postgres is the default for python and the library situation is much better over there as well.

I am a little confused by your post. Moving to Django is the wrong way?

Yeah Django feels terrible to me compared to .net core. I already wrote a similar answer to someone else so I will just paste it here

"The developer experience in python was just way worse for me. Autogenerating swagger docs didn't seem possible without manually adding annotations to all my code (for flask at least), the battle-tested python libraries like django and sqlalchemy aren't built for async, mypy is obviously way worse than an actual type system and having static types really helps with understanding a large new code base, model validation (the thing forms/serializers handle) was more tedious, vscode was worse than visual studio (code navigation was very hard to do in python. I could jump 1 layer into library code, but when I tried to jump deeper vscode couldn't find anything.)"

I've seen people say you should only choose rails or django for a startup, but .net core mvc provides the same batteries included approach and is built with the modern web ecosystem in mind. It also runs on linux and easily integrates with postgres so its just as cheap now as well. I think people that write off C# haven't really worked with it in its present form. I didn't find python significantly more terse than C# (other than having to define properties on types, which I already stated I prefer), just less feature rich.

I've tried working with Django and it's always seemed much too heavyweight to me. Flask is my preferred web framework for Python; much easier for me to use and feels like it's helping me where needed and getting out of my way where needed, not weighing me down.

Django is very good if you want your end-site to be "CMS like". It comes will a lot of accessories to help in that goal.

Flask is lighter and it is more DIY but easier if you have something that is more generic.

Yeah I've heard that before, but from my (admittedly brief) time it seemed like you could just use the utilities you want and ignore the stuff that forces you into a CMS'y box. Feels like if you really wanted to you could just use a form/serializer to validate input data and to return a domain object, pass the domain object to a service layer, and then use the django orm (or honestly anything if you really want to stray from the django way), to handle database interactions. I understand flask might be nicer because you get to feel like you dont have a bunch of wasted code bloat in your deployable, but there are some pluggable systems that come with django that are nice (user system, pluggable authn/authz, caching framework, etc.) I didn't really mind the bloat because a lean deployable isn't that important to me and it all felt sluggish to me compared to .net core anyway.

Yes you can definitely pick and choose the parts of Django you want. Sometimes they have interdependencies but you don't need to use it all. Or for example, use Jinja instead of its templating system

Yeah I actually wrote a prototype in Django and flask and just felt like I was reinventing the wheel too much with flask. e.g. marshmellow = serializer from DRF, sqlalchemy = django orm, etc. I wrote password hashing with bcrypt for user logins and then realized I should salt the passwords and just went back to django since it already has stuff for everything I was doing.

I actually don't mind the heavy framework thing because .net core mvc is the same deal. The developer experience in python was just way worse for me. Autogenerating swagger docs didn't seem possible without manually adding annotations to all my code (for flask at least), the battle-tested python libraries for web dev aren't async, mypy is obviously way worse than an actual type system, model validation (the thing forms/serializers handle) was more tedious, vscode was worse than visuals tudio.

Same here, for me it has been mostly about .NET, Java or anything else with JIT/AOT compilers out of the box, after my AOLServer like experience.

Back then we already had Zope and it wasn't blazing fast.

I disagree. I tried to learn F# a few months ago and the experience was horrible.

I followed the tutorial using the aspnet CLI tool and got nothing but errors. I Google’d for a while until I gave up.

Can you link to the tutorial you followed?

Of course it’s gonna be hard if you are trying to learn a language plus a framework at the same time, especially given that F# is treated as a second class citizen by Microsoft. An easier path is to break that learning into more manageable tasks: on one hand getting familiar with F# (which is already some work as it has both a functional side, and a CLR one; you need more the OOP part for doing ASP.net) and in the other ASP.net, for which the golden road is C#.

Otherwise the doc is quite good for ASP.net Core, and it’s rare I get stuck on a problem for too long with it.

> I tried to learn F# a few months ago and the experience was horrible.

Switching to a programmig language based on an entirely different programming paradigm is not comparable to switching to a language based on the exact same programming paradigm to develop the exact same application using the exact same design patterns.

I’m completely comfortable with the language coming from Elixir. The problem was the tooling/docs didn’t match the experience.

This was from the official site and many of the commands threw errors.

I think it was a case of incomplete Mac support, which should have been called out.

IIRC I posted on HN about it and got an apology for the state of tooling

> How hard is it to get up to speed on any other tech stack?

If I find myself debugging python tools, I usually just add debug statements to figure out WTF it is trying to do, and reimplement it in bash. It invariably is less than 10% as many lines of code, and also more debuggable / readable than the original.

Granted, most of the python scripts I see these days are build processes or cluster coordinators.

> reimplement it in bash

> more debuggable / readable

More debuggable / readable for you, perhaps.

Hey, for me too. But then again I used to write bash frameworks for fun so there's that.

Given how badly bash goes wrong if spaces in filenames sneak in, or if you want to do error handling, at several places I've had a policy of rewriting scripts from bash into python. It has the advantage that it's usually a better cross-platform solution than running bash on Windows.

I'll find out myself. Just got handed a codebase that is a mix of Python/Gunicorn/node when this sprint started off. I've seen the word gunicorn in the repo... but still don't know what it does yet. First time for this old back end Java/C++/XQuery programmer, so how hard can it be?

(On the plus side, seems my Jetbrain kit includes PyCharm, so I've even got an IDE!)

Gunicorn is just a WSGI server, basically used to spawn a pool of webserver processes for your python backend. A python webserver is more optimal when used in multiprocess configuration (as opposed to multithreaded configuration, where python sorely suck at), and gunicorn will do that for you automatically, routing each http request to available worker process in the pool.

> Gunicorn is just a WSGI server, basically used to spawn a pool of webserver processes for your python backend.

Or a pool of asynchronous "green threads" using any of a number of libraries (gevent is the one I'm most familiar with). The thing to avoid is mixing the two (multiple processes and green threads).

> A python webserver is more optimal when used in multiprocess configuration

For CPU bound worker tasks, yes, this is true. For I/O bound applications, not so much; asynchronous I/O can handle the same I/O load with much fewer resources (particularly as forking Python processes uses a lot more memory because so much of the memory in the Python interpreter is dynamic, so you don't get a lot of benefit from what in a compiled language would be shared read-only code that doesn't need to be copied for every fork).

> (as opposed to multithreaded configuration, where python sorely suck at)

Yes, the limitation of the GIL is one of Python's worst warts. (It looks like there are finally efforts to remove it, but it's taken a long, long time.)

Ah yes, I haven't used gevent for years and now associate gunicorn with wsgi. I have far easier experience with nodejs and golang when I need to do async parts on the backend (usually for websocket stuff), then use some message queue/passing system with celery or zeromq to communicate back and forth with the python backend.

That's because I'm more productive with Python (been using it for ten years to implement many backend services), never hit any of its performance limitation yet (I mostly use it to develop b2b app with medium traffic at most, and always leverage distributed task queue for anything that might excessively blocks the webserver process), and also didn't want to maintain a fleet of windows servers (at least in the past when .net was windows only).

It does not take that much time to get up to speed on another tech stack, but it does take some time, and servers are really cheap. Even if the learning curve is close to none, it's just cheaper to rent a dozen machines than have an engineering team spend a day or two looking into a new ecosystem.

Learning curve close to none? I like that many times there's "the way" of doing things, but when I have to do something slightly outside the recommended way, I feel that the effort required outweights all the benefits.

Isn't that true for all frameworks?

“The way” of doing things depends on the language, and for example for me Python is very hard. It’s not my main stack but I use it regularly and after years I still can’t find "the way" of doing things by myself, I feel like I end up on stackoverflow way too often because of anxiety of not being idiomatic. It’s like there is an enormous meta knowledge very specific to the language. That’s not something I experience with other stacks.

1) .NET APIs change very frequently.

2) C# is a very verbose language, that requires a lot of typing.

3) F#, the best language in .NET, is largely ignored by the .NET community.

I can't speak from recent experience, but on point 1) when I was teaching myself to code, I focused on .NET due to its prevalence in my local market. I took part in the .NET user group (called DUG, natch), attended the meetups etc.

And this was at the time when MS would announce a new blessed way to do things on a reasonably frequent basis. When I started, the blessed way to access data was DAOs, then it was ADO.NET (note, those could be around the wrong way, I have trouble figuring it out now in hindsight), then it was Linq2Sql, then that was deprecated for Entity Framework (which, I'll give credit, they seem to have stuck with, even if it does feel like a half-cribbed NHibernate).

I was frantically trying to learn the blessed thing, because the MS shops I was familiar with only used the blessed thing - the server was IIS, the database was SQL Server, the language was C# (I was the only member of the DUG who coded in F#, and few others knew of it), and you used the blessed patterns and the blessed frameworks.

Incidentally, this is why FOSS has had such a hard time in .NET, as soon as MS releases something that is reasonably feature complete, a lot of single-vendor minded companies switch to it.

And I met a lot of developers in their early 40s who were quietly terrified of getting left behind on the MS technology treadmill, and trying just as frantically to learn the new blessed thing as I was.

And then I got hired by a Java shop, and faced a paradigm where shit code cough java.util.Calendar, java.util.Date cough stuck around for yonks because it was good enough and replacing it had to be done very thoughtfully and gently.

Point #2 isn't super relevant in the age of IDEs, but I agree wholeheartedly that F# deserves a lot more love than it gets.

> 1) .NET APIs change very frequently.

ASP.NET Core 2.1 was released in 2018 and will be supported until late 2021.

ASP.NET Core 3.1 was released a few months ago and there is no end of support in sight. Moreover, the changes between 2.1 and 3.1 were not that many. I've migrated a whole ASP.NET Core 2.1 web service to 3.1 in less than 1 hour.

> 2) C# is a very verbose language, that requires a lot of typing.

Nonsense. The only added verbosity to C# when compared with Python are the type declarations, which arguably are a problem plaguing Python. The first class support for events and async programming and properties in C# more than make up for it.

> 3) F#, the best language in .NET, is largely ignored by the .NET community.

I fail to see what point you were trying to make.

Compare C# with Kotlin or Scala and you'll find that involves a lot of redundant, boilerplate code.

I wonder when was the last time you used C# then.

> ASP.NET Core 2.1 was released in 2018 and will be supported until late 2021.

Which is way too unstable, especially for the kind of corporate environment c# has typically been used in. Getting those places to upgrade to stable supported versions of the framework has always been a battle even when backwards compatibility was great, if they have to deal with breaking changes every few years they will never upgrade.

This is why so many companies stick with their ancient COBOL systems, most modern alternatives don't offer the stability they need.

> Which is way too unstable,

ASP.NET Core 2.1 is the LTS release of ASP.NET Core 2, which was released in 2017. I fail to see how a first class framework with a LTS that was released years ago can be described with a straight face as "way too unstable".

> Getting those places to upgrade to stable supported versions of the framework has always been a battle

ASP.NET Core 2 is stable since at least 2 or 3 years ago, depending on how you decide to count.

> This is why so many companies stick with their ancient COBOL systems, most modern alternatives don't offer the stability they need.

This assertion is simply wrong at so many levels. Don't confuse "why waste money maintaining working software" linesof reasoning as a sign of respect for stability.

More importantly, it's disingenuous to even think of the technical debt that keeps cobol on the map as relevant to the world of web services.

> I fail to see how a first class framework with a LTS that was released years ago can be described with a straight face as "way too unstable".

I fail to see how you can call 2 years of support an LTS with a straight face, it's taking the piss out of the term. The LTS of the OS I'm likely to run it on is supported for 8 years. 2 years isn't even enough time to finish many projects on the same LTS it started on.

At work we've got 30 year old c/c++ code bases that still run, they'll probably run for another 20 at least, we've got 20 year old python code that still runs (for now) and we've got 20 year old c# projects that still run. That last one will never get rewritten in .net core, in part because they've pissed away the stability the framework had. It would be crazy to use tools with 2 years of support for any of those projects.

> Don't confuse "why waste money maintaining working software" linesof reasoning as a sign of respect for stability.

Why should they waste money maintaining working software when there are stable options available? What does upgrading to asp.net core get them? Why should tens of thousands of companies waste money modifying working software just because someone on the core team thought the existing API was inelegant or too hard to maintain compatibility?

C# doesn't require all that much typing if you're using Visual Studio, and even less if you have Resharper.

And you mention this as if it was a good thing.

It is, not using IDEs in 2020 is like cutting trees with a knife.

You will eventually cut it down, but I see better ways to use my time.

The strengths of any programming language can also be their weaknesses, so there isn't really a "best" language.

Notice a language can't be everything. They're either too verbose, too terse/cryptic, or trade ease of development for lack of control/performance/efficiency.

IMO, C# strikes a good middle ground on such matters...

A lot of developers never get proficient on one stack. Much less more than one.

You don't need to master one stack to get up to speed. For that investment to pay off you only need to be almost as good as you were with your old stack, which isn't hard if you already are not proficient.

I'd argue that time would be better spent actually learning any single stack properly.

Writing code is one thing though.

Now you have to figure out how to run a CI on a build server. Deployment on your production systems, be it containers or even not. Monitoring, alerting, profiling, tuning under load. You have to support a new database connection library with new shenanigans. In general, you need to integrate the new language into the existing ecosystem. The latter may even be impossible depending on the stack and solution chosen. Just look up those weird Java only caching servers.

All of that is possible, sure. Due to company acquisitions and specialized teams in some areas, we're kinda running the full bingo card of languages.

But there's little denying: We overall spent less time handling language runtimes and language-specific monitoring back when we were java+mysql and that's it.

> Now you have to figure out how to run a CI on a build server. Deployment on your production systems, be it containers or even not. Monitoring, alerting, profiling, tuning under load. You have to support a new database connection library with new shenanigans.

Most if not all of those items are either trivial or non-issues.

In fact, I would argue that deploying a Python app is a far more convoluted process than getting an ASP.NET Core app up and running.

With Docker, the problem simply disappears.

Getting it to build and test on a CICD pipeline is as hard as typing $ dotnet build, or $ dotnet test.

> All of that is possible, sure.

Not only it is possible, it's laughably easy.

We are supposed to avoid our problems, not perpetuate and aggravate them as eternum because we are too lazy to look for ways to make our life easier.

> It's totally reasonable for a company to choose the Python/Gunicorn option

Yes, if done properly. The issues with the particular Python/Gunicorn setup the author described in an earlier article (linked to in this one) were not so much Python/Gunicorn issues as "not understanding how to properly use Python/Gunicorn" issues, or more generally "not understanding how the tool you are trying to use actually works" issues. (I actually shudder to think what such a group would have done trying to program the same application in C.)

Twitter ran on Rails until 10 million users.

Ah yes, the Fail Whale era.

They should have kept the fail whale instead of offering "Sorry the tweet couldn't be loaded. Please retry" which I get 3/4 of the time on my phone.

Missing an order of magnitude? Twitter was claiming to be still running Rails in the mid-2010s, when they apparently had 200+ million active users. The service was finally sunsetted during the big Scala rewrite.

They abandoned their message queue (Starling) written in Ruby early on. That abandonment was often misattributed to them abandoning Rails, I guess because of the shared usage of Ruby. The confusion was compounded to back when someone at Twitter (may have been Odeo at time?) posted a message board rant about how Rails does not scale because ActiveRecord did not allow connecting to multiple databases out of the box. That event seems to be the origins of the "Rails does not scale" mantra that swept the internet for a time. But, humorously, someone replied with a solution a few minutes later.

But how many instances did they have running that app, and at what cost? Did they have to build a ridiculous amount of caching in? And wasn't that the time period where they were incredibly unreliable to the point that their "fail whale" server error page became a running gag?

Twitters architecture was a bad joke. Many to many communication in a scalable manner has been a solved problem from decades: you federate. You assign users to buckets, you assign buckets to servers. You route messages like you'd route e-mail. Been there, done that. The federation does not need to be outwardly visible.

Twitters problem was a too centralized architecture not Rails.

And I say that as someone who at the time hated Rails and who still hates Rails. I find it bloated and over-complicated. It may even have led them to make bad architectural choices because of how it was structured.

But they still did make bad architectural choices, and they fixed those choices at the same time they moved off Rails.

Until they had to switch to a mix of Java, Scala and C++.

Had they started with them and there wouldn't exist blue wales.

Not to mention that python is plenty fast compared to the time it takes to write stuff to the network. Of course heavyweight frameworks like django don't help the equation, but writing fast network code in python isn't exactly hard either.

> python is plenty fast compared to the time it takes to write stuff to the network

With the proliferation of microservices, I find this increasingly not true. Sure, python definitely is plenty fast when you need to send something to a user many miles away. But with microservices, writing to the network might mean writing to a machine in the same data center, or even the same host in a different container. That's as fast as a few memory copies and a few context switches.

This is something a lot of people don't get with most higher level languages.

My first commercial use of Ruby was in 2005. Not web facing, but messaging middleware. As in a pub-sub type passing of messages between various endpoints.

We had a C version. It was about 7k lines to support the bare minimum we needed. As an experiment to teach myself Ruby I wrote a Ruby implementation. With the usual caveats (it's often easy to make a rewrite better in all kinds of ways, including size), it was ~700 lines, far easier to read, and supported far more functionality, so I put it in production.

Was it slower? As usual that depends what you mean by "slower". It consumed 10x more CPU, but it also did much more work (e.g. supporting more flexible routing of messages etc.). The throughput, however was the same, and 10x more CPU means that maxing out the network connection took 10% of a single core instead of 1% of a single core.

For some types of tasks CPU is the most important thing, but for a lot of tasks you'll be IO limited. And for a lot of tasks that people think are CPU limited are really down to poor IO handling (causing excessive context switches e.g. through lots of small reads is a common one)

And other people don't get it is possible to have high level languages and almost C like performance.

You don't need to give up on JIT and AOT compilation to use high level languages, and this is where current tooling for Ruby and Python ends up losing.

Fair enough, I've been playing with D a lot lately, which is basically "what if python was a C dialect instead". It's an incredibly simple language to learn but it's no less easy to write in than python. For most things I still reach for python though, probably because I've grown comfortable with duck typing.

side-note when starting with D: make sure to install dub. it's the package manager and basically eliminates makefiles from the compilation process. Just "dub init" and "dub run" and you're off to the races.

>python is plenty fast compared to the time it takes to write stuff to the network.

Give us some numbers.

Inside the data centre you have 10G, 40G, 100G ethernet connections. I know for a fact that you will struggle to soak a 10G connection using a single thread so I know you can't do this in Python without multiple processes using SO_REUSEPORT.

So use multiple processes with SO_REUSEPORT then. Or find yourself a wsgi server that does, because it's not exactly an unsolved problem.

That said by far most programs don't need to worry about saturating a 10G connection. I'm not writing a file server in python, I'll leave that to nginx or S3. I'm writing business logic in python which tends to be bottlenecked by a database in any case.

Python is great for plumbing together other functionality, which it turns out means most backends you'd be writing anyways. Python is less great at handling a large quantity of data, though most of the time you can get away with handing the data handling to some library (e.g. numpy or libuv or any one of thousands of libraries).

Worst case you can easily plumb in some C-calling-convention code into python. With FFI it's a matter of copying the header definition and you're off to the races. That way you can still write the bulk of the program in python, delegating the bulk data wrangling to C or D or rust or go or whatever you prefer.

The only reason to use Python for anything more than few hundreds lines worth of utility is if you're working on a codebase that's already in Python, and even then it's debatable.

There simply isn't an excuse for using Python for any infrastructure. It does nothing particularly well - or even right - other than very purpose-specific scripting. It can tie things together well enough. And your codebase becomes a liability rather than an asset.

I always say this opinion when it comes to Python discussion in HN and I always get downvoted but hey, "all it takes for evil to triumph...".

> The only reason to use Python for anything more than few hundreds lines worth of utility is if you're working on a codebase that's already in Python, and even then it's debatable.

One important reason for using Python is that it almost always forces people and companies to release the source code.

I will take a shitty Python script over shitty C/Rust/Go/Java code any day for that reason ALONE.

With source code, I can fix your shitty program (and all programs are shitty--even mine). If it's compiled, that path is blocked.

Any which language do you use for web apps if I may ask? Let me guess. Rust.

I primarily do all sorts of systems programming, but the few web backends I did were C# on Azure Functions and it was spectacular. I’m a very big proponent of Microsoft’s tooling and language development.

The difference in productivity between C# and Python is such that they might as well have been developed by different civilizations.

(I have occasionally used Python since about 2003 and C# since 2018)

> The difference in productivity between C# and Python is such that they might as well have been developed by different civilizations.

I think they were. Is that a bad thing?

So you always get downvoted to hell whenever you post this and you've decided everyone else is the problem? Can you even understand how you sound? I cringed with sympathetic embarrassment just from from reading this. Seriously, rethink your life choices man.

This coming from someone who has literally never written a line of python in his life.

Trading internet points for potentially helping someone make a better engineering decision is a fair enough deal.

"Rethink your life choices" — because they have opinions that are unpopular and they express them anyway? I might not agree with those opinions, but I fail to see how discouraging them from expressing them is in any way good.

People expressing unpopular opinions and defending them with evidence is how we find out when the popular opinion is wrong, and how, when our discourse is functioning properly, we can gradually change the popular opinion to be less wrong. You, and the people downvoting the comment, are throwing a monkey wrench in those works.

I love Python and use it most of the time when I need to get things done quickly, but its runtime efficiency cost is becoming increasingly unappealing for three reasons: the end of Moore's Law, the concurrent rise of manycore and SIMD, and the steep rise in Python's footgun count. And now there are better alternatives.

— ⁂ —

In 2000 or 2005 Python was a simple, consistent, practical language with a policy of strict error handling that was very useful for producing reliable code: "Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it." Its runtime cost was significant but bearable, about a factor of 20–40: if you wrote your code in C instead of Python, it would run 20 to 40 times faster, and that was all the machine could do.

In 2020 Python is an overcomplicated, inconsistent, slow, unreliable language with a persistent schism resulting from the core developers' poor choice to make backwards-incompatible changes to simplify the language. It has metaclasses, superclass method resolution order linearization to enable mixins (two different ones, in Python 2), a lazy sequence construct that has been gradually Frankensteined into a general coroutine construct (with an additional lazy sequence construct added on top), two different incompatible language constructs to compensate for the lack of block arguments or full-fledged lambdas (decorators and context managers — I'm excluding generators here since they're more powerful than block arguments), and on and on. The reference documentation for "import" alone is 20 pages, and that's in Python 3, the simplified version of Python.

Python's performance cost has not increased in absolute terms — in fact, it's even improved a bit — but it's increasingly painful. In 2000 we could rest easy knowing that whatever we wrote in Python would be sped up by Moore's Law and Dennard scaling, roughly a doubling in speed every 18 months, so in three years it would be four times as fast, and in three more years it would be 16 times as fast. That, together with a little judicious implementation of inner loops in C, was a small price to pay for getting things done sooner and not having to open core files in a debugger.

But then Dennard scaling slammed into a wall around 2006 and Moore's Law sank into a swamp around 2016. Meanwhile, manycore meant that without multithreading, or at least multiprocessing, your program suffered an additional order of magnitude slowdown. Even US$40 hand computers now feature quad-core CPUs. Today, the gap between what the machine can do in absolute terms and what it can do when saddled with Python is a gap of 1000 or 10,000, not 20. If you can cope with the limitations of PyPy (it supports Numpy now! Since 2017) then you can get up to the speed of single-threaded C, which is about 3% of what your computer is capable of. But it's not going to get faster just because hardware progressed: computers will maybe be twice as fast in five years, at best, and maybe not. If it's too slow today, it'll probably be too slow then too.

But that's not the worst part. Python's completely botched Unicode handling introduces bugs into most Python programs that handle strings from the outside world, latent bugs that only surface once those strings contain non-ASCII characters — similar to the situation with bash scripts and filenames containing spaces, although that can be detected by purely local analysis (missing doublequotes around a $var, red alert!). Plan 9 had already demonstrated one correct way to handle the situation (the one used in Golang and Rust) and Markus Kuhn's UTF-8B proposed another, one which was eventually partially implemented in Python as PEP 383 ("surrogateescape") but turned off by default. I've had bugs in on-orbit satellite control software that I couldn't track down because Python generated a UnicodeDecodeError when it tried to log the stack trace.

— ⁂ —

At the same time, other alternatives got a lot better. Java grew into a mildly reasonable language, and Kotlin and Clojure are outstanding ones. Haskell, defying everyone's expectations, became practical. Microsoft started trying to embrace and extend free software, so now we have F# on Mono, which is almost OCaml — almost as convenient and concise as Python, but enormously less bug-prone. Mike Pall, a superhuman intelligence from the future, wrote LuaJIT, which gives you performance on par with C in a language as friendly as Python — not modern overcomplicated Python, old Computer Programming For Everybody Python. 100 million people, including little kids, program computer games in Roblox using Lua. (It's bug-prone as hell, though. Lua has a footgun for each toe, as Sean Palmer says.)

Even C++ has been tamed somewhat. And of course we have Rust and Golang. Golang is only a little bit uglier to program in than Python, and both of these new systems-programming languages make it a lot easier to take advantage of manycore, though not SIMD.

Switching from Python to Golang is as easy as switching from Perl to Python, but with a lot more benefits. And that's a big reason why a lot of the important infrastructure software written over the last decade has been written in Golang.

On the horizon, we have things like arcfide's Co-dfns APL compiler, Matt Pharr's ISPC, and GLSL to show us how massively parallel programming, including SIMD, can become accessible to mere mortals. They aren't yet practical options as alternatives to Python (except that GLSL is practical for its original purpose, of making nice graphics), but they might be pointing the way to something that is.

— ⁂ —

So I think it's eminently defensible that Python should now be consigned to "a few hundred[] lines worth of utility". Python is great for scripting TensorFlow, and it's a far superior substitute for MATLAB. But writing infrastructure in Python in 2020 is like writing infrastructure in Perl in 2005.

...which I was also doing. I think I probably owe an apology to a lot of folks at Aruba Networks who are maintaining that code today.

Reply on news.ycombinator.com