WSGI and the Pluggable Pipe Dream | Hacker News Reader

WSGI and the Pluggable Pipe Dream

lucumr.pocoo.org

121 pointsDasIch20 comments

That was quite a rabbit hole, but a good read nonetheless.

I was pretty surprised at all the criticisms that Pump received yesterday (http://news.ycombinator.com/item?id=2810373). The general response was defensive and critical just because it was trying to replace WSGI. However, it would seem that WSGI isn't perfect for all applications and indeed there is a place for Pump. My question is: how can we resume encouraging people to contribute? Quite honestly, if I had posted that thread and received the response it got, I would be utterly discouraged from contributing to the community in the future.

But the thing is the problem Pump solves, is already solved by all WSGI wrappers at application level: Werkzeug and WebOb. I don't think it's really worth to have another standard just to solve something that has been solved at the application level (and may led to even more confusions in the future.)

Armin Ronacher (who also is the author of Werkzeug) did designed something similar, called Webapi[1] and as he mentioned in the article, it's not a very good idea.

[1]: https://github.com/mitsuhiko/super-secret-wsgi-replacement

That is a good point. However, the thing that's attractive to me about Pump is how exceedingly simple it is. A lot of things (like Django) try to do a LOT. The complexity of frameworks like WebOb seem to be growing, and that's a turnoff.

I did take a look at Werkzeug and it's one that is definitely awesome. But I think there's a false expectation that choice is bad. There's nothing wrong with Pump. It's another choice, and the response from the community was "WE ALREADY HAVE THIS" instead of "good job".

There's nothing wrong with Pump (at first glance). And instead of giving it a chance, the reaction is "no thanks". And that's sad.

The thing wrong with it is it presents itself as a replacement to WSGI without solving all of the problems that WSGI goes (or any new ones, AFAICT). And as a result, no, I won't use it in replacement of a standard for which I have tools aplenty.

> the thing that's attractive to me about Pump is how exceedingly simple it is

The problem with that is that "your simple" != "everyone else's simple". Maybe my "simple" is to have the ability to yield chunks of data as it gets generated using chunked transfer. It is a simple thing, WSGI can handle but Pump can't.

Pump is trying to replace WSGI but WSGI is a standard, if it tries to replace a standard API it has to prove that it can do all the things WSGI can do, so WSGI can be kicked to the side and everyone can switch to Pump.

So to summarize, it is ok if Pump just claimed to be a simple HTTP interface but as soon as it claims to be a better, less complex WSGI, then it starts to play the "standard" game and it show how the author doesn't really understand what WSGI is and how it got to be that way.

> Maybe my "simple" is to have the ability to yield chunks of data as it gets generated using chunked transfer.

I'm curious, have you ever actually had to do this? Even Rails only just got this feature in 3.1 (http://weblog.rubyonrails.org/2011/4/18/why-http-streaming).

Yes I had to do this for an interactive media site.

WSGI is _not_ a framework. It's a gateway interface, like CGI. Django is a web application framework. You don't (usually) write apps using a raw gateway interface like WSGI or CGI. You write web applications in a framework like Django or Pump.

The framework then needs to work with the web server -- but how? To be portable and useful, a framework it should support CGI, SCGI, Fast-CGI, ISAPI, HTTP, etc. That lots of work to do for every single framework. Big ones and little ones all need to support all those interfaces, and you know they won't. That's the whole point of WSGI: to be one single abstraction to bind all web servers to all frameworks.

If you want to write a framework, WSGI sets the bar pretty dang low. It supports pretty much everything you need to do HTTP without too many awkward quirks. You don't need to worry about how to implement a fast-cgi->nginx adapter. You just make your framework talk WSGI and plug it into existing nginx->fast-cgi and fast-cgi->wsgi libraries. Done.

WSGI was carefully designed to be as simple as possible, but no simpler. It is simple, but powerful and flexible. It's possible to write powerful composable middleware that works with any webserver and any framework, because middleware is just a WSGI app that wraps a WSGI app. Middleware can transparently add sessions, caching, compression, chunking, etc, that don't need to be tightly coupled to the framework or the webserver. So, just like WSGI can bind all webservers to all frameworks, any middleware can insert itself between any framework/webserver pair (I'm sure there are exceptions, of course). The framework shouldn't have to screw around with chunked encodings or compression. WSGI's design allows one good library to easily do that work regardless of the rest of your stack.

This article spoke about how awkward send_request() is. What it does is make it trivial for middleware to read, add, and modify headers without having to parse HTTP. Suppose you want to write a javascript obfuscator. One way to do that is to write some middleware that detects Content-Type="text/javascript" and obfuscates the response. Because of the way send_request() works, you detect javascript by looking at a variable, not writing an HTTP parser. Also, if you write that middleware for Django, it'll work unchanged when you get fed up and switch frameworks.

The whole write() vs yield vs return thing can is awkward, but it makes it easier to write a wsgi adapter for older cgi-based frameworks that respond via sys.stdout.write().

Only framework, middleware and gateway authors have to care about WSGI. If a framework author is complaining about WSGI, he or she almost definitely don't understand where WSGI came from or what problem it's really solving.

WSGI is minimal, composable, and elegant, even considering that it's backwards compatible with older frameworks. If everything supports WSGI, then everything can just be plugged into everything else without much screwing around. Composability and loose coupling are very powerful, especially at the levels of the stack where WSGI lives.

I agree with you, but Pump is not a web framework. You are not supposed to write your application with Pump. Picasso is a framework I built on top of Pump (https://github.com/adeel/picasso) that's more suitable for that.

(I can see how you could get a mistaken impression from reading the discussion on HN, though.)

it was still a good contribution because it prompted this discussion, this blog post and it might lead to something new again

"But before you blindly throw it away you have to understand why it was created in the first place."

This is an excellent piece of advice that I wish more people would use. If you find something that you disagree with (a law for instance) it is a good idea to first see how it came to be. The decisions people usually have some reasoning behind them that can't just be discarded when being evaluated.

The last part was the most insightful for me.

Personally I am pretty damn sure that WSGI no longer carries the importance it had a few years ago. I think it no longer makes sense to merge different applications on the WSGI level together, it should be done on a higher level and JavaScript is a nice way to do that.

Just think about Google's gray bar. You can totally throw such a bar on top of different independent parts of the application by emitting a tiny piece of JavaScript that generates that bar and handles your user session.

I started thinking about it and was like "Right, you can make multiple async requests to hit multiple apps!"

I like the conclusion about using http. This goes against the approach advocated in monolithic frameworks such as django. It would be interesting to see a web framework that separates components and connects them with http.

That's exactly what WSGI does. Well, except for one brilliant improvement: It doesn't pass around raw HTTP, it passes around pre-parsed HTTP data structures. There are three structures: a response code ("200 Ok"), a list of headers, and an iterable that yields the body of the response (or older frameworks write to a filelike object that you can reimplement if necessary). Dealing in pre-parsed (or rather, not-yet-serialized) HTTP makes it much easier to write middleware.

If you really want to plug stuff together with HTTP, there exist HTTP servers that serve wsgi apps, and I'm sure there is a wsgi app somewhere that is an HTTP client. So you plug your framework's wsgi-app (which you didn't write and don't have to worry about) into an HTTP server. Then you point a WsgiHttpProxy at your HTTP server. Plug that WsgiHttpProxy app to your middleware and plug your middleware into another instance of your HTTP server. Rinse and repeat for as many middleware layers as you want. Now everything is HTTP, like you want.

Or, you know, just plug all your middleware together, avoiding parsing, serialization, IPC, context switching and all that crap.

The author isn't arguing that this is a bad idea per se, just that it doesn't actually work that easily.

>> But what I do not believe in is that magical plug that is called “framework independent pluggable application”. I don't know where this idea came from that it might work, but it does not. The idea that you can reused code on top of WSGI to work with Framework 1 and Framework 2 is not working out.

If it's not working out because wsgi.input can't be read twice, then write a middleware that turns it into a StringIO, and write another middleware that rewinds it.

Nothing is going to save you from the fact that you either need to buffer your response, or don't throw an unhandled exception after it's started. Yes it's a problem, but buffering is the only magic that'll help.

If the problem is that middleware authors can't be bothered to implement the whole spec because it's too hard, then submit a patch. I promise you that WSGI is easier to implement than HTTP/1.1 In fact, I would argue that it's as easy as possible. If it were easier, we probably wouldn't be having this discussion, because it wouldn't have been powerful enough to have been adopted.

Or write a middleware container that transforms the WSGI api into something you think is more sane, then implement middleware in thate.

Build the next great thing on top of WSGI, because in the end, the answer is either WSGI, something that's architecturally equivalent, or else each framework has to have its own compression, session, and other libraries. And no answer is going to save you from having to do a little bit of work to put things put together correctly.

I just don't understand why we need a single monolithic product that is at the core of the system. When you use any language specific API you are locking out possibly useful tools.

The overhead of asking each component to use HTTP is an acceptable price to pay for native compatibility with the Internet.

There are a few things I don't understand in this post

> neat nod to async systems that can use this neat trick to yield empty strings to signal that they are not ready yet.

Can't the system just do nothing to signal it's not ready yet?

An empty string is "nothing".

> Can't the system just do nothing to signal it's not ready yet?

In an async system that means you either have to suspect execution (for instance by yielding something) or to block. And blocking is bad.

Reply on news.ycombinator.com