I've never met Dan McKinley
> ... adding a Python middle layer.
Yup, that POS was my baby. "Emid" the Etsy MIDdle layer.
> They had asked a Twisted consultant what to do, and the Twisted consultant said they needed a Twisted middle layer (go figure).
That's inaccurate.
Glyph was the Twisted "consultant" (for those who don't know, he's one of the main authors of Twisted.)
But he was called in later, the middle layer was not his idea, I had already written most of it by the time we hired him. I do not know why they wanted to hire him. Pretty much the only thing he did was write the Twisted Memcache interface, and that took him about 20 minutes? He live-coded it in front of me, less than 100 LoC.
The thing was, Haim (the sysadmin & DBA) blamed all problems on the middle layer as a scapegoat. It wasn't actually the problem (how could it be? It did almost nothing!) but since no one else knew that it was easy to blame it. People talked about "the Emid Problem" but it was really the DB on fire all day every day because Haim wouldn't let any business logic escape his control (as stored procs in the DB.)
> The initial release of this resulted in two full weeks of downtime, and an infamous incident where one of the investors had to drive to Secaucus to physically remove the other engineering founder from the cage.
I don't remember that particular incident but then they wouldn't have told me. Haim's meltdowns in the data center were legendary though.
All that stuff about teams must have been after I got fired. I was the whole Emid team when I was there.
> people were understandably really mad at this middle layer, but nobody agreed on the reason to be mad
Right, because it wasn't actually the middle layer that was causing the problems. It wasn't helping, but it wasn't hurting either. (For one thing, it wasn't like it was spewing tracebacks or anything like that. It worked fine, it was just pointless.)
It was very frustrating having my work blamed for problem it wasn't causing, and then being unable to get anybody to listen. (Rob didn't have the knowledge to evaluate the technical details, Haim was lying to him, and Chris was sitting next to Haim playing along with it.)
Anyway...
> The theory of the drop-in team was that the existing middle layer was bad because although it used Twisted, it still used a threadpool. (Twisted is a reactor loop like nodejs, but in Python.) For Twisted acolytes, this situation was heretical.
Again, this didn't matter because the middle layer wasn't the bottleneck.
> The middle layer didn’t really do anything. It was consultant-provided speed holes which (it was believed) would make things faster by (counterintuitively) adding network hops.
Yep. I told them this at the time. (Again, it wasn't "consultant-provided", the three founders hired me to make this thing. Glyph came later and was only there for a week or so IIRC.)
> The middle layer just received RPC and invoked postgres stored procedures. Which, if you have a superficial understanding of things, seems like the kind of boilerplate you can replace with an abstraction.
Yep. Like I did: a single decorator handled 85% of all the API methods, the method bodies were just single docstrings to satisfy syntax. Not even a "pass" statement!
> Then they proceeded to discover that here in reality every single existing endpoint did things differently.
That would be the other 15% of the API, yes.
The rest of that thread is beyond my time, but it sounds horrible. Godspeed.
> So over the course of a few weeks we frantically rewrote the drop-in replacement to use a threadpool instead, exactly like the original heretical one.
> Leaving us with literally the same code as the thing it was dropped in to replace, plus a ridiculous declarative framework, plus some tests. It was around this time that everyone got fired (but not me).
Okay, that is fucking funny.
I regret the waste, but this makes me feel a little bit better about the whole mess.