I spend most of my workday writing Go, so my monkey brain immediately assumed that this was an epic treatise on crispy chocolate from Russ Cox. I'm...slightly disappointed that it's not.
636 karma · joined March 16, 2012
I wonder whether they're also queueing for reads from the TCP connection? At least per their public comms, Facebook uses an interesting combination of CoDel and adaptive LIFO processing [0]: under load, they preferentially serve queued requests in last-in, first-out order to prioritize users who are likely still waiting for the response (vs those who gave up b/c the service is slow). It'd be amazing to have a widely-available webserver expose similar control knobs.
Anyhow, kudos to the .NET team!
Seems to me that this is yet another way that YC has pressured traditional VCs into being more founder-friendly.
Edit: I worked at Microsoft, which also uses tons of tiny repos (at least within Azure). I didn't encounter any good cross-repo management tools, though; apart from having a Jira-like ticketing system built in, Azure DevOps seemed quite a bit worse than GitHub.
Medium-term, the more closely aligned with the rest of Microsoft’s technology they can become, the better - not many MSFT folks understand the ins and outs of sharded Percona, but many of them do understand SQL Server and .NET.
It's hard to get brilliant jerks to accept new ideas. It's hard to motivate your engineers to invest deeply in toxic, unfriendly ecosystems. Of course, extreme technical success can outweigh many of these factors, but that's the exception and not the rule.
Does fx.As[0] help match structs to interfaces? It was added long after my time, but seems to target this problem.
The documentation for Fx, along with nearly all the applications I saw internally, use the dependency injection container only in main - once the application starts successfully, there's no more interaction with the container. For Uber at the time, this struck a useful balance between safety and the difficulty of distributing yet another versioned code gen tool to thousands of repositories.
At the time we wrote Fx, Uber had ~1500 engineers writing Go. The company had 25 million lines of Go, spread across more than a thousand microservices and an unknown number of shared libraries (likely hundreds, perhaps as many as a thousand). Nearly every project was in a separate git repository, with effectively no tools to make large cross-repository refactorings. As an engineering organization, we struggled to make relatively simple cross-cutting changes.
For example, we spent years rolling out distributed tracing. The actual change required was simple: upgrade all your dependencies to something recent-ish, add the tracing library, construct a tracer in main (or something main-adjacent), use the tracer to construct an RPC interceptor, and add the interceptor to your API server. All in, we're talking about ~20 lines of code and a dependency upgrade. It took multiple TPMs, spreadsheets, quarterly planning, and several high-level edicts to get this mostly done.
Why were changes so painful? At root, because nobody cared much about most of the Go repositories. From the perspective of the teams who nominally owned the code (often after several reorganizations over the years), the code worked fine and solved the business purpose - why invest time in changing anything? On the ground, changes were painful. Go's simplicity makes semantic versioning _very_ restrictive, so trying to pull in a year's worth of dependency updates often produced a variety of breakages. (Keep in mind that many of these libraries were used only in a handful of projects and weren't particularly carefully designed or maintained.) Taking on all this pain to change 20 lines of code in main was a difficult sell.
Fx codified some basic back-compat best practices (if you're paranoid) - mostly param and result structs, so constructors have more flexibility to add inputs and outputs. Fx also made most of these problems a negotiation directly between library authors, leaving the microservice team out of the picture: the "standard Uber stuff" package provides a distributed tracer, and the "RPC stuff" package takes an _optional_ tracer and installs the appropriate interceptor. No changes to main or application logic necessary, just a dependency update (which is hopefully safer, since more libraries are forced to follow better semver practices). The reflection-based wiring came with lots of magic and downsides, but the tradeoff was worth it across the engineering organization - it made us _overall_ more able to change our own systems.
Bluntly, IMO Fx made individual codebases less understandable (especially codebases carefully maintained by engineers who like Go). It made the whole company's code more maintainable. The bulk of the Go engineers at the company agreed (the developer experience org tracked NPS, which went from double-digit negative to +40ish).
In the years since Fx, I left Uber and the company has moved most of their Go to a monorepo. I'm not sure what the current cost/benefit tradeoff of this approach is.
Either way, I'd be shocked if this weren't designed top-to-bottom to have excellent performance on Windows.
[0]: https://docs.microsoft.com/en-us/dotnet/api/system.net.socke...
Totally agree. Seems to me that the main benefit is that it's easy to extend in .NET, vs. C/C++/Lua/WASM.
I don't see a way to choose the benchmarking platform in the PowerBI dashboard, so I assume all these numbers were collected on Windows. In that case, it doesn't surprise me that YARP is faster: Envoy uses libevent for cross-platform I/O, but libevent is relatively slow on Windows. It exposes Windows-native I/O completion ports as a BSD-style socket API, and the implementation is both inherently somewhat slow and pushes buffering concerns into the Envoy code. Kestrel, YARP's underlying webserver, uses libuv instead. libuv is more actively developed (because of Node.js) and takes the opposite approach, exposing an IOCP-like API even on POSIX systems. Basically, YARP's I/O model is much closer to Windows' native model. This Envoy issue[0] is really informative if you're interested in the details.
More broadly, the .NET team that builds the Kestrel webserver and contributes to YARP and gRPC is full of performance heavyweights. I'd start by assuming that the benchmarks for their projects are thoughtfully designed. Everyone makes mistakes, but start by assuming that James Newton-King, David Fowler, and their peers are brilliant engineers leading a talented team - because they are.
I say all this as someone who doesn't particularly love Windows as a development environment, .NET as a language, or Microsoft as a company (quit after just 6 months). Credit where credit's due.
[0]: https://github.com/envoyproxy/envoy/issues/4952#issuecomment...
Edit: moogly points out that my info on the Kestrel implementation is out of date. Mea culpa!
Despite webmail's many flaws, I haven't found an MUA that's as good for me as the Gmail web UI.
- Most MUAs choke on my archive. Simple searches take seconds.
- Every modern email service has a tag-like construct. Representing tags in IMAP duplicates messages.
- Decades later, IMAP sync is still slow, unreliable, and annoying to debug.
- A surprising number of MUAs ignore account boundaries when integrating with my contacts.
- Depending on my job, I go through years-long periods with lots of calendar-related mail. Tight integration between Google Calendar and my mail client is very convenient.
- I use snoozes and (especially) delayed sends often.
- I use all these features on my phone, too.
I haven't found an email client that solves all these problems, on both my laptop and my phone, that's better than Gmail web UI + Gmail iOS app. With two exceptions, all the clients I've tried are slower and less reliable, and they often have fewer features important to me. These days, they're often a less-polished, poorly-tested website in an Electron container. I'm happy to pay for GSuite (or whatever they're calling it these days), because I get ad-free clients that work for me and excellent uptime/deliverability.
* The only genuinely good clients I've found are Mailbox (never shipped a stable desktop client, but had useful innovations) and Superhuman. Mailbox was discontinued post-acquisition, and Gmail has implemented their best features. Superhuman is fine, but not worth paying for (though I'd perhaps feel differently if I spent hours each day in email).
[0]: https://github.com/golang/go/wiki/SliceTricks#additional-tri...
I've found nearly every post clear and thought-provoking. Most have also generated good discussion here: https://news.ycombinator.com/from?site=danluu.com
If you're a .NET shop, it seems very useful to have a reasonably fast toolkit for deploying specialized proxies. It's still early days, but I'd expect this to become as fast as anything written in Go or on the JVM, which seems sufficient for many use cases.
IMO, YC catalyzed a lot of that change. I don't always agree with every partner's latest think piece, but I appreciate the changes they've brought to startup funding.
From my perspective, though, the original article is talking about everyday engineers (the "technical team"). If you're an early, senior engineer at a startup, you might be offered 1% of the company. Over the early years of the company, you're often critical to the success and velocity of product development - usually just as critical as the technical cofounder. After four years, your company is acquired for $1B! Your 1% has been diluted to 0.75%, so you walk away with $7.5M. After long-term capital gains, you pocket a bit more than $6M. Objectively, you're now very wealthy - if you're still relatively young, this is enough money to buy a nice house, fund your retirement, and do some angel investing, even in the most expensive cities in the US.
Meanwhile, the technical cofounder walks away from the same story with $200M tax-free. That's dynastic wealth. Does the founder's additional year of work justify this difference? What about the partner from your primary VC, who's likely to personally take home tens of millions despite doing nothing more than investing other people's money and sitting in the occasional meeting?
This is the rosiest scenario - the engineer, founder, and VC all earn life-altering amounts of money. Things get much worse if you're a less senior engineer, you're hired after the first large round of funding, or (especially) the company's exit isn't as good. In all those situations, the founder and VC still get a life-altering windfall, while the senior engineer might barely get enough to retroactively bring their salary up to market rate.
One thing to note is that it's easy to exceed your email provider's quotas with a system like this. For example, Gmail allows a maximum of 3600 emails per hour [1] before incoming mail bounces.
I generally use gopatch instead. It uses the same mechanism under the hood, but the patch files are much less effort to write.