274 karma · joined October 22, 2020
"I bought the top result for 'wheel' on eBay and ruined my life trying to fit it to this aircraft engine, this is such a joke!" Shit has just worked for years if you want it to work.
This was more about cache coherency, and it's not an argument, it's a trait shared by every other Linux-over-X implementation (UML, coLinux). It is fundamental to solving a problem users want - seamless integration, i.e. no opaque ext4 blob. Why doubt the reasons given by Microsoft, when they match observations not just for WSL but every other system in the same class?
Example: Linux local filesystem performance mostly derives from the dentry cache. That cache keeps a parsed representation of whatever the filesystem knows appears on disk. The dentry cache is crucial to pretty much any filesystem system call that does not already involve an open file, and IIRC many that also do. Problem is, that same cache in WSL must be subverted because Linux is not the only thing that can mutate the NTFS filesystem - any Windows program could as well. This one fundamentally unfixable problem alone is probably 80% the reason WSL1 IO perf sucked - because the design absolutely required it.
Solutions are rip out a core piece of kernel functionality, and in the process basically taking over ownership and maintenance for ALL code anywhere in the kernel assuming the existence of said cache, engineer something that is somehow better, and support this in perpetuity, including any semantic mismatches that turn up much later that were never designed for
The idea of merged ps output where Windows binaries would show up in the Linux /proc. How would you even begin to implement that without confusing EVERY process management tool ever written that interfaced with /proc? What about /proc/.../environ? On UNIX that has no character set. On Windows it is Unicode.
A trillion problems like this made WSL1 a beautiful nightmare. Glad it was tried and from watching the tickets, that team bled heroically trying to make it all work, but ultimately, I'm also glad it's gone, because the replacement is infinitely easier to engineer, maintain, and use, and that team earns its quarterly bonus much more easily. Everyone wins, except the astronauts, but as experience teaches us they never do.
Your original question was what is the point. These are the points. As for why not Docker, k8s network effects and strategy of its sponsors mean Docker is on a lifeline, everyone knows that.
> K8S on a single node does nothing for you network-wise
- Container IP auto-assignment
- Container security policy
- Container DNS management
- Ingress management ("custom Nginx config")
- "Environment that feels like a large network and doesn't change if moved to a large network"
What part of this is difficult to understand?
Kubernetes centralizes all the traditional technical nonsense related to providing a robust environment to deploy applications to. I want Kubernetes even in a single node scenario because I want Kubernetes-like packaging, deployment and network services for any app I work on, as they are a net simplification of what was previously an ad-hoc world of init scripts, language-specific supervisors, logging, monitoring, etc etc., and rearchitecting an app deployment from scratch simply because its resource requirements increased.
If people want to continue trying to scale it down further, where is the harm in this? There are plenty of legitimate cases where it makes sense. There's no real limit to that work either, it's conceivable with the right implementation improvements (in k8s and the container runtime), it might eventually be possible to reuse the same deployment model even for extremely small devices.
Lambda requests: ((1.5e9 * 4) / 1e6) * .20 = $ 1,200
Lambda CPU (1536 MB): 0.0000025000 * 1.5e9 * 4 = $ 15,000
API Gateway HTTP reqs:
(count): 1.5e9 * 4 = (6 billion)
(first 300m): 300 * 1.0 = $ 300
(next 5700m): 5700 * 0.9 = $ 5,130
LAMBDA MONTHLY TOTAL = $ 21,630
LAMBDA YEARLY TOTAL = $ 259,560
NLB (2x)
(NLB hours 1 month):
2 * 0.0225 * 24 * 30.45 = $ 33
(NCLU hours):
2 * (2280/50) * 0.006 * 24 * 30.45 = $ 399
NLB MONTHLY TOTAL = $ 432
NLB YEARLY TOTAL = $ 5,184
EC2 YEARLY
(if 1 req/vCPU) = $ 253,926
(if 15 reqs/vCPU) = $ 67,704
(if 30 reqs/vCPU) = $ 33,852
That's an 87% reduction in operational costs in the ideal (and not uncommon!) case, and a minimum of a 59% reduction in the case of a web app from hell burning 250 ms CPU per request.
Price-perf ratio between Lambda and EC2 is obscene, even before accounting for Lambda's 100ms billing granularity, per-request fees, provisioned capacity or API Gateway. Assuming one request to a 1 vCPU, 1,792MB worker that lasted all month (impossible, I know), this comes to around $76, compared to (for example) a 1.7GB 1 vCPU m1.small at $32/mo or $17.50/mo partial upfront reserved.
Let's say we have a "50% partial-reserved" autoscaling group that never scales down, this gives us a $24.75/mo blended equivalent VM cost for a single $76 Lambda worker, or around 3x markup, rising to 6x if the ASG did scale down to 50% its size the entire month. That's totally fine if you're running an idle Lambda load where no billing occurs, but we're talking about the BBC, one of the largest sites in the world...
The BBC actually publish some stats for 2020, their peak month was 1.5e9 page views. Counting just the News home page, this translates to what looks like 4 dynamic requests, or 2,280 requests/sec.
Assuming those 4 dynamic requests took 250ms each and were pegging 100% VM CPU, that still only works out to 570 VMs, or $14,107/mo. Let's assume the app is not insane, and on average we expect 30 requests/sec per VM (probably switching out the m1.medium for a larger size taking proportionally increased load), now we're looking at something much more representative of a typical app deployment on EC2, $1,881/mo. on VM time. Multiply by 1.5x to account for a 100% idle backup ASG in another region and we have a final sane figure: $2,821/mo.
As an aside, I don't know anyone using 128mb workers for anything interactive not because of memory requirements, but because CPU timeslice scales with memory. For almost every load I've worked with, we ended up using 1,536mb slices as a good latency/cost tradeoff.
Even in the latter, full text indexing which every platform has had for years now makes it much less likely that the full directory tree will even get walked, and differences even less likely to be noticed. As a side note, every desktop platform's full text search seems to suffer software performance problems that are largely independent of the underlying disk
Even in the full-tree enumeration case, since Spectre/Meltdown mitigations landed, system call overhead is so high now that even with a lightning fast disk, a large chunk of total time taken to walk the directory tree is lost basically twiddling the CPU mode securely. You can definitely still see the difference between SATA and NVMe, but you can also definitely measure the amount of time during the NVMe run that is spent in software -- incrementally faster NVMe will have quickly diminishing returns.
"What about databases!" This was my original interest in SSDs to begin with. It turns out, despite being a data monkey who loves large databases, since 2013 any time I've worked with a giant dataset like this, it is always in the form of large scans (usually from something like a CSV or XML file), where SSDs don't really have a mind-blowing advantage over magnetic (but of course they are still 5-10x faster a seq io, its just that data parsing and processing is typically the bottleneck now).
"I built an average web application and it attracted an average number of users, it more or less worked if you visited it from mobile" would look fucking awful on anyone's resume.
4.39 MB in 50 requests is a showstopper on any mobile network, especially when alternative solutions do not have that problem. It is fair to assume some first time experiences will involve 7.5 seconds or more additional latency on 3G networks, double that for a poor signal areas.
It's still an instant pass
- It needed to show a loading spinner on a 250 Mbit connection
- It hung the browser while 'booting' the page for a solid 3-5 seconds
- It downloaded 4.39 MB in 50 requests
- Opening web inspector in Firefox while reloading the page was sufficient to cause the boot process to hang indefinitely
- Page looks pretty, but at this point it barely matters
We could conclude either that the project's attention to important details is low, producing this experience, in which case what else might we discover once committed, or alternatively we could conclude that it is high, in which case, this is the best possible experience for any Uno app.
Instant pass
Lambda VM time has a massive markup compared to regular compute. It only makes sense where usage does not exceed some threshold the BBC absolutely certainly do.
There are plenty of alternative options, even on AWS, that don't suffer from such huge markup without requiring any additional ops input. The thing that runs in Lambda is practically a container image already. Does it really cost tangible budget to have CI build a full image rather than a ZIP file that already contains a few million lines of third party JS/Python deps?
IMHO this is the epitome of serverless gone wrong.
You perhaps missed the bottom 30% suggesting there may be better options.
Docker's registry protocol treats layers essentially as opaque slabs, as any user will no doubt be aware, where update of a single timestamp or byte of a file could mean the reupload (and download) of a potentially unlimited amount of data. While that architecture has its perks (simplicity, sequential IO patterns suited to the magnetic disks of 2013), there are obvious downsides.
We have plenty of systems just like Docker capable of managing Docker-like file counts without the obscene bandwidth abuse, we just don't call them registries. They have names like Perforce, Subversion and even (some Microsoft variant) of Git. Eliminating the data gravity angle is impossible, but reducing its potency by several orders of magnitude is not only possible but has already been done repeatedly in many popularly deployed systems.
This is not to suggest replacing registry with Git, but perhaps some graph-like manifest format combined with some new "fetch-multiple-objects" protocol verb would be all required to bring the cost of running a registry down to essentially a non-issue.
Don't be mistaken, actually using this for public images is falling for another "data gravity" marketing gimmick. So they attract enough public images that some folk find one or more of their services absolutely must pull from AWS, pushing their corporate NAT address over the 41GB/day limit. Now your "free" hosting has become a reason for that company to start paying ~$45/month to AWS in egress fees just to use your images, and perhaps with a new ticket in the backlog to consider simply moving the cluster to AWS.
"But it'll always be like that regardless of provider", not necessarily. The reason that Docker registry consumes a ridiculous amount of bandwidth is largely an architectural issue, both in the software and the design of the client's network. Perhaps a better direction is addressing this immense waste of networking rather than turning it into another free marketing opportunity for a near-monopolistic cloud
I'm also concerned this doesn't mention much about how accessible the internals are. Can a kid crack it open?
Meanwhile, totally digging the nostalgia factor. Probably won't buy one, but damn I'd love one
Try out Office online, insanely more featureful. Scold anyone mailing Docs share links by saying you can't open it, insist on a file instead