Hacker News Reader

Comment by atonse | Hacker News Reader

The odd thing about having 20 years of experience (while simultaneously being wide-eyed about new tech), is that I now have enough confidence to read interesting posts (like any post on k8s) and not think "I HAVE to be doing this" – and rather think "good to know when I do need it."

Even for the highest scale app I've worked on (which was something like 20 requests per second, not silicon valley insane but more than average), we got by perfectly fine with 3 web servers behind a load balancer, hooked up to a hot-failover RDS instance. And we had 100% uptime in 3 years.

I feel things like Packer (allowing for deterministic construction of your server) and Terraform are a lot more necessary at any scale for generally good hygiene and disaster recovery.

View on HN

I have, at various times in my career, tried to convince others that there is an awful, awful lot of stuff you can get done with a few copies of nginx.

The first “service mesh” I ever did was just nginx as a forward proxy on dev boxes, so we could reroute a few endpoints to new code for debugging purposes. And the first time I ever heard of Consul was in the context of automatically updating nginx upstreams for servers coming and going.

There is someone at work trying to finish up a large raft of work, and if I hadn’t had my wires crossed about a certain feature set being in nginx versus nginx Plus, I probably would have stopped the whole thing and suggested we just use nginx for it.

I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary. And I kind of feel like Hashicorp missed a big opportunity there. Their DNS solution, while interesting, doesn’t compose well with other things, like putting a cache between your web server and the services.

I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.

> I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary.

To be fair, half of the API Gateways and edge router projects out there are basically nginx with a custom consul-like service bolted on.

You can do get around the nginx Plus requirement by using a module like ngx_mruby to customize the backend selection. I haven't measured the latency, so it may not be suitable for your 10ms example.

Here's a post I wrote on that ~4 years ago that uses an in-process cache [1]. It'd be fairly easy to add an endpoint to update it and pull data from Consul. I agree with you, it's a missed opportunity - there are alternatives, but being able to rely on a battletested server like nginx makes a difference.

[1] http://hokstadconsulting.com/nginx/mruby-virtualhosts

As a fan of nginx, I really liked your comment. In sleuthing after reading I came across this:

https://learn.hashicorp.com/consul/integrations/nginx-consul...

It appears that if the consul client has the right permissions it can restart the nginx service after editing the configuration file. It uses the consul templating engine to generate an nginx config file.

I haven't tried it myself but it looks promising.

> if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary

Airbnb's Smartstack works well for this. It's not built in to nginx as a module, but I think it's more composeable this way.

Blog post: https://medium.com/airbnb-engineering/smartstack-service-dis...

The two main components are nerve (health checking of services + writing a "I'm here and healthy" znode into zookeeper, https://github.com/airbnb/nerve) and synapse (subscribes to subtrees in zookeeper, updates nginx/haproxy/whatever configs with backend changes, and gracefully restarts the proxy, https://github.com/airbnb/synapse).

It's fairly pluggable too if you don't want to use haproxy/nginx.

Then you have a dependency on zookeeper when you already have consul... it seems like consul template + nginx or haproxy is the solution hashicorp went with.

I totally agree, especially about being able to serve content out of a cache instead of files. It would simplify some of my configuration especially for static sites that point to a CDN.

I like what Caddy is doing, exposing their entire configuration through a REST interface.

You should check fabio ( https://github.com/fabiolb/fabio ), it is really awesome ).

I 100% agree with you, I've been using Consul for four years now to run 100s of services in 1000s of VMs across datacenters distributed globally and not once I saw the need for anything else...

Maybe I just don't have the scale to find service mesh or kubernetes interesting. Nomad however is something I am willing to give a go for stateless workflows that I would usually provision a VM running a single docker container for.

> I have, at various times in my career, tried to convince others that there is an awful, awful lot of stuff you can get done with a few copies of nginx.

under the load point of view, yes. absolutely. no doubt.

under the speed of action, no way. if your k8s cluster is properly managed, you can let developers do most of the operations work themselves, confined into their namespaces, touching only the kind of resources that you tell them to touch.

I personally would advise against using DNS for service discovery, it wasn't designed for that.

The few milliseconds that you get though, most likely is due to your local machine not having DNS caching configured, this is quite common in Linux. Because of that every connection triggers a request to DNS server. You can install unbound for example to do it. nscd or sssd can also be configured to do some caching.

> I personally would advise against using DNS for service discovery, it wasn't designed for that.

It was designed for that but the SRV record requires protocols and their clients to explicitly support it. You can argue that this an unreasonable design choice but load balancers like HAproxy do support SRV records.

Why is dns not used for service discovery? The internet as a whole uses it for service discovery currently

Internet as a whole uses it to provide human friendly names.

I'm saying it is not good idea to use DNS for service discovery, there's a way of using it correctly, but it requires software to do the name resolution with service discovery in mind, and you're guaranteed that majority of your software doesn't work that way.

Why you shouldn't use DNS? It's because when you communicate over TCP/IP you need an address that's really the only thing you actually need.

If you use DNS for discovery you probably will set low TTL for the records, because you want to update them quickly, this means for every connection you make you will be checking DNS server providing extra load on the DNS server and adding latency when connecting.

On failure of a DNS server, even if you set a large TTL, you will see an immediate failure on your nodes the reason is that's how DNS cache works. Different clients made the DNS request at different time so the records will expire at different times. If you did not configure a local DNS cache on your hosts (most people don't) then you won't even cache the response and every connection request will go to a DNS server, so upon a failure everything is immediately down.

Compare this to have a service that edits (let say an HAProxy) configuration and populates it with IP addresses. If the source that provides the information goes down, you simply won't have updates during the time, but the HAProxy will continue forwarding requests to IPs (if you use IPs instead of hostnames, then you also won't be affected by DNS outages).

Now there are exceptions to this, certain software (mainly load balancers such as pgbouncer (I think HAProxy also added some dynamic name resolution)) use DNS with those limitations in mind. They basically query DNS service on the start to get IP and then periodically query it for changes, if there's a change it is being applied, if the DNS service is down they will keep the old values.

Since they don't throw away the IPs when a record expires, you don't have this kind of issues. Having said that, majority of software will use system resolver the way DNS was designed to work and will have these issues, and if you use DNS for service discovery, you, or someone in your company will use it with such service and you'll have the issues described above.

>Compare this to have a service that edits (let say an HAProxy) configuration and populates it with IP addresses.

Just edit the hosts file? If you have access to machines that run your code and can edit configuration, and also don't want the downsides of resolvers (pull-based instead of push-based updates, TTLs), DNS still seems like a better idea than some new stacks, plus you can push hosts files easily via ssh/ansible/basically any configuration management software

EDIT: The only issue I see with DNS as service discovery is that you can't specify ports. But usually software should use standard ports for their uses and that's never been a problem in my experience.

You can specify ports using SRV resource records.

You could but there's no integration for that that i know of so it'd be a bit of work to get working, which is why i didnt include it

https://www.haproxy.com/documentation/aloha/9-5/traffic-mana...

It's how mdns works with Avahi/Bonjour/Zeroconf

This is interesting! Do you have some material on load testing the DNS servers and seeing their breaking point? I've heard as much from other people but never experienced it in practice even using Consul with 0 TTL everywhere.

Perhaps the network infrastructure team always scaled it correctly behind the scenes but they never once complained about the amount of DNS queries.

DNS is fairly lightweight and if you have one local on premises, it might be less noticeable, especially if latency is not critical (in previous places I worked that was the setup, we still had a local cache on every host and I would encourage doing that, it increases resiliency). If latency is critical, not having a cache adds extra round trip on every connection initiated.

If you have hosts on public cloud and use DNS server that is also shared with others the latency typically might be bigger and on high number of requests you might also start seeing SERVFAIL on large number of requests.

I can't find the forum post anymore, but people who had applications that were opening large number of connections (bad design of the app imo, but still) had huge performance degradation when they moved from c4 to c5 instances. It turned out that this was because of the move from Xen to Nitro (based on KVM).

Side effect of using Xen was that the VM Host was actually caching DNS requests by itself, from which all guests benefited. In the KVM, all DNS requests were going directly to the DNS server.

> I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.

Don't resolve DNS inline rather on every DNS update, resolve it and insert new IP addresses.

Correct me if I'm wrong, but I believe Consul, lacking a mesh of its own, is leveraging the early 1990's era trick of using round robin DNS to split load over available servers.

Caching those values for very long subverts the point of the feature.

By way of correction: Consul does not simply "round robin" DNS requests unless you configure it in a particularly naive manner.

Prepared queries [1] and network tomography (which comes from the Serf underpinnings of the non-server agents) [2] allow for a much wider range of topologies just using DNS without requiring proxies (assuming well behaved client software, which is not a given by any stretch).

Furthermore, Consul _does_ have a mesh as around 2 years ago - [3].

You are correct though that long caches subvert much of the benefit.

[1]: https://www.consul.io/api/query.html

[2]: https://www.consul.io/docs/internals/coordinates.html

[3]: https://www.consul.io/docs/connect/index.html

Not really - resolve all backend servers to IPs and list all of them as the nginx backends. When a backend server is removed, update nginx backends.

Round-robin balancing using DNS towards a small cluster is silly - you know when any new instance is added to the pool or removed from a pool, so why not push that load balancing onto the load balancer which in your case is nginx?

You're talking about layering the thing on top of Consul that I already identified in my top level comment.

Consul itself advertises DNS resolution for service discovery.

Maybe I was not clear.

Whatever is the technology that you use to register the active backends in the DNS, rather than doing name => ip address lookup per request, you can resolve all those names => ip address maps upon the service being brought up/taken down and push the resolved map as a set of backends into nginx config, thus removing the need to query DNS per request.

Kubernetes has complexity for a reason. It's trying to solve complex problems in a standardized and mature manner.

If you don't need those problems solved then it's not going to benefit you a whole lot.

Of course if you are using docker already and are following best practices with containers then converting to Kubernetes really isn't that hard. So if you do end up needing more problems solved then you are willing to tackle on your own then switching over is going to be on the table.

The way I think about it is if you are struggling to deploy and manage the life cycle of your applications... fail overs, rolling updates, and you think you need some sort of session management like supervisord or something like to manage a cloud of processes and you find yourself trying to install and manage applications and services developed by third parties....

Then probably looking at Kubernetes is a good idea. Let K8s be your session manager, etc.

I would qualify that a little more. If you are using docker and deploying to a cloud environment already, then moving to a cloud-managed kubernetes cluster really isn't that hard.

I've seen too many full-time employees eaten up by underestimating what it takes to deploy and maintain a kubernetes cluster. Their time would have been far better spent on other things.

That story is getting a lot easier but historically was awful

What is a "mature manner"?

For example by covering all of the NFRs.

You don't always find open source programs that have dedicated so much effort to security, monitoring, governance etc. And doing so in a very professional and methodical way.

There's always more than one way to do things, and it's good to be aware of the trade-offs that different solutions provide. I've worked with systems like you describe in the past, and in my experience you always end up needing more complexity than you might think. First you need to learn Packer, or Terraform, or Salt, or Ansible - how do you pick one? How do you track changes to server configurations and manage server access? How do you do a rolling deploy of a new version - custom SSH scripts, or Fabric/Capistrano, or...? What about rolling back, or doing canary deployments, or...? How do you ensure that dev and CI environments are similar to production so that you don't run into errors from missing/incompatible C dependencies when you deploy? And so on.

K8s for us provides a nice, well-documented abstraction over these problems. For sure, there was definitely a learning curve and non-trivial setup time. Could we have done everything without it? Perhaps. But it has had its benefits - for example, being able to spin up new isolated testing environments within a few minutes with just a few lines of code.

> First you need to learn Packer, or Terraform, or Salt, or Ansible - how do you pick one?

You don't. These are complementary tools.

Packer builds images. Salt, Ansible, Puppet or Chef _could_ be used as part of this process, but so can shell scripts (and given the immutability of images in modern workflows, they are the best option).

Terraform can be used to deploy images as virtual machines, along with the supporting resources in the target deployment environment.

> Salt, Ansible, Puppet or Chef _could_ be used as part of this process, but so can shell scripts

I don't see the point of your post, and frankly sounds like nitpicking.

Ansible is a tool designed to execute scripts remotely through ssh on a collection of servers, and makes the job of writing those scripts trivially easy by a) offering a DSL to write those scripts as a workflow of idempotent operations, and b) offer a myriad of predefined tasks that you can simply add to your scripts and reuse.

Sure, you can write shell scripts to do the same thing. But that's a far lower level solution to a common problem, and one that is far hardsr and requires far more man*hours to implement and maintain.

With Ansible you only need to write a list of servers, ensure you can ssh into them, and write a high-level description of your workflow as idempotent tasks. It takes you literally a couple of minutes to pull this off. How much time would you take to do the same with your shell scripts?

As he mentioned, immutable images make those type of tools largely moot.

Yes. Three web servers and a load balancer is fine. Three web servers and a load balancer, repeated 1,000 times across the enterprise in idiosyncratic ways and from scratch each time, is less fine. That’s where Kubernetes-shaped solutions (like Mesos that came before it) become appropriate.

You can get a lot done with a sailboat. For certain kinds of problems you might genuinely need an aircraft carrier. But then you’d better have a navy. Don’t just wander onto the bridge and start pressing buttons.

You're right.

However, a lot of new (or just bad) devs miss the whole Keep It Simple Stupid concept and think that they NEED Kubernetes-shaped solutions in order to "do it the right way".

Many times three web servers and a load balancer are exactly what you need.

And then what about when you need to add monitoring, logging, APM, tracing, application metrics etc.

Suddenly you have gone from 3 instances to 20.

Adding an entirely new instance is not the only way to accomplish each of those things. A lot of those things can be treated just like applications. You don't need a whole new computer to run Outlook, another computer to run Sublime Text, another computer to run Chrome, etc etc.

All of that is irrelevant to my main point though. It's never one size fits all and then all your problems are solved.

You are far better off actually assessing your needs and picking the right solution instead of relying on solutions that "worked for bigger companies so they'll work for me" without really giving it a lot of thought if you need to go that far.

> Adding an entirely new instance is not the only way to accomplish each of those things. A lot of those things can be treated just like applications.

That's what containers are. Containers are applications, packaged to be easily deployable and ran as contained processes. That's it.

Kubernetes is just a tool to run containers in a cluster of COTS hardware/VMs.

I've said it once and will say it again: the testament of Kubernetes is simplify so much the problem of deploying and managing applications in clusters of heterogeneous hardware communicating through software-defined networks thay it enable clueless people to form mental models of how the system operates that are so simple that they actually believe the problem is trivial to solve.

Again .... that might be the best solution for your company, or it might be introducing complexity where it's not actually needed.

It all depends on the situation and needs of whatever problem you are trying to solve.

Almost all of those aren't single binaries like Chrome.

They often have their own databases, search engines, services etc to deploy along with it. And necessitate multiple instances for scalability and redundancy.

They can be shared across a company. Office one team handels those all other teams can just use them.

I personally went from 4 servers to 5, self hosting ELK and it works great.

Aren't those rather trivial to do if you didn't make the mistake of choosing a microservice architecture?

I get the point but I’ve also seen three web servers and a load balancer go terribly wrong at a number of places as well. L8s provides a lot of portability that you would need a disciplined and experienced engineer to match with raw deployments.

> Many times three web servers and a load balancer are exactly what you need.

May be, just may be, they want k8s not to create value but to develop/enrich resumes - in order to signal that they are smart and can do complex stuff.

I think anyone considering these wild setups should read about how stackowerflow is hosted on a couple of IIS servers. It’s a sobering reminder that you often don’t need the new cool.

Joel, if any, has always been super pragmatic and very realistic.

Not to misunderstand. For FogBugz they wrote a compiler/transpiler for Asp and PHP because the product had to run on customers servers - because "clients should not leave their private data at another company".

Google it, great read.

Tried to google for it, couldn't find anything - could you provide a link, please?

For the details on that transpiler: https://www.joelonsoftware.com/2005/03/30/the-road-to-fogbug...

I would recommend going through all of Joel Spolsky’s posts between 2000 and 2010, there are plenty of absolute diamonds. Part of why StackOverflow was so successful was because Joel had built a big audience of geeks and entrepreneurs with his excellent blog posts (he was the Excel PM during the creation of VBA and had plenty of accrued wisdom to share), so they adopted SO almost instantaneously when him and Jeff Atwood built it.

"“don’t you mean translator?“

Let me explain.

In computer science jargon a translator IS a compiler. It’s exactly the same thing. Those are synonyms."

Every time someone says "transpiler", god kills a kitten. Please, think of the kittens.

> I think anyone considering these wild setups should read about how stackowerflow is hosted on a couple of IIS servers.

Apparently in 2019 stack overflow was hosted in at least 25 servers, including 4 servers dedicated to run haproxy.

https://meta.stackexchange.com/questions/10369/which-tools-a...

Nothing much happens if SO goes down, they are not doing a ton of business transactions.

That's right. As opposed to stock exchange software, which runs on complex micro-services cloud k8s thing-as-a-thing virtualized rotozooming engines. Wait, no, it's literally just good-old n-tier architecture with one big server process and some database backend.

Pet food delivery startups use k8s to manage their MEAN stack. Meanwhile grown-ups still have "monoliths" connected to something like Oracle, DB2 or MS SQL server, because that's obviously the most reliable setup.

The cloud/k8s stuff is an ad-hoc wannabe mainframe built on shaky foundations.

> Meanwhile grown-ups still have "monoliths" connected to something like Oracle, DB2 or MS SQL server, because that's obviously the most reliable setup.

More often than not they just crystalized their 90s knowledge and just pretended there aren't better tools for the job because it would take some work to adopt them and no one notices it in their work anyway.

The "Oracle" keyword is a telltale sign.

Nothing much? The world stops producing software:)

He's got a point though. Ironically the world's most highly scaled software are often fairly unimportant. Think Facebook - people would get annoyed if it went down for a day, but it'd soon be forgotten. Your banks are built with less scalable software but is much more critical.

A typical PHP application that does a bit of database updating per request, gets some new data from the DB and templates it should handle 20 requests per second on a single $20/month VM. And in my experience from the last years, VMs have uptime >99.99% these days.

What made you settle on a multi-machine setup instead? Was it to reach higher uptime or were you processing very heavy computations per request?

Higher uptime and more computation, although this was mostly C# so very efficiently run code. It was an e-commerce site doing +100MM a year.

There was little to no room for error. I once introduced a bug in a commit that, in less than an hour, cost us $40,000. So it wasn't about performance.

Also this was 9 years ago. So adjust for performance characteristics from back then.

100 000 000$ a year with only 20 requests a second? That's some crazy revenue per request, 100 000 000 / (365 * 24 * 60 * 60) = 3.17$ per request!

What were you selling?

Insurance policies :)

Good point, actually the 100MM may have included brick and mortar.

Forgot to multiply by 20, it's actually around $0.15 per request. Still high but certainly not as crazy.

I worked on a site 20 years ago that got even better revenue per request (though with fewer requests).

It did analytics on bond deals. Cost $1k/month for an account. Minimum 3 accounts. Median logins, ~1/month/account.

On the other hand people would login because they were about to trade $10-100 million of bonds. So knowing what the price should be really, really mattered.

Wall St can be a funny place.

That's quite interesting! That's why I asked, I wasn't exactly doubting the figure, just curious about the market that allowed that kind of customer targeting.

It should be (365 * 24 * 60 * 60 * 20) which brings it to $0.15/req if I did my mental math right. Still a high amount of course.

Oh thanks, I actually did the math right the first time, kept only the result and then when I was about to hit send I thought it was too good to be true, thus I did it again and got 3.15$, which was even crazier, but couldn't find why my math was wrong.

A single server isn't redundant. 3 behind a load balancer, where each is sized to handle 50% of the volume lets you take systems offline for maintenance without incurring downtime.

Heck, Raspberry Pis have more horsepower than the webservers in the cluster I ran around Y2k.

For performance context: nginx on a $5 DO droplet does 600 requests per second on static file.

Serving static files with Elixir/Phoenix has a performance of 300 requests per second.

Python+gunicorn serves about 100 requests per second of JSON from postgres data.

(outside of GP's reply) Generically, life is messy and unpredictable, never put all your eggs in one basket. Your cloud server is sitting on a physical hyp which will need maintenance or go down, or even something in your VM goes wrong or needs maintenance. Using a basic N+1 architecture allows for A to go down and B to keep running while you work on A - whether that's DNS, HTTP or SQL etc.

If your physical hyp dies, how do you redirect your traffic to a different one?

Replace "your" with "the" - the hyp can be run by your provider (Linode, DO, Vulture, AWS, GKE, whoever). Most cloud providers have virtual/shared/managed load balancers to rent time on as well, such that you don't have to maintain N+1 of those (let them do it). You could even use basic round-robin DNS, it's a possible choice just not generally suggested.

For the load balancer solution, a lot more is needed then to just rent a load balancer.

Example: What do you expect to happen when the server with your DB goes down? Just send the next UPDATE/INSERT/DELETE to DBserver2? Which is replicated from the DBserver1? When DBserver1 comes back, how does it know that it now is outdated and has to sync from DBserver2? How does the load balancer know if DBserver1 is synced again and ready to take requests?

Even if you set up all moving parts of your system in a way that handles random machine outtages: Now the load balancer is your single point of failure. What do you do if it goes down?

Respectfully, I am not designing a full architecture here in HN comments and have not presented a full HA solution for you to pick apart. Your leading questions seemed basic, you received basic answers - going down the rabbit hole like this is just out of left field for this comment thread.

Load balancing--and yes that can become a source of failure too.

Two load balancers using either keepalived or bgp anycast.

The odd thing about having 10 years of experience as a consultant is that you know when to write "Kubernetes" into a project proposal, even though everyone agrees that it'll be a sub-optimal solution.

But both you and their tech lead want to be able to write "used Kubernetes" on your CV in the future, plus future-oriented initiatives inside your contact's company tend to get more budget allocated to them. So it's a sound decision for everyone and for the health of the project to just go with whatever tech is fancy enough, but won't get into the way too badly.

Enter Kubernetes, the fashionable Docker upgrade that you won't regret too badly ;)

I worked on a transacted 20-60k messages/s system and am not sure K8S wouldn't be a hindrance there... Imagine writing Kafka using K8S and microservices.

I don't know about "lot more necessary". The images are one part of the equation especially to meet various regulations. There is a ton to running a large scale service especially if you are the service that the people who are posting how wicked smart they are at k8 service runs on. Google found that out yesterday when they said "oh hey people expect support maybe we should charge". That is not new for grown ups.

The cloud existed before k8 and k8's creator has a far less mature cloud than AWS or Azure.

But this thread has convinced me of one thing. It's time to re-cloak and never post again because even though the community is a cut above some others at the end of the day it's still a bunch of marks and if you know the inside it is hard to bite your lip.

The most important video I've ever watched for my career and sanity: https://www.youtube.com/watch?v=bzkRVzciAZg

Who is giving you 100% uptime? All major providers (AWS, GCP, Azure, etc) all have had outages in the past 3 years. And that level of infrastructural failure doesn't care about whether or not you're using k8s.

EKS (AWS Kubernetes) has the control plane across 3 AZs.

It is very rare to have a complete region outage so it is pretty close to 100% uptime.

Not OP but you can achieve higher availability through means like redundancy, availability zones, multi-region deployments, etc.

> Even for the highest scale app I've worked on (which was something like 20 requests per second,

Kubernetes is not for you. 5kQPS times a hundred or more services and Kubernetes fits the bill.

> And we had 100% uptime in 3 years.

Not a single request failed in that time serving at 20 QPS? I'm a little suspicious.

Regardless, if you were handling 10 or 100 times this volume to a single service, you'd want additional systems in place to assure hitless deploys.

> Not a single request failed in that time serving at 20 QPS? I'm a little suspicious.

Things that aren't monitored are also things that don't fail.

Same. I like trying out new things, so I have a feel for what they're good for. I tried setting up Kubernetes for my home services and pretty quickly got to "nope!" As the article says, it surely makes sense at Google's scale. But it has a large cognitive and operational burden. Too large, I'd say, for most one-team projects.

I'm in a similar boat, only my eyes are wide, glazed over, and I'm lost in the word salad...which only seems to be getting worse.

Reply on news.ycombinator.com