Hacker News Reader

Comment by rdw | Hacker News Reader

The hypothesis in the article could be correct (that industry is not adopting new academic innovations because they fail in the real world). Based on my experience in this industry, though, it could just be that there isn't a super strong connection between academia and the people implementing these kinds of systems. I've had many conversations with my academically-minded friends where they're astonished that we haven't jumped on some latest innovation, and I have to let them down by saying that the problem that paper was addressing is super far down our list of fires to put out. Maybe there are places where teams of top-tier engineers are free to spend 6 months every year rewriting critical core systems use un-battle-scarred new algorithms that might have 20% performance improvements, but most places I've worked would achieve the same result for far less money by spending 20% more on hardware.

View on HN

Hardware vs software is a big one. You have to hit a certain scale ("web scale") before improving software is cheaper than buying more hardware. Sometimes there are low-hanging fruits but I don't think "building a new distributed consensus system" is usually one of them

It seems these things usually get implemented when Google, Microsoft, Twitter, Facebook (big web company x) hit the problem and they each develop their own optimized solution before slowly open sourcing and products converge and mature to the point it's practical for other companies to adopt (I think container orchestration is a good example where Google, Facebook, Apple(?) each had container solutions long before K8s started exploding)

It leaves a bad taste in my mouth when an argument in that area starts using fake math to drive a point home. Top-tier engineers spend 6 months to have 20% improvement that could be achieved by spending 20% more on hardware?

Where IS all this coming from? Why is it acceptable to pull out a few random numbers out of our ass and conclude that it's always best to buy more hardware?

I know Google literally invented HTTP/2 and HTTP/3 to shave a few % off their hardware costs. And they had to not only implement this in Chrome and Google.com but to make it a world-wide standard. So surely they must be the idiots putting top-tier engineers for 6 months (or more) on such optimization efforts, instead of spending more on hardware? I don't know.

Google is an outlier among outliers.

20% more hardware for google isn't the same as for your local IT biz rocking a small data center. If google can save 1% on hw cost they can feed an army of devs for 6 months.

I do not buy the economic argument. Academia is aiming at individual excellence whereas the industry is, at large, about employee replaceability. This may be a valid economic argument in itself, but I believe there is another appeal to technical mediocrity in the reassurance it provides to management.

When I was part of academia I was under the impression we were aiming for publishing papers, which is why I left.

I've been to SIGMOD and I've seen plenty of database industry people there, I don't think that the knowledge is not being transferred. I guess it is more about business priorities or choosing the known unknown over unknown unknown for support.

Yes for that kind of improvement I think academia would be better served trying to see if the techniques can be incorporated into an existing well tested implementation.

Reply on news.ycombinator.com