g9yuayon

When I reflect what Netflix did back in 2010ish on AWS:

* The declarative infra is EC2/ASG configurations plus Jenkins configurations * Client-side load balancing * ASG for autoscaling and recovery * Amazing observability with a home-grown monitoring system by 4 amazing engineers

Most of all, each of the above item was built and run by one or two people, except the observability stack with four. Oh, standing up a new region was truly a non-event. It just happened and as a member of the cloud platform team I couldn't even recall what I did for the project. It's not that Netflix's infra was better or worse than using k8s. I'm just amazed how happy I have been with an infra built more than 10 years ago, and how simple it was for end users. In that regard, I often question myself what I have missed in the whole movement of k8s platform engineering, other than people do need a robust solution to orchestrate containers.

I don't know if it's just me or what, but even the vanilla version of FireFox feels slow in responding my mouse clicks. Occasionally FireFox stopped being responsive but didn't freeze. I'd have to kill the FireFox process. I ran into such bug often when using Netscape and the bug was never fixed, so it was quite surprising to me that I got to see exactly the same scenario on FireFox.

The whole article reads like what I don't like is immoral and what is immoral justifies my action (to a certain degree, at least). So, I really don't understand: why do techies still even go to the city? Just leave it alone and come back to the south bay.

I'm not sure why people find studying fundamentals in school useless. And I don't think ChatGPT is the answer to in-depth learning. My experience is that it is the fundamentals that are the most valuable that a college can give to its students. A smart person can learn anything on her own, but for an ordinary student like me, what school teaches is invaluable as it is so hard to learn such fundamentals on my own. A few examples in my line of work:

  - Systems. Yes, you can read papers and case studies and what not. But it will not be easy for one to even tell which part of a paper is the essence, which paragraph needs deep dive, or which claim needs close examination. In a seminar, a professor will work with students to critique papers, to cross examine multiple systems on related ideas, to deeply understand the theoretical bounds and the practical implications, and etc. That kind of experience is just not easily available in other places. Besides, it's just miles easier for someone to tech Lynch's book on distributed algorithms, for instance, with properly designed hand-outs and homework than reading the tomb by oneself. Similarly, it's not that easy to grasp the idea of program analysis if one wants to get in the trench of writing a compiler backend. I got seriously confused in an introduction course on program analysis for all the concepts about lattices, partial orders, abstract interpretations and etc. It's hard to imagine that I'd have the same access or even energy to study such stuff out of school. 

  - High-dimensional stats and probabilities. Again, I'm sure a brilliant student can teach herself, but man, even finding the right accessible material can be hard, let alone digest the fundamental ideas and concepts in such readings without the help of my professors and classmates. 

  - Math, all kinds of math. I'm not sure about you, but math matters in software development. Understanding temporal logic and mathematical logic in general makes doing formal verification much easier. Understanding probability and queuing theory enables me to test and diagnose my systems at a whole new level. Understanding combinatorics makes it really easy to learn data structures and algorithms rigorously. Understanding formal reasoning in general makes it easy to follow the books and papers on distributed algorithms. Linear algebra and numerical optimization and calculus are also important but hard to learn by oneself if a person wants to tap into ML sys. 

The bottom line is, fundamentals expand one's conceptual depth and breadth, as well as the ability to abstract and to dive deep, which in turn gives a person more choices. I didn't start as a system engineer, nor did I know that I would work on internals of ML algorithms. But when opportunities called, I could jump on it. And in the meantime, I regretted that I didn't learn more fundamentals when I could, which made it hard for me to dive to the desired levels.

Besides, even the fun of studying STEM topics is hard to get outside of school. I'm sure one can read about physics and chemistry and what not, but man, having labs and professors who can give you guidance... That makes a whole world of difference.

> how do I give more virtualized compute to people without hiring more expert server and networking admins

I put these into the bucket of productivity and low cost. And this issue arises when we need to scale our business, in particular our computation.

It'll be dull if your team has more people than real work. It'll be dull if your team can't articulate what the problem it solves or why it matters. It'll be dull if you are solving a problem that has better solution somewhere else. It's not about company size, but about culture and project. It was not dull to build the first version of S3 or EC2 or DyanmoDB. It was not dull to come up with Chrome or JAX or TPU. It was not dull to drive the project of Vision Pro or iPhone or Apple Watch. It was not dull to build Github Copilot.

It's just that statistically speaking one has fewer chance per 1000 employee to work on something fun without being dragged by company culture in a huge company.

IBM Mainframe was an answer when computers couldn't vertical scale fast enough. Cloud is the answer when our systems couldn't horizontally scale fast and reliably enough. So, the real question is: can we really scale our need of computation again on a single computer or on a set of computers with low enough cost to render Cloud unnecessary?

I always feel amazed that Euler wrote faster than people could publish or understand his work, even after he became completely blind.

I don't think it will be effective. The root cause is not PE, but human nature: greed, complacency, myopia, and etc. We can outlaw PE, but Boeing can simply invent another way to maximize their profit at the cost of product quality. It looks to me the only cure is to encourage competitors, so that long before Boeing becomes this rotting organization, their talented employees can join another competitor to beat the shit of Boeing. Unfortunately there's so much regulatory capture and intertwined interests between Boeing and governments that I don't think we will see any meaningful competition for years to come.

Ken Thompson in the most recent Sharp Tech podcast also talked about the management style of Boeing after their merger with McDonnell Douglas. They mentioned that when Boeing's HQ was still in Chicago, their CEO boasted that the HQ was thousands miles away from Seattle so that the management didn't have to get involved in day-to-day operations. This is just staggering how arrogant Boeing's the executives could be. Contrasting this with how old Boeing worked, how old Lockheed worked, how Elon Musk worked. No wonder Boeing has been in decline. It's just sad that so many talented engineers are stuck in this monopoly, when they could've leave the company and build something groundbreaking. For that matter, I feel extremely lucky that the IT industry have had a bustling culture of startups, so corrupted companies can die and new companies can innovate. I also feel extremely sad that how aviation industry has become complacent and ossified. I also worry that the US's industries are going the British way, as how British government forced their auto industry to consolidate many years ago, ending up destroying their the entire auto industry.

I'd like to add another one: I don't need a separate NodeJS (or whatever engine) service to build my service dashboard. Before NodeJS got popular, backend engineers like me simply put web assets into a folder in my web app, so my service will have an admin page or a dashboard for per-node administrations. For some reason, such practice has become a taboo. My engineers have been insisting that they set up a separate NodeJS service just to build even a simple admin page. But I fail to see why. The reasons I got are usually these three: 1) a NodeJS service gives us optimized performance, though techniques like server-side rendering; 2) a separate service is easier to scale; 3) a separate service offers separation of concerns. However, 1) and 2) are premature optimization to me. All I need is standardized per-node admin page for my service. The QPS is probably one per day by a human. Why would I care about SSR or scalability at all? And 3) is quite hand-wavy. On the other hand, the overhead of managing a separate service as well as the dependencies brought by NodeJS' ecosystem seems high.

So, what's wrong with the old way of having embedded web assets in a service for building simple admin pages?

A friend of mine works in Boeing as a data scientist. His team has 10 people. Two of them can write code for analytics and models. The other 8 "manage projects", whatever that means. They spend their days creating processes, managing tickets, enforcing specific formats of tickets and stories and what not. Yet, none of the eight knew how to write product specs nor could be bothered with basic things like understanding how git works.

I have a hard time imagining how Boeing could survive in the long run with this level of bureaucracy.

Edit: Saw the summary of the book Flying Blind: "A fast-paced look at the corporate dysfunction--the ruthless cost-cutting, toxic workplaces, and cutthroat management--that contributed to one of the worst tragedies in modern aviation". One has to ask: where did the cost cutting go? What's cutting throat? It looks to me that the management of Boeing is grossly incompetent.

A big concern I have with Tesla is that I have no alternative service providers. When Tesla charges one $2000 to fix a broken gear, the person would have no choice to but to accept it.

This reminds me of Dave Sobel's book Longitude, which tells a fascinating story on how John Harrison created the first reliable marine clock. Such stories humble me and make me deeply appreciate the ingenuity of mankind to conquer the seemingly impossible challenges to build the civilization we enjoy today.

> For me it's not the bad plot decisions, it's the tone. Every movie has the same "feel" to it. It's a milder version of the "algorithm optimized" tone of eighty percent of YouTube videos. Almost like every movie is being made by the same person.

This too!

Thanks, it should be duel. Updated accordingly. I kinda think that quantitive change in energy output will lead to qualitative change in the form of fighting. In addition, it's hard for me to relate to fist fighting among superheroes. If I were to enjoy fist fight, Legend of Fist by Jet Li is a much better alternative.

Me too. It's just that I don't want to escape into the world of the villagers in Wakanda, where owners of stealth jet and energy shield fought on rhinos with spears, and a handful of people lived like royalties but I would live like a peasant.

> 2023 had a lot of big-budget bombs. Ant-Man, Indiana Jones, Mission Impossible, Transformers, The Flash, Fast X, Blue Beetle, Elemental, Shazam!, and the Marvels all underperformed somewhere on the scale from "disappointment" to "disaster".

And rightly so. The so-called special effects in those superhero movies are really getting exhausting. What's the point of having superpower yet battling like a peasant in medieval times? What's the point of having super intelligence yet making so many illogical decisions? What's the point of having the best technologies on earth yet having your tribe living in dirt and fighting with rhinos and using duels to elect your leaders? I can even tolerate such plot holes if the movies have good stories or interesting characters, but do they?

I remember in the hay days of XML, someone made an article that discussed why XML is really just a language of S-expressions. This article seems argue something similar.

P.S., S-expression or not, XML-based DSLs are horrible. I don't know how many people have PTSDs using ANT or a slew of commercial "user-friendly" DSLs for test automation, process automation, and etc. It turns out the companies just didn't know how to design small programming languages and certainly lacked of compiler-writing skills.

A trajectory topic, I find Quora's quality deteriorate over the years. What's bizarre is that the Chinese competitor of Quora, Zhihu, is way better than Quora. Case in point, search "estimate parameter size of transformer model" or a simpler version "parameter size transformer" yield nothing relevant on Quora, yet on Zhihu, we got something like this: https://zhuanlan.zhihu.com/p/624740065. One does not have to know Chinese to see the difference in quality and information density.

I would assume so too, except that initially De Beers was arguing that natural diamonds were more pure than the lab-made. When lab-made beat every criteria, 3C and whatnot, created by the diamond cartel, the cartel started to argue for impurity.

> I, as a European, cannot fathom how anyone would spend a significant amount of money on an engagement ring

It's all thanks to more than 100 years of marketing campaign and tight control of supply of diamond by De Beers. People did not buy diamond rings before De Beers' "A diamond is forever", and the supply of natural diamond is practically unlimited. I have to give it to the ingenuity of De Beers - they got Soviet Union to agreed to a supply deal after Soviet Union found huge diamond mine that could have supplied the market with dirt cheap diamonds for hundreds of years! So, I'm very happy that Chinese entrepreneurs do not give a shit about De Beers and managed to figure out how to mass produce lab-made diamonds cheaply (yes, the process was invented and greatly improved by the west, but it was China, specifically the manufacturers in Henan Province, that didn't succumb to De Beers' heinous control, and gave De Beers a huge middle finger).

It's funny that the traditional diamond industry started to argue that these lab-grown diamonds were too pure and lacked the impurities that natural diamonds have. And what is the response of the Chinese manufacturers? They laughed, as adding impurity to a lab-made diamond can be easy and precise.

Personality, I hate the kind of consumerism and irrationality driven by De Beers. I'm very happy to see De Beers destroyed.

> College will not prepare you for the job

My personal experience says that this is a misconception: https://www.reddit.com/r/programming/comments/267s49/comment...

My college taught me essentials that carry me until this day. Actually, I didn't even know how to use a computer before I got into college, yet I was very well equipped when I graduated. What I was taught was harder and deeper than what I worked on when I started in IBM WebSphere, so getting good at my job, at least to a certain degree, was no big deal. Day-to-day tasks are really not that hard. What's really hard is solving unseen problems, for which the college education becomes really helpful.

>> The college will prepare you for some basics, but what most of the colleges teach is so far away from day-to-day jobs

I depends on what "basics" means. I'd argue that what we need in a job can be easily learned on the job, like a programming language, domain knowledge, CSS this or framework that. What's hard to learn systematically and intuitively (still possible, of course) are fundamentals: matrix calculus for you to understand the basics of deep learning; stochastic process to truly understand diffusion models, OS fundamentals + CPU architecture + parallel programming techniques to truly optimize a complex program, deep understanding of distributed systems to design modern stateful systems, deep understanding of mathematical statistics to come up with new ML models, programming paradiagms so you can learn any programming language easily, end-to-end understanding of how to write a compiler so that you can write a descent compiler or interpreter when you really need to design a DSL instead of thinking YAML + Jinga is the best thing in the world. The list can go on and on, and such fundamentals are really hard to learn systematically on the job.

Case in point, a colleague of mine used to mention how he learned distributed system in his college: his professor would ask them to read a paper before each class, and then the professor would ask them deeper and deeper questions, give them historical context, and challenge them with all kinds of corner cases. How many people could get this level of drilling outside of college?

Another example, in the software engineering classes that I took, we learned a lot of things about formal verification: temporal logics, infinite automata, abstract interpretation, model checkers based on OBDD and on SAT, and etc. So now learning TLA+ for my work is really a delight. I couldn't imagine I would be able to learn all the maths behind formal verification on the job, as I wouldn't have access to professors and TAs for questions, there would be no one to distill a 500-page book into its essences, there would be no seminars for us to discuss papers, there would be no well-designed projects that came with professors' feedbacks, there would be no carefully chosen problem sets that were possible to solve yet thoroughly challenged me...

> the 1 billion row benchmarks are run on a single, uncompressed 50 GB CSV file. 50 GB should be stored in multiple files.

For a generic OLAP db, maybe. In this case, though, a single file fits one of DuckDB's use cases: analytics on the data consumed by or produced by a data scientist. In such scenario, it's not uncommon for a multi-GB input or for dumping GBs of a dataframe into a single CSV file.

I thought the post is #1 on HN because people here are genuinely interested in the topic and would like to at least examine the paper and discuss the merits of it.

Yeah, within AWS. I'm not sure about other parts of Amazon

> AWS support is so nice, you can't believe it.

This reminds me of the fond days of having weekly customers calls. We develop AWS services, and we answer our customer-support calls directly. No middle man. Just techies to techies. And we made promises to customers on the fly, and customers sometimes project managed us.

> At Amazon, practically every service is built on top of AWS - a proper vote of confidence!

Not only this, but practically most, if not all, of the AWS services use DynamoDB, including use cases that are usually not for databases, such as multi-tenant job queues (just search "Database as a Queue" to get the sentiment). In fact, it is really really hard to use any relational DB in AWS. I mean, a team would have to go through a CEO approval to get exceptions, which says a lot about the robustness of DDB.

I owe my thanks to this site too. When my team was designing our hexagonal system for geographical analysis back in Uber, I referred to https://www.redblobgames.com/grids/hexagons/ a lot.

My experience back in Netflix too. Elasticsearch (we didn't use the L or K) plus query engine on S3 with a catalog was more versatile and way cheaper than Splunk. Nowadays we get a slew of performant OLAP storages that can be used for log analysis as well, which further render Splunk unnecessary.