Hacker News Reader

Comment by tananaev | Hacker News Reader

I have a mixed feeling about this.

On one hand I definitely agree that it's good to know lower level technologies and that's something I always ask people in the interviews. I think it's important because I know it.

On the other hand there is no end on how low you can go in the technology stack. Do you need to know how sockets works underneath? Low level network protocols? Do you need to know how hardware works because it probably influenced decisions that were made on the low level of the software stack? I know what socket is, but maybe the only reason I know it is because I'm old and I started coding at the time when raw sockets was the only way to implement network communication. When we have modern libraries and frameworks do we really need low level knowledge?

View on HN

Being able to go all the way down the software stack makes it much, much easier to keep said stack honest. Debugging is often a desperate attempt at establishing ground truth. Without a low-level understanding, you're always a the mercy of your tools, and how they have decided to curate the information they're feeding you. Even the most well-meaning curation can be so frustratingly deceiving as to incite violence, and god help you if something you're interacting with has a smug sense of knowing what's best for you.

There have been a number of cases where I've had to rely on the ability to debug binaries directly at -O3, or to resort to Wireshark to get the dose of reality needed to challenge my [our] flawed mental model. If I hadn't been able to do those things, I'd probably still be there, pondering those defects, if not in body [due to declaring outright failure], at least in spirit.

> you're always a the mercy of your tools

More than tools they are products nowadays :-(

You can make sense of all the back and forth in k8s networking? That needs more than package capturing I suppose. How do you translate a million packages into something useful?

most of k8s networking is actually not part of k8s? They don't really have a integrated overlay network except for kube-proxy. (which is lackluster).

If you are running k8s, you should run a proper overlay network first.

"Running k8s" means also running a CNI of course, otherwise k8s doesn't work.

"When we have modern libraries and frameworks"

I would question the assumption of having "modern" libraries, or at least rephrase it in terms of "performance" and "reliability":

Do we really have performant libraries (do they at least saturate the hardware)?

When it comes to networking, I think the answer is going to be more and more, no, we do not have performant libraries and software is being outpaced by hardware. Hardware is sitting idle thanks to our libraries.

Do we really have reliable libraries?

When it comes to storage, again I think the answer is no. For example, Linux RAID has long suffered from a plethora of faulty logic (pick a random disk sector as the canonical sector, split-brain etc.) and of course Linux RAID is not at the right layer to even have a chance of getting this right. And yet even ZFS lacks algorithms to detect or prevent split-brain. It's possible to detect (with 2 disks) and possible to prevent (with 3 or more disks), by taking quorum from the longest partially ordered set in the topology, but even ZFS (which is still awesome) is not reliable in this regard.

When it comes to networking, even TCP is not that reliable but has huge issues with bufferbloat. For example, a single person uploading a Gmail attachment on a 10mbps line will break the Internet for a second or two for everyone at your office (try it sometime - get everyone running "ping" and then upload an attachment). This is why Apple's background software update uses LEDBAT. Most libraries are not using LEDBAT or similar but are trusting TCP's congestion algorithms to do the right thing.

My personal advice in this cases is: try to be familiar with (at least) the abstraction layers below and above you.

Are you a storage expert? Do not ignore how storage hardware is designed (even on a historical perspective) and frequently think about the system calls it is your responsibility to implement.

Are you developing a sidecar proxy? Be fluent in low level network programming, and be aware of the interface you are offering to the application.

Are you a manager? Focus on your own manager and objectives, but do not forget to empathically look at your team members.

There are specialist subfields of programming nowadays: frontend, mobile app, embedded/microcontroller.

And a frontend sitebuilder doesen't have to know anything about transistors, FinFET, system calls, and only the most curious would know about reflow and painting calls in a browser's rendering engine. Sure, the quest for performance always pushes people toward the metal, but from a website's perspective that's just the browser, and maybe TCP window tuning. (And sometimes someone finds that DNS was slow, but sometimes it was a bad BGP constellation, and there was nothing to do really. Or maybe the load balancer - or the backend - was overloaded because there was mutex contention in the kernel - or in the DB behind the backend, but that's again rather far from the usual sphere of investigation for "Frontend Engineers".)

The imporant point would be to be able to go deeper, if there's a need. Or go higher, more abstract. That's engineering (or software design), recognizing, analyzing, and solving problems.

I can't really blame the people who get by just knowing parts of AWS. They make great money by being a Cloud Consultant. It simply shows the low hanging fruits on the market. And how bad the demand side is when it comes to picking the experts.

> And how bad the demand side is when it comes to picking the experts.

You meant, there is a very strong demand ?

I mean big companies that hire "AWS consultants" are big, slow, dumb, incompetent blobs of IT.

And yes, there's a demand for these specific skills, because big inefficient companies are willing to pay for just this, because their in house people are usually overworked or simply not allowed to do it themselves without an expert on site (or on call, or on skype).

And of course AWS is a big jungle of buzzwords itself, without clear documentation (and that's of course very important for them, they don't want to give away trade secrets, nor it should matter for the guarantees they offer, but their own custom nomenclature and sort of arbitrary abstractions over the standard Xen/KVM/qemu, VLAN, VXLAN, etc stuff makes it very hard to know exactly what's going on), so no surprise that there's demand for this. But as others said, sometimes these consultants have rather narrow skills.

And to add a bit more to this, generalists are few, and always quickly disappear from the market, so the holes/niches are plugged by whoever happens to be around.

Finally, I don't exactly blame these corps, it's just what makes sense nowadays. Need for extremely short turn around, instant risk-elimination, ASAP ROI, all point to "experts" and "consultants". No point in getting someone who understands the full stack top-to-bottom, when the project is in the initial phase and you just need EC2 and a DB on RDS.

Yeah, thanks for clarification. As a technical person I find myself a bit confused as I don't know if I should follow some buzz and try to make money in the short term of go deep and go for the long term. Will it pay off ?

Of course I get more satisfaction in "feeling" a "deep" and "true" technical folk.

> money in the short term [or] go deep and go for the long term

I think these are never contradictory, at least I think software jobs allow for the flexibility to keep working on high-yield things while learning something for depth on the side. (Or vice-versa, if you live off your side job, and do/learn something deeper/longterm.)

Especially with network, virtualization, security, distributed systems (DB, storage), etc. the ideas that are trendy now are spin-offs of older things. (Docker is basically LXC with a sane UI/UX, which is just chroot done better, a'la BSD jails, which is just Solaris zones, which is just IBM mainframe partition shit. Similarly everything on the network, like overlay networking, VXLAN, Geneve, OpenFlow, etc. are just natural layering abstractions and applied trade offs that were considered silly a decade ago, or were simply not needed a decade ago. Now that north-south traffic is dwarfed by east-west traffic in every system [this means that north is the client, south is your "DB", and east-west just means services in your system, and since people nowadays use a lot more complicated systems, the inter-component = intra-system communication is huge, compared to the few MBs sent to the client].)

Yeah, same pattern applies so many times, Slack -> IRC + WEB and many others.

>Now that north-south traffic is dwarfed by east-west >traffic in every system [this means that north is the >client, south is your "DB", and east-west just means >services in your system, and since people nowadays use a >lot more complicated systems, the inter-component = >intra-system communication is huge, compared to the few >MBs sent to the client].)

Never thought about it, in those terms.

I agree that it's not necessary, but:

Sockets are generally accepted as the standard. If you want to stream data to another computer, that's pretty much the only way to do it. The solution has been decided already.

A lot of these frameworks live in a space that's still evolving. We, as a community of developers, haven't agreed on the 1 best way to do it. As such, there are multiple offerings. In these cases, I feel it is definitely necessary to now what's underneath, in order to better evaluate ones options.

> On the other hand there is no end on how low you can go in the technology stack.

Not completely true. I think it's possible to have a complete simplified image of a computer in ones head while working on the newest version of your web app, going into depth in this area where problems arise.

And usually it's the other way around than you might think. Sockets you learn once and can probably even apply in 40 years. The current hip web framework you can't even use anymore in the same company in 2-5 years.

Why not go all the way down to device physics? It does harm.

Reply on news.ycombinator.com