The great thing about Linux as the base layer is that it allows a commodity common ground with a very capable system that also facilitates more specialized layers to be implemented on top of it.
The great thing about x86 instruction set as the base layer is that it allows a commodity common ground with a very capable system that also facilitates more specialized layers to be implemented on top of it. Only a handful of programmers develop at a layer below the x86 instruction set, or even at that level for that matter.
We have to get our heads out of that mire.
Now, what is that about alternate OS systems which can still do everything that Linux can, including handling network or service outages?
(*well, there are BSDs, but I don't think thats what you had in mind)
Unix was specifically designed for small, resource starved machines and did not have the powerful abstractions of mainframe OSes like Multics or OS/360. It's OK, but as modern CPUs and IO systems have grown and embraced the mainframe paradigms that had been omitted from the minicomputers (e.g. memory management, channel controllers, DMA, networking etc) unix and linux have bolted on support that doesn't always fit its own fundamental assumptions.
That's fine, it's how evolution works, but "cloud computing" is a different paradigm, and 99.99999% of developers should not have to be thinking at a unix level any more than they think of the micromachine (itself a program running on a lower level instruction set) that is interpreting the compiler's output at runtime.
If you have two processes on same machine, locking a shared data structure takes microseconds. You can easily update hundreds of shared maps and still provide great performance to user.
If you have datacenters in NY and Frankfurt, the ping is 90mS and fundamental speed of light limit say it will never be below 40mS.
So "lock a shared data structure" is completely out of the question, you need a different consistency model, remote-aware algorithms, local caches, and so on.
There are people who are continuously trying to replace Unix with completely new paradigms, like Unison [0].. but it is not really catching up, and I don't think it ever will. Physics is tough.
And an OS it the same: it used to be quite common to write for the bare iron, to which an OS is by definition an abstractional interface. I still do that, but it's an arcane skill frankly not in huge demand. Which is probably a good thing.
Nowadays most code is written at nosebleed levels of abstraction, which frankly is a good thing, even if I don't like doing it myself. But still, as developers do it, they are often dragged back down the stack to a level that these days few understand.
I think the person/company that cracks this will be the dominant infrastructure play of the decade.
Educate me.
It's much more meaningful and useful than "something simple" "running little functions floating in the void" which just sounds like some vapid marketing pitch.
That was a quick response to what I thought would be better than administrating a UNIX system on an Amazon machine. Don't be an ass. I've no experience with the real distributed computing systems created before UNIX.
The great thing about Linux as the base layer is that it allows a commodity common ground with a very capable system that also facilitates more specialized layers to be implemented on top of it.
No, that's the vapid marketing pitch. The Linux kernel randomly kills processes when it starts exhausting memory. It's garbage.
Hey, Brainfuck also facilitates more specialized layers to be implemented on top of it. Now, what's stupid about doing that, however?
Not sure if I'm being trolled... It finds a remote machine by name, and routes a connection to it. It authenticates and establishes a secure connection with the machine. It sends a command to the remote machine, the remote machine executes it, and the result is returned. It then puts the result into a form that can be used programmatically by the local shell.
> That was a quick response to what I thought would be better than administrating a UNIX system on an Amazon machine.
It was content-free.
> Don't be an ass.
What's good for the goose...
> I've no experience with the real distributed computing systems created before UNIX.
And already the expert. Impressive.
> No, that's the vapid marketing pitch.
No, that's the reality. That's why Amazon, Google, Azure, and everybody else offer it in their clouds and use it on their internal infrastructure.
> The Linux kernel randomly kills processes when it starts exhausting memory. It's garbage.
I'll take that over "running little functions floating in the void", it actually exists and works.
It then puts the result into a form that can be used programmatically by the local shell.
That does sound better than returns one octet with two well-defined values, sure.
And already the expert. Impressive.
I know UNIX is shit with a legion of cultists.
Hey, if we live in the best of all possible worlds, explain the market dominance of Windows. Did MicroSoft give Windows away for nothing, to poor unsuspecting university students who decided to hack on it instead of learn what a real computer is, until none remained?
At least read my website before calling me unhinged.
> At least read my website before calling me unhinged.
No. I've seen countless tech prophets peddling snake oil over the decades, and it's always the same. Everything is dumb, poorly versed in the state of the art and history, but this magical ill-thought-out thing will somehow solve everything. It's very boring and predictable.
You forgot an entire direction of input and focused on just a return code.
> The Linux kernel randomly kills processes when it starts exhausting memory. It's garbage.
No it doesn’t. It’s not random and it’s configurable. Turn off overcommit if you want malloc to fail when memory runs out.
You sound like someone who wants some sort of magical fantasy abstraction level where errors never happen and you can just write code without ever caring about said errors or really anything that happens at a lower level or outside of your code. Sorry, but that doesn't exist and never will. Code runs on computers, it doesn't run in a "void".
If you want to disable overallocation and the OOM killer, be my guest. You'll want to install some extra banks of RAM though.
Or is there a better way, provided by other OSes?
You shouldn't get to the point where a perfectly innocent process is killed to reclaim memory. A process can die, or even (in some cases) recover cleanly from running out of memory -- that should be the process's choice.
In Linux you can do that -- disable overcommit (although I'm not sure if that's 100% foolproof). But that's often not preferred because that way does lead to "perfectly innocent" processes being killed: whoever allocates the next page when memory has run out loses. And how do they get to decide? What if a failure path requires paging in one more page of code from disk? Or causes a store to a COW page?
print(os.path.exists("ssh://user@rsync.net:fileThatExists"))
A boolean, a still (relatively) terse line of code. And when you end up wanting to do something even a bit more complicated it'll be a smooth gradient of difficulty.
As with every discussion around shells, though, the amazing terseness of scripts is great, but every form of programmatic behavior beyond "do this then that" and "do string interpolation here" is a massive pain in the butt. The biggest advantage is that stuff like pipes are a very good DSL.
> print(os.path.exists("ssh://user@rsync.net:fileThatExists"))
That's just invoking the whole jungle when all you want is a banana.
I did say Python-esque (I do get that Python in particular has a lot of stuff in its prelude). Really if you want to be super pedantic then the most "banana"-y isn't a line in your shell but some C library calls on both sides of the system.
It doesn't crash because I forgot to import os ?
you're omitting the 500 lines of yaml needed to deploy that, the tens of megabyte for the python base image (of which you'll waste more than 99.99%) and the tens of megabytes in bandwidth to the docker registry and to download it on the kubernetes worker... And the image wight should be accounted at least four times: worker node + 3 in the docker registry (ha and stuff).
it's incredibly inefficient if you just freaking want to know if a file exists.
For example, you can interact with docker in Python with a lib, but it would be amazing if the docker binary just exposed an RPC interface and API for script languages so you wouldn't have to parse out things from output or otherwise try to figure things out when writing shell scripts. This is, of course, the PowerShell thing, but PS's aesthetics displease me and many other people.
Computers are, of course, turing complete. You can do whatevery you want, really. But for example I end up using zsh instead of xonsh because I want to use what other people write as well.
# ssh user@k8s.machine "test -f fileThatExists"
# echo $?
0
I well remember my disgust when I learned AWS was just using UNIX virtual machines or whatever, rather than something simple, such as allowing people to run little functions floating in the void. I know nothing about AWS, but I've not been mistaken, right?
EC2 was a necessary step and though and even that bought things like per second billing, autoscaling and infrastructure as code to the masses.
I ran 1000s of physical Linux servers over the decades using chroots when it was not fashionable to have containers yet with db clusters because I don’t want my laptop to be different than my servers and I don’t want to think or worry about scaling.
We have aws Lambda now, but it is too expensive and, for me, not trivial enough. It actually requires too much thinking (architecting) to make something complex with it and a lot of that is about money; you can stumble into bills that require at least a kidney to settle.
So I still run my own servers, these days with openfaas. With many scripts, it is almost painless and costs almost nothing (we provide for millions of users for a less than 200$ a month in total). But it still is not the dream: I still cannot just forget about everything hosting and abstractions still leak enough but at least I don’t spend more than a few hours a year on it and I don’t go bankrupt like can happen with Lambda.
We are building our own language and runtime (lisp like with types) to fix this: after 40 years of programming, I am pretty much fed up with having computing power and loving programming but having to deal with all this pointless (for programmers!) stuff around it: we went backwards, at least somewhat.
Also I was saving me a phone call with Dell and a sunday evening configuring some hardware.
No new bullshit paradigm to learn, I could shove my code in there with tool I knew and call it a day.
EC2 was a stepping stone and it worked well to get some stuff in the cloud.
Remember the early days when only non-critical, non-PII stuff were in the cloud? And how some companies "just can't use it because of X" ( or worse the dreaded Y )
I have had two types of developers in my teams: one type that understands the underlying technology of the abstractions that they are using, and one type that only understands the layer/abstraction they are interacting with and blissfully clueless about anything underneath.
Neither group worries about transistors or proton decay in their hardware, and both can make 'things'. However, one group is much more capable of effectively building tools and understanding issues when something goes wrong in the chain, whereas the other group is regularly hopelessly lost because they have no clue of what is happening. I'll let you guess which is which.
This is not about worrying or even being bothered with details, but about understanding what you're working with.
I was working with an intern lately, we both had remote machines in the same subnet (i'm omitting some details, of course), and he had to pass me a file.
We was about to download it on his laptop, upload it back to the cloud (slack message) when I would have had to download it on my laptop and send it back to the cloud.
I was able to show this young engineer how to use cat and nc (cat somefile.tgz | nc <ip-address> 8000) to send and then I would use nc (nc -l 8000 > somefile.tgz) to send it in a moment, without third parties involved and without having to make this file go across the globe multiple times.
The thing is: if you know what you actually need to do and have UNIX tools available, you can be insanely efficient.