Sorry, but that's a terrible attitude to have. You don't need to be a full-blown sysadmin to know how to do basic deployments, and learning these things will make you a better developer.
If you just want to do the minimum to run the app, it will make you worse and usually utterly terrible sysadmin, as it usually is going to be "do the minimum required amount of copy-pasting from tutorials to make the stuff running".
Then inevitably don't update shit, nor even set unattended-upgrades package.
And if the server won't get hacked from bad firewall and no updates, the server will ran out of space because dev of course didn't thought about data management and some log file in wrong place without rotation overflowed.
That's from decade+ of fixing shit of developers that think that tutorial on internet made them sysadmin.
It's only going to increase cognitive load, potentially without end (?) which doesn't sound wise long term. :(
I learn new things every day, and one day, Docker was that thing. I am now better at release engineering, cool! But I also lost a day of working on shipping value to users and actually having fun, which is less cool. That's why PaaS has been helpful to certain kinds of people; it makes the curve of progressive disclosure necessary to deploy stuff a little flatter. My hope is that we can make more kinds of things really easy to do the right way, and not yell at people who just want to have some fun.
I do think people who have experience deploying software have a blind spot here, just like I have a blind spot for people who have never learned how to center a div. jUsT uSe FlExBoX
Any cool, beginner friendly resources you can share? Many thanks
Sorry, I detest this attitude of: "Well I was hazed so why aren't you hazed too?"
Did I mention that Russ Cox and I used to write C code with pen and paper? https://news.ycombinator.com/item?id=32874759 If you haven't, are you even qualified to discuss what vital skills others are missing?
Or maybe because I learned perl because it was simpler than tr and awk, I should STFU? Give me a break.
Lol are you me?
I grew up with Slack/assembler/softice/etc and I used to host everything on Slack boxes on Linode (bless them for having a pre-built image). But I left it a while ago because I got sick of dealing with hand upgrades and compiling everything from scratch. I feel the same way about other things. I could probably out-CLI most of the people here saying "you should learn linux!" but sometimes I just need to get something done.
If I'm trying to get something out the door, I don't give a rat's ass about the details. It's fun and interesting when you're a teenager and it's 4am on a summer night and you've got nothing better to do, but these days I have strict time, limited energy, and I'm not going to spend more time fiddling with things than I need to.
Docker, k8s and friends are essentially just level above the "just CM", still just use basic linux primitives but with some coordination.
The "new toys" are essentially abstractions on what sysadmin would do (anyone remember building chroots ? Yeah I don't liked it either, even if it felt "cool" the first time I did it) and every abstraction will leak which means once you are big and complex enough you will need to debug it. Just that random hobby project probably won't get there and even for work it probably might be someone's else job to debug that
> I should STFU? Give me a break
...you probably should, your whining and bragging doesn't add anything to discussion. "Oh I played with cool kids few decades ago, look at me Mr. Important"
I'm joking.
Okay, bringing this conversation back to a normal tone: I have specific needs and want to get specific things done. I agree everyone should learn basic unix. I also think people should get as low to the metal as possible so they understand performance.
But can you acknowledge that there is a near infinite number of things people should learn, and that the priority of that list might be different for other people? That for some, and their technical goals, they might not have the time to prioritize multi-server web app dev when they're focusing on figuring out how to get GPUs to do DDP correctly? I never learned k8s because I all my web apps have super low usage. e.g. data annotation frameworks. I don't know JS and I guess maybe I'd love to, but I'd also like to wait until the field stops moving so correctly so that I could just pick a great framework and not have to relearn everything constantly. HTMX is cool and gives me joy.
Can you acknowledge that building tooling that allow devs to focus on what gives them joy is actually a nice thing? We have enough problems in the work we like that we end up spending endless hours debugging. If someone refuses to learn what they need to solve their main technical pursuits, yeah that's a moral failing, but nonetheless there's a finite surface area we can cover and I prefer to focus on learning the things that come up in the line of duty.
p.s. what are you working on? Maybe we can play together. I am not being snarky. I'm collaborative and maybe you're cool too. :)
/js rant start
I think when that happens it won't be because JS stopped moving, it will be because people will just go "fuck it" and replace it wholly with <favourite language> + WASM.
At least that's how I managed how to not learn JS properly for years. Web frontend work is just drudgery, and I only did it few times when we decided trying to explain a frontend developer ops stuff to make admin page out of is more complex that stitching some shitty code together.
And every single time it was miserable experience and I think only thing that I made and is still in supported framework is fucking jQuery, because every other one seems to shit on backward compatibility and decide to just make new framework. Then call it same name but just increase the number after the name.
/js rant end
> Can you acknowledge that building tooling that allow devs to focus on what gives them joy is actually a nice thing?
I did just say that yes we have those tools and it is fine that dev might not care about internals, I just want to highlight that the whole "I don't care what's underneath as long as it works" might eventually bite, so doing the boring part of understanding something about underlying system might save a lot of effort going forward.
Especially if that bit extra effort allows for overall simpler architecture, as easier debugging usually comes with it. "Just look in logs" or see what process is doing in system directly using installed tools is infinitely easier than fucking with docker/k8s CLI, sidecars and other methods to observe the app in container.
Recent example: A bunch of devs in company we provide infrastructure for got starry eyed for k8s and we just recently had to re-explain that no, the "just give us big POSIX filesystem mounted everywhere" on dozens of nodes isn't gonna work well, and the fact you can tell k8s "give me storage with ReadWriteMany" won't just work. That was after they tried to make shared block device and run XFS on it, which fell apart for obvious reasons
That after same company had year+ long migration from "a big GFS2 volume that was slow as fuck because GFS2 is slow as fuck when you have dozens of nodes for it" to plain S3.
One meeting later and they figured out they don't even need shared storage in the first place, but nobody researched that before and their solution "worked" on test 1 node k8s cluster (for obvious reasons). While a bit of research would have saved the whole ordeal
So less communicative company might just go "right on, we will set up CephFS for you and go ahead" then get more maintenance and the inevitable "this clustered filesystem doesn't work EXACTLY like my XFS partition on ubuntu" problems. It would probably be better for billable hours tho...
More low-scale example: My VPS just have few systemd services for few apps I put there. unattended-upgrades and it is near-zero maintenance for years. Granted, I knew how to do it because that's my day job but I have 100x less complex setup on my VPS. Maybe I should write blog about it...
> p.s. what are you working on? Maybe we can play together. I am not being snarky. I'm collaborative and maybe you're cool too. :)
Well, pandemic/nearby war kinda fucked up every project I started
I was working on midi2sid chip eurorack module. Which was delayed for year+ just waiting to get parts. I think they might be available again, maybe I should resurrect it. It did ran in rust and actually played notes, I just ran out of pins on the MCU I used so I had nice big board with bigger MCU and some niceties added, made it ready to be assembled... right as the chipaggedon started.
I was also working on car datalogger, even got prototype working, except recent economic fuckery put any track days on hold. The plan was to finally get a house, the inflation put any sensible mortgage out of question for at least nearby future so all the fun money are going to the savings now.
The other plan was to replace car's radio with something newer and integrate that with it (via dumb "just send rPi HDMI image thru the infotainment screen" probably), but yeah, can't really do that without garage in the first place...
I experimented with home automation a bit and had some ideas (involving embedded Rust, it is surprisingly palatable) but, well, again, the plans involved having a home, and not really that much possible to do in rented apartment
In meantime like 30 projects that scratch itch nobody else but me have and were not touched after scratching part started to work fine.
I did decide to implement the CPU from The Art of Computer Programming. But it looked annoyingly weird so I just made Go Z80 "emulator". Then rewrote it in Rust as exercise (both in ImGui as frontend). Then realized that getting the other chips than Z80 "right" is a ton of work and.... not that fun work to boot so I called it learning experience for Rust and ImGui and left it alone. It did run enough assembly to do something tho. No longer runs after Rust version upgrade tho. So much for backward compat...
But currently nothing really interesting. I have those cycles, where if work is interesting I spend a most of free time in brainless/semi-brainless fun bracket, and if work is boring I do the interesting hobby stuff, and, well, recent company buyout and some other changes made sure I'm having all the "fun" at work I can take so I haven't been touching hobbies much.
Why make Docker and essential part of a system before you actually need it? There’s a reasonable chance that you never need it. I know this, because a lot of business and hobby systems have been in operation since before Docker existed.
Docker has value in many cases, but it certainly doesn’t have to be a necessity.
Go app which is one blob and some static file ? Sure. Java app that just needs JVM in system ? Go ahead, .service systemd file with some limits is all you need.
Ruby on Rails ? Put that radioactive shit in container, else your server will need to have a bunch of -dev packages just to compile gems and make it running (or alternatively you'd have to compile the gems on same environment server is running, which is more work than just making a container)
I've been deploying Rails since v 2, and never with Docker. The only time I ever had trouble was setting up Rails dev on M1 Mac when M1 was new. In Linux production, it just worked; and pulling/building gems was less a drag than what people regularly experience in the NPM world.
To be fair, I don't use a ton of gems because I generally don't like additional dependencies unless the value is really there. So maybe my use cases have been too basic to experience the pain.
But if you just install it directly via package manager, leave alone and need to say reinstall server or something, the knowledge is lost.
I remember having a lot of ping-pong between ruby devs where they, well, didn't note down what system libs and dev package they needed, just had them installed locally at some point and forgot about, then surprise when app isn't compiling on server.
You don't need to be a full-blown sysadmin to
know how to do basic deployments, and learning
these things will make you a better developer.
- Unix literacy including common tools (awk, sed, grep, etc)
- Docker, etc.
- Relational data stores
- Key/value and document stores
- Base frontend technologies (HTML, CSS)
- Frontend frameworks and the associated language
- Build pipelines to tie it all together
I've grown extremely disenchanted with "modern" software development. Your typical full stack engineer is frankly bad at most of the damn stack and they're not great at any of it. Nobody masters anything anymore. You get people with IQs three standard deviations above normal thinking they need to spin up an armada of AWS crap just to render "Hello World" in 2023. And they still manage to make a mess of it. I can't tell how much of this is brain damage and how much of it is a sociopathic commitment to resume-driven development.
In practicality, I'm a fan of "T-shaped" engineers. Master a piece or two of the stack, and have light knowledge of the adjacent layers and what they do. e.g. If you are an application-level coder you should know what Docket/K8s/etc are and when/why you might use it and when you should eschew them. And honestly I think it should be eschewed fairly often.
The places you can go with a bit of good old vertical scaling are pretty impressive in 2023. A single box with e.g. 24 cores, 64GB of RAM, and fast solid-state storage is a god damn supercomputer when it comes to spitting out web pages. Think about how far Stack Overflow got with that model and at which point you might start to outgrow them.
---
edit: By "mastery" I mean "fairly comprehensive competency" and not "be a recognized global authority." You can fluently use 50-75% of what the tool offers and you know what the other 25%-50% is so you can learn it and utilize it when needed. You can ship code using this tool that is maintainable and performant. For example, perhaps you've never utilized partitioning in Postgres but you know what it is and you understand the appropriate use case for it so you can learn employ it when needed.
mastery of a specific technology adds very
little value
Somebody with Postgres "mastery" should have a strong command of basic normalized table structures, they should know how to construct useful indexes, they should know the downsides of over-indexing, they should know how to find and optimize slow queries, and they should at least know of somewhat advanced topics such as partitioning, materialized views, foreign data wrappers, and so on. It is okay not to have a clue how to set up e.g. partitioning but you should know what it is so that you can learn about it and employ it when needed.
Somebody with "mastery" of an application framework such as Rails should be competent at basic OO design, they should be familiar with Ruby structures and idioms, they should understand what goes where in Rails' MVC paradigm, they should be able to write comprehensive and performant tests, and they should know how Rails' asset pipeline builds and emits front end code. I would not expect them to have mastery of Ruby's metaprogramming, but I would expect them to know what it is and have some idea of how Rails uses it to extend Ruby. (I realize Rails is controversial, but I'm not talking about anything Rails-specific. Feel free to substitute your language+framework of choice)
In short, the sort of working knowledge you might commonly gain with 1-3 years of real-world production experience with a given piece of the stack. Or perhaps less if you were really focused on it and just learning it full time.
Without guidance, folks with less knowledge than this tend to produce absolutely unscalable, unmaintainable spaghetti code for anything larger than a toy project. To return to the screwdriver analogy, they can kind of use a screwdriver but they're constantly stripping screwdriver heads at which point either the project is ruined or they need some help from more practiced craftsperson.
always in danger of becoming more legacy than relevant
The other thing that's going completely over people's heads here is that I made this repo with the intent of helping other people who know less than me. I know I can figure out all the deployment bullshit, but I mentor folks who can't, and also don't have much money, and yet still need to deploy their portfolio projects to the Internet in order to find jobs. I was trying to find a balance of ease-of-maintenance vs monetary cost. If there's an even simpler solution with an even lower cost, that a random bootcamp grad could maintain, I'd adopt it in my template in a heartbeat.
If any of these people claiming it's easy want to work with me on a reboot of the "deploy a web site easy and cheap" template, I'd absolutely collaborate with them on it. I think it's important to democratize all the cheap computing power lying around these days.
maybe that'll make you a worse engineer, but engineers are not human beings.
That's funny, all the engineers I've known have been.
What crowd have you been hanging out with? :)
Hey, I'm the guy who wrote that original response to you. I apologize if that's how you interpreted my comment, but that's not what I meant by it. I wasn't even trying to say that you're a bad engineer (and certainly not a bad human being), I just wanted to say that I disagree with the attitude of that comment. I'm sure if I said it to you in person, you would have picked up on the non-hostile intent and narrow scope of my criticism, but the internet tends to twist things in very negative ways.
Nobody can know everything, and not knowing something doesn't make someone a bad engineer. Software development in particular is more about learning than about knowing, which is why when I see someone saying that they don't need to learn something because <X>, it bothers me. In your comment, you said that you don't want to be a release engineer, but IMO that's a poor excuse since "release engineer" isn't even a role that exists at many places.
And for the record, I didn't even look at the project you linked to. So if you also thought I was criticizing it, then rest assured that I'm much too lazy for that. When I was still learning, online tutorials and resources like yours were extremely valuable to me, so I know how helpful they can be to people even if they're not "perfect" by some snobby asshole's definition...so keep at it!
I also have a wife and a 2.5 year old.
It’s certainly possible if you don’t stop yourself.
It's weird to say that people aren't learning these things because they're "stopping themselves." They're just going in other directions. No one in my life has ever accused me of learning too few things and failing to learn by building things.
Edit: I think I misunderstood the parent comment, sorry for the defensiveness here
But, come now, his point was he’d never worked with a full stack developer and that they don’t exist.
To my reading, that's not what was claimed.
It seemed to be more like "they exist, but most of them turned out to be pretty shallow with each of the technologies".
I'll use myself as an example. I am a "full stack developer." I have fairly deep backend knowledge. But my React skills basically amount to the ability to make small JSX tweaks, and my sysadmin-y skills are also at roughly the same level.
Conversely, the folks who are really sharp at front end technologies tend to write some psychedelically bad backend code in terms of scalability and maintainability.
For some projects this admittedly doesn't matter. Some projects are simple and don't need to be wonders of engineering.
Like others upthread, I am one of them, and probably because I've been paid to do this job for 16+ years.
Honestly my challenge is making potential employers understand that yes, I know my way around Elixir or React, as I do around C or Rust, as I do around sysadmin (not only DevOps) or DBA work, as I do around low-level system code. I'm no world expert at any of them, but most companies need a person that can wear many hats when things go crazy and specialisation is for insects anyway.
Sadly, generalist engineers like me have lost against the trend of "full stack developer" to have been stolen from us by recruiters and less skilled devs to now mean "can use Express and React and maybe deploy on Heroku".
I know my way around Elixir or React, as I
do around C or Rust, as I do around sysadmin
(not only DevOps) or DBA work, as I do around
low-level system code
This is a challenging discussion because the idea of "knowing" or "being good at" a tool is so nebulous.
As I mentioned elsewhere I'm using a definition that is essentially, "the level of skill a solid engineer would acquire after working full-time with a technology for 1-3 years." Not world-class expertise necessarily, but enough time to surpass basic literacy and achieve a real fluency. To encounter edge cases and pitfalls and develop well-supported opinions about best practices.
If you have that level of fluency in all those tools, great! Sounds like you're pretty awesome. In my experience that level of mastery of the entire stack at once is exceedingly rare.
I've written Rust full time for the past 3 years.
I've written C since I was 14 in 2001. I wrote a small operating system (up to reading and running a binary from ext2) around 2004 in C, so I know how a computer works at low level. I might still be able to write x86 assembly.
I've been a MySQL DBA full time for 3 years.
I've administered Linux systems since I was 14 in 2001, and for all my professional career.
Then there's Python, Go, etc.
Probably the one I know the least is React, which means I was PM on a React codebase for a short while, and spent way too much debugging weird issues with Next.js. I stopped paying attention post-hooks, since the frontend world changes too fast, and I'm getting old.
--
Again, this is not to toot my own horn, it's just that if you live and breathe computers, and hate doing the same thing for long (I blame my ADHD), over a long enough time you tend to have quite the repertoire. I think I'm quite average compared to other people that have been around as long.
You'll soon notice there doesn't tend to be anything revolutionary in computing. After your third framework and language, you'll keep finding the same ideas and concept with minor variations.
I've used Elixir full time for the past 6 years.
I've written Rust full time for the past 3 years.
> I know my way around Elixir or React, as I do around C or Rust
Interestingly, I find that I've never been asked to write anything in a compiled language yet. I've been wondering how I could work those skills into my job, but it seems like it doesn't come up much in the things I end up working on
Ran a business. Did everything from building the servers, the code, the marketing, the design, community management, merch, and event planning. Call it "ultra full stack," maybe?
So yeah, it's extremely possible.
But I'll also tell you bluntly I was only really good at a few of those things. Most of those things, I was bad at or perhaps more accurately I just acquired the bare minimal knowledge to get by in my specific and limited use cases.
And many of those skills decayed quickly. I knew a thing or two about SEO and front end development in let's say 2000-2012 but those fields change fast and little I knew back then is relevant now.
That's why I say that in 2023, I think the notion of "full stack developer" has grown untenable. You cannot be good (in the sense of shipping actual maintainable and performant production level stuff) at the entire stack simultaneously now that complexity at each level of the stack has multiplied relative to 2005 or 2015. For example, your own db -- is that a fun (and impressive!) hobby project or is that really production level stuff?
To be ultra clear, I am not knocking what you have achieved. It sounds awesome and I suspect you have dived deeper into more things than I. Kudos, and I mean that sincerely.
I don’t think you have to be a 10 to be a good. But you might need to be 10 to be exceptional. And I will agree that it would be very challenging to be a 7, let alone an 10, across multiple domains at the same time. Certainly, across months/years one’s focus might wander and thus their exceptionality might as well. Being able to adapt and pick up new things at a 7 level though - is the real FSD.
But I see your point and agree with you about the shallowing of the FSD term.
I think it highlights how difficult it is to even discuss these things. We're all using different definitions of "good" and "mastery" and "7/10" and "10/10".
Being able to adapt and pick up new things at a 7
level though - is the real FSD
By my definition, I actually don't think this is possible, at least not for larger languages/tools/frameworks.
My reasoning is this. To be a 7 (my definition) requires time. You need to not just understand the basic premise and syntax of a language, but you need battle scars. You need to have shipped some code in that language, gotten familiar with the ecosystem of libraries, you've troubleshot production issues, and become familiar with common pitfalls and how to avoid them. You've checked out some large codebases in that language and gleaned best practices and things to avoid. You have probably also spent some time in that community, watched/attended presentations from recognized leaders, and have a sense of which way the wind is blowing.
Everything I just described takes time. I don't think even the smartest person in the world can drop in and achieve that immediately unless we're talking about a relatively simple tool.
I mean, could somebody who already knows CSS pick up TailwindCSS and be a 7 quickly? Absolutely.
Could an engineer who is new to Ruby/Kotlin/Python be a 7 quickly? Not by my definition, not by a longshot. In my experience, seasoned developers drop into existing codebases in these sorts of languages and make a mess of things at first until they get used to the ecosystem.
You did all of these in the last 2.5 years? If not, that part seems a bit irrelevant.
But regardless of the response, some kids are way easier than others (easy kids are those who like to eat and like to sleep, imho), and some parents have more help or stricter separation of parental duties.
Of webapps. Will make you a better developer of webapps. It may surprise you to learn that there are more kinds of software than that. Sometimes the final product is something like a binary that runs outside the browser, and deployment is no more complex that putting an tar.gz'd executable on a website.