Or maybe I'm the crazy one.
I do lament that tools have become so complicated, but just about every piece mentioned (except for maybe kubernetes?) actually does a thing that is likely useful to you in production. A run down:
Java - yikes but OK, the JVM is an excellent piece of software, of course you need the actual app you want to run
NGINX - TLS termination, compression, timeouts, rate limiting -- don't have to put this in your app and deal with SpringWhatchaMaCallits if you just do it @ nginx. complexity here means less @ the app level
ElasticSearch - I'm not sure ElasticSearch is the right tool but usually for most apps you'll want search, and you'll want search that is good enough (aka not slow/doesn't suck)
MySQL - A database of some sort is necessary, and most embeedded/single file databases aren't the right fit for a multi-user frequent-concurrent access model, though I do love me some SQLite so I might fight myself on that point.
containers - containers are sandboxed, resource-constrained processes. I believe that it's better to run containerized processes rather than regular processes (i.e. just running your app) because of this isolation. Can't have one app clobber settings/whatever for another if they're properly isolated.
k8s (kubernetes) - a container orchestration tool -- if you're going to run containers on more than one machine then maybe you want to be able to treat all those machines more simply without manually managing all of them. This is obviously not necessary, If you only have to manage 2 machines, maybe just set up systemd properly and make sure two processes are always running and the internet can get to them, whatever.
kops - A tool made for easily provisioning machines on AWS into kubernetes clusters -- only necessary if you've bought in to containers and kubernetes.
cloudformation - AWS's tool for automating infrastructure creation and management -- while cloudformation might be hard to use, time spent automating pays time dividends that approach infinity (don't quote me on this), as long as you actually finish building the automation.
aurora - AWS's offering of a RBDMS, with some scaling advantages and other amazon-secret-sauce stuff. In the example this is deployed with cloudformation so someone on your team doesn't have to go into the actual amazon dashboard and click around for 5 minutes.
VPC - AWS's rendition of a private network for you to use their services on... you probably don't want to not run your stuff in a VPC (I don't even know if that's possible anymore) -- hard to know who else is running in your cloud.
packer - Packer is less important for containers but more for VMs (but I think you can use it for containers too?) -- basically if you have your source code in a folder, it would be nice if you could build a container for it automatically. Packer was used more predominantly for building VM images (AMIs for AWS) IIRC -- this helps to automate the deployment process, you can ensure every machine you start in the cloud or at home has a base level of configuration/software installed. You could even make your deploy artifact (the one thing you need to deploy your app) a VM image or an AMI and all you have to do is spin up a machine and your app is running.
ansible - fantastic automation tool for when you have to perform a task across one or more machines, for example, pushing a container image to a certain machine, or adding a user, or installing docker, or whatever.
jenkins - general task runner and automation helper differing from ansible in that it's used more through it's web interface, and is always-on, so it can do things like running tests whenever code is pushed to a given repository. This speeds up your team by letting them know when something's broken faster. Also, it can do things like deploys, post-deploy smoke tests, etc -- the more things that some random software does, the less I have to worry about jeff/gina fucking up a deploy|test|whatever.
CI - continuous integration is nice -- run your tests automatically so devs don't have to worry about it, make the results visible so no one merges bad code. Even better, make it impossible to merge code that doesn't pass the tests, or ensure that a certain amount of coverage is achieved (though code coverage can be a bad metric)
canary deployments in UAT - This is actually something mostly advanced engineering orgs do. User Acceptance Testing is arguably the only testing that matters, because if your user can't actually do what your app is supposed to do, your app may as well not exist, no matter how well it is built and unit/integration tested. UAT is when you get a person to sit and actually use the app and do what was supposed to be possible. "Canary deployment" is not really the right term here but in context I think wetpaste is referring to spinning up a "fake" version of your app, so that testers can touch something close to the actual thing.
canary deployments - Canary deployments are more traditionally doing a small deployment of a new service (let's say, serve 10% of your users the new version of an app), and observe/watch for errors before letting a deployment go wild. Again, mostly this is only done at really advanced engineering orgs.
Prometheus - Relatively simple and robust monitoring tool for dealing with time series data
Grafana - dashboarding tool that takes input from a few places (prometheus, RBDMS, etc) and charts data so you can easily see how your stuff is doing -- RED (Rate Errors Duration) is a pretty decent monitoring methodology for web services
ELK (ElasticSearch, Logstash, Kibana) - This stack is a little less necessary IMO but logstash pushes your logs to elasticsearch, and kibana makes them visible from the web. Logging is important of course, but you could probably just SSH in and look at the logs or rotate and extract files (or use something like cloudwatch logs if you're on AWS) happily for years.
Jaeger - operation-level tracing so you can easily find out how long different things are taking on your web service -- yes the `/expensive-request` is taking 5000ms, but which parts of what's actually happening are causing it, in production? the DB request? JSON munging on the API side? some other thing?
Service mesh - Much like NGINX, this is a way to outsource things like monitoring code (which you'd have to integrate into every app you wanted monitoring) to the transport layer -- you talk to a proxy (whether per-process or per-underlying-machine), and the proxies ferry your messages to wherever they're supposed to go. They also support service discovery, so now your java app doesn't need to know exactly where it's mysql instance is (which you'd normally feed in with ENV variables), it can just send stuff to mysql://my-mysql or whatever, and as long as the mesh is properly configured, it will go to the right place, and the mesh can do things like telling you how long every request took, or do circuit breaking, or retries, or whatever.
Istio - Service mesh that's built for kubernetes, it does the usual service mesh stuff plus some more, like super configurable routing (the kind that might enable canary deploys) to providing mutual TLS traffic between services automatically, and cluster wide authN/authZ
Calico - Sort of assumes the buy in to containers & kubernetes -- if you're going to run multiple containers on multiple machines but don't want to know every machine's IP and every container's IP, you're going to need a network overlay that simplifies things. In addition to reachability, calico also enables kubernetes's NetworkPolicy controls, so you can restrict intra-cluster communications
That said, I do abhor incidental complexity, and do like simple dependable tools -- however, most of these tools do actually serve a purpose and aren't just bloat. I think these days you can start off as simple as you like, and by all means fight complexity as you see it rear it's ugly head, but at some point, you're going to want to know which requests are slow for your users. At that point, you need to make the choice between a service mesh and a standalone jaeger instance -- that is when it's useful that you know both things exist, so you don't pick the tool that is more complicated (the service mesh) when you don't have to.
"Yeah, no. I have no plans to try and actually make anything with that tangle. Hah."
"Although... I do know what nginx, java, containers, k8s, VPCs, ansible, jenkins, and CI are... so... maybe it's really all the same at the end of the day, I'll see everything in this list similarly 5 years from now, and I should see if I can tolerate it?"
"Hm. There is the small fact that I don't really know what k8s, VPCs, ansible and jenkins actually do, let alone the entire first list, and of all of these I've probably only really used Java, and that only a few times. I think I've had a few minutes' look at GitLab's Grafana dashboard once or twice?"
"I'll bet installing all of these, including all dependencies, would take probably multiple tens of gigabytes of diskspace, and probably consume more RAM that I have installed." (I'm running short on both, I have a few hundred MB of diskspace free right now, and never any free RAM :D)
"I wonder how effectively I can learn these on the job?"
"...I wonder if there are any jobs that don't require $tool_existence+1 years of experience with any of these before they even consider candidates ._."
Goes back to writing PHP script in text editor
React is like one step forwards, 10 steps backwards. Mixing markup and logic, adding a whole front-end compiler, needing imports from third parties to re-implement basic browser functionality - for what? So that users have to see a loading spinner and their back button doesn't work any more?
I'm 90% sure it's an accidental conspiracy to keep front end developer salaries high.
Talking to older heads, apparently they felt the same way when XML and "middleware" came along..
There's no need to mix markup and logic, a good pattern is to have dumb components that just translate state into markup, and keep your logic elsewhere.
Modern javascript (and typescript) is a joy to program in and has greatly improved my productivity. Having to compile it with babel is a small price to pay.
https://github.com/babel/babel/issues/8579#issuecomment-4169...
I pine for the times when we could just place a few <script> tags here and there and call it a day.
I'll cherry-pick these comments, which I think have the best signal/flame ratio:
> ...
> this doesn't seem safe anymore. this stack has too many individuals with no oversight in charge
> i really wish the javascript community would start seeing unnecessary tooling as a vulnerability
> this happens so often in this language
IMO you could totally s/the javascript community/everyone/ here, obviously not for universal values of "everyone" but certainly for many.
It definitely inspires awe.
I mean, what am I supposed to make of this? https://github.com/palantir/tslint/issues/4141#issuecomment-...
That did it; I fired off a report to GH, just to be sure. I decided some of my text veered somewhat off-topic, so didn't send the whole thing. Here's what I said, including the offtopic bit after the 2nd "---".
---
Hi! You've probably gotten a few reports about this by this point, so this is just a redundant make-sure.
https://github.com/lerna/lerna/pull/1616 is a pull request that changed the license of a project with 11k stars and which (IIUC) is also a common Babel dependency. The license change basically extended the MIT to say "but this license expressly forbids the following random list of companies from using this" blah blah. (I don't think this person realizes GitHub is owned by Microsoft...) The rationalization was very political.
The whole debacle got reverted fairly quickly by https://github.com/lerna/lerna/pull/1633, and the user was removed from the project. Yay (and whew)!
The license in question can be found in the user in question's repo, https://github.com/jamiebuilds/license, and it currently has 53 stars. I think it only had 20-30 yesterday. :/
Today, I woke up to https://github.com/palantir/tslint/issues/4141#issuecomment-..., which is reasonably inflammatory.
I have no idea what gets done in situations like this, and will watch with great interest. I say this honestly; I found this over at https://news.ycombinator.com/item?id=17872475 while reading a _completely_ unrelated topic, I have no association/involvement/investment in/with the project.
---
There are 1001 interpretations of course. Mine is that this person may be driving themselves into a frenzy of sorts with more and more outlandish actions. I hope I'm wrong! But unfortunately, because of the scale of the internet, the small number of people that are voicing their agreement with this person's views is large enough that it looks like this person has become completely convinced their mental models are accurate and well-founded.
On the one hand, the above is my rationalization for why I thought it might be a good idea to mute this user's account for a few days. I have nothing against them, but thought that maybe the chance to quieten down would mean they might not harm themselves or others, which I definitely would not want to happen.
On the other hand, I've realized that there's nothing to stop this user using email or Slack or whatever else to continue communicating so muting wouldn't have such an isolating (and hopefully calming) effect at all, and (as so often happens with banhammer situations) would instead probably rile them up and make them 1000x worse. So from a "quiet time" standpoint that wouldn't work at all. :(
---
I saw the 2nd block above is relevant but unneeded reading / commentary.
The only reason I got into web dev were what it offered me as far as ability to make things. But the work itself? I'm just as happy writing reports vs refactoring a test suit. Rails makes software fun for me, I would not have gotten into web dev if i thought React was how things are done.
This is a great idea ( I sincerely mean it), and I too have been toying with this.
Gov programming jobs are also very cushy. The knock on them ( / "cons" ) until recently was that they pay very less, almost 40% less, and that it's full of old people. But that has changed -- atleast in San Francisco and Bay Area where I live.
Esp the City of San Francisco tech jobs[1] are quite interesting and many use modern tech. On top of that almost all Govt jobs accrue pension at the rate of 3% of your gross / year, so if you work 17 years or so you can get 50% of your highest salary at retirement.
The con I see is that the process to get hired is very tedious and involves lots of applications and moves very slowly. I was plesantly surprised to see Sr. Software Engineer jobs that were paying anywhere from 110K to 130K [2] -- all govt jobs have to disclose salary or rate range in JD --, which is not to far off from the average in this area for non-FAANG / non-Startup type job.
[1] SF City Tech Jobs => https://www.jobapscloud.com/SF/?Keyword=&Loc=&DeptNumber=&Oc...
[2] Source / Example: IS Programmer Analyst Principal: Notice salary Range of $105,794 - $133,094/yr in desc. https://jobapscloud.com/SF/sup/bulpreview.asp?R1=PEX&R2=1064...
Now, if you're including the TSP (401k equivalent), maybe? That's 5%/year extra if you also contribute 5%. However, that's what you and the government put into the account, not what's available when you retire. Perhaps it works out to close to 3% average with some estimate of your TSP earnings? I never crunched those numbers.
Government salary ranges can be found at OPM. Look up the GS pay scale. Programmers will be GS-12 to GS-14 positions (most likely 12 or 13, 14 if you're a true subject matter expert, have a Phd, or maybe work at the right facility, otherwise 14s are management).
Still, if you assume something like a 4% return (maybe comparable to the bonds behind them?), that 1.1%/year amounts to another 27.5% on top of your gross pay.
And that's not considering the likelihood that you'll be earning more at the end of your tenure than at the start. You're effectively earning that significant fraction of your final pay on top of whatever you see in your pay check.
So if I work for 20 years and have an average of 100k for my high 3-year, I'm getting $20k/year for the rest of my life.
In the same circumstances but I retire at age 62 (leave work at age 62), I get $22k/year for the rest of my life.
The current President is also talking about eliminating the cost-of-living adjustmetns that have been typical (intended to keep with inflation) for retirees, and changing the 3-year average to a 5-year average.
The other number I gave (5%) is a matching contribution to the TSP (401k equivalent), that is cash given each year and will add up to something more substantial. Even assuming a modest 4%/year return over 20-30 years of contributing.
As an old school embedded programmer, I’ve stopped trying to even figure out how many layers of abstraction, data transfer and transformation, business rules, and libraries are between the user and the metal in modern software. Nobody values small and concise anymore. Big and complicated pays the paychecks.
Would you want to code in your retirement, or do you plan on hanging up your keyboard?
I started programming on a VC-20 doing basic and later assembler. Just this for years. Glorious times.
What helps a little is to go deep into the history of computer programming and see the big picture. The fundamental problems defined in the 30s ~ 50s are still very much relevant. Kubernates, node, react etc are just ripples.
To your list, I would add Docker. I fucking hate that all-pervading obfuscator.
Docker have gone along way but Kubernetes they need more and more version. I feel as I get older I'm more incline to wait a few version for the bugs to be fix and the api to be solidified. The majority of the software out there that come out with version 1 is just marketing to get people to adopt it and they work out the kink as they go on.
If you are then you aren't the only one. A lot of this stuff seems to spill out of companies like Google or Facebook and I'm sure it makes sense at their scale. For most of us we just don't have the same problems and are probably better off avoiding it.
Kubernetes: I've actually not used kubernetes yet; I do hope to, sometime in the future. I do see the value in managing distributing the images/code required for a job to the machine that will eventually execute it, and distributing jobs automatically, as opposed to by hand. I've worked with a system similar to Kubernetes, and having used it, I understand the appeal and miss it greatly. Docker images (which I understand Kubernetes to use) also greatly simplify my life as a developer; generally, the actual VM my stuff is deployed to is significantly out of date (e.g., it's CentOS or Ubuntu) and this allows me to, e.g., get a very recent version of Python 3. It greatly pushes one toward (IMO) not having snowflake (unique, hand-crafted, but can't lose them) machines.
Node: I don't use Node server side; I mostly use npm for managing fetching client-side dependencies. Otherwise, this would be either committing them directly (I dislike mixing third-party and first-party code; it encourages ad-hoc fixes/changes, and doesn't get updated regularly) or a bash script to fetch them (more painful to maintain in the long run than npm, IMO). It also fetches indirect dependencies, so I don't need to figure out that this project requires that dependency, etc. Also, Babel is a godsend of a tool for dealing with browsers that haven't implemented the latest ECMAScript features. In particular, JavaScript modules are wonderful compared to life without.
React: I've not used full react; I've actually not used it at all for lack of time. I would like to use it, as building up UI elements via the DOM APIs is downright painful. Having not used JSX at all, I can completely understand why someone would invent JSX.
Terraform: We didn't use this for a while. AWS resources where managed by hand; nobody would know why something was in a particular configuration, and configurations between production and non-production environments would be different, with no clear rationale as the why. TF at least allows comments and a history of changes (when version controlled), and the language helps keeps things consistent where they should be.
If you've got a kubernetes problem, go for it. But there is a lot of value in simplicity too.