As a blue-collar electrician (although retired), I want to hire anybody that seems to give even the tiniest shit. While working, it took me a decade to realize that "if you're the only person that 'gives a shit,' YOU'RE GONNA HAVE A BAD TIME."
Not only that (which is 100% true) but also you're raising the bar in required work quality for everyone else to get on par with you, therefore expect to make a few friends and many enemies, especially among colleagues.
The older I get, the less I'm willing to help disaster scenarios.
"A lack of planning on your behalf, does not constitute an emergency upon mine."
Agreed. Imo, having "on call" rotations is a clear red flag and bad software practice that has somehow made it in the mainstream.
I understand the idea of "mission critical" when you're running software on the Moon or something, but if your Earthbound software has so many potential bugs that you need to have dedicated people being on call every weekend to fix bugs or restart servers, you just built it poorly.
Most software engineers just happen to be bad engineers. On-call rotations are a band-aid for poor planning and development (to be fair, often imposed by tech-delinquent middle managers or executives).
Let’s all plan our emergencies to 8am to 5pm, Monday to Friday. And don’t forget the scheduled lunch break at 1pm!
It just requires spending more money hiring more people.
In the old days operations tended to be very isolated in much the way you are proposing. The problem with this is that stability depends very much on the software, so over time operations folks would be extremely defensive and impose all kinds of constraints on what software could do, and the software engineers would be frustrated that they couldn't do things efficiently. Imagine how firefighters would feel if construction workers had a tendency to randomly leave explosives and gas cans hidden throughout new construction and then waltzed off to the next job while the firefighters had to deal with the consequences.
At the end of the day, devs need to have some skin in the game or it's a recipe for disaster.
In mature industries, there absolutely are plenty of regulations in place to make sure that builders don't make responders' life harder. That doesn't mean that the responders aren't needed, but the fact that the software industry as a whole decided to go all "response is the only thing we need for most things" is evidence that it is not mature.
The nature of software and physical construction is different.
But, as we know, useful software interfaces are difficult to define well and, once they exist, they tend to be the most inflexible part of a fast-changing system. It is always better (though of course more expensive) to control both sides of an interface for this reason.
The "skin in the game" argument elides this fundamental reason and substitutes one that implies all of this is the fault of lazy devs, which isn't (generally) true IME.
Edit: I missed the part where you say the Heroku customer has their own on-call team. IME this is not true. The whole reason to use a PaaS like this is to avoid having an Ops team. Sometimes orgs outgrow their PaaS and keep using it anyway, but this isn't necessary, only a historical artifact. These orgs would likely save more money and get a better result by going with normal IaaS or racking their own servers and, in fact, are probably actively switching to that model.
Oncall rotations are part of defense-in-depth against bugs and unforeseen circumstances: Most of the companies that survive without a formal one only do so by outsourcing this for the most common cases; to Cloudflare, to Amazon, etc. -- if there's an opportunity cost to being down someone needs to be able to pick up the phone when there's an outage or critical issue.
Nobody engineers software like spacecraft companies- that doesn’t mean that no one else gives a shit, it just means that cost constraints are real things.
Also, the demands on spacecraft software are trivial (“move the camera once a month”, “do a correction burn after a planetary encounter”, “watch this sensor and do this if it changes”) compared to a modern web application at a Fortune 500 company.
I have worked at firms of various sizes and there is always a point where things usually become complex at some point. Software is as much or more so about managing the humans than it is the software. This becomes especially true as the firm grows in size. Like all fields there are certainly some individuals that perform better/worse than others but even for the best engineers out there, mistakes happen, edge cases pop up especially as the potential complexity grows. Of course these mistakes can pop up more frequently depending on the imposed deadlines. Deadlines to me are a healthy balancing act between the different parts of the business. Sometimes they are arbitrary but I think in a healthy relationship it helps to have that pushback/friction to figure out how much effort is required.
That was a long way of saying I think its a pretty naive and dismissive view to just hand wave and say this is both due to bad engineers and tech-delinquent middle managers. You are not asking for it either but I think this also comes down to social ability/skills. If your worldview is that most software engineers I can only imagine this shows up in the workplace.
If you have different constraints, you get a different result.
Shipping things has the highest risk of breaking something. I worked with an SRE team responsible for managing incidents for a few months, they told me that ~80% of incidents are caused by bad code being shipped, and I saw that happen as well.
Modern software applications are complex and interconnected. It's pretty easy to unintentionally break something in a different part of the application, or ship subtly bad code because you aren't intimately familiar with that part of the codebase.
When you're in the middle of harvesting and your tractor breaks down, you want it fixed now, not at somebody else's convenience.
Mechanics and tow trucks also form an oncall for broken down cars.
There's oncalls all over the place for tech of all kinds. I think the biggest difference with software is intellectual property - we've made it so nobody else is allowed to fix whatever's broken, so of course, we need oncalls to fix the problems instead of letting customers go to their preferred mechanic
Mechanical watches have been around for roughly 500 years (about 4x-5x tbe time since the first program, depending on how you count), which is a substantial amount of time to iterate on core functionality. Even then, watches until the 1970s (when quartz was introduced) were often imprecise enough to lose 15 minutes/day.
The Voyager probes have both lost several instruments, were built with substantial amounts of redundancy, all to the adjusted for inflation cost of about $3.94B US dollars. Maintenance per year is estimated to be about $5M, including the occasional software update.
Do you think this company had good CI/CD and automated tests? They did not. There was fortunately a lot of monitoring so at least you knew when the service was in a bad state, but absolutely nothing else other than a ticket and an angry customer.
I would much rather have extensive test coverage, very good CI/CD, make sure not to do releases on weekends and holidays, and have a few people whose job is to do the monitoring and escalate to the right people rather than just putting a target on a random engineer's back and hope they can fix things quickly.
[0] https://www.theverge.com/2021/9/27/22696097/hospital-ransomw...
By far, the most likely thing to kill you in a hospital is not the IT system but errors by the doctors and nurses. Or that you're too sick to save no matter what they do.
Everyone being motivated to develop in a way that isn’t resulting in brittle software and breaking and maybe even use boring tech for stability so even if an on call is a real thing, it’s relatively benign. It’s not a bad thing to rely on the genius of smart people who have been at it for decades and are decades ahead in some realizations.
Having software that self reports and logs multiple occurrences of the same errors and escalating errors that start in the app are one great way to stay ahead of issues.
By the time someone reaches out it’s easy to find the error, session, user, and say ok we see it and are on it. Acknowledgement at this depth upfront quite often let’s the customers to say it’s ok take a look on Monday. It’s reassuring. Also easy to forward such an issue to a distributed team.
With that said, there are some types of software that exist to deal with complexity external to itself. External systems (integrations, screen scraping systems, etc.) can break without fault internally, but need to be fixed ASAP.
One might say, "You built it poorly by deciding to build this at all." And sometimes that's true. But I've been in plenty of situations where the options were 1) don't solve the customer problem, don't pass go, don't collect $200. Or, 2) build finicky software that works but needs to be babied... and hire engineers that don't mind doing this type of work.
That last bit is critical. Some engineers like the heroics and drama of it. Some people can't help but run headfirst into the fire... and there's jobs for them there :-)
So lending a hand with emergencies that even could have been avoided is ok to help someone new to it.
Caring via software or infrastructure for users is often about effort as you have described.
"Just throw more humans at it!"
I thought so too until I gigged for a SaaS company that absolutely needed it. No, it was not moon-shot rocket control software, but it was a money printer with literally thousands of clients around the globe paying six figures for the service. The on-call SWE duty was not onerous: about 2 hour shifts every 2 months, and your job was to attempt to handle it and then escalate it to staff if you couldn't. There were so many CI tests on the way to production that nothing ever happened, until it did, and then you needed to make sure that money printer go brrr.
The best on-call rotations are the ones where you’re rarely paged, but when those pages happen they’re for important urgent work where your involvement is vital (even if your role only needs to be channeling the work to the right person). Ideally you’re also compensated for being available, even when nothing happens (my current team gets a day off [in addition to normal PTO] for every week of on-call time).
My experience is that a dedicated team running a follow-the-sun rotation (so the team doesn't need much if any "out of hours" because it's always in hours for at least one of them) actually leads to flakier services and more alerts. Without the visibility of the impact of the flaky services, fixes are lower priority and also more difficult to validate.
In this manner, on-call is not a lack of planning, nor an emergency, but deliberately ensuring we have a plan in place to triage out-of-hours incidents, mitigate those that need mitigating, and leaving anything else to be fixed in-hours.
On the other hand, unpaid on-call is unacceptable.
It can be made more or less direct to great resonance.
“If you want my help, plan for it and understand I’m not able to help last minute when there’s no planning”
Depends on your relationship with the person who planned poorly: if you truly don't care, then you run the risk of becoming known as the jerk engineer who isn't a team player. The reality is that employees are generally expected to cover for each other to provide the (paying) customer with consistent support.
If the party who planned poorly was the customer, you have to decide whether they're paying you enough to scramble for them, and whether you want their accolades or their ire. My M.O. is to support my customers through thick and thin (within reason, of course), which tends to get me repeat business and referrals.
Yes, going out of your way for your customers is good for business. Give a shit.
People who don’t like anyone working more than them or enjoying life more than them are usually haters who are busy not doing much themselves.
Raising the bar doesn’t mean having to make it more work for everyone.
Those doubt worshipping and self-sabotaging enemies aren’t worth having in your life but it is nice to give a bit of shit about them too.
Move at your pace to wearers what you want and you will find like minded people.
In general, people largely filter out the "caring" of what won't directly affect them personally or their tribe in the short term, blind to the longer term repercussions until they are pointed out with concrete details AND actionable options. Someone doing the "right things" for the current situation, either has foresight or prior knowledge. Those that aren't, either have neither, can't learn, or the options' positive net ffects do not outweigh the momentum of "doing nothing."
Some sort of "change management consulting" or something?
----
I'm looking for some sort of intellectual career which will allow me to capitalize on my inherent "glass half empty" perspectives. I used to do residential home inspections [as primary income, post IBEW], but during recent no-inspections home-buying... my services were required less-and-less.
I have enough runway to last about a decade unemcumbered... but got tired of feeling like the only guy on jobsites that cared about not delivering _turds_ to well-paying customers.
Glad I'm not the only one who thinks this. When I was a teenager trying to decide what to do with my life, I dropped professional programming because every developer I knew at the time just didn't give a shit. Sometimes I wonder if I made the right decision. Recently someone posted an article here about disenchantment in the software industry, after reading it and the comments the feelings of doubt quickly went away.
A sample of my "Giving a shit" checklist includes: Do they not utilize metrics or care when they tank? Do the rank and file engineers have a general malase and use terms like "not my problem" with impunity? Does management continually deprioritize critical "keep the lights on" work eg security patches WHEN they are fully aware of the repercussions (this knowing is key.) List of red flags goes on.
No org is perfect, but why endure the pain when it can be avoided. Especially so when it can easily be recognized early enough in many cases.
Recently the private data center was sold to some international holding company, and it brought a smile to my face recollecting a conversation I had once with [now-former] CEO:
"The data center is just a trick pony show we use to sell our real product."
Glad they did well on the sale to new foreign owners. Definitional evil.
One variant is when Management punishes you for bringing something up by making it your task to then fix it. I think they don't do this on purpose, they often just don't think about it. And it leads to a culture of "don't tell the boss". I've seen some really dire problems swept under the carpet because of this.
Head of agency only gives a shit about billable hours, not sustainable revenue.
CTO gives a shit about shiny new tech objects, not how appropriate they are for the news of the client.
Manager gives a shit about over-committing and shipping gawd-awful soltions, and not the integrity and esteem of the team.
Team members give a shit about not making waves, and bad processes and lack of growth (i.e., learning new things) persist.
Designer gives a shit about the aesthetics of the design, not about how it's going to compromise the UX.
And so on.
I've left more than one job because I'm not wired to be complicit (in someone else self-destructive shit giving), and because I give a shit about clients and users, as well as being able to look at myself in the mirror.
This is so tantamount to "living a good life."
I have two attorney brothers, and the things they fight for... jfc.
Hired bullies. Hound dogs. At the mercy of [well-]funding.
"I gave a shit enough to go out of my way to get these, and carry them. Now I am giving one to you because you gave a shit enough to go out of YOUR way. Thank you."
May a real $2.50 some-time-soon cross your path.
Also, I hope most people don’t believe a gift of less than $10 usd is an insult. It’s not nothing.
I figure that's enough of a lost art that someone might actually think "oh cool" and either pin it to the cork board or add to a shoebox of keepsakes.
You should try and look past the literal aspects of things.
Looks like you can get $100 worth of half dollar coins for $147 from the US Mint https://catalog.usmint.gov/kennedy-2023-half-dollar-200-coin...
Pro-er tip: You can also do this with two dollar bills.
Yes, I'm the "full-sized candy guy" because I don't want the neighborhood kids vandalizing / stealing from my vehicles. Yes, I'm that old. Yes, my neighborhood is that sketchy.
A price I'm willing to offer. So far, no burglaries (and I leave the doors unlocked).
(As well, I'm a Canadian but I believe those are unusual currency denominations so another little plus).
I think for a lot of people, even some on the ever bubbly hacker news, amounts less than $10 are still not considered insulting (context depending).
I did the same with silver dollars for a long time,. Gave them as tips eating out but it sucks if your not autistic because everyone is confused/overly excited.
It's worth about $15 but people would think they won the lottery when I gave them a coin. That's why I stopped.
This makes leaving a Franklin half dollar all-the-more rewarding [to an observant recipient]. For reference, a Franklin half dollar is 90% silver. It is also, IMHO, the most-beautifully-surfaced US coin ever produced. It "tings."
----
It is in my fantasies that I had reason to celebrate a grand-enough "night on the town" as to leave just a single golden buffalo coin as payment [about $2000USD].
A $5 would be 2x better
Does that distinction matters to the person who the $2.00 and $0.50 is handed to? That’s an exercise left up to the reader to try.
Addendum: I interpreted the OPs comment to be regarding appreciation of people who work in roles not typically tipped in the course of their service.
You're right that at this point you'd need to be giving out a lot of them each time for it to count for much. Dollars are worth so little that we're beyond "let's get rid of pennies" and approaching "WTF is the point of coins smaller than the quarter?"
[EDIT] That first bit comes off as harsher than I meant. I just mean that this was a more common thing years back, lots of folks of a certain generation (or couple of generations) did it, and at this point I can definitely see how giving out $2 bills, which was once novel (though when the eighth older person did it in a year, not so novel...) and also a decent amount of money, is getting into "yeah, thanks for the two whole quarters, grandpa, I'll be sure to buy a candy bar with them when I get several more" territory, thanks to inflation.
[EDIT EDIT] I'm struggling with internalizing new prices, too, incidentally. I've finally learned that when my kids get $10 it is not worth getting them excited about going to get a toy with it. They'll need, bare minimum, double that amount, and even then there won't be a lot of options after tax. There's almost nothing they can get with $10 but candy or shitty collectible (ahem, kiddy gambling) card crap.
So ~15 years ago our son had been invited to his best friend's birthday party. Unsurprisingly, the birthday boy had a list of electronic toys on his gift list, as had become tradition.
While my wife and I were out shopping for said gift, she located one of the electronic items. Probably a game cart for whatever was popular at the time.
Meanwhile, I had come across something that had stirred up fond memories of my own: a sleeping bag + flashlight (+ some other related item) combo. Woo :)
We decided to get both the game and the sleeping bag combo, just in case the latter wasn't well-received.
At the birthday party, I think literally every other gift was for his gaming system. Once he opened the sleeping bag gift, he, his mom, and at least a few of the other kids had a "wait - okay that's different and kinda cool" reaction.
Even if he never went camping with them, I hope he got many hours of use out of that sleeping bag and flashlight. I know I did with mine.
Oh my goodness. That perfectly describes my last 5 years in academia. The last person on the deck of a sinking ship, directing panicking souls, after everyone smart and agile already jumped in the lifeboats.
But I don't regret it. I helped a lot of kids into lifeboats, and even found a nice lump of driftwood for myself as the university system gave one final creaking groan then plunged into the darkness.
"Nobody wants to be the last one left at the party. You have to help clean up, then..."
We ended our professional relationship over petty words when I felt like the only person giving a shit about maintaining her assets [as her maintenance guy]. Our personal relationship has outlasted any pettiness; but it still hurts seeing her squander her massive inheritance without maintenance.
"Last one out please turn off the lights..." seems a motif not just of the post-war generation, but the whole post-war culture and institutions.
20% of men in this cohort lost their virginity via SWer, typically abroad during/before wartime.
Just out there leaving messes everywhere, like they're paid to do it (or something).
Maybe this is reductive of me, and certainly any $ amount is better than zero $, but I sort of feel like by putting a financial value on your compliment you are devaluing it.
A super heartfelt "Thank you for giving a shit. You truly made my day and inspired me to have faith in humanity. I know your job is hard and noone expects you to go the extra mile, and I genuinely appreciate that you did" in your own words would be meaningful.
A financial tip of significance (e.g. 20$ - several HOURS of worth) would also be meaningful.
That 2-dollar bills and fifty-cent pieces are rare is going to interesting to some, but I think to most the only thing they'll perceive is that you valued their extra mile at $2.50. I don't think that's what you're intending, so I thought I'd offer that alternative perspective.
And then you get to walk out [usually a few days later] with five straps of Jeffersons, feeling like a baller...
This lasts me about one year of "appreciation" for others' "giving a shit"s.
Yes. This is why people stop giving a shit.
I think new people, fresh out of school, mostly give a shit, but slowly figure out that their lives are a lot easier if they don't.