Sometimes it's hard to distinguish resume-driven development from iterative-StackOverflow-driven development.
Sometimes it's hard to distinguish resume-driven development from iterative-StackOverflow-driven development.
The worst thing I've seen is a stack that parses out a file and loads it into a DB. So someone sends us a file via an expensive SFTP+S3 thing in AWS. That is then picked up by some scheduled task using a proprietary in house scheduler process running inside kubernetes. This proceeds to download the file to the local pod. Then it makes tens of thousands of API calls to match up data which cranks the CPU up on a huge database server. This breaks all the other jobs running. Then it writes another file out to S3, consuming 17GB of RAM in the process. Another process picks that up and then batches it and inserts it into the DB with no transactional stuff around it.
The original process this replaced was a copy into a temporary table and then a bit of transaction-wrapped SQL that took about 20 seconds to import + run. They improved that to 7 hours and reduced the success rate from 100% to about 80%
So far I'm feeling pretty good about what I've developed over the years.
Literally instead of a table.
This stuff is everywhere. Microservices made it worse and half legitimized it.
There's a service that watches the transaction log of your regional replica so that you can make long-poll HTTP requests that return when any change matching your filter is committed. (Edit: usually the HTTP result handler is used to invalidate specific memoized results in the data flow graph, letting lazy re-evaluation re-fetch the database records as needed.)
It makes a lot of sense for a financial risk system, where you end up calculating millions of slight variations on a scenario. The data flow model with aggressive memoization makes this sort of thing much cheaper.
However, I saw plenty of systems written where you'd attempt to write your request to the next key matching some regex (and retry with the next key if it already existed), where your request would contain some parameters and the database key and/or filesystem path where results should be written.
Under-experience with databases easily results in rewriting a database using message queue/bus. Under-experience with message queues/busses easily results in rewriting a message queue/bus using a database.
Database are still fairly poorly documented when it comes to administrative work.
There is an incredible amount of tutorial, books and courses on how to write sql queries and stuff... But there is almost zero content on how to properly administer a database.
I mean, from novice admin to DBA-level capabilities.
I said all this before and i'm ready to write this again: i think there's a good market space for dba-style courses.
The most unpleasant codebases I've dealt with are ones that have suffered from a lack of strong leadership, and they are almost uniquely microservice setups that pull in everything but the kitchen sink, usually because it's just trendy to use it. Monoliths can get pretty damn ugly too but at least it's contained in one single codebase.
It went like:
App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka
The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.
Just two examples from my prior gig (fashion e-commerce).
#1 Our hottest dataset (db of current products) stored in DynamoDB. Core dependency for all our code. Easily fits in < 1Gb of RAM. OMG, just make a hashmap. Over a year, I managed to persuade the team to start transition from DynamoDB to Redis.
#2 Tiny (vs micro) service that munged some URLs. Blocker for an important campaign. Prior team of 4 churned for a year, was no closer to delivery. Spring, ORMs, CI/CD pipelines, the works. I spent a week unraveling the requirements (repeated facepalm). A second week banging out a trivial nodejs thing. (My team preferred nodejs, which was their prerogative.) Really trivial. I felt so bad for the biz dev people who'd been dying to get this functionality for so long.
Anecdotally, optimization tasks (in this brain) are multitudes easier than innovation tasks. I spend a lot of time thinking about how to do things differently whereas optimization utilizes many lessons I've learned over and over again with well-trodden patterns. That's to say, I'm grateful for the doers :)
Bringing what used to be the privilege of upper management (wasting massive amounts of resources while getting paid handsomely) down to software developers.
It's that trickle-down effect people talked about, right?
"I dont think you are fitting in here at MegaCorp"
"But we have no users at all right now"
"But we might have hundreds of millions of users in future"
I want to see more lines of code, not less.
If you do not improve next month we are going to have to let you go, we just cant carry a 0.3x employee such as yourself.
Where?
Dare I say, any time I see a function as a service my brain immediately drifts to inspecting the cost implications of said process.
I said the above as a jest, but seriously, simplification of complex stacks has been a good consulting gig.
I don't care any more. I'm just there to tell people what's shit and then laugh when it explodes in their face.
What makes someone a staff+ is finding a path to iteratively evolving towards that end-state without breaking anything along the way and while having progress to show off at each step.
But hey, they can run some docker and git commands and piddle around an AWS GUI, which means they are highly technical.
When developing anything:
1) you don't get to touch anyone else's code. And another department's code? Something another manager's team manages ... that amounts to treason. Never for any reason. MAYBE if they've totally abandoned it and you absolutely need it (but only during unpaid overtime)
2) you don't get to spend ANY time on anything outside of the current project or JIRA ticket. Any time at all. So really, NOT optimizing anything is faster and cheaper. Just look at all the spreadsheets made!
I’ve had enough calls with the “Senior/Technical Lead Azure Cloud Engineers” telling them exactly what they need to do that me and them really really don’t get along.
I don’t do any of that shit and even I can muddy my way through it, but these people cannot. The real kicker of it is how much these people make.
And you know how there are those people who, every time you need to work with them, they answer a teams call and then “need to get to my computer, give me 5” and their status is perpetually set to away? I don’t want to RTO at all, but dealing with this team almost makes me think I’m wrong about that.
That's on the industry for not training and gating well. It would be nice to have glue/plumber positions so expectations are not out of line too.
Agree solidly on the gating aspect though.
The problem is that quality hires continue to be rare and at some level you are doing area-under-the-curve reasoning.
They're not usually software engineers. They're tool users not tool makers.
So they'll cobble things together to accomplish the task, using only available tools and never anything custom that would do it task much more cleanly, because they understand data, not software. They're not computer scientists or programmers, they're just users. And we all know what that means.
I've spent the majority of my career building better tools for data-related tasks, then winning over my users by showing off performance and productivity gains.
Those replacement systems aren't even working, they're failing to deliver on the same simple data pipelines that we had working by the start of 2020. They're cobbled together using a million little AWS pieces and Docker and k8s... I'm glad that I left that role when I did, we were being pushed by a new-hire with a fancy Data Engineering VP title to do all sorts of asinine things. I went and looked just now and I see that he's Senior VP at a different company, he started there this summer. Onward and upward!
And I thought my unholy xmllint -xpath (bad stuff, lots of slashes) ${1} |sed -r -e s/this/that/ -e s/alsothis/alsothat/ -e /ohyeahthistoo/somethingelse/ | grep something | while read AA; do stuff then echo ${COUNTRY},${SIGN}$(perl -e "printf('%.2f', ${VAR}/1000000)"),${ENTRYDATE}; done|sort
was as bad as things get. I need to get my horror code game up. I mean, not only is the code awful, its very purpose is horrifying (XML to CSV with some transformations, bit of math, all without being able to use any external sources due to security, only what's in a baseline RHEL7 (soon 8, yay!) ).
I promise I'll rewrite it in python at some point.
I'd bet there are a 100 different answers to your question though. This is the way we handle it.
If that doesn’t exist yet (it usually doesn’t), we test manually, but only core workflows.
This is why I hate recruiters, I can't even tell you how many times I've had a recruiter call me saying they are looking for service XYZ. The same concept rephrased in my resume. I have to rewrite my resume just to satisfy these people? No thanks.
Never heard that before, but that’s so on point.