Get me out of data hell
ludic.mataroa.blog
ludic.mataroa.blog
Like why didn't anyone catch the issue with the logs? Because it doesn't matter, every data team is a cost-centre that unscrupulous managers use to launch their careers by saying they're big on AI. So nothing works, no-one cares it doesn't work, most the data engineers are incapable of coding fizzbuzz but it doesn't matter.
People always wonder why banks etc. use old mainframes. There's like a 0% success rate for new data projects. And that 0% includes projects which had launch parties etc. but no-one ever used the data or noticed how broken it was. I don't think a lot of orgs which use data as core-infra could modernize, the industry is just so broken at this point I don't think we can do what we did 30 years ago.
For many organizations, the success rate is indeed 0%. A Group of Eight university (our top 8 universities nationwide), for example, sent me a job description a few months ago where they misspelled the word engineer, and left change tracking on in the Word document. This allowed me to walk through the profiles of the people running their data projects, and it was super obvious that many of the people involved aren't going to do a good job. They could have saved millions by having a random HN person eyeball the CVs of their chosen leadership team.
i think it all goes deeper in overall culture/attitude there.
i was in Melbourne in 2012.. with idea to relocate wholesale, 2nd time. Worked 2 months at some "startup", that fired me when i finished the task given.. Seems it was cheaper to hire "permanent" then fire, rather than take someone on 2 months contract. So that's one red light on the dashboard.. There were other redlights from overall "society", feeling something-is-wrong but i did ignore them for quite a while - people have become evil, etc..
Then i started going around places and mailing my cv here or there (with 22y of experience making software, by that time),.. ibm, ernst&young, you-name-it.. to no avail, and more red-lights flashed on me.. And one day visited some kind of meetup, organised/held in some wellknown company. Seemingly it was kind-of "hiring" event or so, we grouped in teams of 3-4-5 people, with half from company, and other half outsiders.. and went solving some problems of theirs. Or that was the "label". Any solution that any of outsiders suggested, was shot down, with somewhat vague reasons, that at the end started to sound like "if we solve this there'll be no job tomorrow". And Smile :) Lots of smiles. Empty ones.
That was one of the Last red lights on my dashboard. Whether it was a financial balloon pressing everyone so they only smiled and did nothing, in order to pay the mortgage, or something else, i don't know. Next day i watched Sacrifice/1986/Offret by Tarkovsky, and.. bought a ticket out. Discontinued my oz dream. For good.
quoting meself, from 2007-8: "with time, places change people. Other way happens noticeably only while coming in - or switching on."
have fun
I know startups that hired data engineers, deployed warehouses,DBT, a BI tool and churned hundreds of reports, and in one case their DBT project has hundreds of files. No one in that company knew why any of it was used.
All said and done the business users wanted three reports.
More often than not data teams are self-serving than anything else.
For a lot of smaller Internet companies, data is just a fluffer. The real business is in image and which VC bbq you get invited to.
i think
Despite no shortage of technical talent on large projects they can still often fail, and it’s because building a technically impressive thing doesn’t matter if it doesn’t do what business needs.
So it’s about making sure you’re building the “right” thing that delivers on business’s actual needs, and the only way to find out what those are is through constant and ongoing good communication between technical and business people.
The kind of communication that lets cross-functional projects be effective is time consuming, and competent people tend to be overworked, no matter what part of the business they’re in.
As time moves on, there are less COBOL engineers. Hell, sometimes their systems have been written in a bespoke language. There is less and less understanding of why something is set up the way it is due to lack of documentation. Updates / changes to the code sometimes have to wait for 2-3 years because the system isn’t flexible enough (literally, not as in “this change will take 2-3 dev years”). Even code that old contains bugs, but due to the age of the code they’re inscrutable.
However, whichever new system gets tooled up has to be 99.999% flawless, or it could cause serious damage to the bank and even its regional market.
When there is that kind of pressure, dev teams are no longer considered a cost sink, money flows, and the world is possible.
Not seen/heard of this person before but reading this specific blog post it all sounds very familiar, it's depressing.
The "CTO" getting on stage taking a bunch of credit and everything being a mess or incomplete or lies is very familiar. Maybe not CTO but higher management. It's all smoke, mirrors, optics, self promoting, it works as these people end up making their way up the ladder when the lonely dev trying to do better work is just a dispensable cog in the wheel.
Someone once told me he, as a form of therapy, rewrote the company he worked at in a few weekends. He never mentioned it to his coworkers, it was strictly a therapeutic effort. They apparently spend years "fixing" things without making any progress.
Success attracts deadweight who (un)intentionally sandbag efforts to reverse this downward trend for their own self-preservation. I don't blame them, doubt there's a fix when the system requires most people work bullshit jobs instead of collecting UBI.
Something that takes a week in company 1 can take a year in company 2 purely because of organizational issues.
Rotting organizations will produce rotting software.
I argue this is a happy conclusion, not a problem to be solved.
What would "modern" bring to a bank except even more pain & suffering? Database technology invented in the 80-90s is more than sufficient for tracking information at the scale that 99% of financial institutions operate at today.
Virtually every core conversion project I've ever heard of has been a failure or is currently a burning wreck on its way to the bottom.
The only new bank projects that touch data and seem to succeed are LOB apps with highly curated experiences that are tightly integrated with the actual front/back office business. Having buy-in from staff regarding your UX is way more important than spinning out a 20 page AWS architecture diagram. The CTO can only take you so far through the vendor approval process at a bank. Retail operations (i.e. the people who are responsible for the brick & mortar branches) typically have substantially more pull in these organizations.
In the most simple term, a future.
Except if your bank is literally too big to fail, at some point you have to either move on from 80s technology or at least bring in an adaptation layer, because your profit center have also moved on or you're facing harder competition.
A typical example is banks getting merged: there will be a fight to see which system stays and which one disappears. If you froze your technology 4 decades ago it won't be your stack winning. [0]
Another is the evolution legal frameworks: EU countries passed laws requiring interoperable APIs to perform standard banking operations. Being a customer of a decent bank or a fossilized one made a huge difference and the market grew a lot more competitive. People would start hedging their bets when legacy banks looked too far behind.
[0] The most interesting and recent example of this is Mizuho bank just miserably failing at that task to the point the gov. intervened and anyone not married to them probably moved out.
https://www.mizuhogroup.com/news/2021/06/20210615_2release_e...
In my experience (small/mid-size US banks), the institution with more assets or branches usually wins. It rarely has anything to do with technology. If a 6 region, 200 branch monster comes in and wants to buy some 4 branch relic in the West Texas desert, it doesn't matter if the smaller institution has achieved AGI and an intergalactic core platform. They're almost inevitably gonna be merging their records into some old boring IBM system.
Almost as many people pay with card as with phone.
Faster record systems, faster transfers, actually do win people over here.
[0] https://www.rba.gov.au/publications/bulletin/2023/jun/cash-u...
If they're doing it for more strategic purposes, the calculation becomes more complex and there will be more "reverse" acquisitions where the entity closer to the target is prioritized.
I’m unclear whether this is bad for the business or just bad for folks hoping to keep their jobs.
And also partly because it's the third biggest bank of Japan and it wouldn't be allowed to not be doing fine ("too big to fail" doesn't even start to describe the impact of a group this size starting to go down)
Do they do "great" ? Arguably no. They shut down a number of consumer facing locations, had a hard time recruiting, and compared to Mitsubishi and Mitsui the gap has kept widening. In their small/middle customer businesses they're starting to face the rise of GMO and Rakuten where the two other are too far ahead to even need to care about it.
The last year I can find figures for has Mitsubishi's assets -5.46%, SM -4.14%, and Mizuho -2.03% (so yes a decline in absolute terms, but that's the best performance in the top 10, and sounds like closing the gap with Mitsubishi and SM rather than widening it). I can't find a branch count but Mizuho has vastly more ATMs (around 4x as many as Mitsubishi) and that number is increasing. They've continued to make consumer-facing improvements recently like their wallet app allowing electronic money payment directly from you bank account, and English support in their main app. Of course like all of the big banks they're facing competition from the rise of the net banks, but as far as I can see they're doing as well as any of the big four, perhaps better.
One particular thing was we had to convert transit #s into branch numbers regularly. We did this by looking at a sheet of paper of course. Eventually I got fed up and wrote a web app so you could just punch in the numbers and have it instantly convert. I checked and people are still using it 10 years after I quit, which means nothing has changed and they're still using the same god awful software.
They did move some data at some point. I know this because they screwed that up too and partially merged my mom's and my bank accounts, which is a pretty bad error. Would be worse if it was some rando. Speaking of... That's exactly what AT&T did.
It probably depends on what "modern" means here. If updating from tons of COBOL to {Julia, Python, Rust, or some other well known language} with an update to an SQLite backend (or perhaps postgres is acceptable for very specific scenarios), that is likely a good choice due to being able to fix old cruft and add maintainability for the future. If it's a switch to some nosql database backend with everything switched to some cypher-based lang or anything that touches javascript in any way, it's probably a mistake.
(Not that we haven't gained a lot from modern database tools, just something to think about that maybe the data siloes were good sometimes, too.)
The team consisted of one guy (who had a business degree) and a lot of empty cubes they were trying to fill. A year later the company had been acquired and the big data initiative had evaporated.
IMO a major factor here is that software engineering is both opaque and esoteric - at least with physical engineering, there’s something people can look at and think they understand.
A lot of data projects are moving and transforming data no-one cares about. They can fail completely silently, a manager can lie and say 'we've successfully built the data platform which is going to enable AI analytics' and it'll be like a misconfigured S3 or something. No-one's checking the end-product or even understands what it's meant to be.
1. Accurate, auditable billing down to per-chargable entity/event
2. Dashboards for each customer that reflect THE SAME numbers as we generate in #1, so customers could see relevant info quicker than just waiting for the bill from #1
The only reason those things were valued and made a focus though was because a HUGE customer threatened to completely drop our company because that customer did an audit and noticed that we had overcharged them 3% because we were actually billing them on estimated numbers. That led to our CEO being personally yelled at by a much larger CEO, and our CEO (to his credit) didn't blame us (we'd raised the alarms that the bills were estimates and not auditable) but did say "this can never happen again, I trust you, do whatever needs to happen to make sure this never happens again."
And once we had #1 solid and tight, we were able to leverage that solid auditable data to generate solid dashboard numbers that always squared with what showed on the bill.
You needn't live in Australia to reach that conclusion.
E.g. if you are a bank ideally you'd like all your processes automated and streamlined with extremly transparent data flows etc — and you want as many of the banks employees to be proficient in these systems and constantly work on improving the systems within an controlled environment.
In practise this is not the kind of thing that allows single managers to come across like heroes — so it doesn't happen that way and you get island solutions with duct-taped connections between.
I also get invitations to "sponsor events" now, since people see "director" on LinkedIn and think I have way more money than I do. Their business model seems to be flattering executives by inviting them to events where they can network with other rich people, then ask me for "sponsorship" money so that I can go into the room and brainwash them with my marketing material. I might even try it at some point to see if that's an accurate read.
So, 43 years later and Putt's law is alive and well.
<https://en.wikipedia.org/wiki/Putt%27s_Law_and_the_Successfu...>
In other words, it's about status, not money.
Most teams and businesses I know who are doing great things with data are mostly smaller companies and tech built up start-ups and hire one of two types of people. Generalists who can fill the gaps and are interested in everything, and staff software engineers who are amazing developers and asked to put together data pipelines with consideration of the whole infrastructure.
When I was looking for work, all the good companies were not hiring data engineers or machine learning engineers. They were hiring Senior Software Engineers with a remit to build data infrastructure, or build machine learning models. And they immediately removes 90% of the noise applicants.
Some places in Melbourne get real work done, even in the data sector. They're hard to find, but they exist.
Yes. Lots of times the most important asset for these companies is actually contractual obligations in terms of exclusive access to data or customers. It doesn’t matter if the product works, you’ll have to buy the company to build a different one that does. But the (broken or nonsensical) product pushes up the value of mergers and acquisitions. If leadership completely makes shit up then they might go to jail, so, they burn X million on “work” and cloud spend as part of an elaborate argument that it should sell for 10X.
> the industry is just so broken at this point I don't think we can do what we did 30 years ago.
Well no, it’s never been easier to do high quality engineering, but mbas are in charge. They don’t think like philosophers or scientists and don’t traffic in common sense.
For anyone questioning their life / career choices because of this, it’s not about you. An individual working in an environment like this can still be a craftsman of integrity if they focus on small problems and solve them well, but you need to be able to get satisfaction from that, not from some overall mission (which again, is probably fake). If you’re most motivated to work directly on architecture, unification, etc, and want to change lots of things then you will probably be miserable.
But if you’re feeling shitty about the whole thing, it might help to realize that the actual nuts and bolts of adtech/martech data pipelines are much the same as the ones for cancer research or particle physics or climate science, so one can at least try and get transferable skills if circumstances are currently holding you hostage. Data isn’t a bullshit job. Leadership and management that just want to play games is the problem.
At least it has tests.
If you have lots of data flowing and have full teams "working" 24/7 on it, does it really matter if that data is junk and that is not processed in a meaningful way? You can still ask AI to generate some nice looking charts with big numbers to show to investors. Investors like nice charts and big numbers. Or so, some businesses people think.
But in all reality the investors will ask questions like: how will this solve problems for customers, how do you intend to sell this to customers, how much does it cost, how will this generate me money. Unless those investors plan an early exit by finding other, more gullible investors than them, kind of like knowingly investing early in a Ponzi scheme.
If I had to summarize though, it's that the median performance in any field will be at much lower levels than outsiders expect, and some fields with hazier results have this level set very, very, very low, especially when they're hyped up. But also that the market is actually at least a little bit efficient, but over long time scales. I think there's a 50%+ chance that the role of Chief Data Officer begins to die off, but also that it'll be replaced by something silly.
But I am SO triggered by this piece. I had that intrusive feeling you sometimes get when driving where you think, "I could just close my eyes and see what happens", "Or that clif is so close and the guardrail doesn't really extend far enough"
Only for my career. Like I should just not show up on Monday. I should get in the car and drive far away and change my name and work at a nice retail joint in a mid-sized town.
I'm going to need to sit and stare into the distance for an hour and 3.
Except that all our other senior engineers got laid off and there's nobody to pair with, I don't give two fucks about bullying because at this point the entire company knows I'll quit on the spot if they try, and our problems are mostly that the remaining team cannot understand the terrifying eldritch decision making process that led to fun little patterns like "wrap every API call in a try/catch and then ignore the errors".
I am seriously considering doing a TAFE course and becoming an electrician.
This describes the project I'm working on at $job with eerie accuracy :sob:
Dare I ask, was that project written in Java?
I am so very sorry, both that you are suffering through this too and that the infection has spread.
I am currently dealing with a system involving four separate serverless functions that call each other. There's no reason whatsoever why any of them need to be network calls. The fourth function just calls the first function again. One is in a different region for no discernible reason.
From his post about burnout and mental health. Also worth a read.
https://ludic.mataroa.blog/blog/on-burnout-mental-health-and...
Any day now I'll be ready for the grind again. Any day now.
sounds a lot more like learned hopelessness making it harder to respond to stress with radical change because of (normal and human) fears of the unknown.
at some point though responsibility for the circumstances, the feelings, the stress -- the good, bad, and ugly or easy, hard, and nearly impossible -- has to be taken.
there's only one life to live. we owe it to ourselves and others to do more than -- to try not to -- just "roll over and play dead", so to speak.
humans have survived a lot and have adapted to just as much if not more.
if i ever allowed myself to even stay at any of my former jobs coming up in my life when i was paycheck to paycheck because of not making rent or just being flatout broke and homeless, i would have not progressed my career, or life, in any meaningful way, and just fed the negative feedback loop influencing what feels like a miserable existence (even privileged as it were).
can't hold myself hostage. and also, i can't hold those around me hostage as consequence of my non-action, either.
L’appel du vide
Example: a blog post like this one, with the author's real name, that acknowleges it front and center: https://ebb-and-flow.blog/2023/07/23/another-scan-lappel-du-...
There’s a polish restaurant near where I live that makes amazing food. The owner is always out and about, chatting with customers and making sly jokes. Turns out he used to be an oracle sql consultant of some sort, and he turned it all in to run his restaurant. You can tell he’s thriving. I think he’s got the right idea.
Not saying you shouldn't quit, just that it's not so simple.
Money is money, but money comes and goes.
The work you put into finding a healthy source of income is worth every minute.
You deserve to feel good. Life is too short to be a cog in a broken machine.
Oof, that hits a little close to home.
After a person I liaised with on another team left, I asked his superior if there was someone else I should build bridges with. We started talking about one of the team members and he said, “I don’t want you to talk to him. We like him, and if you talk to him he’ll leave.”
This was on Slack so I don’t know if this was a jest or he was serious/mad. But it’s entirely true. I’ve convinced at least half a dozen people that we should expect better from a team environment and ourselves, and that this org (not the whole company, just this division) is a cult of stupidity.
I was trying to recruit collaborators to fix the bullshit but apparently they decided it would be much easier to just start over.
I was lucky enough to get ~6 years running my own tech company after 6 years as a waiter. Then I sold it, yadda yadda, went to Google 6 months later, got ~7 years in there.
It really, really, really, disturbed me how approximately every situation, in every division, with any people, ended up being boiling down to "how do we muddle through one more day without challenging anyones preconceptions", 95% of the time it was tribal antisocial stuff, and no one would speak up about it.
Direct example, for posterity.
I don't wanna speak too directly to it, so lets imagine Google Division A (hereafter, dApps).
New division lead (ex-dApps) joins dBytes with apparent bias against partnering division (dConsumer). Despite the project being previously framed as top priority, new lead consistently undermines dConsumer in meetings and shows little interest in understanding their work. Team adopts leader's negative attitude, becoming obstructive and uncooperative. I ended up carrying a critical launch, virtually alone, for 6 months. At performance review time, my boss questions why I didn't get more team involvement - despite the hostile environment that prevented exactly that - and speaks glowingly about how we need to support peer going for promotion based on their excellent job on part Y...which they didn't do. They spent 2 days on it then said it was impossible. And they were definitively the most cooperative because at least they tried, and wouldn't actively be aggressive in meetings with the outgroup.
Everything, always, came down to: A) don't cause conflict at all, at home, or you will be buried B) we'll bend over backwards to accomodate conflict you invent, as long as we can clearly define them as an out-group with 0 ability to affect us day to day.
But it’s trickier to coax people still on the fence to come out for multiple coffees.
Exactly why I burned-out at work, worked at most 2 hours per day on a good day and finally was ejected from the project after a PM that graduated last year from school noticed and went after my head. Author is a wizard for describing the situation this well.
It's been 3 days I've been free from the tyranny of Jira and project managers, and I worked more on my personal projets than I did in a week at my former workplace.
A beautiful epigram.
But on the other, the same sentence could be written about software deployed to traditional servers. "Because of course, how can you hurt yourself without the joys of badly configured servers?".
This affects amateurs just as much (if not more) as experienced chefs.
I don't think the analogy is very good, since juggling katanas is always a crazy idea, while choosing whether to go serverless or not is always a respectable discussion.
You haven’t worked in conservative industries I take it? Late adopters, every one.
OP is still trying to replace Cobol. I know an insurance company that started that process 15 years ago. Fifteen years.
I just pointed out that personally I would not consider Lambda - which has been a stable and popular technology for 10 years - to be cutting-edge. It's not old but also not cutting-edge imo. I would reserve that term to newer technology. Apparently a controversial view on HN, which is interesting.
To respond to your question, I did work for a bank in 2017 with moving certain burst-type processing to a set of Lambdas.
The cutting edge bit is a nice quip, though I agree not exactly accurate anymore.
All of it could probably been handled by larger node application in docker container somewhere but AUTO SCALING, FAILOVER, SERVERLESS!
Once I started as SRE for a new team, we built a larger monolith using Node and docker on EC2. We would get massive complements for our uptime and reliability but there were some architects extremely unhappy when I revealed in division presentation that it was just Docker + m4.xlarge running Ubuntu 18.04. When I left, more and more Lambdas were being broken down into docker running on EC2. They are probably on some container managed solution now.
Or maybe I am too old with this shit. Still haven't found a use for "serverless" knives.
My day job is writing a desktop/file-based ETL system. I have just added in a schema version feature to cover these sort of issues. It was one of the most requested features, because most people aren't able to control the schemas of the data they receive.
The latest generation of LLMs are pretty, actually very, good at this kind of situation, where somebody has renamed something - but kept some semblance of the meaning - and also moved it so a basic, or even a fuzzy, comparison might not be able to make a good match.
But a model like GPT-4o-mini will eat a problem like this for breakfast, and it's now incredibly cheap to use it for this kind of thing as well.
And it's almost impossible to get them to stop: the hostname should either be a random UUID or a random name from a pronounceable list depending on scale (or a syllabic UUID thing).
Because every other factor has one answer: you look up the other data you need in your CMDB. If that's too hard, you fix that so it's easy (DNS TXT records can be surprisingly useful here).
Oh, then I'm asked to "give this a code review before so-and-so does a deployment tomorrow." Uh, it's a little late to address any of the fundamentals, but there are hard coded paths everywhere...
Once your data is safely inside the database (temporary load tables or otherwise), there really isn't a good excuse for pulling it out and playing a bunch of circus tricks on it. Moving and transforming data within the RDBMS is infinitely more reliable than doing it with external tooling. Your ETL code should be entirely about getting the data safely into the RDBMS. It shouldn't even be responsible for testing new/deleted/modified records. You really want to use SQL for that.
You'll also be able to recruit more help if everything is neatly contained within the SQL tooling. In my scenario, business analysts can look at the merge commands and quickly iterate on the data pipeline if certain customers have weird quirks. They cannot do the same with some elaborate set of codebases, microservices, etc.
One specific thing that really sold me on this path was seeing how CTEs and views can make the T part of ETL 10000000x easier than even the fanciest code helpers like LINQ.
SQL server is where this breaks though. You'll get yelled by DBAs for bad db practices like storing wide text fields without casting them to varchar(32) or varchar(12), primary keys on strings or no indexes at all, and most importantly taking majority of storage on the db host for tbese raw dumps. SQL Server and any traditional database scales by adding machines, so you end up paying compute costs for your storage.
If you use a shared disk system with decoupled compute scaling from storage, then your system is the way to go. Ideally these days dump your files into a file storage like s3 and slap a table abstraction over it with some catalog and now you have 100x less storage costs and about 5-10x increased compute power with things like duckdb. Happy data engineering!
Five varchar(12) columns is more storage overhead than one varchar(60). There's a lot of great use cases for varchar(MAX) and everyone I ever had tell me that varchar(MAX) wasn't allowed didn't understand the internals of DB storage that they thought they did and somehow still believe in their internal model of the DB that varchar is just spicy char and fixed column size allocation.
In other news, I haven't seen a dedicated "DBA" at a company in over a decade.
Yeah, there's still the very rare need to performance engineer out a fixed char field "to the left" of the table to speed up common table scans, but also so many of the reasons you might table scan strings have moved into proper full text search indexes or now all the rage is in vector embeddings.
> In other news, I haven't seen a dedicated "DBA" at a company in over a decade.
Yeah, anecdotally from LinkedIn and other sources it does seem like all the dedicated DBAs that have stayed that way have stuck to very specific niches and/or Oracle Products (including MySQL and derivatives these days; the "Oracle Effect" is strong). Especially in Amazon RDS and Azure SQL Server/Cosmos DB today, Postgres and Microsoft's SQL Server mostly run themselves and day-to-day administration is minor/trivial.
It’s amazingly productive and incredibly cheap to operate.
I’ve seen it mean anything from “distributed computing expert” to “knows SQL”
If by that you mean, “knows the commands to create, fill, and select from a table,” then yes. If you mean, “knows how to create a performant schema and queries that will serve them well into the future,” then no, absolutely not.
OTOH, IME data folk are much more cheerful and willing to change things than devs when I point out the innumerable ways their DB choices are choking them. Devs more often fall on the side of “we don’t have time for that on our roadmap; can’t you just fix it?”
There are still ways to go. I've seen it means "spends the days filling excel spreadsheets".
Ah, clearly you’ve worked with a better class of data scientists than I.
It is one of the few types of jobs I have worked were someone credulously offering adding five more layers to fix an issue with latency is a normal operating procedure though.
The author’s experience is not far off from my own.
1. Any solution in place can only be understood by the person who created it
2. ”No, we can’t change that because then we’d have to validate everything from scratch again”
And therefore, as the author says:
> ”we'll continue with the work instead of fixing the critical production error”
I’m honestly not sure how to address it either. With traditional software dev we’d write tests, incorporate those into CI/CD, and start to course correct. We can use sample data to validate the code does what we think it does and that we didn’t break it.
But in these data projects, it’s not only the code that’s changing, but the data is also a moving target. You can write a test with sample data, but tomorrow your data might change because someone in sales added a custom field to the CRM, or IT upgraded the accounting software and all of the unique IDs changed, or someone upgraded their Excel version, or whatever.
And your code that works on the sample data needs to handle all of this, which obviously it can’t. You can try to validate the data somehow, check the schema, check if the number of rows hasn’t doubled or halved, and so forth, and then stop it from importing until you look into it, but also you can’t stop inbound data because an exec has a meeting in a few hours and expects their report to be updated.
I heard something about “data contracts” that’s supposed to address this, but it sounds like the next in a long line of buzz words intended to get management to buy another data product.
Has anyone worked in this kind of project that went well?
The thing that is hard at big organizations isn't that executives need the data for meetings. The issue right now is that many organizations are already 3-4 years into building their analytics platforms, and Chief Data Officers worldwide are trying to prevent their role from disappearing. They're already very much "Mom, we have CTO at home" in many companies, as evidenced by the fact they're usually reporting to the CTO or CFO.
So at this stage, they've already told the business that the platform is "ready", and they are onboarding data sources. With no way to measure data quality, the only thing visible at the organization level is number of data sources onboarded. The fastest way to onboard data sources is to have good CI/CD and a solid developer environment, but this would probably result in slowdown for 1 - 2 months even if you had executive backing to bulldoze all objections from the IT department.
That's the sort of thing I can commit my team to as a business owner, but most executives don't have the nerve to slow delivery down and aren't losing money out of their pocket due to the inefficiency - I get to talk with a lot of them due to the blog's success these days, and many of them really are just employees with more status, with the same incentives. And to make it worse, the loss of nerve is actually understandable, because the type of team that would build something this bad will also waste those two months then still deliver slowly! But most people aren't thinking in terms this complex, and yes, I know it isn't that complex.
I'm expecting to pick up some work in this area at larger orgs in a few years when these leaders rotate out and new leaders rotate in and go "what the hell IS this?", but for now we're mostly aimed at helping smaller places do it right from day one.
We're gradually forming our own, complex opinions in this. In the consulting context, this is essentially our product. A fascinating realization moving from software to marketing is that a sales pitch or marketing strategy can be built in a way that isn't entirely dissimilar to code, and that it has second-order effects. They aren't the same because... they aren't the same, but there's an artistry combined with principles to doing it "right".
And then as a consultant there's additional complexity, as each team is different. Some are high-performing and need a bit of an external jolt. Others need the help the most, but are in politicized environments so they're almost inaccessible until a new executive comes in who can admit there are problems (or indeed, even see that there are problems).
Joe Reis has some great stuff in Fundamentals of Data Engineering, which includes advice on early objectives when rolling out a new practice.
Disclaimer: Joe has hosted me on his podcast, and we are in the mutual-marketing whirlpool together. But I've been recommending his book long before I met him.
Wish I had a LinkedIn and were an Executive, so I could connect with you.
And on it goes.
What would I give to be able to compose sentences like this one.
I see questions like these a lot and every time I feel that people immensely underestimate the effort required for curating data. In my experience data can only ever be as good as what it's being used for and in this story the logs haven't been used for this purpose before so they're not going to be any good.
It's some sort of data variation on the second law of thermodynamics - entropy is winning. Going in with the expectation that things should be better will only lead to frustration.
Seems to be "observability", for anyone else seeing that for the first time.
We shouldn't just have wide events/big spans emitted... We should have those spans drive the pipeline. Rather than observability being a passive monitoring system, if we write code that reacts to events we are capturing, then we shuffle towards event sourcing.
Given how badly coupled together with shoestring glue & good wishes so many systems are, how opaque these pain zones are, it feels like the centralization upon existing industry standard protocols to capture events (which imo include traces) is a clear win.
(Obvious downside, these systems become mission critical, business process & monitoring both.)
Monitoring, dashboarding and alerting should leverage other BI-class tooling.
I resigned due to the night terrors caused by the cyber security issues I saw everywhere. The more I explored and understood the more sleep I lost.
Thank you for this phrase; I'll quote it at every opportunity.
I’ll stick heavy hitters from Goldratt, Fowler, Feynman et al in there, but there’s going to be a “dark humor” and “snarky” category and this will definitely go into one of them, along with some Ambrose Bierce.
For real, a fun and refreshing read (if also a little haunting).
I feel this comment in my bones.
Add: for people who sharer the feeling -- you can work in a place where velocity isn't all, managers are not assholes and you can dedicate yourself to craft.
"…what they actually needed to do was fire most of the staff in every team, leaving behind the two people who actually had good domain knowledge, then allow them to collaborate with good engineering teams to build sensible processes and systems.
Instead, they hired a bunch of Big Firm Consultants. You can see where this is going already."
I’m on month ten of my tech sabbatical and it’s been great. I’m no closer to wanting to return to the industry (former FAANG data jockey), not in a position where I can never go back but am in that middle-state where I want to use tech to contribute to the betterment of society, the community, nature and sustainability. I have a couple more months to try and figure out how.
I came into the industry exactly as a previous AI Trough of Disillusionment was in full swing so I’ve been vaccinated.
I’ve found this to be one of my largest day-to-day problems even in a relatively functional organisation. Particularly when it involves something I can’t run on my own machine, like an AWS service.
In a previous role I often found myself constructing elaborate hypotheses about what was going on inside systems I couldn’t see into. I’d then need to try to verify it with someone on another team, in another timezone, who had the access but not necessarily the development background. Which usually meant getting on a screen share and asking them to click various things I wasn’t allowed to. If I was wrong, back to the drawing board and start again.
> you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it
I once sat down with a data engineer to try to fix a specific problem they had and that was 100% accurate. They were left to die by Ops and CISO.
For example, Operations might demand that Engineering develops an increasingly-byzantine approvals process, to stop Sales from over-promising impossible or unprofitable projects.
Immensely frustrating and draining where you can have accomplished ~nothing in a full day of work to fix what should have been a five minute change.
"I would like to make this systemically better by addressing our second order problems that are causing our very visible first order problems"
You will be told that we absolutely do not have time for that. The only actions you're allowed to take are fighting the fires closest to you, not turning off the pumps that spray the gasoline everywhere.
Typically this is only even possible because nothing that you're doing is actually used or scrutinized since if it was, someone would have immediately noticed that nothing works. Usually this is at places running on varying levels of investment dollar three-card-montey.
And — not to take away from the piece – If noone noticed the logs have been garbage for two years, they can't be that important.
The logs are stupid (sorta) but imagine how many other issues can exist if we uncover something like this every time we open something.
The people with experience and knowledge get pushed aside in favor of someone who can talk management-speak and (often) looks the role. Suddenly meetings where Things Get Decided only include managers who don’t know what they’re doing.
The best is when they make a mess and then get tremendous accolades for half-fixing it.
I’ve even seen it happen within relatively small startups. It’s a sign of rotting culture, but sometimes you have a mortgage to pay and have to get comfortable with the situation until you can find a better gig.
This is the first I've heard anyone say that. Can you elaborate? We've got more leave (separate holiday and sick leave), none of the "at-will" stuff, right to disconnect, less hours, etc.
Fully remote. I can expense $120/month for phone and internet, and a few lunches each month, too. I can get a new laptop and/or monitor sent to me just by asking.
When I do visit the office, the trip is fully expensed. Free daily lunch. Coffee and drinks and snacks everywhere, free. Private desks in a semi-open office with couches scattered around. Lounges with hundreds of board games, nearly all of which have seen table time during work hours.
Primary projects are tracked in a knowledge-sharing system, but I can mostly work on what I want to. I’m encouraged to merge small fixes and refactors without any ticket-pushing at all. Yelling by managers or anyone would not be tolerated.
“At-will” is more FUD than reality in my experience. Most companies, when firing or laying someone off, give something like 2 weeks of severance for every year of service.
Are there specific protections that are lacking in the US that you would expect to result in worse employee outcomes?
Again, I'm not arguing about who has it better. I am asking a very narrow question about worker rights differences between the two countries. If you want me to say that US software workers have it better, then sure. But that's not really what I'm asking.
Can someone, please, tell me this is a joke. Because I can't be certain, but it doesn't look like one.
(But yeah, this one I know is for real. The other one that feels like a joke, but looks real is harder to decide.)
They would almost always feature a "I sip my coffee. It tastes like <feelings at the time>" Example: https://www.reddit.com/r/talesfromtechsupport/comments/203qx...
That's why you build data platforms and name your team accordingly. This is much easier position to defend, where you and your team have a mandate to build tools for other to be efficient with data.
If upstream provides funky logs or jsons where you expect strings, that's for your downstream to worry about. They need the data and they need to chase down the right people in the org to resolve that. Your responsibility should be only to provide a unified access to that external data and ideally some governance around the access like logging and lineage.
Tldr; Open your 'data' mandate too wide and vague and you won't survive as a team. Build data platforms instead.
If you want an answer to a specific question, we can spin up a read replica and a Metabase and write a query in an afternoon, cool. I’ll get you a chart, we’ll move on. If you want “a data analytics platform to enable blah blah blah” I’m out, I can’t do it. My eyes won’t focus, my hands stop moving.
Developers sometimes tell me stuff like “Kubernetes is too complex”, “jeez React is a pain”. I send those quotes to my friends stuck writing 195 step DAGs to transform log files from s3 into s3 so they can eventually land in s3 - ah yes but they’re parquet somewhere in between, and that matters for some reason. We laugh together, but I can see it hurts them more than I intended.
Life is too short to faff about doing nothing. Go join a company with less than 100 engineers and learn to be happy again. Let the enterprises burn, we’ll all be better for it.
Anyways this was a fantastic piece, I hope this person writes their book after all.
But but then I won't be able to lease a new panzerwankmobile every 2 year to impress people I don't like!
That’s half of Big Data to me. Maybe more. No idea what we’re looking for but we will tell you after the checks clear.
sounds similar to what I do! on my team, it does actually serve a purpose, it's just hard to see the forest from the trees sometimes
AU dev scene is not great. Really heavy with POs, and PMs and CTO's without the background.
Ow
I recently re-read it for the third time while on holiday and was taking notes on pages worth including in a book review. I ended up with about 1/3rd of all pages logged and decided no review was better than just telling people to read the book, but I might write a blog post on gumption traps.
Brilliant
He is writing as if the engineers all knew how to fix the systems, but were just powerless to do that. But I’ve also seen projects lead by engineers that only added to the overall complexity.
There is a paradox in this - the people who seem the most confident about fixing the systems usually only make things worse. Chesterton fences and stuff.
This article triggers me because everybody who reads it will always believe that they would fix the mess if only they got the power, but in practice when they get power they would only add new complexity to the whole mess.
Because there were no automated tests. If the company needs something to work, that thing needs a, preferably automated, test.
Very happy to see this here, realise it's the same person and that this is "a thing", and then to rollick in the author's backlog. A joy! Raucous real-life laughter has exploded from me on numerous occasions along with most articles. I think I've read 5 in a row there, and my brain is buzzing happily.
Thank you to the author for having the courage to write about real experiences. A breath of fresh air. I look forward to future books and articles, and reading more previous work, and cross my proverbial fingers hoping they can keep it real in the face of what will presumably be an avalanche of grifters looking to leech off the attention.
https://www.documentjournal.com/2024/05/technical-images-fil...
I have a fantasy of quitting my job to write books on the [modern] theory and practice of information systems engineering. Not 'how to write software', that's been done; I mean all the forms of engineering around software/information systems. In my dream, I write the books, everyone reads them, and starts doing their jobs right.
But then I remember, I, a person arrogant enough to believe he knows how to do things right, still can't get shit done right. Maybe if I were a one-man company, I could 'do everything right', and feel good about the result. But I depend on an entire company of people to do the right thing, in the right way, at the right time. That's hard even with the best people. No company is made up of the best people. It's always a mix of the best, worst, and in-between.
Strangely, a company can put out a decent product, despite the company being a tire fire. This is some comfort when you get older. You realize that everything being shit is okay, as long as the bills are paid. I have PTSD from when the thing that paid the bills was on fire, every week, for years. Lately at every job I have, I internally panic and scream at how horrible everything is. Because I'm haunted by what might happen. But it's not happening yet. So I muffle the screams, smile and nod along with the stand-up-meeting-cum-status-update.
The sad thing is, I forget that it's okay that the stand-up is shit. I forget that I'm still getting a fat paycheck just to sit in meetings that could have been an e-mail. I forget that, despite the company bleeding cloud costs [no savings plans, RIs, serverless, right-sizing, etc], we seem to be making a profit. Despite the terrible designs, bad process, ineffective leadership, absentee management, lack of security, and all the rest, the bottom line is fine. The shit is fine. Currently, and probably for the unforeseeable future.
I get craftsmanship. I'm a crappy woodworker. I enjoy making things well, and getting better at it. But our jobs are not fine woodworking. Our jobs are construction. We are banging rusted nails into shitty, twisted, racked, cupped, knotty-ass studs. If we're lucky. Yeah, this building is going to be shit. But somebody's still going to pay for it. And there'll be another job after. If we really wanted fine woodworking, we never would have taken this job, and we know it. We'd be struggling to sell a cabinet that took us two 80+ hour weeks, too tired to appreciate its beauty, too defeated by flaws only we notice.
So let's stop beating ourselves up. Let's stop beating each other up. We don't, can't, won't, find meaning in this monument to mediocrity. No comfort from the pain zone. No pride to take home. But we are paying the bills, with more left over than most have. No broken backs and long hours. No lack of health care, no abuse from customers or the public. Not even that big a worry about job security. We are the lucky ones. We are blessed with a golden shovel. So let's do like those blue collar laborers we often idolize, and get to this annoying, bloody awful work that we are blessed with.
For example, what would happen if carpenters just eyeballed every measurement, and just shrugged their shoulders when the walls didn't line up. If they found out the way the wiring was planned will not actually end connecting where they thought it would and just shrugged and did it anyway.
What if they had hour long meetings about how to drive in screws and their workdays consisted of putting up a beam, and then going home because they think they've done enough for the day.
If all of society was ran like that, it wouldn't be running for long.
I definitely think it isn't viable for many people, but I've also met people who could totally do it that are scared because they're around a lot of people with ability/financial constraints that they simply don't have.
This is why I reflexively chafe when I see "Software|Data Engineering".
I've never witnessed it in practice anywhere I've worked.
Sure, embedded systems for a pacemaker/defense industry or something is probably proper engineering, or graphics programming where you're concerned with creating approximations of real light transport with a time budget per frame or whatever, but most of this over-architected junk involving putting strings in databases and sending clients 10MB of JS to render a list is the farthest thing from "engineering".
A decade+ of inflated compensation really went to people's heads with these silly titles.
But as you (hopefully) progress in your career, the expectation is that you are increasingly the one making and justifying those lists, which is much more soul crushing and harder to "background" and pretend that it doesn't matter.
> we're serverless, because how can you hurt yourself without a cutting-edge?
Just perfect.
^^^ Then do it... and then strangler fig the original.
I've sought advice from various people on this, some who are famous-ish or quiet sales powerhouses in the US. My question was "What are executives buying when they hire consultants?", and the answer is consistently "Comfort". No one is actually comforted by Deloitte, KPMG, whoever.
The moment I had confidence in an ethical consulting practice is the moment someone said "I don't even know where I would hire good consultants". This was someone with 30 years or something absurd of industry experience, including mentoring people that went on to become staff engineers. After processing that, I realized I don't know where to hire a consultancy that isn't going to bait-and-switch me with mediocre talent. They obviously exist, but they probably can only support something like 1 to 10 clients each.
Even Thoughtworks, a place that I used to hear mentioned positively, was flagged by the CTO of a >$1B company over lunch last week as "shifting to bait-and-switch" tactics.
tl;dr Pretty sure you can absolutely do better and make a living off that, but you have to think very carefully, do research, read a bunch of sales/marketing books, have great communication skills and at least adequate engineering skills. I still don't know if I have some of those, but if I fail it'll be a skill issue, not because the problem is not tractable.
The older the company is, the more likely one finds this morass.
It won't change absent powerful technical leadership.
Right there ^^^
If you're already good, just have coffee with executives/leads at places that take engineering seriously and you'll earn well, but there's no need to enter data hell as an employee at the typical company.
Mike Judge did such a great job of making that character into Dollar Store Steve Jobs.
> coated in grass which rends those who tread upon it like a legion of upraised spears,
and there's more. Get to the point, whatever it is.
I've tried to come up with some heuristic to determine whether or not a team is competent, good, or doomed. I've been exposed to all over the last... 8-10 years, and one of the key things I've noticed is the ratio of competent/skilled developers to the unskilled ones is a big ... indicator(?). Predictor?
Colleague of mine has been working with a team - dev team has ranged from 5-8 people over the last few years. Few people seem to have any grasp of programming at all. Only two people - my colleague and one other - have ever taken projects from ideas to delivery, or even taken features from requests to successful rollout of already functioning software.
The arguments that people get in to there - days or weeks of people 'researching' whether or not OAUTH 'really' requires 'refresh tokens' or whether it's really supposed to be a JWT. Management has some notion of 'every voice is legitimate and should be heard - we don't support bullying' and so on.
If you have a team of 10, and 1 or 2 people are simply bad at having the ability to think somewhat abstractly, you can survive.
If that number hits, say, 4-5... the team will struggle. A lot. You can keep things going, but it will be slow. And everything becomes a battle.
If that number becomes 7 or 8, and you only really have 1-2 developers who are actually competent developers... things will continue to spiral downward.
On the other side - I worked with a team of about 8-10 people on a 6 month contract. The larger org had another 40 or so folks, handling other projects, and support. Onboarding was great - I pushed production code in the first week. Everyone on the team was competent, including the juniors. I had more development experience, but they had more company experience, and it was really a relatively enjoyable engagement overall.
It was refreshing to be able to ask anyone on the team questions, and either get a workable answer, or an "I'm not sure, let's check with XYZ" to get working answers. The "oh, yeah, it's ABC" when ABC is clearly not the answer stuff never happened. People committing code and pushing to production without ever having run the code at all - I've experienced that - didn't happen - that's happening to my colleague.
The problem with a plurality of tech-incompetent folks in a tech group is that they honestly can not determine that they aren't competent. The only examples of competence are in the minority, and tend to not be trusted (even though that minority is the only portion that turns out working/functional code).
Leaving ends up being the only option in those cases. My colleague is only at his place part time, and has hung around because they've gone through some restructuring where new folks were brought in, and... you hope that things might get better in a few months, then realize they don't.
That is a _technical solution_ to a fundamentally _social_ problem.
- Negotiate with different teams to figure out what a field means
- Be told that a field should be converted from one format to another, but oh wait it's causing errors somewhere downstream because it was told the wrong instructions
- People come to you with some issue about the code you maintain, and you dig enough to realize the root cause is another team's code
edit: https://ludic.mataroa.blog/blog/the-violent-role-of-relentle...
In any case, the book deal had constraints like "no swearing", and it was implied they'd find their own artist for the cover. I didn't even intend to swear, but I care too much to let them assign an editor and impose arbitrary constraints. I ended up chatting a bit with Ed Zitron after the AI rant article went super viral, and he told me that I have the audience to just publish my own books.
So I'm doing just that! I'm starting with ten short stories, at the advice of someone in the local Melbourne scene that has helped many people publish. Then I'm aiming to write one book containing a series of essays on IT work, and one fantasy/fiction book. I'm a reasonably good judge of popularity, and because neither will be particularly angry, I will probably only sell a few hundred copies. But I'll have a book with my name on it, which is very special to me.
Many of us build systems to manage automatic actions to take care of this stuff, if I had an engineer I had to take 100+ steps to get something done I would definitely be considering 1) What the hell am I paying for and 2) Why the hell am I paying for it?
Also, once the people who speak up about a problem leave, all you are left with are idiot yes-men in management, old timers doing their job as minimally as possible to not to be noticed by management, and fresh new engineering grads ready to be grist for the mill. When those sorts of people are writing all of the code around you, no matter how good you are, you will be driven insane.